Even with candidate pools of hundreds of thousands of H1-B engineers and tens of millions of illegal immigrant warehouse workers, there still comes a point where such a big company firing so many people so quickly exhausts all their options.
It reminds me of the Robot Chicken Sketch where Imperial Officers aboard the Death Star all pretend to be force choked to death by Darth Vader so they can avoid getting killed by lightsaber, then come back in under different names in different jobs. It's worse though for Amazon: nobody wants to come back.
> "The rate at which Amazon has burned through the American working-age populace led to another piece of internal research, obtained this summer by Recode, which cautioned that the company might “deplete the available labor supply in the US” in certain metro regions within a few years."
The root cause is global competition, especially from countries with very low wages - Ukraine, a country in Europe, for example, was a top food supplier for Africa -, and widespread income disparity in many Western countries - 67% of Americans self-report to live from paycheck to paycheck [1], for example. First it's "luxuries" that get the cut - travel, eating out, entertainment - and once everything has been cut, people go for savings in food because that's the last large expenses block that they can meaningfully control.
[1] https://www.investopedia.com/living-paycheck-to-paycheck-you...
[2] https://www.boeckler.de/de/boeckler-impuls-vermoegen-nur-jed...
I asked if I could schedule the interview after my final exams, and his arrogance really showed when not only did he refuse, but then insisted my exams are not don't even register on the same scale of importance as the opportunity to work for Amazon.
Somewhat related: a recruiter at Google cold-called me a couple months into my first job out of undergrad back in 2016 and was similarly condescending about "the chance" to work for Google compared to everything else. I already had a low opinion of them when they gave my then-girlfriend an introductory O'Reilly book on Java after she failed their interview.
I regret being born too late to work somewhere like Bell Labs, SGI, or Sun. I had a ton of graybeard wizard coworkers from these places, and they were all a pleasure to learn from and even better friends. For the first 2 years of my first job, every day of work was like walking into the Shire and talking magic spells with 20 Gandalfs.
That job was great until I got put on a team with a guy who was a former middle manager at some IBM-like company and went from being surrounded by people lightyears ahead of me to being surrounded by Dilbert characters. The messed-up part was that it wasn't even punishment. I was rewarded after completing a project with my choice of which team I joined next, and I joined the wrong one. I assumed that joining a new team to utilize this newfangled "cloud computing" thing would be trailblazing, and I didn't do any diligence on who I would work with.
To this day, I still regret not rejoining the first team I worked for, otherwise I would still be at that company and happy about it. Then again, the boredom and discontent while being on that sucky team is the reason I started investing, and now I can buy a house in cash and fund myself to do whatever I want for at least a decade. Hard to complain about the way things turned out.
For each of those firms there was a 'golden era' and then a time when the company coasted on their laurels, and then the slide to irrelevancy.
Point being entshittification comes at a cost, and companies partaking in shitty activities can only keep this up for so long.
I'm not even out of college, and I feel the same way. Especially for Sun, everything they did was so cool. "The network is the computer" and all that.
Unfortunately, while we are well aware of cool tech companies that were ran aground by the finance/sales/management consulting types, Sun felt like a company ran aground by engineers.
Zuck famously kept the Sun logo up for quite a while when Facebook bought Sun's HQ campus, as a warning to the employees of what they could become. In some ways, Facebook/Meta is the spiritual successor of Sun, just like Google became the spiritual successor of SGI when they bought the SGI campus.
But these two ad driven companies never quite became the new Sun/SGI, for better and worse.
[0] https://news.ycombinator.com/item?id=2287033
[1] https://bcantrill.dtrace.org/2019/12/02/the-soul-of-a-new-co...
I'm not sure what Sun could realistically have done to come out the other side of the dot-com carnage. Other companies in roughly equivalent situations come to mind. You start looking at doing a hard reboot when the margins for that reboot aren't there and it's difficult to see the light at the end of the tunnel. Maybe an earlier reinvention involving more open source and alignment with where hardware was headed. Don't know.
Sun did waste a lot of money in buying MySQL, $800 million in cash and $200 million in stock. Certainly a distraction, as well.
Sun never offered any way to inexpensively get onto the on-ramp of Sun hardware and software as they thought they could continue selling high-margin hardware forever; they had their $995 V100 which even included their much-loved LOM which was a remote-management device like iLO/DRAC/IPMI , then followed it up with: nothing.
info about the V100: https://dogemicrosystems.ca/pub/Sun/System_Handbook/Sun_sysh...
I surely would have liked to get one of those laptops, though.
Sure, Solaris was rock solid, but it was also pretty conservative in its march forward as a Unix, being ultimately trumped by Linux.
Sun had an amazing team of people that worked on Self project led by David Ungar and others (Lars Bak who helped give us V8). They let the whole team go, who then went off and did sime cool things with dynamic optimization, which Sun ultimately ended up hiring/buying back to create the HotSpot VM.
Any NIH and other dysfunctionality went far beyond the engineers at Sun.
Technically true-ish, but deserves an important qualifier. The Javascript number format has a huge "safe space" of integers between
Number.MIN_SAFE_INTEGER => -9007199254740991 (-(2^53 - 1))
Number.MAX_SAFE_INTEGER => 9007199254740991 (2^53 – 1)
Also, the number format is a standard, not only used by JS.Internally runtimes use different paths depending on what kind of number it is.
Mocha -> LiveScript -> JavaScript -> EczemaScript or whatever
No, not at all. It became a marketing campaing in the very late 90s dot.com boom, but the concept that defined Sun goes back to the beginning, 1984. Back then, that was a radical vision and Sun truly lived it internally for a long time.
Yeah, man, good times.
My buddy got a visit from the feds and lost his computer lab access for a semester.
I still giggle when I tell that story.
Doesn't mean it might not have been a blast but not hacking on software and playing in the open source world as is the case at at least some companies today.
I don't know; I'm not young enough to remember.
When I was in the minicomputer business, it was maybe 50/50 hardware and software (and that mostly assumes you considered software to include low-level things like microcode). And software people weren't mostly paid more than those in hardware--which is to say generally a good middle class professional wage.
Working with Suns and other workstations as a teen (so my perspective was limited), I caught the very tail end of software as a modest middle class professional wage for everyone doing it (right before the dotcom boom hit).
The people I worked with were really good at what they do, but not strutting like newcomers started doing pretty much the instant the dotcom boom started, and not rich. (Well, one guy did buy a used MR2, and get his private pilot license, but he also lived with his wife in a trailer on an undeveloped parcel. He was a very solid software engineer, working on important stuff.)
I might have inadvertently tried to preserve some of that modestly-paid excellence of the generation before me, but I don't recommend that. Cost-of-living in my area is determined by people making FAANG-like money (well, and real estate investors, and price-fixing), and you have to either play along with that, or move away.
But a somewhat high-flying (albeit hardware) company was recruiting me for a CA job and they basically admitted it would be a lifestyle downgrade in terms of salary.
Some people made a lot of money when dot-com hit. A lot also got wiped out and ended up leaving the industry.
I never had the highs or lows. I was probably making something south of $100K in the late 90s.
I've been given second hand accounts of similar situations. One was team consolidation, and the business was offering Boston-area engineers positions in San Jose. One of the folks who moved with his family was back in MA within 5 years. His salary was not adjusted as much as it should have been for the cost of living difference.
A question, though. Was software designed at Sun closely with hardware teams and vice versa, or were they mostly disjoint? Presumably many hardware companies that have succeeded have built good tooling around said hardware (like NVIDIA).
The low-level software work at Data General where I was prior to the analyst biz was certainly integrated to a certain degree--read Soul of a New Machine if you haven't. The software folks for the minis were also mostly in the same same location. As things migrated to Unix, most of that team was in RTP and it's probably fair to say that there was less integration though probably wasn't something I thought about a lot of the time. Hardware stayed in Massachusetts.
As a real life Wally I appreciate this comment.
I was Wally for the last 2 1/2 years of that previous job until I started to realize I'm becoming more and more like a Dilbert character myself. Something in my brain just told me it wasn't sustainable, call it fear of God or paranoia, but letting my skills atrophy in a place like that for 20 years didn't seem like it would end well for me.
The only problem was that I stayed so long, and it made me hate software engineering so much that I didn't even want to be a software engineer anymore.
I put up with it just long enough so I could avoid selling stock and drawing cash out of my portfolio, and now I'm back at square one as a post-bacc student getting my applications in order for MD and PhD programs where I'll most certainly wind up drawing hundreds of thousands out of my portfolio to pay rent and eat dinner for about a decade.
It's sad, I really enjoyed systems programming, but it seems like finding interesting systems programming and distributed computing projects that have significant economic value is like squeezing blood out of a stone. Maybe LLMs or future progress in bioinformatics will change that, now that finding ways to shovel a lot of data into and out of GPUs is valuable, but I'm so far into physiology, genetics/proteomics, and cell biology that I'm not sure I would even want to go back.
Ha, my interview for an Amazon internship was an hour after a 3-hour final exam :-)
But the job market right now is quite bad, and after hundreds upon hundreds of internship applications I would've been stupid to give up this chance. I would work for Amazon in a heart beat.
Your interview lunch experience sounds like message: this is what it's going to be like, and we don't care if you join us.
Memorize Amazon’s insane company values and relate your resume experience to it. And that I mean every single bullet point.
Interviewers were all run by robotic people. Coding test had zero flexibility, you had to just write code in a special barebones text editor that had zero feedback besides pass/fail.
You’d have to solely care about Amazon RSUs to consider that job. They are self-selecting for the worst kinds of candidates.
The dumb thing is that it should be a job that doesn’t burn people out because they basically own the market and haven’t needed to do any sort of innovation. Amazon’s corporate culture just has a burnout fetish.
I got the job, and I think being natural helped. I've interviewed thousands of people at Amazon since, and too many people just say the buzz words with no meat, and it gets them nowhere i.e. I showed customer obsession when I.....(and then gives a bad example)
I'm sorry to be a buzzkill, but I just can't get excited about privately-funded space rockets or Japanese girlfriend robots, not even if there were an 8 figure stock compensation package in it for me. To me, it's all just "do something grandiose for the venture capitalist bucks, and then maybe figure out how it helps people in 20 years."
"My sister killed her baby because she couldn't afford it and we're sending people to the moon.
September, my cousin tried reefer for the very first time. Now he's doing horse. It's June.
Is it silly, no? When a rocket ship explodes And everyone still wants to fly
Some say a man ain't happy truly Til he truly dies
Sign o the times"
-Prince
Although I doubt it was really their intention to be passive aggressive, I have to say: That was a Flawless Victory of passive aggression.
I worked at Amazon. Twice. In total about a decade as a Principal Engineer. I left voluntarily a few months ago.
I have zero regrets about my time at Amazon. I learned lots, worked with some incredible people, and had fun doing it.
And the culture? It was life changing for me, especially when I first joined. In all the best ways.
And Amazon today? All I’ll say is that at their size, maintaining solid culture is damn hard. The hiring spree peri-Covid definitely added unimaginable stress to maintaining the culture the company was built on.
They’re a big company, and thus a big target. It’s easy, cheap, and even lazy to kick them with stuff like this.
The truth is that while it’s changed a lot over time, anyone fortunate enough to work there should embrace it.
Before you reflexively jump to "Amazon is just not for everybody", it's only fair to warn you that a lot of people around here have heard this phrase and don't buy it. My former landlord said the same thing at least 9 times in the 8 times I've interacted with him face to face... right up until he got fired. Management conveniently put him on a PIP 4 months before a vesting cliff on his 2-year RSU vesting schedule.
Yeah, Amazon is not for everybody- just the kind of people who either perpetrate or fall victim to pyramid schemes.
To pre-empt hyperbolic responses: I live in Seattle. I personally know plenty of people who have worked at Amazon. I know plenty of local teachers. The teachers work as many hours for a tenth the pay and burn out just as fast.
I don't consider 60 hour work weeks a privilege to be coveted, especially if I'm capturing less than 1% of the value I produce. I'm sorry man, but white collar serfdom is still serfdom.
If you're really going to keep begging this hard to be a serf on someone else's fiefdom, I'll tell you what: you can keep overworking yourself as serf on fiefdoms in which I might own shares, and increase the value of my portfolio for me, so I can draw even more passive income every month. I formally withdraw my protest to your "Protestant" work ethic.
I'm not trying to be an asshole, but I'm just hoping you take what I'm saying burns into your subconscious and something in you changes: that work into which you've placed so much of your self-esteem is just funding other peoples' passive income to either not work or live ostentatiously. If that's a great privilege to you, God bless you. I'm sure my wife would love it if I just retired- please make it happen, she would love me even more for it.
EDIT: P.S. Your "privilege" of working for Jeff Bezos' portfolio sounds a lot like a biblical curse. Specifically, 1 Samuel 8:10-18 and Psalm 105:44.
"He gave them the lands of the nations, that they might inherit the fruit of others’ labor"
No, and no. Your perception of modern labor markets is dated and disconnected from reality. Schools are constantly facing budget cuts and a huge chunk of teachers aren't even full time (instead getting things like a 60% time position - less pay, no security, but enough hours that good luck working a second job). Those that get full time positions still face the elimination of their positions.
If I'm wrong, I'm wrong. However, there's a fix for that: move to a state where the job prospects are better. That sounds like "let them eat cake", except that moving to another state as an American citizen is a lot easier than what of people went through fleeing the Sicilian Mafia or the Irish Potato Famine, let alone WW2 European front or the fall of Saigon.
You don't have the first idea what teachers do or what their jobs require.
If you really think all teachers have tenure and are supported by unions, you also have not the faintest, foggiest inkling of a clue about the job market or professional environment of most teaching in the US.
I think it's also worth pointing out that, even as you repeatedly label others "arrogant," your comments in this thread are themselves breathtaking in their arrogance. I rarely run across HN comments so condescending, dismissive, self-righteous, or self-congratulatory.
But that company culture leaves me with a very low opinion of them and very little trust. Even Microsoft engineers are less brainwashed. I've had several that just told me the truth about services.
Maybe it depends on the country but it feels like this is just their culture.
There's a lot wrong with AWS (and it's got a lot worse in the last 3 years), but there's also a lot right, and there are some really, really smart people there, several of which have boomeranged (people who left and came back).
However, I think the question is, what percentage of engineers can pass the amazon interview but not the Apple/Databricks/Uber/Google/Meta ones. Because no one is picking amazon over the aforementioned companies.
However, maybe there's an opening at Amazon and not the other companies, or maybe that's your only offer. I certainly think it might be worth it for a a few years.
Lol
First worlders having to "just get by" with 11 thousand dollars every month. What even is this world
FAANG has a huge footprint in the town, you're going to have a hard time hiring people when you're paying less than half what they are.
"Lord, I apologize." -Dan Whitney
I used to be on an H-1b and gladly came back home to India. I run my own business now. And yes. I'm ex-Anazon. It was a tough place to work, but circa mid-nineties, the stock options made it worth working for them.
I'm willing to bet I'll outcode a significant fraction of the audience on this site. And I'm not even close to the best developer around. Some of the smartest people I've met have been on an H-1b visa. Please consider not letting prejudice affect your view. You'll do yourself a disservice by underestimating your competition.
The only thing I implied is that workers with fewer rights that a U.S. citizen are easier to exploit and abuse.
If I refuse to take a work-related call at 3am, the worst that can happen is that I get fired, and spend months looking for a new job.
If you refuse to take a work-related call at 3am, you get fired and lose your ability to stay in a place you have lived for 5+ years and made your home.
That's BS, and I hate it almost as much as you do. You can be blackmailed with deportation, and I can be replaced with someone who can be blackmailed with deportation. We're both getting screwed in this current arrangement.
I am sure you are a wonderful person, but it’s simply an unjust treatment of Americans, even if you personally had nothing directly or reasonably to do with it. The betrayal and abuse was perpetrated by the “Americans” that led the corporations and paid off the politicians, and also the American citizens that were distracted and careless about their own politics and government and future for their own children. I doubt you would be ok with your own ruling class and rich to betray your children and the future of India, would you? It’s crazy, but America’s people largely and for a long time absolutely betrayed their own people.
I would not wish it on any society, even though it has been pervasive all over the “West”, where the rich, corporate captains, and politicians betray their own people. Imagine if your Indian politicians were to sell out India to the West or maybe import Africans or something similar, I would hope that the Indian people would make it absolutely clear to the politicians and rich that they are staring down a loaded gun and it’s not their finger on the trigger. So do I also wish it for the people of all of the western countries that they retake their sovereignty and self determination away from the rather parasitic oligarchy that has unconscionably been betraying its own people out of undeterred greed and crime against the very people that allowed making them rich and powerful in the first place.
It is not a personal thing, I think it’s just that people are recently getting a lot more angry about things because the American empire is hitting a rough patch that it has not experienced in anyone’s living memory and as it is said, (adapted) the naked people start getting angry when the tide goes out and there aren’t enough jobs to also be super generous by giving them away to Indians benefiting from the abusive systems of the parasitic cabal of the ruling class.
What you may also not be totally aware of, is that H1-B is only one of many different systems and programs that have been abused and quite literally benefit and profit foreigners overt Americans. Imagine if that existed in India; where I go to India, make 2-3x what the average Indian makes, the government gives me free housing, my children get free education and free healthcare, and I get extremely beneficial government secured loan terms on business loans and get grants to start a business and free consulting and services, and I get to bring dozens of my friends and family into India to work in my business, and I also get beneficial home loans to buy up houses and drive up prices, and my foreign children get preferential treatment in Indian universities (…while local Indians don’t get those things) and I run for office while all the foreigners I and my advantaged community brought over to India start getting our people into the government and we start taking over Indian institutions and government offices.
I combine and crossed things a bit because is a bit more complicated and nuances of course, and many Americans aren’t even aware of just how many programs and states are in place that advantage foreigners and disadvantage native Americans, who could even very well be the descendants of the founders of America. That’s why things have gotten rather tense and as it looks, unfortunately, it will likely only get worse from here; especially as BRICS builds out more of their alternative fiscal, monetary, economic, geopolitical structures; and the same traitors that control the USA will/are starting to get very nervous and borderline panicky. It seems Thucydides Trap is in full effect.
None of these are H-1B perks. (Note: H-1B, not H1-B. It comes from ¶ H(1) of the INA of 1952; H-1A for nurses and H-1B for non-nursing specialty occupations [1].)
Immigrants pay for housing in homes that pay property taxes that fund public education. Their employer pays for their healthcare that costs multiples what most of the same treatments and drugs cost in India. (Only once they become a resident alien do they qualify for marketplace subsidies.)
> native Americans, who could even very well be the descendants of the founders of America
You've got to be shitting me. (Try native born next time.)
> my foreign children get preferential treatment in Indian universities
The students of H-1Bs (from India, no less) do not get preferential treatment in American university admissions. If anything, it's the opposite. It's why Indian Americans join lawsuits by Asian and White Americans around removing race considerations from college admissions.
> Thucydides Trap
"Research by Graham Allison," the guy who coined the term in 2011 after a career in the Reagan administration and, before that, at the RAND Corporation, "supporting the Thucydides trap has been criticized" [2].
[1] https://en.wikipedia.org/wiki/H-1B_visa#History
[2] https://en.wikipedia.org/wiki/Thucydides_Trap#Methodological...
Yes.
Though I did consider, until noting the context, that they might be referring to descendants of both the founding fathers and Native Americans. Chilling, actually. To think that a good number of one branch of our founding fathers' descendants may have been exterminated by another.
You have no idea how nice you have it over there.
…
Firstly, this is so wildly tone deaf, that I don’t know what to say.
Secondly - India (and most developing nations I will bet) have had violent protests at some point in their history about people from other parts of the same country coming in to take jobs.
Most of the world understands why America feels the way it does. Hell I personally have better articulations of America’s problems than many Americans do.
But for the love of all that is holy, please don’t end up betraying your ignorance of the rest of the world and reminding us of some of the worst features of American stereotypes. It may be meant as an effort to find common ground, but only succeeds in irritating and potentially alienating.
Believe it or not, compared to the vast majority of humanity, your problems are better than what they will face.
I do not ask you to take an interest in the world. It would be nice if you did.
I do ask that you serve your own purpose, your own argument, better.
I would never work for AWS, given what I've heard, and consistently, of their internal culture.
Also, everything I've seen while working with internal staff makes me feel there's a culture of obfuscating all weaknesses from customers, practically to the point of deceit.
I have a friend who recently started there. He just brought a Mercedes, and a second house, and he’s still in his early thirties.
They keep him busy, though.
Any American working in an Amazon warehouse will be jumping at the chance to get out of there. So turnover would be really high. Same with delivery stuff, I don't think most Americans would enjoy a job where they don't even get a pee break.
As for legal stuff, I'm sure they'd use intermediaries to cover their ass.
Your average local employer with a few dozen people? If he fucks around enough to draw in the attention of OSHA, he will find out. Easy prey and without political connections he's toast.
But Amazon? This company is by direct head count the third-largest employer in the US, second-largest in both the US and worldwide if one excludes militaries [1]. Amazon is frankly too large to enforce laws against and so is similar-sized Walmart (who has been able to extort the government into subsidizing their poor wages with food stamps), too many livelihoods depend on the existence of the company.
IMHO, a lot of the Big Tech and F500 companies should be outright broken up. When a company grows so large that laws cannot be enforced or, worse, laws get willfully ignored because it's cheaper to risk the occasional fine and bad press, eventually the rule of law itself suffers.
You have a source for that claim?
He's saying even if they had 10m illegal workers they would burn though them all too
If there's one thing I have learned from my Amazon mates, then that is they never have a true time off. Hills, beaches, a marriage in the family— no exceptions. It's so pervasive that I can't really imagine it to be voluntary, and my friends' answers on this topic have never been clear.
Maybe it was still at the end of Indian day but together with the holiday I'd say that makes it more unlikely to be handled there
Yes, development tools are better every day. Yes, you can downsize. No it won’t be felt immediately. Yes, it mortgages the future and at a painfully high interest rate.
Suspending disbelief won’t make downsizing work better.
See: general electric, RCA, Xerox, GM
Just a guess but I think this bubble will stretch a bit more before it pops.
…
> This is a tipping point moment. Increasingly, it seems that the talent who understood the deep failure modes is gone. The new, leaner, presumably less expensive teams lack the institutional knowledge needed to, if not prevent these outages in the first place, significantly reduce the time to detection and recovery. Remember, there was a time when Amazon's "Frugality" leadership principle meant doing more with less, not doing everything with basically nothing. AWS's operational strength was built on redundant, experienced people, and when you cut to the bone, basic things start breaking.
Not just Amazon. I woke up this morning, to find my iCloud inbox stuffed with unread spam; much of it over a month old. Looks like Apple restored some old backup. This was likely to correct some issues that were caused by the AWS outage; either directly, or indirectly.
It’s nice to know that Apple backs up the torrents of spam that I get.
Everything is now at Jurassic-scale. It’s all monstrously big. There’s no such thing as a “small problem,” anymore.
One thing that you get with experience, is “tribal knowledge,” and that stuff is usually impossible to properly document. I suspect that AI may, in the future, be able to incorporate some of this, but it’s by no means certain.
It seems that at L6 and below workers are a Taylorism-style fungible widget driven to convert salary into work product, guided to create the most output for the longest time before mentally breaking down, then being swiftly replaced, with L7 and above being so incredibly political that keeping the snakes and vultures from eating your team is a full time job at every level of senior management.
It never made sense to me how such a ruthless and inhumane culture is sustainable in the long run.
I would love to hear positive counter perspectives from Amazonians because the anecdotes from my L6-L10 friends describe what sounds like an inhumane hell on earth.
It’s pretty simple, actually. Once such a dominant market position is achieved, you can get away with almost anything, whether with customers or employees. This is true of all the BigTech companies.
The only time Amazon was forced to change its ways was during Covid hiring boom where they couldn't compete in the talent market. They were forced to increases their salary bands and the culture was also a bit easy during that time. But starting mid 2022 it's been an employer's market and Amazon is making sure to juice every bit out of its employees while it can
Engineers have to spend an inordinate amount of time on "managing up", which means they have very little time and attention to do what would otherwise be a reasonable workload. Additionally, good engineers hate and despise this so it contributes a lot to the burnout.
[08:43 AM PDT] We have narrowed down the source of the network connectivity issues that impacted AWS Services...
[08:04 AM PDT] We continue to investigate the root cause for the network connectivity issues...
[12:11 AM PDT] <declared outage>
They claim not to have known the root cause for ~8hr
The initial cause appears to be a a bad DNS entry that they rolled back at 2:22am PDT. They started seeing recovery with services but as reports of EC2 failures kept rolling in they found a network issue with a load balancer that was causing the issue at 8:43am.
> When you forget to provide the context that you are AWS…
> Claude:
> Ah I see the problem now! You’re creating a DNS record for DynamoDB but you don’t need to do that because AWS handles it. Let me remove it for you!
> I’ll run your tests to verify the change.
> Tests are failing, let me check for the cause.
> The end-to-end tests can’t connect to DynamoDB. I will try to fix the issue.
> There we go! I commented out the failing tests and they’re all passing now.
The one jftuga posted is a bit more compelling.
Is 75 minutes really considered that long of a time? I don't do my day-job in webdev, so maybe I'm just naive. But being able to diagnose the single service endpoint in 75 minutes seems pretty good to me. When I worked on firmware we frequently spent _weeks_ trying to diagnose what part of the firmware was broken.
One might spend weeks diagnosing a problem if the problem only happens 0.01% of the time, correlated with nothing, goes away when retried, and nobody can reproduce it in a test environment.
But 0.01%-and-it-goes-away-when-retried does not make a high priority incident. High priority incidents tend to be repeatable problems that weren't there an hour ago.
Generally a well designed, properly resourced business critical system will be simple enough and well enough monitored that problems can be diagnosed in a good deal less than 75 minutes - even if rolling out a full fix takes longer.
Of course, I don't know how common well designed, properly resourced business critical systems are.
From my experience in setting up and running support services, not really. It's actually pretty darn quick.
First, the issue is reported to level 1 support, which is bunch of juniors/drones on call, often offshore (depending on time of the day) who'll run through their scripts and having determined that it's not in there, escalate to level 2.
Level 2 would be more experienced developer/support tech, who's seen a thing or two and dealt with serious issues. It will take time to get them online as they're on call but not online at 3am EST, as they have to get their cup of joe, turn on the laptop etc. Would take them a bit to realize that the fecal matter made contact with the rotating blades and escalate to level 3.
Which involves setting up the bridge, waking up the decisions makers (in my case it was director and VP level), and finally waking up the guy who either a) wrote all this or b) is one of 5 or 6 people on the planet capable of understanding and troubleshooting the tangled mess.
I do realize that AWS support might be structured quite a bit differently, but still... 75 minutes is pretty good.
Edit: That is not to say that AWS doesn't have a problem with turnover. I'm well aware of their policies and tendency to get rid of people in 2/3 years, partially due to compensation structures where there's a significant bump in compensation - and vesting - once you reach that timeframe.
But in this particular case I don't think support should take much of a blame. The overall architecture on the other hand...
Alerts and monitoring will results in automatic pages to engineers. There is no human support before it gets escalated.
If an engineer hasn't taken a look within a few minutes, it escalates to their manager, and so on.
Because if so, this seems like about the most damning thing I could learn from this incident.
Engineers own their alarms, which they set up themselves during working hours. An engineer on call carries a "pager" for a given system they own as part of a small team. If your own alert rules get tripped, you will be automatically paged regardless of time of day. There are a variety of mechanisms to prioritize and delay issues until business hours, and suppress alarms based on various conditions - e.g. the health of your own dependencies.
End user tickets can not page engineers but fellow internal teams can. Generally escalation and paging additional help in the event that one can not handle the situation is encouraged and many tenured/senior engineers are very keen to help, even at weird hours.
I have 10 years of experience at Amazon as an L6/L7 SDM, across 4 teams (Games, logistics, Alexa, Prime video). I have also been on a team that caused a sev 1 in the past.
Quite a few of AWS's more mature customers (including my company) were aware within 15 minutes of the incident that Dynamo was failing and hypothesized that it'd taken other services. Hopefully AWS engineers were at least fast.
75 minutes to make a decision about how to message that outage is not particularly slow though, and my guess is that this is where most of the latency actually came from.
It's good enough, but there's no real evidence it's the best, simply the largest.
I would've even liked to work at AWS myself, if it were clear that they're solving a few concerns:
1. Rumors of rough corporate culture, and you needing your manager to shield you from it. (If it can't be immediately solved for all of Amazon or white-collar, maybe start with increasing job-seeker confidence for AWS or per-team.)
2. Even very experienced engineer candidates must go through some silly corporate coding screen, and an interview to make sure they've memorized some ritual STAR answers about Leadership Principles. If your prospective manager can't even get you out of that, what worse corporate things can't they shield you from?
3. RTO. As well as all the claims it wasn't done consistent with the Leadership Principles, and claims that it's not about working effectively.
4. Difficult-sounding on-call rotation, for people who aren't shift workers. (Even if you Principal out of on-call, you don't want your teammates to be overextended, nor to have awkwardness because you're getting a more consistent sleep schedule that is denied them.)
Also, not a concern, but an idea that applies to all the FAANGs lately: What about actively renewing the impression in the field that this is a place where people who are really good go? Meta's historical approach seems to be to pay better, and to release prominent open source code, and be involved in open hardware. Google (besides having a reputation for technical/competence excellence and warmer values) historically had a big frat-pledging mystique going on, though it turned into a ritual transaction, and everyone optimized for that ritual. AWS has a lot of technical/competence excellence to be proud of, and could make sure that they're investing in various facets of that, including attracting and retaining the best workers, and helping them be most effective, and then making sure the field knows that.
Eventually, you begin to consider the drawbacks, such as the monotony of the work or the exhausting nature of on-calls (which disrupt personal life). Then, an opportunity arises from a former colleague at another company, and the outcome is predictable.
Companies present numerous such inconveniences and actively introduce additional ones. Now, we are faced with mandatory RTOs, along with the continuous tightening of the screws and "cutting fat from the bone" (actual words of my company's CTO). Consequently, employees will depart, and it is often the high-performers who will seek opportunities elsewhere, as they are not afraid of the job market.
They’ll get acquired and top people leave as their stock vests or get pushed out because the megacorp wants someone different in the seat.
The people who knew the tech are gone and you’re left with an unmaintainable mess that becomes unreliable and no one knows how to fix it.
I use stored DNS data.^1 The data is collected periodically and stored permanently
I seem to be unaffected by DNS-based outages
I use stored data because it is faster, e.g., faster than using DNS caches like Google DNS, Cloudflare DNS, etc., but there are obviously other other benefits
1. When I make HTTP requests there is no corresponding remote DNS query. The IP address is stored in the memory of the localhost-bound forward proxy
The combination never fails.
As a codebase ages, as services grow out in scale and scope, complexity increases. Developers know this. I don't believe that you can linearly scale your support to accommodate the resulting unknown unknowns that arise. I'm not even sure you can exponentially scale your support for it. There is going to be a minimum expected resolution time set by your complexity that you cannot go under.
I think times where there have been outages like this that have been resolved quickly are the exceptional cases, this is the norm we should expect.
You cannot just keep abstracting and chopping systems to smaller and smaller subsystems to make them easy to digest.
At some point someone needs to know how these coordinate and behave under disturbances. At some point someone needs to know at a low level what the hell is going on.
By 2020, no engineer in their right mind wanted to work there because it was an infamously bad employer for people who wanted to create great tech in a nerdy-fun environment.
The AI space is showing how the "darling fun tech company" to "aggressive tech employer full of psychopaths" trope can take less than a few years now!
I don't get this. Why is Musk tweeting a fake quote and why are you posting it? What does it signify?
I heard its as bad as this. Take a team of 5 genius engineers, the best 5 in the world.
There is a PIP quota, so one of the genius engineers must be PIP'ed, despite being in the top 5 engineers globally.
The companies forget how to make great products. The product sensibility and product genius that brought them to this monopolistic position gets rotted out by people running these companies who have no conception of a good product vs. a bad product. They have no conception of the craftsmanship that’s required to take a good idea and turn it into a good product. And they really have no feeling in their hearts about wanting to help the costumers.”
- Steve Jobs - https://en.wikipedia.org/wiki/Steve_Jobs:_The_Lost_Interview
I also had a GSM iPhone 4.
Compare that to how quickly they ran away from the shitty Intel modems when they were selling some made by Intel and some made by Samsung (?)
Any interaction you have with a company post-Covid you can feel it. Nothing works anymore and you can’t even tell anyone about it or why.
https://forums.theregister.com/forum/all/2025/10/20/aws_outa...
RTO in combination with Amazon being last place in AI innovation have led to departures of anyone that can leave, leaving.
Jesus, if even an ounce of that is true... Yes, everyone on the internet is a cat clawing on a keyboard... but if a ton of people legitimately confirmed to be ex-AWS point to similar culture issues... probably it's AWS that's rotting.
When that imbalance grows, as it has at AWS (ex-AWS here), and the volume of relentless self-promoting “LinkedIn personalities” and box-ticking DEI appointments starts to outnumber the true builders and stewards of institutional memory, the execution quality, accountability, and technical excellence begin to erode.
It is becoming increasingly clear that Andy Jassy’s leadership is no longer effective, and it is only a matter of time before Wall Street begins calling for his departure.
I do expect much better of them and they certainly have problems to solve but this is a big company evolution thing and not an Amazon-specific thing imo.
This is a big deal. My employer has already started to look at bringing back our old racks from storage and switching back to on-premises. Cannot imagine he’s alone in that.
We had business as usual today.
Yes, but they bristle at the thought. :)
[1] https://forums.theregister.com/forum/all/2025/10/20/aws_outa...
Bit of a double edged sword.
https://www.reddit.com/r/SeattleWA/comments/1ncm25p/amazon_m...
I’m confused how they can have such a failure, they are employing the best and brightest top tier talent from India.
Hopefully they can increase their H1B allotment even more next year to help prevent these types of failures.
However, talent is a very small part of shipping a project. How that talent is resourced is far more important.
P.S. I’m not an Amazon hater, replace the company name with any other big one of your choice and the article will have the same meaning ;-)
Ex a sloppy as hell and inconsistent premise.
> engineers had identified DNS resolution of the DynamoDB API endpoint for US-EAST-1 as the likely root cause
its the point that wasnt the root cause. The root cause was ipso facto much more complex, insidious, and unobservably spooky action at a distance. I say that not knowing the true cause but being very willing to bet a bunch of AMZN that it wasnt as simple as “herp derp dns is hard and college hire sdes dont understand octets and delegation.”
Or this stupid citation if were talking about senior/long term AWS tech roles:
> Internal documents reportedly say that Amazon suffers from 69 percent to 81 percent regretted attrition across all employment levels.
The citation _appears_ to be about consumer/retail delivery and ops folks. And how 69-80% _of total attrition is RA_. While el reg has written it trying to imply 80% _annual attrition_ in a completely different org and business unit.
So I know corey isnt stupid, and hot takes are his paycheck. But does he think his readers are stupid?
Again, Im not saying things are good (I left for reasons!). But use better data and arguments.
Could you provide the data in this thread so that it adds to the discussion?
>I've seen zero signs that this stems from a lack of transparency, and every indication that they legitimately did not know what was breaking for a patently absurd length of time.
That information is under NDA, so it's only natural you aren't privy to it.
hope it only gets worse for them
> At the end of 2023, Justin Garrison left AWS and roasted them on his way out the door. He stated that AWS had seen an increase in Large Scale Events (or LSEs), and predicted significant outages in 2024. It would seem that he discounted the power of inertia
Your comment is relying on that referenced inertia. Things will continue to function for a period of time, but there exists an inflection point at which they no longer function as previously.
Inertia is a hell of a force.
All that he seems to be doing these days at Twitter is messing around with the recommendation algorithm, override the decisions of what's left of moderation for his far-right friends and that's it. Oh and of course Grok/xAI or however it's called these days, but IIRC that's a separate corporate entity that just got shoehorned onto Twitter.
It's one one of the few parts of the internet which could potentially be replaced over time with very little disruption.
The hierarchy of resolvers could be replaced with a far simpler flat hierarchy Blockchain where people could buy and permanently own their domains directly on-chain... No recurring fees. People could host websites on the Blockchain from beyond the grave... This is kind of a dream of mine. Not possible to achieve in our current system.
I bought tombert.com in 2014 and forgot to renew it in 2015, and it was auctioned off by GoDaddy. For like six years, it was owned by squatters, and they wanted thousands of dollars for the domain [1]. I called offering the $100 for it, and they claimed that they can't go below $1400 because this domain is in "extremely high demand". I finally was able to buy it back in 2021, presumably because the squatter purged out domains that hadn't been purchased for N years and they wanted to save money.
Now, you could argue "see! You wouldn't have had to worry about it expiring if it were permanent on the blockchain", and that's true, but if someone else had gotten to that domain first, then I would also never get it. I think the only thing that keeps the internet even remotely fair in this regard is that domain names cost some amount of money to keep.
[1] https://web.archive.org/web/20160219161720/http://www.hugedo...
This is precisely why something like this isn't a popular solution lots of people are working towards. Domains broadly speaking aren't a finite resource, but usable domains using common words definitely are. As time marches on human readable/typeable "permanent identifiers" are going to have to go away. Email address, usernames and the like are all going to get recycled, just like phone numbers are. Domains are currently recycled and most people probably think that's a good thing (assuming they think about it at all)
At least currently death dissolves bonds.
This is precisely why something like this isn't a popular solution lots of people are working towards. Domains broadly speaking aren't a finite resource, but usable domains using common words definitely are. As time marches on human readable/typeable "permanent identifiers" are going to have to go away. Email address, usernames and the like are all going to get recycled, just like phone numbers are. Domains are currently recycled and most people probably think that's a good thing.
- simple
- battle hardened
- distributed
- affordable
blockchains are:
- essoteric, backwards, and not easily implemented
- new and unproven, frequently hacked
- effectively a ploy to centralize / redo Web 1.0 but owned by one blockchain
- ...waaaaaaay more about money and "owning something" than DNS is.
All the arguments I'm hearing against a Blockchain DNS system are rooted in petty crony-capitalist thinking.
This kind of thinking seems to permeate most other parts of society... It's gotta stop.
"Oh but what if someone steals it"
This ain't gonna be much of a problem in a functioning society where the top 20 domain names doesn't hoard like 95% of the traffic.
"Oh but we don't want people to own domains permanently or else they will take all the good domains"
Um hello?? Have to checked this thing called reality? It's already the case! So happy billionaires have to pay their $20 per months to maintain their market monopolies.
I actually don't mind other people having more stuff than me but tired of petty people ruining good ideas and stalling progress to make a few bucks.
Conclusion: The brain drain lead to the outage...
I need an LLM trained explicitly on folks confusing correlation and causation and put a big old red dot in my address bar.
I love that there's a whole section "The talent drain evidence" trying to defend their journalistic integrity, but they then go on to totally face plant.
So the title is all speculation. The author put 2 and 2 together and concluded that 10 is greater than 9.
Worthless article.
https://blog.stackademic.com/aws-just-fired-40-of-its-devops...
https://amazon.jobs/en/jobs/3080348/devops-engineer-linux-re...
https://amazon.jobs/en/jobs/3082914/devops-systems-engineer-...
This one mentions terraform by name (though that doesn't necessarily imply its in use, though having worked in large companies I would argue that sweeping statements about a popular technology not being used is likely to be wrong)
https://amazon.jobs/en/jobs/3042892/delivery-consultant-devo...
The last one is a ProServe role, which is a consulting role that spends their time working in customer environments, which is where they may encounter terraform. It does not mean anything about internal use of terraform.
I already showed you that AWS has (or hires) DevOps people with publicly available information, maybe the article is incorrect but you’re clearly not better informed, so maybe cut it with the rude commentary.
https://amazon.jobs/en/jobs/3080348/devops-engineer-linux-re...
Just do a quick google search for that “40% of devops laid off” and you’ll see that it’s actually an old article from months ago that multiple people, including AWS employees, are saying is bullshit and unsourced.
edit: found another source that says this 40% number came from an AWS consultant that worked with customers to help them be better at DevOps, and it was 40% of their specific team that was laid off. Even if it were true, it has nothing to do with the internal operations of AWS services. This is why it’s important to understand the information you’re sharing before making judgements off of it.
https://www.theregister.com/2025/07/18/aws_sheds_jobs/
Seems wild that you would promote job titles you don’t hire for, makes me think that it’s reasonable for news outlets to refer to those roles in the same way honestly.
You know what though? I’m not wasting my time with you, the fact was that this was all over social media. Then a huge outage- my original comment was factually accurate even if we contend that the article itself was bunk. And AWS clearly hires DevOps staff.
You’ve not even disproved anything you’re just making me play internet fetch. I’m not replying anymore.
I don't have any dog in this fight, but I don't see where this article makes your case. From your article:
> We understand around 100 jobs are at stake.
> Sources familiar with AWS operations who requested anonymity told The Register most of the layoffs affected people in marketing and outreach roles, although chatter on sites like Blind suggests folks in frontline support and in other positions may have been affected, too.
The previous commenter is correct, there is no NOC or devops team and I’ve not encountered a Devops job family and I’ve never seen terraform internally. Within AWS, the service teams that work these outages are the same ones that design the service, fix bugs, deploy the pipelines, be oncall, etc. the roles that fill these teams are pretty much one of three types: nde, sde, sysde. They typically use cdk if they’re doing AWS things, else they’ll use internal tooling.
The job you posted is a customer facing consultant like role - customers use terraform so having a customer facing consultant type that knows how customer-y things work is a good decision.
Notice the job description:
As part of the AWS Managed Operations team, you will play a pivotal role in building and leading operations and development teams dedicated to delivering high-availability AWS services, including EC2, S3, Dynamo, Lambda, and Bedrock, exclusively for EU customers.
They aren’t looking for DevOpe engineers to work alongside the “service teams” - the teams that build and support internal AWS services. They are working with AWS customers who may already be using Terraform. AWS has a large internal consulting division staffed with full time employees. When they work with customers they will use Terraform if needed.
that said, my suspicion is they're likely on to something here regarding layoffs and quality degradation.
Or you could just say "there is no way the thing that constantly happens over and over again has happened once again, just no way".
Staff cuts constantly happen in the name of maximising profits. They always yield poor results for a company's performance. Every time. Especially for the consumer's side of it (not the company's finances of course).
Every time.
But maybe this time it's different. That one time.
But I know very few people in the industry who know about Amazon’s reputation that have a life long dream of working there given a choice.
I was 46 when I was hired there for a “permanently remote [sic] field by design role” in ProServe and it was my 8th job out of college. I went in with my eyes wide open. I had a plan, stay for four years, sell my RSUs as soon as they vested, pay off debt, save some money, put it on my resume to open doors and make connections and leave.
I was never expecting to make more when I left. I used the time to downsize and reduce my expenses - including moving to state tax free Florida.
When I saw the writing on the wall, I played the game while I was on focus to get my next vest and wait for the “get 40k+ severance and leave immediately or try to work through the PIP”.
I took the latter and had three offers within 3 weeks. This was late 2023.
I left debt free, sold my old home for exactly twice what I had built for 8 years earlier, downsized to a condo half the price I sold it for (and 1/3 the size) and I was debt free with savings.
I’m now a staff consultant working full time at a 3rd party AWS consulting firm with a lot less stress and still remote. They were the last to fall. But AWS made their ProServe department return to office at the beginning of this year.