Comment by neofrommatrix

> The pressure to announce more and more at reinvent while the quality of what was being announced dropped was the specific feeling I'm talking about.

Yea, I can certainly see this making one feel claustrophobic.

That's an extremely sensible idea in multiple dimensions. It prioritises clarity of thought over rambling discussion in conference calls. I wonder if there's a feasible path to gradually steer an existing organisational structure in that direction.

This is Bezos talking about this: https://youtu.be/e47wAgIhZ7o?si=LKZxoabqrCSCQ1ki&t=64

^ exactly, thanks for taking the answer perfectly.

It's basically panel 2 from this:

https://xkcd.com/568/

Beyond the initial publication of the doc, the peer review process is much more sane than trying to review a bunch of power point slides. Similarly, it's much much easier to refer to a well written document when it comes time to implement or reevaluate an idea than going over some power point slides and maybe an associated recording, to say nothing about searchability, discoverability, and maintainability of an actual written document vs PowerPoint slides.

Also, idle side speculation: I wonder how much (if any) one of the underappreciated early employees @ Amazon had a hand in proselytizing this, given she (MacKenzie) is an author.

Oof, must suck to work for a company who doesn't use technical design docs well. I quite like Oxide's RFD model, based off Joyent's I assume, given who their CTO is. https://rfd.shared.oxide.computer/rfd/1

In my experience these documents were actually never good. I’ve never seen anyone ask for estimates either. Surely it’s better than some other companies but if it were good they wouldn’t need this absolutely horrific oncall

I believe most tech focused companies do it and it's called design docs / RFCs.

I heard it expressed years ago, when the role was still call sysops, that it was the one job where the better you were then the less work you did.

It was attached to a similar anecdote about someone being yelled at for crafting a well-oiled system.

Yet here we are talking about the person… they sound pretty influential to me. (I’m only half-joking)

Perhaps more accurate is that there is no one root cause, but a bunch of contributing factors.

I had a quick search to find my favourite article on this one, it might be https://willgallego.com/2018/04/02/no-seriously-root-cause-i... as it demolishes Five Whys but https://www.kitchensoap.com/2012/02/10/each-necessary-but-on... is a classic

In addition to other comments see https://en.m.wikipedia.org/wiki/Ishikawa_diagram or the “new school” of safety engineering thats trying to get away from ideas like “RCA.” Sidney Dekker publishes an older version of his seminal work https://www.humanfactors.lth.se/fileadmin/lusa/Sidney_Dekker...

Im ex AWS and the internal tools _try_ to get away from the myth of human error and root cause. But its difficult when humans read and understand in a linear post hoc fashion.

https://en.wikipedia.org/wiki/Swiss_cheese_model (kind of), pretty standard nowadays way in SRE to talk about root causes (plural) because usually it's more than one specific thing.

Amazon culture was still really rather root cause oriented when I left. The COE process they followed to do post-incident analysis was flawed and based on discredited approaches like the "5 whys". I don't know if it has changed since I left, I rather hope it has.

I genuinely believe there is no such thing as a root cause. The reason I believe that is both grounded in personal experience, and in the 40+ years of academic research that demonstrates in far greater detail that no failure comes about as a result of a single thing. Failure is always complex, and that approaches to incident analysis and remediation that are grounded in it are largely ineffectual. You have to consider all contributing factors if you want to make actual progress. The tech industry tends to trail really far behind on this subject.

A couple of quick up-front reading suggestion, if I may:

* https://how.complexsystems.fail/ - An excellent, and short, read. It's not really written from the perspective of tech, but every single point is easily applicable to technological systems. I encourage engineers to think about the systems that they're responsible for at work, their code, incidents they've been involved in, as they read it. Everything we deal with is fundamentally complex. Even a basic application that runs on a single server is complex, because of all the layers of the operating system below it, let alone the wider environment of the network and beyond.

* If you're willing to go deeper: "Behind Human Error" ISBN-10: ‎9780754678342. It's a very approachable book, written by Dr Woods and several other prominent names in the academic research side in to failures.

My favourite example to use when talking about why there's no such thing as a root cause has been the 737-MAX situation. Which has only become more apt over time, as we've learned just how bad that plane was.

With the original 737-MAX situation, we had two planes crash in the space of 5 months.

If we follow a root cause analysis approach, you'll end up looking at the fact that the planes were reliant on a single sensor to tell them their angle of attack (AoA). So replace the single sensor with multiple, something that was already possible to do, and the planes stop crashing. Job done. Resilience achieved!

That doesn't really address a lot of important things, like, how did we end up in a situation where the plane was ever sold with a single AoA sensor (especially one with a track record of inaccuracy)?

Why did the Maneuvering Characteristics Augmentation System (MCAS) keep overriding pilot input on the controls, which it didn't do on previous implementations of MCAS systems? Why was the existence of the MCAS system not in the manuals? Why could MCAS repeatedly activate?

If the MCAS hadn't behaved in such a bizarre fashion, where it kept overriding input, it's arguable that the pilots would have been able to correct its erroneous dropping of the nose. If you're sticking with a "root cause" model, there's a pretty strong argument that the MCAS system was the actual root cause of the incident.

Given the fairly fundamental changes to the controls, and the introduction of the MCAS, why were pilots not required to be trained on the new model of plane? Is Boeings attempts to avoid pilots needing retraining the actual root cause?

Why was an MCAS system necessary in the first place? It's because of the engine changes they'd made, and how they'd had to position them in relation to the wings that tended to result in the planes nose wanting to go up and induce a stall. The MCAS system was introduced to help with counteracting that new behaviour. Was the new engine choice the root cause of failure?

and so on and so forth..

If you look at the NTSB accident report for the 737-MAX flights, it goes way deeper, and broader in to the incident. It takes a holistic view of the situation, because even the answers to those questions I pose above are multi-faceted. As a result it had a long list of recommendations for Boeing, the FAA, and all involved parties.

Those 2 crashes were the emergent property of a complex system. It's a symptom. A side effect. It required failures in the way things were communicated within Boeing, failures in FAA certification, failures in engineering, failures in management decision making, the works.

No one at Boeing deliberately set out to make a plane that crashes. A lot of the decisions about how they do things are reasonable, and make some sense, in isolation. But they all contributed together to make for a system to which it became a question of "when" and not "if" a major disaster would occur.

If resilience and incident analysis just focuses on a singular root cause, the systems that produced the bad outcome, will continue, and will inevitably make more bad decisions. The only thing that would improve is that they would probably never make a decision to sell a plane with a single AoA sensor.

As we've seen over time, the whole broken system has resulted in a lot of issues with the MAX, beyond that sensor situation.

Presumably your comp changes if you get promoted in that time though?

When I was there, they weren’t creative. They outright said that doing certain things didn’t help them get promoted.

Relative hardship comparisons generally suck.

"hah you think retail is hard, imagine working for a drug gang in Africa! You'd go back to retail in a snap!"

Stress is stress. It makes you unwell. The remuneration isn't the point.

Shit to Salary ratio.

Its almost universal that we will deal with more crap for more money. Some people wont, but most will.

There is a line for everyone though where the shit is too much for any salary.

For software engineering, I have heard stories from all around about the PIP culture due to an 8-12% mandatory URA quota. This results in gaslighting and forced firings to engineers that have a history of performing well.

I personally know a SDM looking to leave now because he said he was basically told to fire one of his reports who was performing fine, or he would be the one fired. I've read stories of "hire to fire" where they hire ICs with the intention of letting them go in 6-12 months so another member of their team doesn't have to.

Culture is self-serving, hunger games style. Management is cut-throat and not empathetic at all.

The NYTimes did a big expose about the toxic culture of working at Amazon, and Bezos' public response was literally along the lines of "we move fast and know that our competitive culture is not for everyone. Our employees are up for the challenge and we are proud of what we accomplish."

In fact, during the hiring boom in 2020-2022 their reputation of SWE ICs was so bad that they were having trouble hiring. Many (myself included) would receive 1-2 emails from different recruiters desperately trying to get us to apply. We all knew to stay away.

The sad part is that many of the toxic managers and even ICs have infiltrated much more of FAANG (and big tech) and with the economic recession especially in tech, Amazon's cut-throat management styles like "performance culture" and PIP quotas have been happening in many other places now.

For many, many people, (majority of programmers, really) morals don't have a price tag. That number could be anything. When you make a programmer choose between some number and customer damage, it becomes increasingly harder to damage the end-customer. This is known as "prioritizing customer obsession" over business function (which is what we're supposed to do).

Please, everyone, if there has to be a choice, save us (civilization) not them (the CEOs). Please. Think critically.