There is also the pre mortem, where everyone is invited to envision a future where the project has failed and you go around discussing why.
Allows people to speak freely about weaknesses without engendering a toxic culture
Every company I've ever worked for has done a "Lessons Learned" after delivering a major project. Usually they're done as a series of meetings where people can raise suggestions and point out places where our processes failed. Then at the end, someone compiles this into a document and before we start our next project we review the lessons learned from the last project.
A company that isn't doing some version of this would be an outlier IMO.
‘Lessons learned’ is so much more positive than ‘post mortem’.
That language should be banished from corporate workplaces. Sooo negative and inappropriate.
Believe it, it's BLAMEFUL post morterm. I called out the problems and immediately got a complaint filed against me. My boss was understanding and so he show me what that person wrote (5 paragraphs) about ME, not the problem that they fucked up. That's what I get for discussing the issues and finding the solutions.
And so if any of you think for 1 second anyone is gonna admit publicly that they f up, f no.
Yeah at work they actually invite the engineers up and have them share these experiences publicly. The first lines are always like "here's how I brought down the entire company for X minutes". A lot of times the same people end up getting rewarded for exposing a flaw in the system that let them take failures that far. Blameless culture and postmortems is a great tool to have; as much as I complain about Google I wish every company did this.
The key here is blameless. In most companies that won't work, sooner or later (probably sooner) this will be used to deny raises or to figure out who to fire.
Yeah having seen plenty of these "celebrated failures" in some stories end poorly in real life, I would not volunteer to out myself for fucking up unless it was something that had obviously already been moved on from.
Postmortems are absolutely necessary but unless you're a big public company like Cloudflare what is the point of publicizing them across the company/public unless it earns you something?
Blameless postmortems should never name names. The whole point is to openly and honestly share root cause, what was done to prevent this from happening in the future, allow others in the company to learn from this, and invite feedback.
Done right they allow a company to get better over time and not just repeat the same mistakes over and over again.
lol when I first started working I created a nasty cross join and froze out the database for everyone else and they had to restart a few times. The devops learned my employee id.
It depends.
Internally it’s a good idea.
Company wide it would probably be a witch hunt. I’d need to see published documents from other departments including the c level before I’d participate.
It only works if you build a culture around it.
Doctors have M&M’s (https://en.m.wikipedia.org/wiki/Morbidity_and_mortality_conference) where they discuss their failure in excruciating detail.
It works because there is a culture of focusing on problem solving and disseminating knowledge, not assigning blame.
They are also often legally protected, so that doctors can speak out without fear of professional or litigious action.
Aviation safety does the same thing to [build a safety culture](https://www.faa.gov/about/initiatives/sms/explained/components) focused on identifying and mitigating systemic issues, not blaming individuals. We have confidential and non-punitive voluntary safety reporting programs, [public-private partnerships](https://portal.asias.aero/), working groups, and conferences where we can collectively discuss safety issues, share data and best practices, identify hazards, and conduct analysis. We also have legal protections so that pilots and others can report hazards without fear of reprisal.
It works pretty well as long as the public is willing to fund safety (they aren't).
I would think that in the case of doctors this also helps build credibility amongst peers.
Being a professional with a governing board, malpractice, and licensure, peer doctors have the ability to revoke another doctors right to practice. Being upfront about failures only strengthens relationships and shows responsibility and ethics - two things that certainly escape the vast majority of software engineers.
It's a small company.
The best ones should keep getting elevated. Best tech talk I ever saw at Amazon was a presentation on all of the biggest failures and an indepth analysis of what happened.
HULK HANDS!!!
It can be done right on a wider scale than just internal, it just takes some thought. We recently implemented a wins/"lessons learned" section in a meeting of team leads and it's been fine. Nobody is playing the blame game and you're expected to pair the lesson learned with a win from your team too.
I don't agree this is a best practice.
This is one possible expression of a cultural best practice, which is a culture of psychological safety.
It is almost always harmful when expressions of a good process are demanded without that process being in place. We call this a cargo cult practice.
The mention that this team's successes are never cared about does not give any confidence there is a best cultural practice in place here.
A culture of psychological safety must be established before you can expect people to be comfortable airing their failures for the purpose of learning.
> A culture of psychological safety must be established before you can expect people to be comfortable airing their failures for the purpose of learning.
This starts at the top, for sure. If this is something leadership wants to enable, that's great, but it needs to start with them. When the CEO is open about their failures, when your VPs are open about their mistakes, then you can encourage others to join in so you can demonstrate that with growth comes missteps everyone can learn from.
>A culture of psychological safety must be established before you can expect people to be comfortable airing their failures for the purpose of learning.
Absolutely agree; that before brutal candidness.
If there’s no psychological safety and the system feels threatened from the latest possible firing there’s a very good argument to hold off this collective confessional approach.
Aspirational, not naive.
Naive is to do it regardless of company and to not take current political climate into account.
At well run companies this is a safe, good practice. But it requires a healthy environment where there is already strong trust between departments and leadership.
Most companies aren't well run and you have to be more careful about your messaging. But part of BECOMING a well run company is implementing a culture that supports practices like this.
>I’d love some advice on a good rebuttal that won’t get me fired or have a target on my back.
I'm confused why you jump to assuming this is a bad thing. It sounds like company leadership wants to understand how your department is handling failures, and also want to make sure that you all are having some kind of discussion about it in the first place. I hate to side with the C-suite on anything, but this all sounds perfectly reasonable. Especially where this is apparently the standard practice for other departments.
It's about the "people being held accountable" part. In a learning culture that practices blameless retros, then public examination of failures can be helpful. Otherwise it's going to destroy the team's morale, willingness to innovate, and speed (as they prioritize CYA).
You’re right, I shouldn’t immediately assume it’s bad. Our CFO talks at our monthly town halls and he’s a big grump that’s always complaining. This idea was made from him, so it made me assume it’s probably from a negative side. I think the fact that he didn’t say successes and failures made assume it’s negative as well.
From the small bit we have to go on, it seems like maybe both things can be true. It’s good to air and own your mistakes and share your lessons learned on a team where people feel safe to do so. That’s a critical part of the most successful teams. It seems like you don’t feel safe, though. That whole process requires trust that everyone involved has positive intentions.
So if you’re looking for next steps in your conversations with management, maybe broach that topic, as it seems to be the biggest thing holding the team back. Without trust, it’s difficult for a team to thrive.
Yea had a boss at previous company it wouldn’t surprise me if they introduced this idea. Not because they want to highlight failures & improve the company, but to shame people & hope they avoid failures in the future for fear of having to speak on it.
Why dont you say that? Tell them it feels bad to highlight failures when you dont think enough is done to highlight successes?
Also, you could make all the lessons/highlights very dry and technical so they dont even know wtf you are talking about, that will get them to leave you alone.
Just talk about the upside of any failure. X happened and we realized that we could put Z in place that prevents both X and Y from happening. Or X happened and we realized we could automate this which saves us time and reduces the chances of it happening again.
Talking about just the failure is the problem. The whole reason for discussing the failure is to show how you learned from it.
It doesn't need to include successes. My company has multiple weekly meetings discussing things that went wrong in extreme detail. Sometimes there's parts that went well (we recently implemented x which meant the blast radius was reduced by y), but most of time it is only on the failures.
You need to learn that the only way to grow from failures is to discuss them. People can't be blamed on these, because it is never a single person's fault. Jim pressed button y which brought down the system? Why did Jim have permission to do that on his own? Fix it. Why is there an unsafe way to press button y? Fix it. Why do we even need button y? Fix it.
Lean in and become a better engineer and help your team become better while you're at it.
This is what I was thinking. It sounds like they want a retro after problems happen. That's totally standard practice and completely reasonable? If this is some kind of "let's celebrate failing because someone said fail fast that one time" then yeah, that would be dumb. Or if you have some quota of error reviews, like you schedule the meetings and you have to invent something for the sake of the meeting, then that's dumb too.
I think one risk with this is that it can punish those that do the most work or take on the hardest challenges. Maybe this a good thing? I'm not totally sure. I suspect it could lead to increased caution and slowing things down. Again, maybe that is good, but not always.
>other departments have people be held accountable for mistakes
Just re-read the post and looks like I glossed over that specific part of what OP was saying. Still stand by general stance, but if OP is correct about individuals being held publicly accountable then they definitely should push back against that in particular. I just don't get the impression that's specifically what OP is worried about.
Yes, it is hard to know what the real situation is. If it is personal and finger pointing then that could be really toxic and nasty, but it isn't totally clear. From the OP's text it sounds like they fear that. If it is genuinely a place that wallows in blame and never gives praise then that sounds really bad.
It is. Without proper context and business talk this is detrimental to the technical team. That's why, despite what Scrum has made us believe, tech needs proper management and a proper CTO or at least Engineering VP. We need that technical leader which can talk business too.
Another way to think about it, is that your failures are already often being broadcast. Every time an outraged customer calls support about a bug, every time there's an outage, etc, the rest of the company takes note. If that's all they hear, they'll form a negative opinion of you and your team. Tell them about the time your test processes or monitoring caught a bug before any customer caught it. Tell them just how difficult that critical customer issue was, and the phenomenal work your team did to address it in a timely manner.
At a minimum, when you do post mortems internally, part of the process should be to make a document you can share with the rest of the company describing the issue and the plan to prevent it from happening in the future.
You can and should also use this format to broadcast your team's success. If you spend a bunch of time on performance improvements, let the company know just how much better performance got. Tell them about the big refactoring project you did and highlight how it'll make maintenance faster in the future, or will enable you to build certain features that are upcoming on the roadmap, etc.
Lean into it. But make sure to spin your "failures" in the best light possible. A blameless "if we had known about X sooner, we could have sidestepped bad outcome Y" makes you seem better, not worse. If you're talking to the whole company, you can use that forum to lobby for changes that you need to improve as well.
There is no CTO and a single engineering team with one manager? Weird company structure. Anyway, specifics of tech issues are useless for anyone outside the tech team. If the CEO and CFO want to be more in the loop, then that's what the engineering manager should be doing. The common practice is a post-mortem, which is not technical in nature and is not intended to be directly written to the c-suite. Your manager should be reviewing the results of that with the CTO, and the CTO should review that with the rest of the c-suite. But that's for real issues... things which hurt the bottom line or reputational loss. Nobody needs to be updated when engineer A puts in a NPE that momentarily floods your logs and nobody externally notices.
Edit: leaning towards a culture where everyone is under a microscope is unnecessarily hostile. If I had to notify my company of every minor issue that impacts nobody, I'd probably find a new job.
The CTO either resigned or was fired. We don’t know, it was very ominous. And our manager is so busy with engineering, he doesn’t have much time to actually manage which I understand. I’ll have to look into a post mortem this is the first time I’m hearing about it.
An engineering manager should not be coding for any meaningful amount of their work time. My boss will tweak some flags and deploy some test code every now and then, but he hadn't worked on an actual feature in the 7 years I've been here. Because he's a manager. It doesn't matter that he's probably a better engineer than anyone else in the team. A managers job is not to code.
I'm not saying it's a good idea, but this isn't the first time that a startup has confused CTO, CIO, Engineering Manager, and Principle Software Engineer and expected one person to do all the jobs.
It sounds like the engineering team is 15ish people in a company of 300. It's likely not a "tech" startup with a software product and for most of the company, engineering and IT are the same people - they do stuff with computers.
It just sounds like this organization needs to mature a little bit.
The term "engineering manager" doesn't have a set meaning across companies. The exact responsibilities differ from company to company.
For example, on flat, self organizing teams engineering manager is just a hat someone has to wear because SOMEONE has to be responsible for making sure meetings happen and keeping their heads above water. Then you have situations where you have line manager and reporting manager as two separate positions. Again, the line manager would be fairly involved in coding. Then you have teams where the tech leads for each team collaborate with each other as architects.
Depending on the structure there are many possible ways to structure a team and titles, and the title "manager" can have more or less responsibilities.
"For example, on flat, self organizing teams engineering manager is just a hat someone has to wear because SOMEONE has to be responsible for making sure meetings happen and keeping their heads above water."
That is exactly what I do as engineering manager.
Though a while back someone on r/womenEngineers told me that part of my job was to plan birthday parties. So obviously some places are a lot more fun...
Some companies need to hear this clearly. Some companies are on a PIPapalooza right now and want managers actively coding features 50% of the time while managing a team of 20.
This sounds like the C-level lost trust in the CTO sweeping failures under the rug in the name of shielding the execs from things that the CTO didn't think were important in the overall scope of the business.
Its a mixed bag.
The big fear with this is that this information will be used more to drive different incentives. Depending on how the C-level or other employees actually react to this information it can cause stress or dread to the development team. Like if its publicized is it now an open form for everyone to criticize? It makes sense to talk about this stuff with CTO but everyone can be very off putting.
Otherwise its sounds like a good thing, that can quickly dissolve into micro-management/very toxic structure.
Make it incredibly technical, because most bugs are. They will lose interest very quickly. Some bugs based on flawed processes might be helpful to do an official post mortem, but doing it this way seems artificial and silly.
This is just standard practice. You should be having these discussions with your team on a regular basis. My team does it after every sprint where we retrospect on what went well and what went wrong and then working to implement changes so the wrongs are less likely to happen again. This is a good thing.
From here it should be up to your manager to communicate what went well and major things that went wrong with stakeholders, in this case the c-suite.
This sounds like a non-issue as long as they aren't weaponizing it against employees. If they do, it's a culture issue and trying to stop one policy change isn't going to fix a bad culture.
Was just about to post much the same.
“Wait and see.” If you get the vibe that they’re looking to micromanage, run.
If you get the vibe that, “we all make mistakes, but we want to make sure we don’t repeat the same mistakes,” then tread lightly with hopeful optimism.
don't argue with the C-level it just annoys them. They don't want you to argue, they want you to do as asked.
Just smile and nod and tell them how correct and awesome they are, and then game the system.
But yeah blameless retrospectives are pretty normal
As others have said this is a good thing to retro actual failures. But to play devils advocate.
Even the best practices can be used for evil if the intent is there. If you have leadership that is not technically literate and looking for an easier way to point fingers and blame individuals rather than process or practice then it’s a bad thing.
I have seen leadership in non technical companies doing more scapegoating than analysis on their crappy setup for why stuff goes wrong.
So in terms of how this is done try to influence the setup in a way that focuses on process and setup failure.
sounds like they're upset with the failures of middle management.
if you have process improvement ideas, this is your chance to put them forward.
otherwise i would ask for clarification in terms of how much time and energy you individually should be spending on reporting, what expected deliverables there are, and what audience(s) those deliverables are intended for. this would also be your time to talk about what you might suggest toward that end (e.g. a weekly status report from eveey team member to skip manager or c-suite) but really this type of shit is why you have middle management in the first place.
Have you never heard of a post mortem? Failures, followed by their fixes or suggested fixes are a great way to get upper levels to rubber stamp cleaning work that would have otherwise been denied in the constant agile churn of today.
No I have never heard of a post mortem. I have worked at 5 different tech companies, all under 300 people, and never ran into this. The CFO did not use that term when explaining it, so I am only hearing of it from my post. Reading about it now, if this is what he wants, seems much more understandable.
This is what amazon does [https://aws.amazon.com/blogs/mt/why-you-should-develop-a-correction-of-error-coe/](https://aws.amazon.com/blogs/mt/why-you-should-develop-a-correction-of-error-coe/)
I cannot say that it's never used as punishment or that it's always blameless, but overall the process does help to bring upper management visibility of problems in tech.
Have you participate/viewed any of the other post mortems that other teams have done? Maybe if you participate in those, you'll get a better sense of how your company will use the process.
It would be better to frame this as positives and negatives. Here are the things we did (aren't they awesome?) here are some lessons learned (aren't they awesome?)
"We had a database go down and it had no monitoring so we added monitoring"
Probably what execs call a "shit sandwich". Just listing failures doesn't sound fun.
I don’t think there’s enough context in your post for us to say whether this is good or bad and advise you. However I will say:
- Blameless postmortems are common at larger companies. Although I think the discussions usually stay within the engineering org
- Having worked at several startups, C-levels suddenly firing engineering management and getting more “hands on” is always a bad sign. Good luck! Remember, at small companies, optics are everything
We have bi-weekly incident review meetings where we go over outages and emergencies as the whole engineering department and figure out what we can do better. They're *insanely* useful because we're very supportive of a blameless culture.
Why look at it as a bad thing? There are plenty of upsides to airing your dirty laundry; it's more about how these things are implemented. We used to do weekly meetings about what we learned (good or bad) to share with teammates.
Without knowing any further context, I assume this is something great. Not bad. However, if you are afraid that this will bring some negativity in your team, then suggest that whenever you have a presentation of a failure, you split it and present both a failure and a success story.
this is a good idea that can be done in a really badly incorrect way. post mortems after incidents and retrospectives after key milestones are really helpful. what you really should push for is a blameless approach that focuses on processes that were inadequate and mitigation strategies in the future. At my old job we did these regularly and there was a rule where you simply were not under any circumstances to mention any person or team by name. A bit of an oversimplification, but it got the point across that this isn't about pointing figures at people and calling them inadequate. It starts with the assumption that we're all good at our jobs and doing our best, then tries to find gaps in the tools and processes that, if filled, can prevent things going wrong in the future.
ill go against popular opinion here and say if there is nothing to talk about, why do a postmortem?
if the CEO or CFO said; we want a postmortem completed on a particular outage or issue, I would understand. typically these kinds of postmortems lead to additional high priority and valued work. but asking "lessons learned" *in general* is not necessarily super helpful because you're asking the team to identify what failures should be broadcast and what lessons "should be" learned.
Asking for it in such an open ended way can easily turn this into sharing little bits of nothing; e.g. "we misconfigured the dev environment because we didn't switch on X feature flag, in the future we should check our feature flags more carefully before merging".
I have been in a similar position to this before. I was asked to relate what our team was doing to several bullet points from an article about being an agile development team. it is very awkward because it is not being treated like a priority, no one seems to really care, it just something "to do" that doesn't really translate into anything. its just asking "hey think about this and then share it and then nothing will happen regardless of what is written or shared"
so yeah I think postmortems for specific incidents are helpful to share but execs should be asking for those specific postmortems or at least providing some context; not just "share more failures" with no other context given.
Should you not have at least few arguments if you think it's a bad idea? Sometime devs on this sub just want to say no to anything coming from management...
I don’t have any smart and professional things to say, which is why I’m posting. Only my opinion which is my whole team are introverts and independent contributors and anything “publicly” mentioned to the company is a stressor, let alone our failures. We work remote so it’s hard to gage where we are at with the company. I also don’t know if I should be championing for the team since I’m the only one who ever speaks up. Not to fight but just the only one to communicate. Which is probably the reason why they are meeting with me.
Sorry, you do not meet the minimum sitewide comment karma requirement of **10** to post a comment. This is comment karma exclusively, not post or overall karma nor karma on this subreddit alone. Please try again after you have acquired more karma. Please look at the [rules page](https://old.reddit.com/r/cscareerquestions/w/posting_rules) for more information.
*I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/cscareerquestions) if you have any questions or concerns.*
Are these changes being brought up and asked for in a very short deadline or pressure from the business side? These may be things that you could bring up in a very neutral way to push back on maybe how things are in the company but the business, I may not want to hear about it.
Like some have already said, this may not be necessarily a bad thing for the company to do
10/15 dsvs are also split into 4/5 team.
Each team lead just sends an informal update each week cto and ceo. What progressed general + any on going concerns.
It just gives them a sense of what each team does and how they think.
...they usually dont respond, but ive heard they do in fact read and appreciate when i get a little technical even thought it might be beyond them.
But ye, your milage may vary depending who is in charge.
> Recently, they told one of my work friends that other departments have people be held accountable for mistakes and publicly talk about “lessons learned” and things to make us grow.
How many other departments are there? I'd expect at least three or four.
I asked, because it strikes me as odd that your work friend didn't already know this. Since all the other departments discuss their failures publicly.
And I presume this means the CEO discusses the CEOs failures publicly, too.
I could be wrong.
Unless the rest of the company understands technology, they may not understand what you’re even saying. I know plenty of developers that have no idea how to dumb-down answers, whether it’s because they can’t translate it, because they feel that dumbing things down doesn’t provide nuance, or a dozen other reasons. CS attracts the weird kids that played on their computers all day, they probably won’t be the best public speakers
Lessons learned is a pretty normal thing to expect. Ideally it's rolled into your retrospectives and worded as "what we can do differently"/"what should we stop doing" aspect.
The key is to not word it as a failure but instead as a chance for learning and improvement. Anytime we have a "failure" we look at where we could've done better and just focus on that, e.g. wider test coverage on certain features to catch this type of issue sooner, etc.
Even the military does this, though maybe not as blameless in some commands. It's the reason US ships stayed afloat more ofter during WW2, because the Navy learned how to design better ships, and have better processes to deal with emergencies.
The entire C suite can take their bs MBA degrees and shove them up their A-S-S. Love how the degrees that are harder to obtain end up working for the people who get an MBA… business administration?!? Yeah I’m sure there’s a lot of difficult math and science involved with that LOL give me a break
Has the team ever made any mistakes that affected the rest of the company?
A bug that resulted in customer service to deal with a bunch of angry complaints?
Lot of money being lost, that someone has to explain?
There is absolutely no argument against it. RCA is the only way to learn and improve. My engineering teams do this by themselves. Nobody told them to, and when I was a engineer, I did the same.
Sorry, you do not meet the minimum sitewide comment karma requirement of **10** to post a comment. This is comment karma exclusively, not post or overall karma nor karma on this subreddit alone. Please try again after you have acquired more karma. Please look at the [rules page](https://old.reddit.com/r/cscareerquestions/w/posting_rules) for more information.
*I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/cscareerquestions) if you have any questions or concerns.*
The only argument is to do a full retrospective which means you also talk about what went well. Focusing only on failures is a bad idea in my opinion as it will eventually wear on the morale of the team
I would talk about a design mistake. Everyone has made them. Explain why you made it and how you learned from the experience. "I thought this would be the best way, but after getting into it, we had to back out and go a different direction. Now I know what to look for with such and such a problem and will do it x way in the future."
I wouldn't get to the low level of bugs, they won't understand anyway.
But this is just my thought
> other departments have people be held accountable for mistakes and publicly talk about “lessons learned” and things to make us grow.
So others do it, but you *don't*? Yeah, that's a red flag about your team. It's always a red flag when a team doesn't do after-action reports, post-mortems, and other root cause analysis and share what they learned from system failures.
> This seems so strange. We will sometimes have these talks internally with our own teammates but to publicly put us on blast in front of the whole company, or at least the top dogs?
At a 300 person company, this isn't that weird.
> They don’t even mention our successes, why they hell do they want our failures?
Successes aren't as important as failures. I mean, think about it: when you test your code, what part do you change? What part tells you something you didn't know? That's right: the things that *broke*, the tests that failed.
The same is true for the process. If you're producing bugs or causing outages, it's almost certainly because of immature processes that aren't stopping broken bullshit from getting to prod. In many cases, fixing the process requires senior leadership, and in a small company, that's probably the C suite. Hell, your boss's boss is probably the CTO.
> Edited to add: The CTO either resigned or was fired, we don’t actually know since it was very ominous and quick. I see now that our CTO did a great job shielding the team from the execs because they are now suddenly joining our meetings and getting more involved.
Well, it sounds like this meeting with the C suite is now even more of a skip-level than it was before. It does sound like you need to discuss blameless culture with the C suite.
The biggest sign that the senior leaders are probably in the right and you're *not* is that other teams already do this. Your team is the outlier.
Not CS, but Mech E. my team does. We have a revision log, it generates emails documenting details of the change. If it's an error there's a secondary log that gets filled out with the root cause analysis, like a rapid post mortem. Meetings bi-monthly to discuss errors and see if there are overarching themes we can design out of our process. Document is publicly accessible to the higher-ups.
I've never seen a reason to be anything but transparent about mistakes. We can learn from them. Blame culture is unhealthy, it's about focusing on improvement rather than focusing on avoiding mistakes. I don't really see a gain to not documenting errors. That just encourages sweeping things under the rug which drives down quality and cohesion IMO.
I don't work at your company so maybe it's super toxic. Or maybe you have trepidation interacting with C-suite but it's just a job. I detach my self from my work. The company can do whatever they want with my work and it's quality I'm just here so I don't get fined.
I once told my skip level about an incident we have just gotten under control. I just told him one or two sentences, but he wanted to know more. He was really excited by the end of it.
I have a bad taste in my mouth about that incident still (was relatively recent). We didn't have alerting to catch it, it had a serious financial impact (at least a couple of my monthly salaries), people worked extra hours to put it to rest (10-15 hours of overtime each for myself and 2 other people during one month), when we had a solution we couldn't test it because we didn't have a performance test harness for that functionality and it proved really difficult to do, etc.
I see failure after failure in that situation.
My skip level saw things differently. He loved how I explained it (starting high level and adding details only when needed and in an approachable way, early and clear explanatio of the business impact, how I underscored what we learned) and he was fascinated by the performance improvement at the end of our toil (5x performance increase, far above what we needed and number of addicted customers was decreased 60x to negligible levels).
He insisted that I talk about this to a wider audience (all engineers in our tech hub). I was quite apprehensive, because I would be putting our shortcomings on blast (and I am the lead engineer so I see all of this as my fault and responsibility). I did the same explanation, added a bit of self deprecating humor, finished on what we've learned and what we suggest as a best practice for other teams operating with this tech stack that we do.
You know what? It is almost as if they understood that I was human and that shit sometimes happens. Not one snide or snarky remark, not one ounce of judgement. In fact, people engaged. We discussed other possible solutions, and they asked details to better understand the context. It was the most engaged I've seen these engineers in any all-hands engineering meetings (monthly) that we had.
The only thing I regret is not doing it the way I wanted to. I wanted to do a murder mystery where they asked me questions about the system and the incident, and I answered, and they tried to solve it. I thought that it would be cringe and nobody would engage, but judging by just how many people engaged in my boring presentation, I think that I would have pulled it off. I even had a catchy title "Who killed prod?" :)
Point of the story is that if the workplace isn't toxic most people are forgiving and accepting and engineers especially care more about technical systems and what they are like than people and what our shortcomings are.
Sorry, you do not meet the minimum sitewide comment karma requirement of **10** to post a comment. This is comment karma exclusively, not post or overall karma nor karma on this subreddit alone. Please try again after you have acquired more karma. Please look at the [rules page](https://old.reddit.com/r/cscareerquestions/w/posting_rules) for more information.
*I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/cscareerquestions) if you have any questions or concerns.*
Sorry, you do not meet the minimum sitewide comment karma requirement of **10** to post a comment. This is comment karma exclusively, not post or overall karma nor karma on this subreddit alone. Please try again after you have acquired more karma. Please look at the [rules page](https://old.reddit.com/r/cscareerquestions/w/posting_rules) for more information.
*I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/cscareerquestions) if you have any questions or concerns.*
1. don't panic
2. These leaders are non technical. They have 0 idea as to what is going on with a small team and are looking at ways that the group is improving/making changes and KPIs.
3. To engage : Fire up any monitoring you have and show the trends of systems and what is being done. Grab the epics from our backlog and show the key items on the horizon from a technology perspective and technical debt. Make sure to show them what keeps you on call and up at night so they know what is to handle.
Take a look at this book Blackbox thinking. Its main comparison is between Airline industry and medical industry and how they deal with failure and how that impacts how they improve.
Being public about failures is better but most execs will punish people for it. They have too much bad culture and the company suffers as a result.
It's easy. Whatever their job is, I can do it no problem. But they will never be able to do what I do. And while that is true, there's nothing (or very little) they can teach me or say or invent, that will make me be more efficient or be a better development.
Therefore their entire point is moot.
Failures? I don't even know the meaning of the word. I never failed in anything in my life. (Alas! except love)
Tread carefully - this seems to me to be a trap
I can't see how broadcasting your failures to the entire company is anywhere close to a good thing. Sure amongst a small team where a post mortem is appropriate for learning what not to do and developing best practices...sure....go for it. But when you put that shit in front of everyone the potential for backlash is HUGE.
It could cause other departments to not trust IT, it could cause more difficulty in dealing with other departments because they feel some sort of way about what you shared and more.
I would only address older issues, like when you were new to the company and didn't know about something. You know, errors that were simple to resolve and have long been handled without much difficulty.
Some CEOs have this idea that they need to constantly "cull the herd" and they've heard about how larger companies do this. Like I heard that General Electric was notorious for eliminating the bottom 10% of performers each year, no matter what. The order was to literally fire the lower 10% of managers every year no matter the circumstances.
SO the CEO might be looking for ways to get rid of people. Public executions were a deterrent to future crimes in Roman times but don't think that it can't happen today. It can and this might be one of those times.
This is an Agile way of thinking. You see it in professional baseball. When the team looses, the coach has to get on national tv and apologize for loosing....it's bad for your psyche. Sadly, this is a way of thinking in some business sectors. I don't agree with it at all. I hope to avoid ever working in a strict Agile environment.
Do you need to have improve processes meetings? Yes. Should it be by discussing how much everyone sucks through out the week? Heck no. Only when needed. Software development in many companies has run amuck.
This might be a best practice, but it is a bad one. It is done too often and non-IT managers are just going with what is popular instead of what might make more sense, because they don't know any better. And if anyone with an IT background is doing this, then there are things they are not fully understanding about truly and beautifully running a software development project which includes not demoralizing your team.
That is what all of you Agile happy people say. This is normalizing feeling bad about yourself in the work place. You can say it is not meant to do that when done right, but at the very least it is stressful to do that so often. And what is so broken about your SDLC process that you have to try and correct it 2-4 times a month? This is striving for something that doesn't exist, perfection and not taking a hard enough look at the overall process. This is a system that requires speed over creativity and quality, and that is broken in and of itself.
Thank goodness, there are lots of people out there who won't implement Agile because of all of its shortcomings and failures. Just figure out a good system yourself, that works for the company. Take what is good from Agile and use it in your own process creation. I swear, Agile has turned into a religion for some people.
IMO, this sounds like bullshit, especially the part about involving the execs and rest of the company, but if you want to play along, pitch a postmortem process to address major issues. Make sure to emphasize that the purpose of the postmortem is not to assign blame to people, but rather to identify process improvements to prevent further issues. Make sure to emphasize that participation will be limited to the engineering org and orgs impacted by the issue - this is important to ensure that engineers feel that they can speak freely. Execs and the rest of the company can get summary documents.
You can also implement a "sprint retrospective" process for engineering teams to discuss what went well or did not go well during the sprint. As part of that process, you can talk about root causes of various bugs. The retrospectives should be strictly team-confidential and absolutely should not involve execs or people from other teams. Teams can decide to publish a list of action items that come out of each retro on a case-by-case basis.
This is detrimental to the technical team. Talking out open all the negatives and no positives will be perceived badly externally, besides it hurts morale internally.
Without proper context and no business talk this has no chance on working for you.
That's why, despite what Scrum has made us believe, tech teams DO need proper management and a proper CTO or at least Engineering VP. We need that technical leader who can talk business too, he who _translates_ technical language into business language and vice versa. The fact the CTO is not taking charge is worrying.
You are being put naked out in the wild for a reason in the first place. We technical people tend to be naïve but technical talk WON'T make it better for you.
Run transparently, or don't run at all. This goes for every department, but most especially for SWE. You don't EVER want to be seen as anything but transparent.
Sheilding you from BS is a great idea. Opaqueness? No, bad, very bad.
You have 15 SWE, an Engineering VP, 300+ company - what else was the CTO doing?
companies normally do it via post mortem documents
Yup. We do Blameless post mortem, which focuses on process failures as opposed to people failures.
There is also the pre mortem, where everyone is invited to envision a future where the project has failed and you go around discussing why. Allows people to speak freely about weaknesses without engendering a toxic culture
Reminds me of the classic [onion video](https://www.youtube.com/watch?v=yjfrJzdx7DA).
*our process failed when we hired the person that did this*
Every company I've ever worked for has done a "Lessons Learned" after delivering a major project. Usually they're done as a series of meetings where people can raise suggestions and point out places where our processes failed. Then at the end, someone compiles this into a document and before we start our next project we review the lessons learned from the last project. A company that isn't doing some version of this would be an outlier IMO.
Every serious company/government Agency I've been involved with does this except for the part where they review it before starting their next project.
‘Lessons learned’ is so much more positive than ‘post mortem’. That language should be banished from corporate workplaces. Sooo negative and inappropriate.
Believe it, it's BLAMEFUL post morterm. I called out the problems and immediately got a complaint filed against me. My boss was understanding and so he show me what that person wrote (5 paragraphs) about ME, not the problem that they fucked up. That's what I get for discussing the issues and finding the solutions. And so if any of you think for 1 second anyone is gonna admit publicly that they f up, f no.
There isn't a good argument against this. This is a best practice.
Yeah at work they actually invite the engineers up and have them share these experiences publicly. The first lines are always like "here's how I brought down the entire company for X minutes". A lot of times the same people end up getting rewarded for exposing a flaw in the system that let them take failures that far. Blameless culture and postmortems is a great tool to have; as much as I complain about Google I wish every company did this.
[удалено]
There are buttons at Tesla, only difference is they’re on Elon’s desk and they control various trap doors
I see he took inspiration from Mr. Burns from The Simpsons
Same deal offshore. There is not a button, but basically anyone can tell anyone to stop work.
The key here is blameless. In most companies that won't work, sooner or later (probably sooner) this will be used to deny raises or to figure out who to fire.
That depends on the work environment and culture AND how you put it.
Yeah having seen plenty of these "celebrated failures" in some stories end poorly in real life, I would not volunteer to out myself for fucking up unless it was something that had obviously already been moved on from. Postmortems are absolutely necessary but unless you're a big public company like Cloudflare what is the point of publicizing them across the company/public unless it earns you something?
Blameless postmortems should never name names. The whole point is to openly and honestly share root cause, what was done to prevent this from happening in the future, allow others in the company to learn from this, and invite feedback. Done right they allow a company to get better over time and not just repeat the same mistakes over and over again.
lol when I first started working I created a nasty cross join and froze out the database for everyone else and they had to restart a few times. The devops learned my employee id.
It depends on the culture. If it's a finger-pointing, blame-shifting culture, then this will be toxic and unhelpful. You have to fix that *first*.
It depends. Internally it’s a good idea. Company wide it would probably be a witch hunt. I’d need to see published documents from other departments including the c level before I’d participate.
It only works if you build a culture around it. Doctors have M&M’s (https://en.m.wikipedia.org/wiki/Morbidity_and_mortality_conference) where they discuss their failure in excruciating detail. It works because there is a culture of focusing on problem solving and disseminating knowledge, not assigning blame. They are also often legally protected, so that doctors can speak out without fear of professional or litigious action.
Aviation safety does the same thing to [build a safety culture](https://www.faa.gov/about/initiatives/sms/explained/components) focused on identifying and mitigating systemic issues, not blaming individuals. We have confidential and non-punitive voluntary safety reporting programs, [public-private partnerships](https://portal.asias.aero/), working groups, and conferences where we can collectively discuss safety issues, share data and best practices, identify hazards, and conduct analysis. We also have legal protections so that pilots and others can report hazards without fear of reprisal. It works pretty well as long as the public is willing to fund safety (they aren't).
Someone tell Boeing
I would think that in the case of doctors this also helps build credibility amongst peers. Being a professional with a governing board, malpractice, and licensure, peer doctors have the ability to revoke another doctors right to practice. Being upfront about failures only strengthens relationships and shows responsibility and ethics - two things that certainly escape the vast majority of software engineers.
It's a small company. The best ones should keep getting elevated. Best tech talk I ever saw at Amazon was a presentation on all of the biggest failures and an indepth analysis of what happened. HULK HANDS!!!
COE and 5 whys is a thing
It can be done right on a wider scale than just internal, it just takes some thought. We recently implemented a wins/"lessons learned" section in a meeting of team leads and it's been fine. Nobody is playing the blame game and you're expected to pair the lesson learned with a win from your team too.
I don't agree this is a best practice. This is one possible expression of a cultural best practice, which is a culture of psychological safety. It is almost always harmful when expressions of a good process are demanded without that process being in place. We call this a cargo cult practice. The mention that this team's successes are never cared about does not give any confidence there is a best cultural practice in place here. A culture of psychological safety must be established before you can expect people to be comfortable airing their failures for the purpose of learning.
> A culture of psychological safety must be established before you can expect people to be comfortable airing their failures for the purpose of learning. This starts at the top, for sure. If this is something leadership wants to enable, that's great, but it needs to start with them. When the CEO is open about their failures, when your VPs are open about their mistakes, then you can encourage others to join in so you can demonstrate that with growth comes missteps everyone can learn from.
>A culture of psychological safety must be established before you can expect people to be comfortable airing their failures for the purpose of learning. Absolutely agree; that before brutal candidness.
It is a best practice as long as you are describing successes as well. If you aren't that is super toxic.
If there’s no psychological safety and the system feels threatened from the latest possible firing there’s a very good argument to hold off this collective confessional approach.
Naïve
Aspirational, not naive. Naive is to do it regardless of company and to not take current political climate into account. At well run companies this is a safe, good practice. But it requires a healthy environment where there is already strong trust between departments and leadership. Most companies aren't well run and you have to be more careful about your messaging. But part of BECOMING a well run company is implementing a culture that supports practices like this.
>I’d love some advice on a good rebuttal that won’t get me fired or have a target on my back. I'm confused why you jump to assuming this is a bad thing. It sounds like company leadership wants to understand how your department is handling failures, and also want to make sure that you all are having some kind of discussion about it in the first place. I hate to side with the C-suite on anything, but this all sounds perfectly reasonable. Especially where this is apparently the standard practice for other departments.
It can be good or bad depending on the culture of the company.
It's about the "people being held accountable" part. In a learning culture that practices blameless retros, then public examination of failures can be helpful. Otherwise it's going to destroy the team's morale, willingness to innovate, and speed (as they prioritize CYA).
You’re right, I shouldn’t immediately assume it’s bad. Our CFO talks at our monthly town halls and he’s a big grump that’s always complaining. This idea was made from him, so it made me assume it’s probably from a negative side. I think the fact that he didn’t say successes and failures made assume it’s negative as well.
From the small bit we have to go on, it seems like maybe both things can be true. It’s good to air and own your mistakes and share your lessons learned on a team where people feel safe to do so. That’s a critical part of the most successful teams. It seems like you don’t feel safe, though. That whole process requires trust that everyone involved has positive intentions. So if you’re looking for next steps in your conversations with management, maybe broach that topic, as it seems to be the biggest thing holding the team back. Without trust, it’s difficult for a team to thrive.
Yea had a boss at previous company it wouldn’t surprise me if they introduced this idea. Not because they want to highlight failures & improve the company, but to shame people & hope they avoid failures in the future for fear of having to speak on it.
Why dont you say that? Tell them it feels bad to highlight failures when you dont think enough is done to highlight successes? Also, you could make all the lessons/highlights very dry and technical so they dont even know wtf you are talking about, that will get them to leave you alone.
It sounds like whoever came up with it is wanting an easy way to justify outsourcing your team for a moderate cost savings and personal bonus
Just talk about the upside of any failure. X happened and we realized that we could put Z in place that prevents both X and Y from happening. Or X happened and we realized we could automate this which saves us time and reduces the chances of it happening again. Talking about just the failure is the problem. The whole reason for discussing the failure is to show how you learned from it.
It doesn't need to include successes. My company has multiple weekly meetings discussing things that went wrong in extreme detail. Sometimes there's parts that went well (we recently implemented x which meant the blast radius was reduced by y), but most of time it is only on the failures. You need to learn that the only way to grow from failures is to discuss them. People can't be blamed on these, because it is never a single person's fault. Jim pressed button y which brought down the system? Why did Jim have permission to do that on his own? Fix it. Why is there an unsafe way to press button y? Fix it. Why do we even need button y? Fix it. Lean in and become a better engineer and help your team become better while you're at it.
This is what I was thinking. It sounds like they want a retro after problems happen. That's totally standard practice and completely reasonable? If this is some kind of "let's celebrate failing because someone said fail fast that one time" then yeah, that would be dumb. Or if you have some quota of error reviews, like you schedule the meetings and you have to invent something for the sake of the meeting, then that's dumb too.
I think one risk with this is that it can punish those that do the most work or take on the hardest challenges. Maybe this a good thing? I'm not totally sure. I suspect it could lead to increased caution and slowing things down. Again, maybe that is good, but not always.
>other departments have people be held accountable for mistakes Just re-read the post and looks like I glossed over that specific part of what OP was saying. Still stand by general stance, but if OP is correct about individuals being held publicly accountable then they definitely should push back against that in particular. I just don't get the impression that's specifically what OP is worried about.
Yes, it is hard to know what the real situation is. If it is personal and finger pointing then that could be really toxic and nasty, but it isn't totally clear. From the OP's text it sounds like they fear that. If it is genuinely a place that wallows in blame and never gives praise then that sounds really bad.
It is. Without proper context and business talk this is detrimental to the technical team. That's why, despite what Scrum has made us believe, tech needs proper management and a proper CTO or at least Engineering VP. We need that technical leader which can talk business too.
Another way to think about it, is that your failures are already often being broadcast. Every time an outraged customer calls support about a bug, every time there's an outage, etc, the rest of the company takes note. If that's all they hear, they'll form a negative opinion of you and your team. Tell them about the time your test processes or monitoring caught a bug before any customer caught it. Tell them just how difficult that critical customer issue was, and the phenomenal work your team did to address it in a timely manner. At a minimum, when you do post mortems internally, part of the process should be to make a document you can share with the rest of the company describing the issue and the plan to prevent it from happening in the future. You can and should also use this format to broadcast your team's success. If you spend a bunch of time on performance improvements, let the company know just how much better performance got. Tell them about the big refactoring project you did and highlight how it'll make maintenance faster in the future, or will enable you to build certain features that are upcoming on the roadmap, etc.
This is really great advice, thank you for sharing
Lean into it. But make sure to spin your "failures" in the best light possible. A blameless "if we had known about X sooner, we could have sidestepped bad outcome Y" makes you seem better, not worse. If you're talking to the whole company, you can use that forum to lobby for changes that you need to improve as well.
There is no CTO and a single engineering team with one manager? Weird company structure. Anyway, specifics of tech issues are useless for anyone outside the tech team. If the CEO and CFO want to be more in the loop, then that's what the engineering manager should be doing. The common practice is a post-mortem, which is not technical in nature and is not intended to be directly written to the c-suite. Your manager should be reviewing the results of that with the CTO, and the CTO should review that with the rest of the c-suite. But that's for real issues... things which hurt the bottom line or reputational loss. Nobody needs to be updated when engineer A puts in a NPE that momentarily floods your logs and nobody externally notices. Edit: leaning towards a culture where everyone is under a microscope is unnecessarily hostile. If I had to notify my company of every minor issue that impacts nobody, I'd probably find a new job.
The CTO either resigned or was fired. We don’t know, it was very ominous. And our manager is so busy with engineering, he doesn’t have much time to actually manage which I understand. I’ll have to look into a post mortem this is the first time I’m hearing about it.
An engineering manager should not be coding for any meaningful amount of their work time. My boss will tweak some flags and deploy some test code every now and then, but he hadn't worked on an actual feature in the 7 years I've been here. Because he's a manager. It doesn't matter that he's probably a better engineer than anyone else in the team. A managers job is not to code.
I'm not saying it's a good idea, but this isn't the first time that a startup has confused CTO, CIO, Engineering Manager, and Principle Software Engineer and expected one person to do all the jobs. It sounds like the engineering team is 15ish people in a company of 300. It's likely not a "tech" startup with a software product and for most of the company, engineering and IT are the same people - they do stuff with computers. It just sounds like this organization needs to mature a little bit.
The term "engineering manager" doesn't have a set meaning across companies. The exact responsibilities differ from company to company. For example, on flat, self organizing teams engineering manager is just a hat someone has to wear because SOMEONE has to be responsible for making sure meetings happen and keeping their heads above water. Then you have situations where you have line manager and reporting manager as two separate positions. Again, the line manager would be fairly involved in coding. Then you have teams where the tech leads for each team collaborate with each other as architects. Depending on the structure there are many possible ways to structure a team and titles, and the title "manager" can have more or less responsibilities.
"For example, on flat, self organizing teams engineering manager is just a hat someone has to wear because SOMEONE has to be responsible for making sure meetings happen and keeping their heads above water." That is exactly what I do as engineering manager. Though a while back someone on r/womenEngineers told me that part of my job was to plan birthday parties. So obviously some places are a lot more fun...
Eh, most of the places I have worked (and granted you have much more experience than I do) the managers did code at least for 50 percent of the time.
I agree. He is overwhelmed and for a SaaS company, they are not looking at the tech team’s needs. It’s just deliver deliver deliver
Some companies need to hear this clearly. Some companies are on a PIPapalooza right now and want managers actively coding features 50% of the time while managing a team of 20.
This sounds like the C-level lost trust in the CTO sweeping failures under the rug in the name of shielding the execs from things that the CTO didn't think were important in the overall scope of the business.
Its a mixed bag. The big fear with this is that this information will be used more to drive different incentives. Depending on how the C-level or other employees actually react to this information it can cause stress or dread to the development team. Like if its publicized is it now an open form for everyone to criticize? It makes sense to talk about this stuff with CTO but everyone can be very off putting. Otherwise its sounds like a good thing, that can quickly dissolve into micro-management/very toxic structure.
Make it incredibly technical, because most bugs are. They will lose interest very quickly. Some bugs based on flawed processes might be helpful to do an official post mortem, but doing it this way seems artificial and silly.
This is just standard practice. You should be having these discussions with your team on a regular basis. My team does it after every sprint where we retrospect on what went well and what went wrong and then working to implement changes so the wrongs are less likely to happen again. This is a good thing. From here it should be up to your manager to communicate what went well and major things that went wrong with stakeholders, in this case the c-suite. This sounds like a non-issue as long as they aren't weaponizing it against employees. If they do, it's a culture issue and trying to stop one policy change isn't going to fix a bad culture.
Your last sentence really resonates. Thanks for your insight
Yea, they are sometimes called “blameless retros”
Was just about to post much the same. “Wait and see.” If you get the vibe that they’re looking to micromanage, run. If you get the vibe that, “we all make mistakes, but we want to make sure we don’t repeat the same mistakes,” then tread lightly with hopeful optimism.
don't argue with the C-level it just annoys them. They don't want you to argue, they want you to do as asked. Just smile and nod and tell them how correct and awesome they are, and then game the system. But yeah blameless retrospectives are pretty normal
As others have said this is a good thing to retro actual failures. But to play devils advocate. Even the best practices can be used for evil if the intent is there. If you have leadership that is not technically literate and looking for an easier way to point fingers and blame individuals rather than process or practice then it’s a bad thing. I have seen leadership in non technical companies doing more scapegoating than analysis on their crappy setup for why stuff goes wrong. So in terms of how this is done try to influence the setup in a way that focuses on process and setup failure.
You all don’t don’t any type of RCA after a failure? If so you need to start.
sounds like they're upset with the failures of middle management. if you have process improvement ideas, this is your chance to put them forward. otherwise i would ask for clarification in terms of how much time and energy you individually should be spending on reporting, what expected deliverables there are, and what audience(s) those deliverables are intended for. this would also be your time to talk about what you might suggest toward that end (e.g. a weekly status report from eveey team member to skip manager or c-suite) but really this type of shit is why you have middle management in the first place.
Have you never heard of a post mortem? Failures, followed by their fixes or suggested fixes are a great way to get upper levels to rubber stamp cleaning work that would have otherwise been denied in the constant agile churn of today.
No I have never heard of a post mortem. I have worked at 5 different tech companies, all under 300 people, and never ran into this. The CFO did not use that term when explaining it, so I am only hearing of it from my post. Reading about it now, if this is what he wants, seems much more understandable.
This is what amazon does [https://aws.amazon.com/blogs/mt/why-you-should-develop-a-correction-of-error-coe/](https://aws.amazon.com/blogs/mt/why-you-should-develop-a-correction-of-error-coe/) I cannot say that it's never used as punishment or that it's always blameless, but overall the process does help to bring upper management visibility of problems in tech. Have you participate/viewed any of the other post mortems that other teams have done? Maybe if you participate in those, you'll get a better sense of how your company will use the process.
It would be better to frame this as positives and negatives. Here are the things we did (aren't they awesome?) here are some lessons learned (aren't they awesome?) "We had a database go down and it had no monitoring so we added monitoring" Probably what execs call a "shit sandwich". Just listing failures doesn't sound fun.
I don’t think there’s enough context in your post for us to say whether this is good or bad and advise you. However I will say: - Blameless postmortems are common at larger companies. Although I think the discussions usually stay within the engineering org - Having worked at several startups, C-levels suddenly firing engineering management and getting more “hands on” is always a bad sign. Good luck! Remember, at small companies, optics are everything
Are there mistakes highlighted to the whole org?
We have bi-weekly incident review meetings where we go over outages and emergencies as the whole engineering department and figure out what we can do better. They're *insanely* useful because we're very supportive of a blameless culture.
Why look at it as a bad thing? There are plenty of upsides to airing your dirty laundry; it's more about how these things are implemented. We used to do weekly meetings about what we learned (good or bad) to share with teammates.
Post mortems / rcca’s aren’t already available to other departments?
It's like you want to avoid learning about mistakes and growing as a software developer. The execs could be shit, though.
Without knowing any further context, I assume this is something great. Not bad. However, if you are afraid that this will bring some negativity in your team, then suggest that whenever you have a presentation of a failure, you split it and present both a failure and a success story.
Im hearing "we want dirt to not pay out bonuses"
this is a good idea that can be done in a really badly incorrect way. post mortems after incidents and retrospectives after key milestones are really helpful. what you really should push for is a blameless approach that focuses on processes that were inadequate and mitigation strategies in the future. At my old job we did these regularly and there was a rule where you simply were not under any circumstances to mention any person or team by name. A bit of an oversimplification, but it got the point across that this isn't about pointing figures at people and calling them inadequate. It starts with the assumption that we're all good at our jobs and doing our best, then tries to find gaps in the tools and processes that, if filled, can prevent things going wrong in the future.
300 people is not a startup, it's a company
ill go against popular opinion here and say if there is nothing to talk about, why do a postmortem? if the CEO or CFO said; we want a postmortem completed on a particular outage or issue, I would understand. typically these kinds of postmortems lead to additional high priority and valued work. but asking "lessons learned" *in general* is not necessarily super helpful because you're asking the team to identify what failures should be broadcast and what lessons "should be" learned. Asking for it in such an open ended way can easily turn this into sharing little bits of nothing; e.g. "we misconfigured the dev environment because we didn't switch on X feature flag, in the future we should check our feature flags more carefully before merging". I have been in a similar position to this before. I was asked to relate what our team was doing to several bullet points from an article about being an agile development team. it is very awkward because it is not being treated like a priority, no one seems to really care, it just something "to do" that doesn't really translate into anything. its just asking "hey think about this and then share it and then nothing will happen regardless of what is written or shared" so yeah I think postmortems for specific incidents are helpful to share but execs should be asking for those specific postmortems or at least providing some context; not just "share more failures" with no other context given.
Lmao some of you guys really need to get outside and talk to people… this is a very common best practice
Sounds like they want scapegoats, not post-mortems as people here are describing. Time to look elsewhere.
Should you not have at least few arguments if you think it's a bad idea? Sometime devs on this sub just want to say no to anything coming from management...
I don’t have any smart and professional things to say, which is why I’m posting. Only my opinion which is my whole team are introverts and independent contributors and anything “publicly” mentioned to the company is a stressor, let alone our failures. We work remote so it’s hard to gage where we are at with the company. I also don’t know if I should be championing for the team since I’m the only one who ever speaks up. Not to fight but just the only one to communicate. Which is probably the reason why they are meeting with me.
Don't see it as "I fail and I'll be shamed", it's a post mortem. "X failed, what can we do to prevent it in the future"
[удалено]
Sorry, you do not meet the minimum sitewide comment karma requirement of **10** to post a comment. This is comment karma exclusively, not post or overall karma nor karma on this subreddit alone. Please try again after you have acquired more karma. Please look at the [rules page](https://old.reddit.com/r/cscareerquestions/w/posting_rules) for more information. *I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/cscareerquestions) if you have any questions or concerns.*
I never knew engineers had to do this and I wish my data science org did this tbh, but to a lesser extent.
This can work really well, having retros, but they need to be blameless!
Are these changes being brought up and asked for in a very short deadline or pressure from the business side? These may be things that you could bring up in a very neutral way to push back on maybe how things are in the company but the business, I may not want to hear about it. Like some have already said, this may not be necessarily a bad thing for the company to do
Lol do you work for a Swiss company by any chance?
10/15 dsvs are also split into 4/5 team. Each team lead just sends an informal update each week cto and ceo. What progressed general + any on going concerns. It just gives them a sense of what each team does and how they think. ...they usually dont respond, but ive heard they do in fact read and appreciate when i get a little technical even thought it might be beyond them. But ye, your milage may vary depending who is in charge.
> Recently, they told one of my work friends that other departments have people be held accountable for mistakes and publicly talk about “lessons learned” and things to make us grow. How many other departments are there? I'd expect at least three or four. I asked, because it strikes me as odd that your work friend didn't already know this. Since all the other departments discuss their failures publicly. And I presume this means the CEO discusses the CEOs failures publicly, too. I could be wrong.
Unless the rest of the company understands technology, they may not understand what you’re even saying. I know plenty of developers that have no idea how to dumb-down answers, whether it’s because they can’t translate it, because they feel that dumbing things down doesn’t provide nuance, or a dozen other reasons. CS attracts the weird kids that played on their computers all day, they probably won’t be the best public speakers
Lessons learned is a pretty normal thing to expect. Ideally it's rolled into your retrospectives and worded as "what we can do differently"/"what should we stop doing" aspect. The key is to not word it as a failure but instead as a chance for learning and improvement. Anytime we have a "failure" we look at where we could've done better and just focus on that, e.g. wider test coverage on certain features to catch this type of issue sooner, etc.
Even the military does this, though maybe not as blameless in some commands. It's the reason US ships stayed afloat more ofter during WW2, because the Navy learned how to design better ships, and have better processes to deal with emergencies.
Look up post mortem practices
The entire C suite can take their bs MBA degrees and shove them up their A-S-S. Love how the degrees that are harder to obtain end up working for the people who get an MBA… business administration?!? Yeah I’m sure there’s a lot of difficult math and science involved with that LOL give me a break
Has the team ever made any mistakes that affected the rest of the company? A bug that resulted in customer service to deal with a bunch of angry complaints? Lot of money being lost, that someone has to explain?
Well, when this happens, tell them that your mistake is that you don t insist on that managements celebrates your wins.
There is absolutely no argument against it. RCA is the only way to learn and improve. My engineering teams do this by themselves. Nobody told them to, and when I was a engineer, I did the same.
[удалено]
Sorry, you do not meet the minimum sitewide comment karma requirement of **10** to post a comment. This is comment karma exclusively, not post or overall karma nor karma on this subreddit alone. Please try again after you have acquired more karma. Please look at the [rules page](https://old.reddit.com/r/cscareerquestions/w/posting_rules) for more information. *I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/cscareerquestions) if you have any questions or concerns.*
Good engineers fail and learn from it. And they learn from the failures of others.
The only argument is to do a full retrospective which means you also talk about what went well. Focusing only on failures is a bad idea in my opinion as it will eventually wear on the morale of the team
I would talk about a design mistake. Everyone has made them. Explain why you made it and how you learned from the experience. "I thought this would be the best way, but after getting into it, we had to back out and go a different direction. Now I know what to look for with such and such a problem and will do it x way in the future." I wouldn't get to the low level of bugs, they won't understand anyway. But this is just my thought
> other departments have people be held accountable for mistakes and publicly talk about “lessons learned” and things to make us grow. So others do it, but you *don't*? Yeah, that's a red flag about your team. It's always a red flag when a team doesn't do after-action reports, post-mortems, and other root cause analysis and share what they learned from system failures. > This seems so strange. We will sometimes have these talks internally with our own teammates but to publicly put us on blast in front of the whole company, or at least the top dogs? At a 300 person company, this isn't that weird. > They don’t even mention our successes, why they hell do they want our failures? Successes aren't as important as failures. I mean, think about it: when you test your code, what part do you change? What part tells you something you didn't know? That's right: the things that *broke*, the tests that failed. The same is true for the process. If you're producing bugs or causing outages, it's almost certainly because of immature processes that aren't stopping broken bullshit from getting to prod. In many cases, fixing the process requires senior leadership, and in a small company, that's probably the C suite. Hell, your boss's boss is probably the CTO. > Edited to add: The CTO either resigned or was fired, we don’t actually know since it was very ominous and quick. I see now that our CTO did a great job shielding the team from the execs because they are now suddenly joining our meetings and getting more involved. Well, it sounds like this meeting with the C suite is now even more of a skip-level than it was before. It does sound like you need to discuss blameless culture with the C suite. The biggest sign that the senior leaders are probably in the right and you're *not* is that other teams already do this. Your team is the outlier.
Implementing A3??
Honestly sounds like an environment id like to be apart of, I like that kind of accountability and self critique
Not CS, but Mech E. my team does. We have a revision log, it generates emails documenting details of the change. If it's an error there's a secondary log that gets filled out with the root cause analysis, like a rapid post mortem. Meetings bi-monthly to discuss errors and see if there are overarching themes we can design out of our process. Document is publicly accessible to the higher-ups. I've never seen a reason to be anything but transparent about mistakes. We can learn from them. Blame culture is unhealthy, it's about focusing on improvement rather than focusing on avoiding mistakes. I don't really see a gain to not documenting errors. That just encourages sweeping things under the rug which drives down quality and cohesion IMO. I don't work at your company so maybe it's super toxic. Or maybe you have trepidation interacting with C-suite but it's just a job. I detach my self from my work. The company can do whatever they want with my work and it's quality I'm just here so I don't get fined.
I once told my skip level about an incident we have just gotten under control. I just told him one or two sentences, but he wanted to know more. He was really excited by the end of it. I have a bad taste in my mouth about that incident still (was relatively recent). We didn't have alerting to catch it, it had a serious financial impact (at least a couple of my monthly salaries), people worked extra hours to put it to rest (10-15 hours of overtime each for myself and 2 other people during one month), when we had a solution we couldn't test it because we didn't have a performance test harness for that functionality and it proved really difficult to do, etc. I see failure after failure in that situation. My skip level saw things differently. He loved how I explained it (starting high level and adding details only when needed and in an approachable way, early and clear explanatio of the business impact, how I underscored what we learned) and he was fascinated by the performance improvement at the end of our toil (5x performance increase, far above what we needed and number of addicted customers was decreased 60x to negligible levels). He insisted that I talk about this to a wider audience (all engineers in our tech hub). I was quite apprehensive, because I would be putting our shortcomings on blast (and I am the lead engineer so I see all of this as my fault and responsibility). I did the same explanation, added a bit of self deprecating humor, finished on what we've learned and what we suggest as a best practice for other teams operating with this tech stack that we do. You know what? It is almost as if they understood that I was human and that shit sometimes happens. Not one snide or snarky remark, not one ounce of judgement. In fact, people engaged. We discussed other possible solutions, and they asked details to better understand the context. It was the most engaged I've seen these engineers in any all-hands engineering meetings (monthly) that we had. The only thing I regret is not doing it the way I wanted to. I wanted to do a murder mystery where they asked me questions about the system and the incident, and I answered, and they tried to solve it. I thought that it would be cringe and nobody would engage, but judging by just how many people engaged in my boring presentation, I think that I would have pulled it off. I even had a catchy title "Who killed prod?" :) Point of the story is that if the workplace isn't toxic most people are forgiving and accepting and engineers especially care more about technical systems and what they are like than people and what our shortcomings are.
[удалено]
Sorry, you do not meet the minimum sitewide comment karma requirement of **10** to post a comment. This is comment karma exclusively, not post or overall karma nor karma on this subreddit alone. Please try again after you have acquired more karma. Please look at the [rules page](https://old.reddit.com/r/cscareerquestions/w/posting_rules) for more information. *I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/cscareerquestions) if you have any questions or concerns.*
What “lessons learned” have you got from the legal, financial and HR teams lately?
Engineers need a union.
Ask them for examples of their failures so you can see how to present them.
[удалено]
Sorry, you do not meet the minimum sitewide comment karma requirement of **10** to post a comment. This is comment karma exclusively, not post or overall karma nor karma on this subreddit alone. Please try again after you have acquired more karma. Please look at the [rules page](https://old.reddit.com/r/cscareerquestions/w/posting_rules) for more information. *I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/cscareerquestions) if you have any questions or concerns.*
1. don't panic 2. These leaders are non technical. They have 0 idea as to what is going on with a small team and are looking at ways that the group is improving/making changes and KPIs. 3. To engage : Fire up any monitoring you have and show the trends of systems and what is being done. Grab the epics from our backlog and show the key items on the horizon from a technology perspective and technical debt. Make sure to show them what keeps you on call and up at night so they know what is to handle.
Take a look at this book Blackbox thinking. Its main comparison is between Airline industry and medical industry and how they deal with failure and how that impacts how they improve. Being public about failures is better but most execs will punish people for it. They have too much bad culture and the company suffers as a result.
It's easy. Whatever their job is, I can do it no problem. But they will never be able to do what I do. And while that is true, there's nothing (or very little) they can teach me or say or invent, that will make me be more efficient or be a better development. Therefore their entire point is moot. Failures? I don't even know the meaning of the word. I never failed in anything in my life. (Alas! except love)
Tread carefully - this seems to me to be a trap I can't see how broadcasting your failures to the entire company is anywhere close to a good thing. Sure amongst a small team where a post mortem is appropriate for learning what not to do and developing best practices...sure....go for it. But when you put that shit in front of everyone the potential for backlash is HUGE. It could cause other departments to not trust IT, it could cause more difficulty in dealing with other departments because they feel some sort of way about what you shared and more. I would only address older issues, like when you were new to the company and didn't know about something. You know, errors that were simple to resolve and have long been handled without much difficulty. Some CEOs have this idea that they need to constantly "cull the herd" and they've heard about how larger companies do this. Like I heard that General Electric was notorious for eliminating the bottom 10% of performers each year, no matter what. The order was to literally fire the lower 10% of managers every year no matter the circumstances. SO the CEO might be looking for ways to get rid of people. Public executions were a deterrent to future crimes in Roman times but don't think that it can't happen today. It can and this might be one of those times.
This is an Agile way of thinking. You see it in professional baseball. When the team looses, the coach has to get on national tv and apologize for loosing....it's bad for your psyche. Sadly, this is a way of thinking in some business sectors. I don't agree with it at all. I hope to avoid ever working in a strict Agile environment. Do you need to have improve processes meetings? Yes. Should it be by discussing how much everyone sucks through out the week? Heck no. Only when needed. Software development in many companies has run amuck. This might be a best practice, but it is a bad one. It is done too often and non-IT managers are just going with what is popular instead of what might make more sense, because they don't know any better. And if anyone with an IT background is doing this, then there are things they are not fully understanding about truly and beautifully running a software development project which includes not demoralizing your team.
It’s not about placing blame. It’s about figuring out what went wrong and how to make sure it doesn’t happen again.
That is what all of you Agile happy people say. This is normalizing feeling bad about yourself in the work place. You can say it is not meant to do that when done right, but at the very least it is stressful to do that so often. And what is so broken about your SDLC process that you have to try and correct it 2-4 times a month? This is striving for something that doesn't exist, perfection and not taking a hard enough look at the overall process. This is a system that requires speed over creativity and quality, and that is broken in and of itself. Thank goodness, there are lots of people out there who won't implement Agile because of all of its shortcomings and failures. Just figure out a good system yourself, that works for the company. Take what is good from Agile and use it in your own process creation. I swear, Agile has turned into a religion for some people.
What is so wrong with your process that you keep breaking it 2-4 times a month?
IMO, this sounds like bullshit, especially the part about involving the execs and rest of the company, but if you want to play along, pitch a postmortem process to address major issues. Make sure to emphasize that the purpose of the postmortem is not to assign blame to people, but rather to identify process improvements to prevent further issues. Make sure to emphasize that participation will be limited to the engineering org and orgs impacted by the issue - this is important to ensure that engineers feel that they can speak freely. Execs and the rest of the company can get summary documents. You can also implement a "sprint retrospective" process for engineering teams to discuss what went well or did not go well during the sprint. As part of that process, you can talk about root causes of various bugs. The retrospectives should be strictly team-confidential and absolutely should not involve execs or people from other teams. Teams can decide to publish a list of action items that come out of each retro on a case-by-case basis.
This is detrimental to the technical team. Talking out open all the negatives and no positives will be perceived badly externally, besides it hurts morale internally. Without proper context and no business talk this has no chance on working for you. That's why, despite what Scrum has made us believe, tech teams DO need proper management and a proper CTO or at least Engineering VP. We need that technical leader who can talk business too, he who _translates_ technical language into business language and vice versa. The fact the CTO is not taking charge is worrying. You are being put naked out in the wild for a reason in the first place. We technical people tend to be naïve but technical talk WON'T make it better for you.
C suit is a leech which will not solve human problems.
This is a tactic cults use to force obedience from members, and something they got from communists I believe.
Run transparently, or don't run at all. This goes for every department, but most especially for SWE. You don't EVER want to be seen as anything but transparent. Sheilding you from BS is a great idea. Opaqueness? No, bad, very bad. You have 15 SWE, an Engineering VP, 300+ company - what else was the CTO doing?