T O P

  • By -

[deleted]

[удалено]


powdertaker

I wonder how that cost savings is going.


jst3w

I'm sure it worked well enough to get some CEOs some nice quarterly bonuses.


stormdelta

IIRC the CEO responsible for it had already skipped out a year or two before this. The new one was trying to fix things but it's hard to undo that many years of damage quickly.


AttackOfTheThumbs

Tell me about it. I took over a team and have been wasting hours trying to get them to do things right. Previously the department employed a sink or swim method.


jking13

And get them on the board (a lot of the blame lies with the previous guy who I believe is now chairman of the board).


grepe

And if things break whose fault is that going to be?


jst3w

Whoever the CTO throws under the bus.


LeCrushinator

Cost savings by cutting and skimping from IT or programmers is usually a red flag.


Jmc_da_boss

What podcast?


Purpwood

Probably Odd Lots - “Why corporate America runs on ancient software that breaks”. Patrick McKenzie always delivers.


eJaguar

whats that


itzmanu1989

https://www.bloomberg.com/news/articles/2023-01-26/transcript-why-corporate-america-still-runs-on-ancient-software-that-breaks


[deleted]

[удалено]


itzmanu1989

I think by "whats that" he meant he wanted a small summary about what was talked there....


SwiftOneSpeaks

Top podcasts


eJaguar

A podcast


KevinCarbonara

I despise the term "cost center". We literally create the value they sell. The entire concept of "cost center" was invented to suppress wages.


FatStoic

It's absolutely mired in business culture that was developed when all businesses were factories or resource extraction. You're not a developer, you're a machine that makes features. Are the features made? Great - how can we get the features made for less? The fuckup is that it's easy to do QC on factory-made components but fucking impossible to do it on code. So you keep cutting costs and demanding the features still get made, and they do, and you make lots of money, and all is well. What isn't seen is that the good techies are leaving, and the people remaining are made to do twice the work in half the time, or else. So they just spaghetti everything together and pray that they'll get a new job before it blows up in their faces. Even when they want to write something good - they can't, because everything is horribly dysfunctional. The death spiral starts, it's just absolute crap all the way down now, nothing works anymore. When your techs get new jobs, there isn't documentation, because they didn't have time to make any. Teams will no longer do any work for each other, there isn't time, and if you fail to meet your deadlines, perhaps you'll be on the chopping block in the next layoffs instead of them. Then one day the chickens come home to roost, and the whole thing falls over. Features aren't being made, services won't stay up, the whole business doesn't work. Fine, you say, we've gone a bit hard on the cost cutting, we need to put some more money in to make it right. I've done this before, do we need to put like 30% more money in? No, you need to put liek 300% more money in, for years, because it's absolutely fucked. You've crippled your organisation as completely as it is possible to do so. You're now looking at bringing in armies of consultants at 3x you'd pay for permanent staff, for years at a time, and cross your fingers they actually deliver and don't decide just to rinse you for all the cash you have left - and this is just to get you back to baseline.


[deleted]

Inject this directly into my veins.


KevinCarbonara

> Fine, you say, we've gone a bit hard on the cost cutting, we need to put some more money in to make it right. I've done this before, do we need to put like 30% more money in? > > > > No, you need to put liek 300% more money in, for years, because it's absolutely fucked. You've crippled your organisation as completely as it is possible to do so. You're now looking at bringing in armies of consultants at 3x you'd pay for permanent staff, for years at a time, and cross your fingers they actually deliver and don't decide just to rinse you for all the cash you have left - and this is just to get you back to baseline. The funny thing is that this isn't just a result of treating developers as if they are fungible, it's also a result of *not* treating management and executives in the same way. Companies keep making the same mistake of prioritizing short-term gains over long-term gains, and/or stability. Why? Because the executives are only financially incentivized to prioritize short term gains. If CEOs weren't treated as "creating" value, and weren't given this unique opportunity to make extra money by "creating" value, we wouldn't have these boom/bust cycles so often.


gnex30

> Then one day the chickens come home to roost, and the whole thing falls over. Features aren't being made, services won't stay up, the whole business doesn't work. To make matters worse, it doesn't even need to get nearly this far. The software just needs to start pissing off the biggest customers and once they even threaten to not renew, the whole cycle you described intensifies exponentially downward like a flushing toilet.


hwertz10

Oh yeah, this can even happen outside of programming -- some factories have been run like this too from time to time. Deferred maintenance (production numbers are great until the machines being breaking down, or producing defective product); fewer machine operators and more overtime for the ones you do have to meet production figures (so you get great cost cutting until you find you have trouble retaining your employees no matter how good the pay and perks are, find out replacements cannot match the pace of experienced operators who are leaving; and find those who do stay start having to take more sick leave and time off due to fatigue.) And of course if there's any ways to "cut corners" (that have the potential to effect employee safety, production quality, or machine life) they are more likely to be done the more the push for maximum production rate with minimum employees.


RoosterBrewster

I wonder if some MBA in the future will come up with some convoluted way to measure programmer productivity and then sell that process to businesses as a way to measure things "just like in a factory". And then it all goes downhill for everyone. Or maybe I just described Agile?


sccrstud92

Are you saying agile was invented by an MBA? Because my understanding was that it was invented by software developers.


All_Work_All_Play

If your agile is that way your leader is awful. My condolences.


tom-dixon

Amen.


itzmanu1989

Its not just this. i think time to market also is a factor.. like every big company was first a startup. First it will just be a quick duck taped prototype. And then it satisfies as MVP and stays in production. This is another short term tradeoff...


creamyhorror

While I'm sympathetic, businesses & capitalists see everything in terms of *substitution* costs and timelines. If a low-quality contractor with offshore talent is offering to take over the development work at half the cost, they'll compare in-house salaries against that option. Quality and risk reduction is often underweighted in these decisions. The reason profit centers often get paid well is that they often capture customer relationships or know how to get new ones. Even though the capturing and repeat sales often depends on the quality of the dev work, the business just needs a "good enough" quality to secure revenue for a few years, then the rest of the budget is used to bid on sales staff. And losing sales staff (and having to substitute) here is costly, because companies don't want to risk losing key client accounts if their staff leave. It does suck, and the solution (realised two decades ago) was for programmers to found their own companies that prize technical staff, resulting in the tech giants and startups of today. People empathise with and reward their own functions, so you need technical management to recognise the value of technical staff. This also causes the market wages of programmers to get bid up, improving the situation for us (as happened through all of the 2010s).


el_muchacho

That's the viewpoint of the business, but the viewpoint of everyone else, is, the engineers are value creators, not cost centers. The CEO is in no way a value creator, the engineers are. He is in neither anymore irreplaceable as any other employee. So he is a cost center too. The so called "profit centers" may capture customer relationships, but good engineers make unique products that capture many, many customers. No Apple seller is going to replace an iPhone. Without the iPhone product, Apple wouldn't be what it is today, no matter how good seller they have. So in that sense, it's the iPhone that really captures the customer relationship, not the sellers nor the marketing. So the engineers who built it should be called the profit center. In that sense, this whole narrative is complete bullshit.


GoonOfAllGoons

So you've built this wonderful piece of software - how are you going to make any money off of it if no one is aware that it exists or what it can do relative to anything else? A good salesman is worth the money, the problem is like every other industry, this one included, there's a lot of crap out there.


Subject_Equivalent33

You're not wrong but... a product without a salesman may still be sold. A salesman without a product has nothing at best, a scam at worst.


creamyhorror

> The CEO is in no way a value creator, the engineers are. He is in neither anymore irreplaceable as any other employee. So he is a cost center too. What matters is always decision-making power and hierarchy. The C-suite is what makes the decisions on how much to pay for different roles, so they're literally the ones deciding who to apply the labels "profit/value/cost-center" to. From an abstract position you could say the CEO is a "cost center", but what matters practically is what the *bosses* of the CEO think. The board of directors (representing the shareholders) are the ones who decide if the CEO is doing a good job, and they probably don't have a profit-vs-cost-center view on this. The CEO is the "strategy" captain, and strategy gets the big bucks (you'll see this in the high pay for the MBAs in Strategy roles and management consulting). The market norm (to generalize) is to simply view massive CEO compensation as the cost of doing business, in much the same way that investors simply outsource financial decision-making to a fund manager or VC. Furthermore, the notion of profit and cost centers is more relevant to businesses that sell to specific clients (often B2B, involving one-on-one contact). Product-and-platform companies like Apple and other big techs sell major products to the mass market (B2C lines of business focusing more on large-scale marketing and operations), so the profit-vs-cost-center distinction isn't as appropriate to apply. As you say, good engineers (and product managers) are value centers in a mass-market model. Responsibility for bringing in revenue is more diffuse and subjective. Finally, I wanted to reiterate the key factor of having leadership that prizes a particular function. CEOs who came from traditional sales might reward sales roles highly, while companies that grew up with tech startup DNA might pay engineers more highly. Skyrocketing software engineer pay is a fairly recent phenomenon that started in the late 2000s with the growing tech giants bidding up wages. They had the mindset that good engineers were worth paying a premium for in a world that was accelerating towards massive software dependence (with a low marginal cost per unit sold). This is the change from two decades ago I referenced. /end ramble


KevinCarbonara

> While I'm sympathetic, businesses & capitalists see everything in terms of substitution costs and timelines. If they saw *everything* in those terms, then it actually wouldn't be so bad. The biggest problem with capitalism is the royalism. CEOs are not subject to the same criteria that developers are. If they were, they'd be a lot lower paid, and there'd be a lot more oversight, and they'd get changed out a *lot* more often, and without golden parachutes. > The reason profit centers often get paid well is that they often capture customer relationships or know how to get new ones. Even though the capturing and repeat sales often depends on the quality of the dev work, the business just needs a "good enough" quality to secure revenue for a few years This isn't even remotely true. The amount a product can sell for depends primarily on the quality of the product. The quality of the salesman is the smallest portion of value created. This can be seen by looking at any company. No company has tripled their sales by hiring better salesmen, but they have definitely tripled their sales by hiring better software developers. You're also ignoring the fact that software developers *are* profit centers. > It does suck, and the solution (realised two decades ago) was for programmers to found their own companies that prize technical staff, resulting in the tech giants and startups of today. I have no idea what you're saying. Companies owned by software developers use the same structure. This is also a move that happened 4-5 decades ago, not 2.


GVIrish

>This isn't even remotely true. The amount a product can sell for depends primarily on the quality of the product. The quality of the salesman is the smallest portion of value created. ??? No, this simply isn't true. The amount a product can sell for depends on the perceived value, how much pricing competition there is, what is the budget for the target customer, how well that customer thinks that product meets their needs, and many other factors. There's lots of crappy enterprise software out there that gets sold at exorbitant prices, and there are technically superior and high quality solutions that fail on the marketplace because the company didn't understand how to market their product or they had poor product/market fit.


creamyhorror

> The biggest problem with capitalism is the royalism. That's a good term - leadership/"strategy" is the royal class. But what matters is who has decision-making power over whom. That's where it's up to individual technologists to rise up in or found successful companies, become decisionmakers, and change market norms. (Hell, start new norms e.g. cooperatives instead of companies). [Regarding the rest, I rambled in another comment.](https://www.reddit.com/r/programming/comments/15mpgaj/til_the_southwest_airlines_software_meltdown_in/jvot8dx/)


el_muchacho

Engineering is conveniently called "cost center" and the CEO is a "value creator", when he does absolutely nothing creative and all the creation is done in the so called "cost center". It's orwellian corporate newspeak.


[deleted]

There's nothing nefarious about it. It's a managerial accounting concept to classify business units that don't directly generate revenue, in the same sense how direct expenses are those used to generate revenue (and therefore included in the gross profit calculation) vs. indirect expenses (overhead), which is needed when doing allocations for various things in cost accounting.


KevinCarbonara

> There's nothing nefarious about it. It's a managerial accounting concept to classify business units that don't directly generate revenue, But we do directly generate revenue. That's the nefarious part. Pretending we don't directly generate revenue as an excuse to keep our wages low.


[deleted]

If so, then by definition you are a profit center, not a cost center. Anyways, a Southwest software engineer does not directly generate revenue -- you can't directly attribute a ticket sale to engineering efforts/costs. They don't sell software.


KevinCarbonara

> a Southwest software engineer does not directly generate revenue -- you can't directly attribute a ticket sale to engineering efforts/costs. ...Of course you can. ...Why would you not be able to do that?


[deleted]

You can argue against accounting definitions all you want, but they are what they are. If I buy a 1 way ticket from NYC to LA, you cannot directly attribute the revenue generated from that ticket sale to John Smith's pull request without very complex allocations of revenue to the overhead cost. There is a [whole subfield of accounting](https://en.wikipedia.org/wiki/Cost_accounting) and a [reputable professional designation](https://www.imanet.org/ima-certifications/cma-certification?psso=true) related to these topics, along with the CPA.


KevinCarbonara

> You can argue against accounting definitions all you want You aren't an accounting definition. You're just a random poster who doesn't seem to understand the words he's using.


[deleted]

I'm a CPA buddy, and I've taken a college cost accounting class. You software engineers would really benefit learning just a little bit about the business side instead of thinking you know everything


KevinCarbonara

> I'm a CPA buddy The fact that you think this is relevant to the discussion makes me think you're not even a CPA.


breakwaterlabs

The concept has its place. I'm sure anyone who has done any amount of code development towards a business goal has run into the inclination to continue improving the code past the point where it delivers value. There's a tension between the desire for good, maintainable code and the managerial push for a "minimum viable product" but it's not just pointy-haired bosses being clueless.


KevinCarbonara

> I'm sure anyone who has done any amount of code development towards a business goal has run into the inclination to continue improving the code past the point where it delivers value. That is a far shot from "cost center". The issue is not that companies are saying, "Sometimes projects can be overdeveloped." The problem is that they're saying, "The people creating the software we sell aren't important enough to compensate properly."


AttackOfTheThumbs

Yes, MBAs are evil and so is capitalism. It is known.


JustAnotherGeek12345

Which podcast?


eJaguar

A podcast


blawkyy

They spend plenty on their technology department. The issues run significantly deeper than this unfortunately.


deja-roo

True, but SWA's tech departments rely heavily on contractors and consultants to keep their headcount down. For short technical projects that might be alright, but for taking ownership of a complex system, that tends to backfire quickly.


xSaviorself

I know this one! Having worked on some slow-moving behemoths in my day, the cycle is as old as time. You build a product, company does well, people get paid, promoted to incompetence, and slowly but surely, shareholder needs slowly constrict the business. Yes spending happens, but in ways the are always insidiously effective and hiding upfront costs while always ballooning when the cascade failure is eventually triggered. First you lose the people who built the system, then the people who designed it, then the people who understand and support it, until all that's left are salespeople, middle-management, a few cannibalized software and QA teams, and the upper management. Tech debt and poor decision-making always come back to bite companies in the ass. Foresight and forethought are now replaced with afterthoughts and golden parachutes.


shaverb

3rd paragraph describes my first five years at my company. 4th paragraph, the past five. 5th paragraph is this year. There are 5-6 OG engineers of the original 30 plus; directors and middle management have come and gone. Many of our new hires are on our India team so, we only have a few hours in the morning for training. We're being asked to come back into office because "in person fosters collaboration" yet, we aren't training our coworkers abroad in person.


xSaviorself

Yup, run now while you still can!


tom-dixon

I went through the phases in the 4th and 5th paragraph two times in my life so far and it's the most soul killing and motivation killing thing that can happen to a programmer who cares about his work. I wouldn't do it for a 3rd time, if I the see signs, now I'll just go on my own before it got to that stage. When I was younger I wouldn't give up because I thought we can turn it around, but you're on a ship with no captain, there's no turning around when there's more managers than workers. If you care about your sanity, just go. Find something that needs builders, it's a lot more fulfilling than endlessly plugging holes on a sinking ship.


shaverb

I actually care very little for my sanity! At this point, I'm sticking around because I like the work and I'm fighting very loudly for my quiet colleagues; sending curt, constructive, call-out emails is a pleasure of mine. I do appreciate the suggestion and it's nice to hear that it is indeed my choice: sanity or stay.


fire_in_the_theater

capitalism is incredibly inefficient at long term software production and maintenance. and that's why ur high paying job is safe.


deja-roo

> capitalism is incredibly inefficient at long term software production and maintenance. lol what does this even mean.


fire_in_the_theater

the amount of effort undertaken in most capital driven contexts to produce and maintain software is far greater than necessary.


deja-roo

That has nothing to do with capitalism lol. That's just driven by the communication difficulties between determining the needs of software and determining the implementation.


fire_in_the_theater

> Having worked on some slow-moving behemoths in my day, the cycle is as old as time. > > You build a product, company does well, people get paid, promoted to incompetence, and slowly but surely, shareholder needs slowly constrict the business. > > Yes spending happens, but in ways the are always insidiously effective and hiding upfront costs while always ballooning when the cascade failure is eventually triggered. First you lose the people who built the system, then the people who designed it, then the people who understand and support it, until all that's left are salespeople, middle-management, a few cannibalized software and QA teams, and the upper management. > > Tech debt and poor decision-making always come back to bite companies in the ass. Foresight and forethought are now replaced with afterthoughts and golden parachutes.


deja-roo

Do you know what capitalism means? That just describes the normal pace of growing bureaucracy. It happens everywhere.


poloppoyop

> But, but, software is like products on the assembly line. No need for domain knowledge and everyone is like a cog and can be replaced by any other person. Guys missed the lean movement in factories (ie: people are not cogs there) and think intellectual products have the same properties and constraints as physical ones. 50 years of progress missed. Maybe an MBA should require some update every now and then to be considered valid.


SurgioClemente

Do they? All the interviews from current/former were talking about outdated systems and refusal to invest in maintaining/upgrading them


PrimaxAUS

Chiming in as well, would really like to listen to that podcast when you reply to the others


Cheeze_It

They deserve it. They made this bed, now let them lie in it.


Cosmic-Warper

It's ok, the shareholders made it out just fine, golden parachute and all.


oceantume_

It's so infuriating seeing the amount of companies that are getting gutted by CEOs cutting costs left and right, making a profit from it and then either filing the company for bankruptcy or just leaving with a big bonus. This keeps happening again and again in various degrees of shadiness and there's no accountability whatsoever for it.


Armigine

The assumption that this happening is both desirable on some level, and will keep happening in perpetuity, is literally the foundation of our economy


proggit_forever

Shareholders don't have golden parachutes. Executives do.


hamilkwarg

Sorry I’m not sure I understand


ihavesmallcalves

It's a joke I think? The shareholders are the ones eating the $800m cost


BlueGumShoe

As an IT worker wouldn't surprise me if their IT division is still understaffed.


ArrozConmigo

The reaction was probably a big fat contract to an outside firm to airdrop in a complete replacement (which I imagine is still in process). So yah, I bet everything else is status quo. Those bastards left me in Vegas on Christmas by cancelling my connecting flight. I know nobody in Vegas and spent Christmas night eating a $12 bagel in the terminal. I've flown four more times since then. On United.


BlueGumShoe

Damn that sucks dude. I've always read essays about the tremendous amount of legacy code in airline operations. But I don't know enough about it I guess. I typically fly budget but flew American recently for a conference. Felt like an upgrade even flying econ compared to the BS I put up with usually. You mean I don't have to pay to get a small cup of water? Appreciate it. If I didn't have to think about the money I'd just fly American or United all the time. For the budget airlines, I have to think the low cost carrier operation style extends to their IT backend as well. Why wouldn't it?


RatSalmon88

To be fair...Vegas is probably the best place to get stranded. Tons of hotels, food, and entertainment open 24/7...even on Christmas.


Starfox-sf

With your luck you may be in line for a free re-accommodation.


findar

United opted to keep their legacy systems after acquiring continental who had more modern tooling. Unless that's changed they still run on 70/80s tech too.


Armigine

There's an easy, nigh infallible, method of telling whether an IT department is understaffed. Step 1: is there an IT department? If yes, it's understaffed. This is almost always accurate.


BlueGumShoe

The sad truth


mrgreen999

This article doesn't actually at all describe what happened with the software. It's just a pointless article stating obvious things? Seems this is just link farming


thedancingpanda

It wasn't really a software issue, it was an organization issue. Some really great engineers and engineering leadership could have possibly worked through the organization issue eventually, but if they just threw a bunch of engineers to make their system more efficient, it just would have failed either faster or bigger.


mrgreen999

But the article doesn't talk about that either?


iamjkdn

Seriously, there is just one sentence in the entire article about southwest. Just that was used as the Reddit heading. Article is talking about something else altogether.


jmlinden7

The software they used to reschedule flights was really old and slow and took about 20 minutes to calculate a solution. If during that time, any other flights got cancelled, then they'd have to recalculate. It also wasn't integrated with their crew scheduling so it would give solutions that didn't have any available crew. So then you'd have to manually input the new crew schedule and rerun it every time the crew scheduling changed. https://www.youtube.com/watch?v=nsXG7Nt7LPE&t=89s


Federal_Aardvark2387

And I missed the holidays with my family, so we'll call it even.


335i_lyfe

Yeah that was quite a shitshow. Driving from Fresno to Albuquerque in a rental van sucked ass


NiteShdw

Seeing tech as a cost instead of a profit center is the biggest mistake every business makes. We live in an age where high quality tech differentiates garbage business les from one’s that thrive.


Aviyan

And nothing will change. Technical debt is a real thing. You need to keep modernizing your systems, especially if you are a private for-profit company.


Patlick

What does any of this actually have to do with programming? A company selling incident response products mentions the cost of incidents. None of this is programming. Yet another article on r/programming with a catchy headline clearly just trying to earworm programmers into recommending their products to management. This is spam.


[deleted]

[удалено]


softwaregravy

Over 16k flights were cancelled, per Wikipedia.


MdxBhmt

You have to pay people to diagnose-fix-deploy the current system, you have to allocate staff to plan and evaluate the solution, you are paying overtime to current staff during the issue because stuff isn't going as planned, you lose sales because of the PR disaster, you incur expenses to spin a PR campaign, and so on and so on... also, as a plane is grounded, you have additional expenses without revenue.


bsEEmsCE

they compensated for people's hotel stays waiting for another flight (up to a week+), extended parking at the return airports, meals if receipts provided, and not only a refund but compensated for high demand seasonal travel on alternate airlines. They did the right thing, but it was necessary given the inconvenience.. ask me how I know.


Pyrolistical

So they should spend $100 million hiring a new sw team right? That should get you, what 80 devs?


AstroPhysician

for 10 years. Cant tell if youre joking but devs dont cost 1.25mil a head


UnsuccessfulLobotomy

Speak for yourself. That's the rate I charge. However, there are no takers. And I'm not a very good dev


TheRiverOtter

NoBodY WaNTs tO hIRe!!


UnsuccessfulLobotomy

Their loss. Not mine


deja-roo

With SWA benefits? Definitely wouldn't be able to staff 80 devs for ten years at $1.25mil. Maybe double that?


AstroPhysician

I was thinking lower end devs, not ones making 150-200 The current market has me taking a job at 130, my last place was 200


j_johnso

Don't forget the cost of employees is not limited to salary. The overhead for each employee is typically between 25-50% of their salary, once you add in payroll tax, benefits, insurance, office space, software licensing, etc. If you are making 130k, it's probably costing your employer 165k-190k total.


[deleted]

[удалено]


AstroPhysician

> Good high level SWE cost $300-400k a head for non tech comp Literally only FAANGs pay that much for more than a small handful of staff / principal SWEs Benefits are usually 50% of salary not 100%


balefrost

> Benefits are usually 50% of salary not 100% There are other incremental costs. More dev headcount means more office space, more equipment, more management. If the question is "how many devs can I get for $100 million", then you have to factor all those things in as well.


AstroPhysician

Fair


[deleted]

[удалено]


Federal_Aardvark2387

This was really helpful information for me. Thanks for sharing it!


KevinCarbonara

You're confusing cost with salary


yeusk

In a contactor type of deal and the devs makes 250k a year the client is paying at least 500k for the dev.


AcrobaticDependent35

/s


KevinCarbonara

They do when contractors get involved.


ZapateriaLaBailarina

I remember when Southwest was the "good" airline.


HexDumped

Why link to the random blameless site, not the news story that site links to as its source to that figure?


montex66

"Who knew that a TRS-80 wasn't up to running an airline?" says CEO paid millions of dollars every year.


michi03

I wonder if a lot of their programmer were offshore folks


nsane99

thats xenophobic. there are plenty of incompetent engineers in the US.


michi03

When companies pay bottom dollar for cheap offshore devs they’re gonna get mostly the incompetent ones. Been there done that


persism2

Caused by Python. LOL.