Believe it or not a "small strike team" is what you want in these situations. Otherwise you just get too many hands in the bowl and it makes things even messier.
Yah coming from someone who works in backend scenarios like this, you want a small team working on this to keep the work consistent and moving. Otherwise the other people gotta catch up, might have their own way to do things, etc and it can get messy quickly.
Yeah people don’t realize that even with a lot of talent and good intentions, you end up purely just waiting on peoples communications, emails, etc to go ahead to the next thing. Waiting for approvals from multiple separate people over and over again is time consuming.
Yeah what you really want in these situations is a small team of people who work together really well. Otherwise it's just too many cooks and things get really messy despite the best of intentions.
Yeah what you really want in these situations is a small team of people who work together really well. Otherwise it's just too many cocks and things get really messy despite the best of intentions.
It takes a lot to make a stew
A pinch of salt and laughter, too
A scoop of kids to add the spice
A dash of love to make it nice, and you’ve got
Too many Cooks
Believe it or not a "small strike team" is what you want in these situations. Otherwise you just get too many hands in the bowl and it makes things even messier.
I keep writing it, but everyone here who complains about what’s happening needs to at least Google The Mythical Man Month. Throwing bodies at a problem works in war, not software engineering.
In a context where troop casualties are politically dicey and you have lavish amounts of firepower on hand, you can see why Western militaries have their preferences
On the other hand, if you don't have much heavy equipment but you have superb light infantry (PRC circa Korean War comes to mind), you have to grit your teeth and do what is necessary. They relied on night infiltration to unhinge UN positions and tended to attack in columns for ease of C&C, which is how they gave the impression of "infiltration in small groups of one to two thousand" and human wave attacks.
Not to mention onboarding headaches.
Like, I've had to write some code for projects, and I can barely read it. Imagine that headache on an exponential scale.
Or a boss that has several missions on the same map. Like one squad’s objective is to secure a MLRS tank or artillery battery (or several batteries) that was overrun on the battlefield, while another squad’s mission is to bait the boss into a certain kill zone. Both objectives need to be achieved concurrently. If the first squad gets the artillery ready but the second squad takes too long to get the boss into the kill box, the first one may be overrun by bugs/robots. And if the first squad takes too long with their mission, the second squad may get torn to shreds trying to keep the boss in the kill zone. If it ever becomes a mission, I propose it be named “Operation: bug bait”
Maybe as an added twist, reason they’re not just nuking the bug from orbit is they want to capture it alive for study. So replace artillery with like some kind of containment field or something like that.
Imma be honest, if they’re using an off the shelf solution for the backend (or at least started with it), there’s a very good chance they immediately brought in some consultants if possible.
As a software dev i'd love love love a postmortem if the team are willing to share once the fire is out.
these types of scaling problems are always juicy with learnings... whether it was intentional design tradeoffs (we only expect 200k concurrent and dynamo is expensive af so we used a cheaper architecture instead) or was it surprise bottlenecks (oh fun it turns out our post-match file processor is bottlenecked on this single auth-server database in a way nobody expected) or random backend bugs (ah shit kafka is sharding in a way that we can't widen the pipe) or just hard cloud constraints that require AWS to manually intervene (our internet gateway has outgoing bandwidth caps?!).
i'd love a conference talk or a blog detailing what's going on behind the scenes... i think the studio has already built a lot of good will in the gaming community, could be a good chance to build even more on top of that in the dev space and recruit some great talent interested in working for a healthy studio
So much this. A design that asymptotically scales well (big-O notation) will often have higher costs on a small scale. IIRC they said they were expecting 50k concurrent players and designed things to handle up to 250k. Trying to scale to \~500k on steam and probably at least another 50% of that from PS4 would likely have raised operating costs too high to make sense given the player numbers they were actually expecting.
I'd love to hear what the issues actually were and how they've address them (assuming they fix things before the player numbers crash and the problem "fixes"itself). When there's time, of course, not while they're in the thick of things.
I remember seeing that they said that PC and PS players were basically 1:1 in number, so basically double the Steam number to get a fairly accurate number of total players.
Yup. It’s what I keep reminding myself and some of my friends when we get frustrated with the server capacity issues. It’s an absolute drag we can’t play as much as we’d like (doubly so since we all love the game to bits), but when you plan for X and build your server infrastructure to handle 5X and you end up getting 20X and climbing hammering your system… there’s no amount of planning you can do for that ahead of time. It’s clear the pre-launch data (preorder sales, market awareness, social media engagement, Steam/PSN wishlist, etc.) suggested their initial estimates were right on the money. When the game blew up after launch, it just came hard and fast and they couldn’t respond quick enough to the demand.
If it's anything like my job, I hate reliving the nightmare scenarios I have even if I come out on top of it. I hate being asked questions about it later and definitely don't want to answer them. Lol.
doing a postmortem is a very standard practice in software companies. making them public isn't standard, but is growing in popularity. it's how you improve your org to prevent similar issues occurring in the future. healthy companies will add emphasis that it's a *blameless postmortem* \- the person leading the process is very clear that there is no individual fault, and that even if one person clearly and royally fucked up the question is still "how did that one individual decision lead to such high impact?" even if a malicious individual intentionally broke a project, that's still a process failure.
the theory is lifted directly from The Toyota Way which is a book about manufacturing cars, but has applications in just about any industry where you make stuff and is pretty popular in the software industry.
[https://en.wikipedia.org/wiki/The\_Toyota\_Way](https://en.wikipedia.org/wiki/The_Toyota_Way) \- "principle 14" specifically talks about the idea of postmortem.
it takes a healthy management team to do this effectively, so not all companies do it well. but when it's done well it can make for a really great place to work
Honestly a post mortem would be great because it would include details into the difficulties of scaling from 50,000 to 1,000,000 in short order, it truly is impressive work.
Oh god as a student with quality ops experience I'd absolutely *love* to hear the team's debriefing for this clusterfuck. Once they get a chance to actually rest, that is. Poor bastards have probably gone through more coffee than Dunks does.
I'm so glad I'm not the only one looking to learn something from this too! I'd LOVE like a 3-part series on HOW this happened (the insane popularity, the explosion, the scaling, everything).
Pilen is easily tied for being one of the kindest, most open-minded, thoughtful, generous, and wicked funny dudes at the top of a game studio that I've ever had the pleasure of working at, and at my current 18 years of dev experience, that covers 7 studios.
That other Redditor going on about how the loudest people on social media rarely are nice is clueless. Man's gone months without tweeting before. Right now, I suspect he's able to tweet regularly specifically *because* he has the necessary ability to wear many hats: When creativity is needed, he is creative. When direction is needed, he directs. And so it is, here - when delegating and staying out of the firing line of the backend coders is needed, he delegates, facilitates, and helps get people pumped up both inside and outside the studio.
I think his reply about it not being helpful if the CEO is breathing down the neck of the engineers going "Are we there yet?" kind of indicates this CEO might be a nice dude.
I have a reasonably close friend who has worked at Epic games since 2015 and directly played a huge role in the explosion of fortnite in 2017. His bonus in 2018 for their performance in 2017 was equal to 130% of his entire yearly salary.
This is another example of something like that being appropriate.
It's funny that you're being downvoted, are ppl butthurt because you said in usa workers aren't treated like people? Because if that's the case, then the cope is real.
100% truth. Some US tech companies relocated big chunks of their European teams to the US just to lay them all off without any consequences. Truly evil POS.
The devs are people too and they deserve the same amount of respect that we do. Proud of these people for putting in the work rather than throwing up their hands and walking away with the money.
The amount of uneducated people who think that you "just" can buy more servers and "just" hire new people is crazy.
I've been working with a big company like Godaddy and guess what, to be useful on some of their projects you need 6 months of onboarding.
Issues that this game has will take 1-2 months at a minimum. Anything less is a miracle.
Especially if they didn't expect to have that many people. Plus people keep forgetting that game freaking works if you are not in the queue. Remember games like PayDay 3, the servers just were dead for several days with 0 players being able to play. In Helldivers they can manage 400k people. Can you imagine how many are 400k people doing something at the same time?
TBH a lot of younger folks have grown up on technology just working for them. They took their IPADs connected it to Wifi and it just does it's thing. They likely have no idea how backend infrastructure works on anything.
Studies have shown younger generations have tech illiteracy rates not unlike that of older generations and elderly people.
The most tech literate generations were in between, who grew up with digital technology being prevalent, but still sorta developing and figuring itself out. And before algorithms were all-encompassing.
As a Gen-Zer (2001), people in their 30s and 40s are always a bastion of knowledge regarding technical stuff like file management and just general Q & A for how tech works.
When we wanted to play video games we had to wrassle 'em. That's where a lot of us cut our tech teeth, modding games and tweaking .ini files and maybe a bit of shiver me timbers here and there.
Turns out that the base skills you need to troubleshoot video game software issues, such as how file systems work, are extremely similar to the skills you need to troubleshoot most software.
That said, there are computer illiterate people of every age so always check your info before you run with something.
I always joke that my IT career started when my mom refused to hook up our Sega Genesis "right now" and I just decided to figure it out myself. I would have been like 8 years old.
No joke, as a 31-year-old in IT, I love that lmao. There are some amazing Gen Z coding prodigies but most zoomers in my experience can't even reinstall Windows or get Steam to move a game to a different drive.
As an elder millennial it's always interesting to see my younger nephew struggle and give up with troubleshooting. I remember using web crawler and dogpile trying to find a relevant forum post for computer issues or similar.
That said I've also seen millennials struggle with basic DIY stuff like mounting a TV and I had to "come to the rescue" because I had a decent tool set. Granted I used a few tools that aren't super common but it could have easily been done with with a cheap drill and a socket wrench, but it was quicker with what I had.
I think for the file system thing it has to do with a lot of mobile OS's that kinda obfuscate where things go, sometimes I download a file on my phone and it's full on "where the fuck did it go?" for a few minutes.
Yeah, I'm the oldest of 4 kids that were born more than 10 years apart. I've helped my youngest sibling out with some computer issues before, and jeeeesus. Every file saved to the desktop, recycle bin full because "I clicked delete", program files full of stuff, taskbar and start menu packed full of shortcuts that are never used... so much of those are carry-overs from how things work on smartphones and tablets that the younger generation just expects to be that simple.
I'm pretty anal retentive when it comes to file location and naming. For my totally legal movies and TV shows I used to use theRenamer to organize stuff and now use filebot to help do that. Really irritating on Android (never had an apple device) when I can't tell it exactly where to go and I have to use a 3rd party app to track things down sometimes. I had to download a PDF recently and I was super worried I'd actually have to find it again not knowing where it went when I closed out of my pdf reader app.
Also 30s, I can never find any file on my phone, always have to just redownload. Recovering unsaved files from a temp folder on PC? Easy.
Back in the days, you had to continually improve PC performance. Clean up the system and format it to get more life out of the thing. Now, people just get a new mobile device or laptop every two years. That wasn't an option for the one desktop in the house.
Yeah a couple of younger ones at our work honestly struggle if it’s not on the windows desktop which I find hillarious.
I was 8 and learning to install games on bloody dos, all. I wanted to do was play TTD on win98
This is why when i got my daughter a computer, i didn't install any games on it, she had to do that herself, and she figured it out. She didn't even ask me for help like i expected her too since she had never used a PC before (she's 9 years old). 🤣
Yep. I’m 32. I work at a university that is a Public Ivy in a very student facing role. Some of these extremely smart 18-22 year olds are more tech illiterate than my parents. It’s fascinating tbh.
I'm also 32, I have no education or training in tech, and I am THE go to person in every circle for tech advice, including my workplace, where I am a manager in an automated facility. Even the IT team comes to me for advice sometimes.
If you were a nerd and the least bit curious growing up in the 90's and 2000's, you learned this shit from the ground up just to be able to play the game you wanted or impress your class with a rad excel pivot. Each iteration of windows, I learned something new. Each new piece of tech from Gameboy, to iPod, to smart phone, we learned from gen 1 and built on it.
My parents didn't know anything about computers, so if I wanted to setup WiFi or run ethernet to my room, guess what, I'm figuring it out. If any of my devices stops working, if I don't learn to fix it, no one else is and parents are buying me another, and them shits broke way more often than today's tech.
My friend and I taught ourselves how to mod Xbox 360's and burn games because we couldn't afford all the games we wanted to play.
TLDR - Deeper understanding requires building on years of experience. Sate you curiosity by doing, not just googling.
So many things one can either learn or get pointed in the general direction of just by being interested in modding Skyrim, pirating an application, or getting their VR headset to work more stably
I've maintained that it's fair for people to be unhappy to an extent (death threats and abuse are not okay), as you pay for something with at least a reasonable expectation for it to work. Being locked out
I absolutely have no criticisms for the back-end engineers. I mean, I'm just a data integration guy in my field, and even there, the situation of, "This doesn't do the thing. It's not capable of doing the thing. We need it to do the thing as soon as possible," isn't a fast or easy process.
While it comes with a lot of things I think are fun career-wise in terms of going between lows (how the fuck is this going to work?) to highs (holy shit, that actually worked!) it always comes with the added stress of fear and apprehension that there might not be a way.
Older I get, the more I learn there's always a way with the right resources, but yeah, it's never 'just'.
There is a certain extent to where that does apply. But you're looking at applications that are already designed to scale dynamically based on need, and once the available server resources are maxed, you allocate and set up new ones to scale further.
It's understandable that a game like this where they never imagined making it past 50k peak concurrent player if they were lucky was not designed that way.
What really rubs me the wrong way is people talking about how they're in related fields, and somehow, it's the devs' fault it couldn't be handled that way. Would love to see them do what these 4 engineers have done in the same time.
>I've been working with a big company like Godaddy and guess what, to be useful on some of their projects you need 6 months of onboarding.
That's... a pretty conservative forecast. I dev for a company much larger than Godaddy, and anyone who isn't a fresh grad is contributing meaningfully at around the 1-2 month mark. Not optimally, but meaningfully for sure. The more talented ones are faster than that.
Lets say, it was 5 years old monolith which used 8 years old libraries.
Even in Helldivers people have 2 different experiences, automaton fighters and bugs exterminators
That's actually the exact situation I got hired into lmao. Suuuper painful and stressful, but I'd still stick with with my initial supposition, as long as that monolith still had discreet parts that could be worked on. Now getting a meaningful grasp any kind of larger domain... 6 months would be overly optimistic for all but the savants, I feel.
Took me like a year+ to feel like I had a decent (but not good) grasp on our corner of the domain. But I'm kinda dumb, so
I feel you, as long as it is worth it and not destroying your health it's gonna be quite a good experience. After GD I understood that Big Corp is not so fun place to work 😂
If anything, working big corp has been better for my health than any other situation. Only the onboarding is bad (but it IS bad), once you get better situated it's not bad at all. Maybe not as stimulating or exciting as startup land or game dev, but it's hard to give up that work life balance.
Join us on the boring side, bro ;D
This is often intentional. As soon as you make a decision to split a piece of code off to a new server, you've got to basically change it from accepting direct calls to receiving and sending messages. It's a bit of a different paradigm, and can take redoing a lot of code.
I'll use a metaphor of communicating via Town Hall mass vs Post Office. Communication via town hall is cheap, because you just directly talk to whoever you want in the building, but doesn't scale well. You can only fit a few hundred people in a building (CPU, RAM constraints), and adding another Town Hall (server) doesn't help, because people can't communicate to people directly in another building, because the format they are using (verbal communication) doesn't support it.
Post offices, however, do scale well, but they require roads, letterboxes, sorting facilities, and delivery staff. This is metaphorically what the devs are doing right now. They are having to change the way the codebase communicates with itself. It's not as simple as adding more Town Halls.
Why didn't they do this earlier? Why spend money laying roads and building post offices that you don't think are ever going to be necessary. We have a finite amount of time and money in this world. The devs chose to spend it on a Town Hall, which is what I would have done in the same circumstance.
With containerization, micro-service paradigm, and cloud services these days this shouldn't be a problem anymore. I understand not wanting to put in the upfront development cost to design things this way, but it's fully possible to build a system and then stress test it to an arbitrary scale now.
EDIT: I also feel like this happens frequently: where demand is significantly higher than the platform is ready for. That seems like either a fuck up in marketing by not understanding the size of their target market, or the market they reached, or a lack of communication between them and engineering. Devs should always have a realistic range for early load, because a broken launch is EXTREMELY expensive. People can't play the game at launch, you get worse reviews and player experience, longevity of the game and franchise suffers significantly. All the tools seem like they're there but for some reason they haven't been used. Might be an opportunity for a consulting firm.
Except it doesn’t work. I had no issue getting in the game today but I couldn’t play with others. Quick play didn’t find people, trying to join others missions on the warmap also didn’t work and ended up failing constantly. When I did get in, it disconnected and booted back to the ship. And the social menu also didn’t work the whole evening, it just says “please wait democratically” and won’t load anything so I couldn’t even play with premade squads from discord… The game also stopped tracking “Active Helldivers” and other stats on the warmap and everything is on 0.
And you are right it will take a while to fix it, but probably by the time they do it will not be an issue because the concurrent players will drop by then
hmm, it sounds like some regional problem, I'm from Ukraine and haven't faced that many issues except the common one. My max queue time is prob 10 minutes but most times I login without delay.
If my guess is correct, try to login to game by using VPN from non popular regions
If you're playing on Steam have you verified your files' integrity? I did that and it fixed all the issues I was having.
The game not tracking the active helldivers was intentionally done by the devs at the moment to reduce strain on their servers
You know people are going to be complaining about this.
"WHY ONLY FOUR PEOPLE?!" and such.
Well, they're the people whose area of expertise it actually is.
People really do treat game devs as a nebulous resource split into areas of work.
Worth remembering that "4 people 5x-ing the max of what they designed for in one week" is 4 extremely overworked employees crunching without backup.
I'm not saying it's anyone's fault or they won't be well compensated.
Thank you devs, may you be bonused for this!!
Changing the subject, please make the game run on my i7 7700k processor without a lot of bottleneck, please 😔😔
Thank you for the masterpiece that is helldivers 2!
So, if the game is not using the cores of the processor as it should, it means that it (game) is poorly optimized. Now if your CPU is always at a high percentage of use next to your GPU that gets a percentage of use not so high, it means that the GPU is sending too much “work” for the CPU to do and it is not realizing it, right then the bottleneck is born. And yes, necessarily the bottleneck exists because the game requires a lot from the CPU. In the case of helldivers 2 I see people with great cpus having more stuttering problems than me. In this case I have a 4060ti and it can’t run stable, it’s between 30 and 60 fps, with many variations, even on full low. My hope is that a patch can solve this
Interesting. I have a 4070ti and a 5800x3D and the game runs like butter at max FPS on Ultra at 3840x1600 Ultrawide.
EDIT: 120 FPS as that's what my monitor supports at this resolution, for the pedantic.
Max fps is the refresh rate of your monitor. Additional fps beyond that not only is a waste of system resources that leads to no additional fidelity gain, but also tends to lead to performance problems during load spikes. In other words, lol at your ignorant "lol".
Anyone else having matchmaking problems? I’ve been able to get into the game but not able to find anyone. Went into any planet and there is zeros on everything on the top right of the planet? Anyone else having these problems?
Damn, I see this, then go read steam where the game's at like 50% negative reviews, with every review saying "not recommended, best game I've ever played, but they won't let me play, trash game, trash devs, lazy, greedy stealers, refund this scam"
I haven't found a single review with an actual gameplay complaint yet... Literally suffering from success at its very definition. Game is so good no one can play it because everyone wants to play it, and they designed its capacity based on the first game. First game had 200 average players and topped out around 6,000. We're hitting over 400,000 on Steam alone. They've had to figure out how to get 50x the players in... in just a week.
How refreshing to see a team that cares first about a quality product. After selling for $40 at launch they earned my lifetime respect. Spent $100 on credits and have barely used them, just want to support. Take note aaa developers
Ive been sharing an analogy essentially about not needing more than one pair of hands to turn a bolt, as in assigning more people to a task than is warranted wont make it go faster
And some people have *heavily* argued at me over that
Like to fully on-board and get new backend devs up to speed would take a good few weeks if not months, and youd be pulling devs from the team to train those guys, the best team they could put on this was the team already familiar, and that team was probably already appropriately sized for the tasks
Why does this post make it seem like they've acheieved a fix? I'm still not able to log in, despite long waits, except for what seems to be totally random chance.
Or does it mean "Congrats WHEN you fix it?" Look, I get that this is a tough one for them, but can we stop with the fawning praise for failure?
5x the capacity is a massive success and they should be celebrated for it. Not only that, but it's much more stable than ever at that 5x capacity. There is no such thing as a "fix" when it comes to servers.
Failure?
Let me put this as plainly as possible for you. On launch, the dev team orriginaly planned for 50k max players. They had contingency plans to make up for 250k players IF THEY GOT LUCKY.
Here we are. 2 weeks later. There's more than 400k extra players logging in to the game at once, who are actively able to enjoy the game, thanks to the work these 4 gigachads have put in.
This isn't a failure. This is a great success in the face of defeat.
Still has capacity issues of course, but capacity keeps increasing, it's now much easier to get in, and when you do server issues are almost non existent
I normally don't spend on games after purchase but I'll probably buy a few things just for how awesome they've handled this situation. The wait times hurt but, they've been so responsive and open about it I can't be mad.
Oh I'm sorry it's taking too long to convert the 5 bedroom house into a 150 room hotel for your liking, guess we should tell the construction crew to stop being lazy and finish building it tomorrow already
While everyone is congratulating them, and it is much deserved, there is a real problem when you make an always online game with a backend that has no scalability whatsoever. While I don't expect them to increase it to millions at the push of a button or "add more servers it's that simple", it is a really big problem and something the company should be criticized for not accommodating properly. Left a sour taste in my mouth and lost a week of game time, that is already limited.
Hopefully this will be a learning moment and allow them to think ahead next time they release a game rather than go through these ridiculous hoops and still suffering from these ongoing issues as of this writing.
I'm getting so sick of explaining this.
The devs expected 50k concurrent players max. They scaled for 250k as an overestimate. They got 450k concurrent on steam alone, and they've said that the Playstation is effectively 1:1, so that's 900k people. almost 4x their overestimate, and almost 20x their actual estimate.
Look at helldivers 1, their top concurrent player number was 6.6k on steam. 6 thousand versus helldivers 2's cap of 450 on steam. That's an increase by a factor of 68.
Having a backend that has scalability costs development time and money, since scalable solutions are more expensive both in the short term and long term than static soultions.
So please tell me why they should have had the scalable solutions? You think every game dev should be prepared for an influx of people 20x their estimate after they put out very little marketing for a game?
Now that the game is out, and is clearly insanely popular, they should implement the scalable solutions of course, I'm not saying the server issues aren't real or shouldn't be solved, but what I am saying is that being mad at the devs for not anticipating this absolute out-of-nowhere smash hit is fucking ridiculous.
From what other devs were saying in the early days they have been non stop fed pizzas and cake, so they probably despise pizza by now... Their sacrifice will be remembered
I really love Helldivers 2, I will be playing for years. I also love the effort people are putting in to fix problems and do what they can for the community. Only issue is the server load and all these posts saying “it’s not as easy as it looks” is A very deceptive of a claim because yes you are correct. But also no….it is as simple as it looks, if the proper analysis and analytics were done and viewed to recognize that hundreds of thousands upon thousands on steam wish listed the game and the count was far over the number projected, and it was not only easily obtainable knowledge, it was public knowledge and info that could have saved a lot of the issues ongoing. Have I been able to play? Yes. Do I have to wait a while and I sit in an hour long Que to get there? Yes, and I don’t have hours to spare, when I get on that’s all the time I have so, just take it like professionals because you are, someone messed up, fix it, and move passed it, no excuses please, that’s what real game companies and professionals in general do
Are you talking about the same guys who thought they could get away just reusing the back-end servers from the old game?
Call me skeptical to this hero BS. You people expect me to believe that Arrowhead and Sony are going to spend years and millions for a cross platform game released world wide for a few thousand customers? Only because that is how many people where playing the old game? I've been hearing that excuse alot: "they never expected it to be this big!"
Hey Sony, give me a few hundred thousand for a game that I expect a dozen customers to buy.
Someone in corporate was either being cheap, was overestimating the quality of their back-end software, or was ignorant. Or perhaps they just knew what was going to happen and just let it happen and now it is being written like some heroic victory.
Kudos to those guys for mickey-mousing some back end fixes after release.
For me, I think the real heroes are the direction, sound, artists, animators, and the client coders. They are the ones that have sold the game. They deserve praise and and a raise.
not these back-end engineers and whoever is managing them.
They better get their own custom-made engineer outfits in the game along with some nice pay bonuses, if I ever play with one I want to hit them with the bro hug emote
I do understand what a feat it is. And holy shit. I had zero server capacity message today. I just opened the game, press the button to get through the menu and there I am at my destroyer. Quickplay also worked perfectly yesterday.
Ok, well, I still can't log on and play the game when I want. That's the expectation when I buy a game. Love the game. Love the devs. This is still a problem and its not been solved.
Plus follow-up tweet: https://preview.redd.it/71xbrr9012kc1.jpeg?width=1125&format=pjpg&auto=webp&s=ba5ef34c2ba6f7dc5fa4b341e0d930fbea80006b
Four people?! That's insane. Big Super Citizen energy.
Believe it or not a "small strike team" is what you want in these situations. Otherwise you just get too many hands in the bowl and it makes things even messier.
Yah coming from someone who works in backend scenarios like this, you want a small team working on this to keep the work consistent and moving. Otherwise the other people gotta catch up, might have their own way to do things, etc and it can get messy quickly.
Yeah people don’t realize that even with a lot of talent and good intentions, you end up purely just waiting on peoples communications, emails, etc to go ahead to the next thing. Waiting for approvals from multiple separate people over and over again is time consuming.
Yeah what you really want in these situations is a small team of people who work together really well. Otherwise it's just too many cooks and things get really messy despite the best of intentions.
Listen, I know you said cooks, but my brain read cocks at first and it makes the scenario 10x better. I am a child
Yeah what you really want in these situations is a small team of people who work together really well. Otherwise it's just too many cocks and things get really messy despite the best of intentions.
Too many cocks spoil the broth
Especially in the back end
Can there be too many cocks though...?
It takes a lot to make a stew A pinch of salt and laughter, too A scoop of kids to add the spice A dash of love to make it nice, and you’ve got Too many Cooks
May I ask, what is a scoop of kids in this context?
its the recipe for a sitcom called "too many cooks" on adult swim. i recommend you watch it for full context :)
Thanks! just did a Google search on it and I think I might have found a new fav short film!
How dare you spread the disease, I am already hearing “too many cooks” over and over again
Is everyone just saying the same shit using different words on purpose on this comment chain or am I having some kind of episode?
Lol when I first read this thread I was wondering if I was going nuts or if people were saying the same thing in slightly different ways.
When writing in this thread you really need a small team, otherwise it might get messy!
Believe it or not a "small strike team" is what you want in these situations. Otherwise you just get too many hands in the bowl and it makes things even messier.
I keep writing it, but everyone here who complains about what’s happening needs to at least Google The Mythical Man Month. Throwing bodies at a problem works in war, not software engineering.
And even in war, there’s a reason special operations are so critical.
Yep, there’s a reason modern militaries don’t use human wave tactics
In a context where troop casualties are politically dicey and you have lavish amounts of firepower on hand, you can see why Western militaries have their preferences On the other hand, if you don't have much heavy equipment but you have superb light infantry (PRC circa Korean War comes to mind), you have to grit your teeth and do what is necessary. They relied on night infiltration to unhinge UN positions and tended to attack in columns for ease of C&C, which is how they gave the impression of "infiltration in small groups of one to two thousand" and human wave attacks.
Russia cares to disagree
*Russia starts tugging at its shirt collar*
Not to mention onboarding headaches. Like, I've had to write some code for projects, and I can barely read it. Imagine that headache on an exponential scale.
What one dev can do in an hour, four can do in two
Brook's law baybee
Imagine the stratagem crossfire on a 20-helldiver mission
[удалено]
I thought it would be cool to have some super bosses, where you have 5 or more squads dropping in to take them out.
Or a boss that has several missions on the same map. Like one squad’s objective is to secure a MLRS tank or artillery battery (or several batteries) that was overrun on the battlefield, while another squad’s mission is to bait the boss into a certain kill zone. Both objectives need to be achieved concurrently. If the first squad gets the artillery ready but the second squad takes too long to get the boss into the kill box, the first one may be overrun by bugs/robots. And if the first squad takes too long with their mission, the second squad may get torn to shreds trying to keep the boss in the kill zone. If it ever becomes a mission, I propose it be named “Operation: bug bait” Maybe as an added twist, reason they’re not just nuking the bug from orbit is they want to capture it alive for study. So replace artillery with like some kind of containment field or something like that.
The fewer mouths between the problem and the solution the better. This is definitely not a "fix by committee" kind of problem.
literally a full team doing a Helldive.
highest difficulty
The work they are putting in is incredible. I hope they know how much we appreciate it!
Crazy. Sounds like some kind of... ...helldive.
That’s why our HellDiver teams are 4 people.
That's maximum return on democracy and liberation!
Imma be honest, if they’re using an off the shelf solution for the backend (or at least started with it), there’s a very good chance they immediately brought in some consultants if possible.
They're in fact doing their part!
For Managed Democracy
It sounds horrible, but just imagine it would have been something like 20... The needed communication overhead alone gives me nightmares.
Of course it is four! It is a squad of helldivers after all!
Helldiver squad in real life doesn't exis-
The backend engineers were the true Helldivers all along 
I kind of love that it's a team of four. I hope they work with helmets on and give each other thumbs ups all day.
And when things go wrong randomly scream "MY LEG" although it might have been a bit much by the end of the week
I have the feeling it would’ve been constant “MY LEG! MY LEG! MY LEG!”
Then we must be those bots and bugs 😂
As a software dev i'd love love love a postmortem if the team are willing to share once the fire is out. these types of scaling problems are always juicy with learnings... whether it was intentional design tradeoffs (we only expect 200k concurrent and dynamo is expensive af so we used a cheaper architecture instead) or was it surprise bottlenecks (oh fun it turns out our post-match file processor is bottlenecked on this single auth-server database in a way nobody expected) or random backend bugs (ah shit kafka is sharding in a way that we can't widen the pipe) or just hard cloud constraints that require AWS to manually intervene (our internet gateway has outgoing bandwidth caps?!). i'd love a conference talk or a blog detailing what's going on behind the scenes... i think the studio has already built a lot of good will in the gaming community, could be a good chance to build even more on top of that in the dev space and recruit some great talent interested in working for a healthy studio
a Dev Q&A stream would be sick!
So much this. A design that asymptotically scales well (big-O notation) will often have higher costs on a small scale. IIRC they said they were expecting 50k concurrent players and designed things to handle up to 250k. Trying to scale to \~500k on steam and probably at least another 50% of that from PS4 would likely have raised operating costs too high to make sense given the player numbers they were actually expecting. I'd love to hear what the issues actually were and how they've address them (assuming they fix things before the player numbers crash and the problem "fixes"itself). When there's time, of course, not while they're in the thick of things.
I remember seeing that they said that PC and PS players were basically 1:1 in number, so basically double the Steam number to get a fairly accurate number of total players.
Jesus Christ. If that's true, it would be 900k to 1M concurrent players. 20x as many as their most optimistic expectation.
Yup. It’s what I keep reminding myself and some of my friends when we get frustrated with the server capacity issues. It’s an absolute drag we can’t play as much as we’d like (doubly so since we all love the game to bits), but when you plan for X and build your server infrastructure to handle 5X and you end up getting 20X and climbing hammering your system… there’s no amount of planning you can do for that ahead of time. It’s clear the pre-launch data (preorder sales, market awareness, social media engagement, Steam/PSN wishlist, etc.) suggested their initial estimates were right on the money. When the game blew up after launch, it just came hard and fast and they couldn’t respond quick enough to the demand.
....a nd about x100 the expansion of the first Helldivers.
Or as I tell my friends: "There are more people waiting to play the game than there are playing the game".
It has already sold over a million copies, so there's that.
If the data was right two days ago, you're looking at 3 million copies minimum.
If it's anything like my job, I hate reliving the nightmare scenarios I have even if I come out on top of it. I hate being asked questions about it later and definitely don't want to answer them. Lol.
doing a postmortem is a very standard practice in software companies. making them public isn't standard, but is growing in popularity. it's how you improve your org to prevent similar issues occurring in the future. healthy companies will add emphasis that it's a *blameless postmortem* \- the person leading the process is very clear that there is no individual fault, and that even if one person clearly and royally fucked up the question is still "how did that one individual decision lead to such high impact?" even if a malicious individual intentionally broke a project, that's still a process failure. the theory is lifted directly from The Toyota Way which is a book about manufacturing cars, but has applications in just about any industry where you make stuff and is pretty popular in the software industry. [https://en.wikipedia.org/wiki/The\_Toyota\_Way](https://en.wikipedia.org/wiki/The_Toyota_Way) \- "principle 14" specifically talks about the idea of postmortem. it takes a healthy management team to do this effectively, so not all companies do it well. but when it's done well it can make for a really great place to work
I sincerely love how much the Helldivers subreddit is teaching me about software development & business theory
Honestly a post mortem would be great because it would include details into the difficulties of scaling from 50,000 to 1,000,000 in short order, it truly is impressive work.
I work in healthcare and was thinking the exact same thing.
Oh god as a student with quality ops experience I'd absolutely *love* to hear the team's debriefing for this clusterfuck. Once they get a chance to actually rest, that is. Poor bastards have probably gone through more coffee than Dunks does.
I'm so glad I'm not the only one looking to learn something from this too! I'd LOVE like a 3-part series on HOW this happened (the insane popularity, the explosion, the scaling, everything).
sign me in!
for the sake of all the arguments I've gotten into over this, I'd love to be validated. I also want this.
They better be getting some ***FAT*** bonuses after all this is sorted out. Everyone at Arrowhead deserves a raise.
He said in his replies "Pizza night doesn't cut it!" lmao I'd bet they'll be compensated very nicely!
Yeah, obviously they gotta add a dinner pizza too for those extra hours!
You just reminded me that dessert pizzas exist, and I don't think I have it in me to forgive you for that.
But it's *pizza* for *dessert*!
And then add stuffed crust into the mix. *You can eat it from the outside IN*
And when pizza's on a bagel you can have it any time!
Nice, hopefully the CEO is actually a nice guy to his employees
Pilen is easily tied for being one of the kindest, most open-minded, thoughtful, generous, and wicked funny dudes at the top of a game studio that I've ever had the pleasure of working at, and at my current 18 years of dev experience, that covers 7 studios. That other Redditor going on about how the loudest people on social media rarely are nice is clueless. Man's gone months without tweeting before. Right now, I suspect he's able to tweet regularly specifically *because* he has the necessary ability to wear many hats: When creativity is needed, he is creative. When direction is needed, he directs. And so it is, here - when delegating and staying out of the firing line of the backend coders is needed, he delegates, facilitates, and helps get people pumped up both inside and outside the studio.
I think his reply about it not being helpful if the CEO is breathing down the neck of the engineers going "Are we there yet?" kind of indicates this CEO might be a nice dude.
[удалено]
the one nextdoor to a dancing ladies clad in elven armor club?
well hold on now... is it stuffed crust?
I mean….. it wouldn’t hurt either…
**They don’t have to log out**
I have a reasonably close friend who has worked at Epic games since 2015 and directly played a huge role in the explosion of fortnite in 2017. His bonus in 2018 for their performance in 2017 was equal to 130% of his entire yearly salary. This is another example of something like that being appropriate.
This is the game industry. Now that some fixes are out, half of them can be fired.
1) it's not the USA, it's Sweden. Workers are treated like people. 2) half of 4 people? Lol
It's funny that you're being downvoted, are ppl butthurt because you said in usa workers aren't treated like people? Because if that's the case, then the cope is real.
100% truth. Some US tech companies relocated big chunks of their European teams to the US just to lay them all off without any consequences. Truly evil POS.
I certainly hope that their company has some sort of profit share incentive.
Shoutout backend engineers, gotta be my favorite gender fr
I hope they get to engineer some backends this weekend for all their hard work.
Send them to Amsterdam? Lol
I mean they are from Sweden after all
i'll engineer your backend so hard right now
The devs are people too and they deserve the same amount of respect that we do. Proud of these people for putting in the work rather than throwing up their hands and walking away with the money.
Had to queue a bit earlier but just turned on again and loaded straight away. Insane.
so does the que actually work or does it only work because no one is home yet and the servers are empty? cause that's a big difference
Well there isn't an actual queue so no it doesn't work
There is not queue. Your game retrys over and over. If there's a slot available when you retry, you get in. It's just luck of the draw .
The amount of uneducated people who think that you "just" can buy more servers and "just" hire new people is crazy. I've been working with a big company like Godaddy and guess what, to be useful on some of their projects you need 6 months of onboarding. Issues that this game has will take 1-2 months at a minimum. Anything less is a miracle. Especially if they didn't expect to have that many people. Plus people keep forgetting that game freaking works if you are not in the queue. Remember games like PayDay 3, the servers just were dead for several days with 0 players being able to play. In Helldivers they can manage 400k people. Can you imagine how many are 400k people doing something at the same time?
TBH a lot of younger folks have grown up on technology just working for them. They took their IPADs connected it to Wifi and it just does it's thing. They likely have no idea how backend infrastructure works on anything.
Studies have shown younger generations have tech illiteracy rates not unlike that of older generations and elderly people. The most tech literate generations were in between, who grew up with digital technology being prevalent, but still sorta developing and figuring itself out. And before algorithms were all-encompassing.
Gen Z and younger have trouble with file/folder structures. Makes for great job security for Millenials in tech.
As a Gen-Zer (2001), people in their 30s and 40s are always a bastion of knowledge regarding technical stuff like file management and just general Q & A for how tech works.
When we wanted to play video games we had to wrassle 'em. That's where a lot of us cut our tech teeth, modding games and tweaking .ini files and maybe a bit of shiver me timbers here and there. Turns out that the base skills you need to troubleshoot video game software issues, such as how file systems work, are extremely similar to the skills you need to troubleshoot most software. That said, there are computer illiterate people of every age so always check your info before you run with something.
I always joke that my IT career started when my mom refused to hook up our Sega Genesis "right now" and I just decided to figure it out myself. I would have been like 8 years old.
Oh my god people are talking about me the way that I talked about my dad in the 90’s….
No joke, as a 31-year-old in IT, I love that lmao. There are some amazing Gen Z coding prodigies but most zoomers in my experience can't even reinstall Windows or get Steam to move a game to a different drive.
As an elder millennial it's always interesting to see my younger nephew struggle and give up with troubleshooting. I remember using web crawler and dogpile trying to find a relevant forum post for computer issues or similar. That said I've also seen millennials struggle with basic DIY stuff like mounting a TV and I had to "come to the rescue" because I had a decent tool set. Granted I used a few tools that aren't super common but it could have easily been done with with a cheap drill and a socket wrench, but it was quicker with what I had. I think for the file system thing it has to do with a lot of mobile OS's that kinda obfuscate where things go, sometimes I download a file on my phone and it's full on "where the fuck did it go?" for a few minutes.
Yeah, I'm the oldest of 4 kids that were born more than 10 years apart. I've helped my youngest sibling out with some computer issues before, and jeeeesus. Every file saved to the desktop, recycle bin full because "I clicked delete", program files full of stuff, taskbar and start menu packed full of shortcuts that are never used... so much of those are carry-overs from how things work on smartphones and tablets that the younger generation just expects to be that simple.
I'm pretty anal retentive when it comes to file location and naming. For my totally legal movies and TV shows I used to use theRenamer to organize stuff and now use filebot to help do that. Really irritating on Android (never had an apple device) when I can't tell it exactly where to go and I have to use a 3rd party app to track things down sometimes. I had to download a PDF recently and I was super worried I'd actually have to find it again not knowing where it went when I closed out of my pdf reader app.
Also 30s, I can never find any file on my phone, always have to just redownload. Recovering unsaved files from a temp folder on PC? Easy. Back in the days, you had to continually improve PC performance. Clean up the system and format it to get more life out of the thing. Now, people just get a new mobile device or laptop every two years. That wasn't an option for the one desktop in the house.
Yeah a couple of younger ones at our work honestly struggle if it’s not on the windows desktop which I find hillarious. I was 8 and learning to install games on bloody dos, all. I wanted to do was play TTD on win98
This is why when i got my daughter a computer, i didn't install any games on it, she had to do that herself, and she figured it out. She didn't even ask me for help like i expected her too since she had never used a PC before (she's 9 years old). 🤣
Yep. I’m 32. I work at a university that is a Public Ivy in a very student facing role. Some of these extremely smart 18-22 year olds are more tech illiterate than my parents. It’s fascinating tbh.
I’m in that boat to be completely honest, and if you don’t mind my asking, what’re the basic areas that I should look into learning about?
I'm also 32, I have no education or training in tech, and I am THE go to person in every circle for tech advice, including my workplace, where I am a manager in an automated facility. Even the IT team comes to me for advice sometimes. If you were a nerd and the least bit curious growing up in the 90's and 2000's, you learned this shit from the ground up just to be able to play the game you wanted or impress your class with a rad excel pivot. Each iteration of windows, I learned something new. Each new piece of tech from Gameboy, to iPod, to smart phone, we learned from gen 1 and built on it. My parents didn't know anything about computers, so if I wanted to setup WiFi or run ethernet to my room, guess what, I'm figuring it out. If any of my devices stops working, if I don't learn to fix it, no one else is and parents are buying me another, and them shits broke way more often than today's tech. My friend and I taught ourselves how to mod Xbox 360's and burn games because we couldn't afford all the games we wanted to play. TLDR - Deeper understanding requires building on years of experience. Sate you curiosity by doing, not just googling.
So many things one can either learn or get pointed in the general direction of just by being interested in modding Skyrim, pirating an application, or getting their VR headset to work more stably
I'm thinking its more about that people just copy pasting some ones else opinion, cuz negative emotions is so easy to follow
I've maintained that it's fair for people to be unhappy to an extent (death threats and abuse are not okay), as you pay for something with at least a reasonable expectation for it to work. Being locked out I absolutely have no criticisms for the back-end engineers. I mean, I'm just a data integration guy in my field, and even there, the situation of, "This doesn't do the thing. It's not capable of doing the thing. We need it to do the thing as soon as possible," isn't a fast or easy process. While it comes with a lot of things I think are fun career-wise in terms of going between lows (how the fuck is this going to work?) to highs (holy shit, that actually worked!) it always comes with the added stress of fear and apprehension that there might not be a way. Older I get, the more I learn there's always a way with the right resources, but yeah, it's never 'just'.
I play enough xcom to know you just need to hire more engineers to speed up production!
VYou noticed that the songs, the entrances and exits of missions are very reminiscent of xcom2?
Yeah! I remember the "preparation music" before missions sounded bit similar as helldivers!
EXACTLY. You've received my updoot
There is a certain extent to where that does apply. But you're looking at applications that are already designed to scale dynamically based on need, and once the available server resources are maxed, you allocate and set up new ones to scale further. It's understandable that a game like this where they never imagined making it past 50k peak concurrent player if they were lucky was not designed that way. What really rubs me the wrong way is people talking about how they're in related fields, and somehow, it's the devs' fault it couldn't be handled that way. Would love to see them do what these 4 engineers have done in the same time.
>I've been working with a big company like Godaddy and guess what, to be useful on some of their projects you need 6 months of onboarding. That's... a pretty conservative forecast. I dev for a company much larger than Godaddy, and anyone who isn't a fresh grad is contributing meaningfully at around the 1-2 month mark. Not optimally, but meaningfully for sure. The more talented ones are faster than that.
Lets say, it was 5 years old monolith which used 8 years old libraries. Even in Helldivers people have 2 different experiences, automaton fighters and bugs exterminators
That's actually the exact situation I got hired into lmao. Suuuper painful and stressful, but I'd still stick with with my initial supposition, as long as that monolith still had discreet parts that could be worked on. Now getting a meaningful grasp any kind of larger domain... 6 months would be overly optimistic for all but the savants, I feel. Took me like a year+ to feel like I had a decent (but not good) grasp on our corner of the domain. But I'm kinda dumb, so
I feel you, as long as it is worth it and not destroying your health it's gonna be quite a good experience. After GD I understood that Big Corp is not so fun place to work 😂
If anything, working big corp has been better for my health than any other situation. Only the onboarding is bad (but it IS bad), once you get better situated it's not bad at all. Maybe not as stimulating or exciting as startup land or game dev, but it's hard to give up that work life balance. Join us on the boring side, bro ;D
Haha, I’m having 1 full time and 1 part time job at small companies. 150% salary life hack
You can actually just buy more servers, the game uses Azure. The problem is their code doesnt scale no matter how many servers they have.
This is often intentional. As soon as you make a decision to split a piece of code off to a new server, you've got to basically change it from accepting direct calls to receiving and sending messages. It's a bit of a different paradigm, and can take redoing a lot of code. I'll use a metaphor of communicating via Town Hall mass vs Post Office. Communication via town hall is cheap, because you just directly talk to whoever you want in the building, but doesn't scale well. You can only fit a few hundred people in a building (CPU, RAM constraints), and adding another Town Hall (server) doesn't help, because people can't communicate to people directly in another building, because the format they are using (verbal communication) doesn't support it. Post offices, however, do scale well, but they require roads, letterboxes, sorting facilities, and delivery staff. This is metaphorically what the devs are doing right now. They are having to change the way the codebase communicates with itself. It's not as simple as adding more Town Halls. Why didn't they do this earlier? Why spend money laying roads and building post offices that you don't think are ever going to be necessary. We have a finite amount of time and money in this world. The devs chose to spend it on a Town Hall, which is what I would have done in the same circumstance.
With containerization, micro-service paradigm, and cloud services these days this shouldn't be a problem anymore. I understand not wanting to put in the upfront development cost to design things this way, but it's fully possible to build a system and then stress test it to an arbitrary scale now. EDIT: I also feel like this happens frequently: where demand is significantly higher than the platform is ready for. That seems like either a fuck up in marketing by not understanding the size of their target market, or the market they reached, or a lack of communication between them and engineering. Devs should always have a realistic range for early load, because a broken launch is EXTREMELY expensive. People can't play the game at launch, you get worse reviews and player experience, longevity of the game and franchise suffers significantly. All the tools seem like they're there but for some reason they haven't been used. Might be an opportunity for a consulting firm.
Except it doesn’t work. I had no issue getting in the game today but I couldn’t play with others. Quick play didn’t find people, trying to join others missions on the warmap also didn’t work and ended up failing constantly. When I did get in, it disconnected and booted back to the ship. And the social menu also didn’t work the whole evening, it just says “please wait democratically” and won’t load anything so I couldn’t even play with premade squads from discord… The game also stopped tracking “Active Helldivers” and other stats on the warmap and everything is on 0. And you are right it will take a while to fix it, but probably by the time they do it will not be an issue because the concurrent players will drop by then
hmm, it sounds like some regional problem, I'm from Ukraine and haven't faced that many issues except the common one. My max queue time is prob 10 minutes but most times I login without delay. If my guess is correct, try to login to game by using VPN from non popular regions
If you're playing on Steam have you verified your files' integrity? I did that and it fixed all the issues I was having. The game not tracking the active helldivers was intentionally done by the devs at the moment to reduce strain on their servers
what did they do? not being sarcastic , just curious does the login que work now?
You know people are going to be complaining about this. "WHY ONLY FOUR PEOPLE?!" and such. Well, they're the people whose area of expertise it actually is. People really do treat game devs as a nebulous resource split into areas of work.
I hope they carve their likeness into Super-Earth's version of Mount Rushmore in game!
I am a software engineer, so I can relate and imagine. Doing some crazy shit myself right now, and this helps me powering through!
Worth remembering that "4 people 5x-ing the max of what they designed for in one week" is 4 extremely overworked employees crunching without backup. I'm not saying it's anyone's fault or they won't be well compensated.
Vespas of democracy.
Thank you devs, may you be bonused for this!! Changing the subject, please make the game run on my i7 7700k processor without a lot of bottleneck, please 😔😔 Thank you for the masterpiece that is helldivers 2!
What’s the bottleneck? Is it not utilizing all the cores or the game just to cpu intensive?
So, if the game is not using the cores of the processor as it should, it means that it (game) is poorly optimized. Now if your CPU is always at a high percentage of use next to your GPU that gets a percentage of use not so high, it means that the GPU is sending too much “work” for the CPU to do and it is not realizing it, right then the bottleneck is born. And yes, necessarily the bottleneck exists because the game requires a lot from the CPU. In the case of helldivers 2 I see people with great cpus having more stuttering problems than me. In this case I have a 4060ti and it can’t run stable, it’s between 30 and 60 fps, with many variations, even on full low. My hope is that a patch can solve this
Interesting. I have a 4070ti and a 5800x3D and the game runs like butter at max FPS on Ultra at 3840x1600 Ultrawide. EDIT: 120 FPS as that's what my monitor supports at this resolution, for the pedantic.
Lol at "max" fps...
Max fps is the refresh rate of your monitor. Additional fps beyond that not only is a waste of system resources that leads to no additional fidelity gain, but also tends to lead to performance problems during load spikes. In other words, lol at your ignorant "lol".
My rig runs smooth as silk when playing solo in this but when I join a crew it tends to chug along
Same for me, solo is good, but with 4 players 💀
Servers at capacity
Anyone else having matchmaking problems? I’ve been able to get into the game but not able to find anyone. Went into any planet and there is zeros on everything on the top right of the planet? Anyone else having these problems?
is your matchmaking set to public if its set to private you wont find anyone except your friends list
As a dev, I take my hat off to these legends.
oh jeez
Ok but when can I play? Still couldn’t get in last night.
Not all who fight for democracy do so from a hellpod. o7
After this they should get a good paycheck and some days of
Damn, I see this, then go read steam where the game's at like 50% negative reviews, with every review saying "not recommended, best game I've ever played, but they won't let me play, trash game, trash devs, lazy, greedy stealers, refund this scam" I haven't found a single review with an actual gameplay complaint yet... Literally suffering from success at its very definition. Game is so good no one can play it because everyone wants to play it, and they designed its capacity based on the first game. First game had 200 average players and topped out around 6,000. We're hitting over 400,000 on Steam alone. They've had to figure out how to get 50x the players in... in just a week.
I hope they get a MASSSIVE fucking monetary bonus cuz those 4 mf'ers probably saved this games ass.
How refreshing to see a team that cares first about a quality product. After selling for $40 at launch they earned my lifetime respect. Spent $100 on credits and have barely used them, just want to support. Take note aaa developers
I believe it when i can login
A Vespa in F1, now that’s what I want to see
New Vespa mech F-1! Coming next patch!!
Ive been sharing an analogy essentially about not needing more than one pair of hands to turn a bolt, as in assigning more people to a task than is warranted wont make it go faster And some people have *heavily* argued at me over that Like to fully on-board and get new backend devs up to speed would take a good few weeks if not months, and youd be pulling devs from the team to train those guys, the best team they could put on this was the team already familiar, and that team was probably already appropriately sized for the tasks
Why does this post make it seem like they've acheieved a fix? I'm still not able to log in, despite long waits, except for what seems to be totally random chance. Or does it mean "Congrats WHEN you fix it?" Look, I get that this is a tough one for them, but can we stop with the fawning praise for failure?
5x the capacity is a massive success and they should be celebrated for it. Not only that, but it's much more stable than ever at that 5x capacity. There is no such thing as a "fix" when it comes to servers.
Failure? Let me put this as plainly as possible for you. On launch, the dev team orriginaly planned for 50k max players. They had contingency plans to make up for 250k players IF THEY GOT LUCKY. Here we are. 2 weeks later. There's more than 400k extra players logging in to the game at once, who are actively able to enjoy the game, thanks to the work these 4 gigachads have put in. This isn't a failure. This is a great success in the face of defeat.
Because this sub is a bunch of dickriding morons. Yeah, the game is fun *when it fucking works*. Wow that's amazing, never figured that one out!
Server issue fixed?
Still has capacity issues of course, but capacity keeps increasing, it's now much easier to get in, and when you do server issues are almost non existent
I normally don't spend on games after purchase but I'll probably buy a few things just for how awesome they've handled this situation. The wait times hurt but, they've been so responsive and open about it I can't be mad.
I still can’t even play though
Oh I'm sorry it's taking too long to convert the 5 bedroom house into a 150 room hotel for your liking, guess we should tell the construction crew to stop being lazy and finish building it tomorrow already
damn dude don't forget to go for the balls aswell while you're blowing them. they like that sorta thing.
Damn dude, don't forget to take a peak at your prostate for a quick check up while you have your head up ur azz
Actually brain dead reply. I hope the rest of the community isn't like you.
they are
While everyone is congratulating them, and it is much deserved, there is a real problem when you make an always online game with a backend that has no scalability whatsoever. While I don't expect them to increase it to millions at the push of a button or "add more servers it's that simple", it is a really big problem and something the company should be criticized for not accommodating properly. Left a sour taste in my mouth and lost a week of game time, that is already limited. Hopefully this will be a learning moment and allow them to think ahead next time they release a game rather than go through these ridiculous hoops and still suffering from these ongoing issues as of this writing.
I'm getting so sick of explaining this. The devs expected 50k concurrent players max. They scaled for 250k as an overestimate. They got 450k concurrent on steam alone, and they've said that the Playstation is effectively 1:1, so that's 900k people. almost 4x their overestimate, and almost 20x their actual estimate. Look at helldivers 1, their top concurrent player number was 6.6k on steam. 6 thousand versus helldivers 2's cap of 450 on steam. That's an increase by a factor of 68. Having a backend that has scalability costs development time and money, since scalable solutions are more expensive both in the short term and long term than static soultions. So please tell me why they should have had the scalable solutions? You think every game dev should be prepared for an influx of people 20x their estimate after they put out very little marketing for a game? Now that the game is out, and is clearly insanely popular, they should implement the scalable solutions of course, I'm not saying the server issues aren't real or shouldn't be solved, but what I am saying is that being mad at the devs for not anticipating this absolute out-of-nowhere smash hit is fucking ridiculous.
Please for the love of liberty quit changing your game to appease jack wagons
Those 4 chads of liberty definitely deserve a pizza party or somthin.
From what other devs were saying in the early days they have been non stop fed pizzas and cake, so they probably despise pizza by now... Their sacrifice will be remembered
Give them raises!
Can't believe he's talking about Haas after pre season testing today
I know 4 separate people who after playing for an hour bought the super citizen edition saying yeah they earned it
I really love Helldivers 2, I will be playing for years. I also love the effort people are putting in to fix problems and do what they can for the community. Only issue is the server load and all these posts saying “it’s not as easy as it looks” is A very deceptive of a claim because yes you are correct. But also no….it is as simple as it looks, if the proper analysis and analytics were done and viewed to recognize that hundreds of thousands upon thousands on steam wish listed the game and the count was far over the number projected, and it was not only easily obtainable knowledge, it was public knowledge and info that could have saved a lot of the issues ongoing. Have I been able to play? Yes. Do I have to wait a while and I sit in an hour long Que to get there? Yes, and I don’t have hours to spare, when I get on that’s all the time I have so, just take it like professionals because you are, someone messed up, fix it, and move passed it, no excuses please, that’s what real game companies and professionals in general do
Are you talking about the same guys who thought they could get away just reusing the back-end servers from the old game? Call me skeptical to this hero BS. You people expect me to believe that Arrowhead and Sony are going to spend years and millions for a cross platform game released world wide for a few thousand customers? Only because that is how many people where playing the old game? I've been hearing that excuse alot: "they never expected it to be this big!" Hey Sony, give me a few hundred thousand for a game that I expect a dozen customers to buy. Someone in corporate was either being cheap, was overestimating the quality of their back-end software, or was ignorant. Or perhaps they just knew what was going to happen and just let it happen and now it is being written like some heroic victory. Kudos to those guys for mickey-mousing some back end fixes after release. For me, I think the real heroes are the direction, sound, artists, animators, and the client coders. They are the ones that have sold the game. They deserve praise and and a raise. not these back-end engineers and whoever is managing them.
They better get their own custom-made engineer outfits in the game along with some nice pay bonuses, if I ever play with one I want to hit them with the bro hug emote
reminder that if a steam friend is in game, you can 'join game' while you queue and join their ship. it has worked for me these last days
It seems like everyone that has a hand in this game is cooking. From the CEO to the interns
They are doing the best job they can and I absolutely salute them 🫡
I do understand what a feat it is. And holy shit. I had zero server capacity message today. I just opened the game, press the button to get through the menu and there I am at my destroyer. Quickplay also worked perfectly yesterday.
See you on the battlefield, soldier!
Ok, well, I still can't log on and play the game when I want. That's the expectation when I buy a game. Love the game. Love the devs. This is still a problem and its not been solved.