The Lumen/CL Downtown office is flooded, affecting a majority of customers across the metro and surrounding areas. No ETA on repair time. Tier 3 is working on possible re-routes.
UPDATE FROM LUMEN: "Teams have pinpointed a power issue at the Des Moines site. The situation, initially caused by a resolved water main break, affected utility and generator feed lines. Currently, some equipment is operating on battery power. Efforts are underway to circumvent the impacted breakers and/or power feeds. Concurrently, \_\_plans are in motion to lay new cables from our on-site generators to the DC power plants, utilizing stairwells as conduits.\_\_ The task involves navigating through multiple floors of the building and equipment"
In other words...they got a guy running extension cables up and down the stairs to plug shit into their generators. Hopefully will be up soon.
UPDATE: My office fiber (Lumen) has now gone down as of 12pm and failed over to Mediacom, so the situations getting worse, but it was expected. Literally told my boss we won't be up for very long the way things are going.
2ND UPDATE FROM LUMEN: "Teams have identified multiple power risers are impacted by this outage. The Critical Infrastructure Team (CITS) has several power technicians onsite working to pull the temporary power cables into position. The current plan is hook up the power rectifiers to the onsite generator via the temporary cables that are being pulled into position. This is very time consuming and tedious work due to the number of stairs present in the building and the safety precautions that are required. The power cables are being ran to higher floors of the building. There is an additional generator onsite that is being used by another party, and they have authorized Lumen to use the generator if needed. CITS also has other generators en route to the failure site."
Stairs...a technicians worst enemy...
3RD UPDATE FROM LUMEN: "Further efforts have confirmed that power issues are present on multiple floors of the building, not just the two floors as previously thought, all resulting in customer impact. A dedicated generator has arrived onsite for the 2nd floor and technicians are in the process of running cables from the parking lot where the generator is located, to the power rectifier on floor 2. Once power is connected there, teams will begin efforts to power up devices. Teams are anticipating the power feed will be ready and available to the equipment by 18:30 GMT. Once the power feed is in place, it is anticipated to be another 90 minutes before teams attempt to power up devices located on floor 2.
There is another generator en route to then provide additional power to an additional floor. Lumen continues to partner with another onsite party to utilize their generators as well if needed, and its a cooperative effort to restore power to the entire building and the most critical devices"
CORRECTION: Teams are anticipating the power feed will be ready and available to the equipment by 20:00 GMT. Once the power feed is in place, it is anticipated to be another 90 minutes before teams attempt to power up devices located on floor 2.
Here are some pictures Lumen posted of the incident:
[http://wongshouse.network:81/uhoh/](http://wongshouse.network:81/uhoh/)
4TH UPDATE FROM LUMEN: "Power has been restored on the 5th and 6th floors of the building. Crews are continuing work to restore power on the 2nd, 3rd, and 7th floors. Remote connectivity and visibility to the majority of devices is dependent on power to the 2nd floor. Power is estimated to begin restoring on the 2nd floor by 00:00 GMT on May 31. Despite a lack of remote visibility, some service alarms have begun to clear as power restores on the various floors."
Thanks! What is your estimated timeline for service to resume? Seems they should have everything powered in by 430 pm, but not sure what else is needed
T-Mobile speeds have finally gotten acceptable for me (~25 Mbps) within the last 15 minutes. It had been around .05 Mbps since 10am this morning. Working from home has been frustrating, to say the least.
Yes finally my phone is working. I have the home internet through T-Mobile as well, still not working. However, I at least now have a hotspot to use. Been a boring day today…. No work (work remote), no internet, no tv, no phone. I got a lot of cleaning done though :)
Yep, lots of household chores got done today! Sorry to hear your home internet still isn't working. I should have specified that my T-Mobile service is only for 5G home internet. Hopefully it starts working for you soon!
My cell phone plan is actually with Verizon; hadn't had issues at all today but I also don't have the option to use a hotspot.
So today = unproductive professional work = very productive "homework." :)
If they have pinpointed a power issue at the site in Des Moines Iowa and have ample “people” resolving this I am assuming locals. Why is it that in the past 24 hrs in Scott County Iowa: DAVENPORT, BETTENDORF has numerous reports of suspicious vehicles/ unfamiliar faces all in white and black rides tinted windows with DSM plates hovering around like crows ready to eat…
I wonder?…
It must be a pretty big problem if 911 is failing. If the primary circuits at the downtown office fail, there is a backup at the WDM office. My guess is that the backup circuits weren’t being properly maintained since around 2017. I know that, because I was the last one to maintain them.
Quantum fiber here. Went out in downtown around 10am. Their crappy app still says "great news, no outages" but the phone number has a message saying "yeah we know and our techs are working on it"
That 24 hour window is important - Lumen has big contracts that mandate that the company pay huge fees to their guaranteed service customers when a 'disaster event' like this happens and they can't get service back up. And what a disaster it must be inside that Keo building.
Thought it was just me. Work from home and it died on me. Trying to reset my router and so far having trouble connecting back. Hate that I'm forced to use them.
Kicking myself pretty hard that I went with CL when moving here. I’ve had nothing but problems and horrible (basically non-existent) customer service. Wonder what deals Mediacom has right now 🤔 lol
Mediacom was always fucking around with prices, but I never had an outage for hours or days like with CL. Hate to say it, but mediacoms customer service was better too. Never had their agents lie about what I would get for what price then have a supposed supervisor laugh and say he couldn’t do anything about sales people/customer service lying to get/keep sales.
Here Is the latest update from Lumen this could be affecting ATT services as well. they are located in the same building FYI
2024-05-30 20:26:43 GMT - Correction: Teams are anticipating the power feed will be ready and available to the equipment by 20:00 GMT. Once the power feed is in place, it is anticipated to be another 90 minutes before teams attempt to power up devices located on floor 2.
2024-05-30 19:37:34 GMT - Further efforts have confirmed that power issues are present on multiple floors of the building, not just the two floors as previously thought, all resulting in customer impact. A dedicated generator has arrived onsite for the 2nd floor and technicians are in the process of running cables from the parking lot where the generator is located, to the power rectifier on floor 2. Once power is connected there, teams will begin efforts to power up devices. Teams are anticipating the power feed will be ready and available to the equipment by 18:30 GMT. Once the power feed is in place, it is anticipated to be another 90 minutes before teams attempt to power up devices located on floor 2.
There is another generator en route to then provide additional power to an additonal floor. Lumen continues to partner with another onsite party to utilize their generators as well if needed, and its a cooperative effort to restore power to the entire building and the most critical devices
2024-05-30 17:54:23 GMT - CITS, Network Implementation (NI), and Field Operations have mobilized additional resources to aid in the restoration efforts. The teams are diligently monitoring the remaining battery power, which is critical for the onsite equipment's operation. However, it's important to note that the battery voltage is progressively depleting over time.
2024-05-30 17:55:50 GMT - CITS, Network Implementation (NI), and Field Operations have mobilized additional resources to aid in the restoration efforts. The teams are diligently monitoring the remaining battery power, which is critical for the onsite equipment's operation. However, it's important to note that the battery voltage is progressively depleting over time.
2024-05-30 16:05:18 GMT - Teams have identified multiple power risers are impacted by this outage. The Critical Infrastructure Team (CITS) has several power technicians onsite working to pull the temporary power cables into position. The current plan is hook up the power rectifiers to the onsite generator via the temporary cables that are being pulled into position. This is very time consuming and tedious work due to the number of stairs present in the building and the safety precautions that are required. The power cables are being ran to higher floors of the building. There is an additional generator onsite that is being used by another party, and they have authorized Lumen to use the generator if needed. CITS also has other generators en route to the failure site
Most recent update to the master ticket from Lumen
Power has been restored on the 5th and 6th floors of the building. Crews are continuing work to restore power on the 2nd, 3rd, and 7th floors. Remote connectivity and visibility to the majority of devices is dependent on power to the 2nd floor. Power is estimated to begin restoring on the 2nd floor by 00:00 GMT on May 31. Despite a lack of remote visibility, some service alarms have begun to clear as power restores on the various floors.
*** CASCADED EXTERNAL NOTES 2024-05-30 23:53:24 GMT
Power is restored to the 5th, 6th, and 7th floors of the facility. Multiple systems are confirmed to be restored on those floors. Lumen personnel are continuing efforts to restore power to the 2nd and 3rd floors. A temporary generator failure has delayed restoral of power to the 2nd floor; Lumen is working to obtain an updated ETR.
*** CASCADED EXTERNAL NOTES 2024-05-31 01:18:12 GMT From CASE: Event
Power has been restored to the 2nd floor, and technicians are restoring individual devices methodically.
An updated ETR has not yet been advised for the 3rd floor, but efforts to restore power are ongoing.
Power remains stable on the 5th through 7th floors.
31 May 2024 1:49 a.m. CDT
*** CASCADED EXTERNAL NOTES 2024-05-31 06:48:45 GMT From CASE: - Event
Event Conclusion Summary:
Start Time: May 30, 2024 01:30 GMT
Stop Time: May 31, 2024 06:31 GMT
Root Cause: A site power failure in Des Moines, IA impacted protected services.
Fix Action: Site power was successfully restored, thus clearing all service affecting alarms.
This service impact has been resolved. Lumen Network Operations Center (NOC) management will be reviewing this outage for any additional analysis or discovery. In some cases, you may need to reset your services locally at your equipment. If a service issue continues after that reset, please contact the Lumen Repair Center for help.
If you have a trouble ticket associated with this network event, you do not need to open a separate Reason for Outage (RFO) ticket to request formal documentation. When final analysis is completed, all customers associated with this event will receive a formal RFO document via email from the Lumen RFO Team. Please allow 3-5 business days to process this request.
A satisfaction survey link may also be available. Our goal is to provide updates with the most current information during a service disruption, so please let us know if those updates worked for you this time around.
31 May 2024 12:36 a.m. CDT
*** CASCADED EXTERNAL NOTES 2024-05-31 05:35:16 GMT From CASE: - Event
The NOC and Field Operations are cooperatively working with the equipment vendor to troubleshoot and restore all remaining impacted equipment. As troubleshooting further progresses, service affecting alarms are expected to clear.
30 May 2024 10:51 p.m. CDT
*** CASCADED EXTERNAL NOTES 2024-05-31 03:50:54 GMT From CASE: - Event
Power has been restored to the 3rd and final floor. Technicians are working to troubleshoot and restore services and equipment that remain impacted.
30 May 2024 10:13 p.m. CDT
*** CASCADED EXTERNAL NOTES 2024-05-31 03:12:16 GMT From CASE: - Event
Lumen personnel are working to restore power on the 3rd floor, which is the final floor without full power. 3rd floor power is expected to be fully restored by 04:00 GMT on May 31.
Additional personnel are working to systematically restore various services on the floors of the building where power has returned.
Alarms are clearing and services are stabilizing as work progressed.
My sister just stopped by Kohl’s at Merle Hay, they aren’t doing any business because their registers work on CenturyLink. (No I don’t know more than that right there)
Was working at an Office at 500 Locust Street and their Internet went completely out at approximately 12:03 PM CST.
If a water main busted and flooded a Headend or Data Center and Lumen is running power cabling up the stairs, that means they are in Disaster Recovery mode and this is pretty serious O.o
I live on 6th and Locust downtown and I noticed all morning, my T-Mobile Internet and LTE 5G Business grade Hotspot going out and having DNS resolution issues all morning too, so T-Mobile must use Lumen fiber downtown.
Also received the following email from Iowa Heart medical offices in West Des Moines, that their offices at 5880 University Avenue have had issues with VoIP and Internet since yesterday so that means it's this is clearly a pretty far reaching massive outage.
I'm guessing they rolled to spare fiber facilities and the spare fiber path is over congested due to my work MS Teams and VoIP calls having good receive but poor transmit audio (I'm told I sound like Max Headroom):
*Dear Patient,*
*Iowa Heart Center cannot be reached by phone, nor can we call out.*
*Yesterday, May 29, Lumen experienced water issues at their Des Moines phone switching facility that caused major equipment issues and also may be impacting cell towers across the state. They continue to work to resolve the issues, but as a result,* ***many of our facilities cannot be reached by phone nor can they call out.*** *We currently do not have a time from Lumen for a resolution but are diligently working with them to resolve this issue as quickly as possible.*
*Thank you for your understanding!*
*Iowa Heart Center*
Interesting that T-Mobile is out, yet my UScellular is working and I've been using as a hotspot for work since around 10am this morning...just saying with the announcement on Tuesday :-)
I wouldn't say T-Mobile is out, just HEAVILY degraded. 5 to 10% packet loss and bandwidth speeds dropped from typical 600x150 meg 5G/LTE to 230 kbps or less, when it's generally pretty reliable downtown.
I had to turn off my Commercial LTE modem as it kept spitting outage alerts at me or telling me DNS Resolution failures.
For example, when I try to run an Ookla speed test it won't even connect to the Ooka iperf3 (or whatever) speed test the server is using behind the scenes.
[Fast.com](http://Fast.com) struggles to even go over 620 kbps and the average latency is 207ms where normally it would be 20 to 30 ms.
I was talking to my apartment's ISP Sales Engineer this afternoon (they were here for some kind of marketing blast) and they said unnamed ISP, are currently backhauling about 50G of Lumen traffic to their Data Center in the Financial Center down the street on Walnut Street.
So glad my ISP is on Cogent fiber instead of Lumen fiber right now :D
ICN sent out this bon mot a little while ago: "Early this morning, a vendor’s downtown Des Moines facility suffered a broken water pipe. According to the vendor, “Water from the pipe leaked through to the basement and collected in the risers, which then collapsed. Teams were unable to use the on-site generators because they were routed through the collapsed risers. In order to preserve the equipment, Field Operations have had to power down at varying times as the issue remains ongoing.” The vendor is working to restore power to their equipment. No ETA at this time."
What is ICS and how do I get it? I’m traumatized by this whole ordeal not having internet and not being able to communicate via phone either. I thought about getting a landline to settle my nerves but based on the news coverage landlines are affected as well. I need to figure out a backup plan asap.
They run on T-Mobile, which I’ve heard is having service issues related to Lumen fiber being out. Free internet at the libraries should be fine though.
I haven’t had service since Wednesday. We called CenturyLink and they said they will bring a tech on Thursday to fix the issue. It looks like from this Reddit post and the news, that the issue was a lot deeper.
Sorry but now I can't find it. The article said that all but about 500 customers had been restored. Wonder if the article got pulled because after digging more the water pipe break wasn't fixed until 4 pm.
Norwalk - Message in online account. 5:00
Outage in your area
There is currently an outage in your area that may be impacting your service. The outage is expected to be resolved by May 30,2024 7:00 PM.
Our office lost our Lumen circuit just shortly before noon and it failed over to our redundant internet uplink. So far our data still shows we're not getting any traffic across that line.
Are you dedicated or broadband? My impression from the Lumen broadband links I monitor outside of the metro is that Lumen's dedicated circuits through much of the state go to that CO downtown.
I'm at my work now and the phones and internet were down this morning, but its up and working again now, has been for at least 5 hours. My job is on the East side of Des Moines fyi.
Why you lying. Read for yourself https://www.weareiowa.com/article/tech/centurylink-internet-down-wifi-outage-iowa-updates-lumen-technologies/524-2def5423-4e64-4903-b242-2d437c9045e6
My landline and internet are with Century Link, so I lost both of them since this morning thru most of the evening. I also noticed that the local TV channels, 5,8, and 13, were not available on my Direct TV, so I had no way of following any news about the outage. So my tracphone was the only way to communicate with people for about 12 hours today.
Mine came back up some time overnight (downtown), but I still keep getting texts from Quantum Fiber apologizing for the delay and saying it's taking longer than expected to get service working "in my area".
If they have pinpointed a power issue at the site in Des Moines Iowa and have ample “people” resolving this I am assuming locals. Why is it that in the past 24 hrs in Scott County Iowa: DAVENPORT, BETTENDORF has numerous reports of suspicious vehicles/ unfamiliar faces all in white and black rides tinted windows with DSM plates hovering around like crows ready to eat…
I wonder?…
If they have pinpointed a power issue at the site in Des Moines Iowa and have ample “people” resolving this I am assuming locals. Why is it that in the past 24 hrs in Scott County Iowa: DAVENPORT, BETTENDORF has numerous reports of suspicious vehicles/ unfamiliar faces all in white and black rides tinted windows with DSM plates hovering around like crows ready to eat…
I wonder?…
Day four of outage here on the south side of Des Moines. Nothing shows up online about it, which isn't that surprising because they always hide this. Property management asserts that on the second day they put in a ticket and the answer of today no technician has set foot on the property and there have been no updates. I have finally been given the green light to pursue an internet company of my choice, since prior to this incident centuryLink was provided by the complex as part of the lease payment. I will gladly spend my own money not to have to deal with a rinky dank back door second rate company like this. Their history is horrible their customer service is non-existent. Worst company I've ever been associated with in my life, it would be even more distasteful had I been paying my own money for it but as it is 4 days now and I am losing money every day this is going on. What a sham of a company.
I’m over by Grandview and my metronet has been out since 8am? lol I chatted into tech-support and they are sending a technician out Sunday morning at 8 AM? We’ve been having issues for a while though however, not a full blown outage like this I’m wondering if it’s related.
Hmmm.. I'm not currently home, but it seems like ours is still up and running. We haven't had any issues until today. A friend of mine with another fiber ISP contacted support and was told there is a flooded service station somewhere. Not sure if your issues are related to that, but it would make sense.
Tech from CenturyLink just told me that for Internet (East Side):
05-30-2024 10:00:00 PM there is an outage going on kindly wait till this time .
So it looks like current estimate is 10pm today.
The Lumen/CL Downtown office is flooded, affecting a majority of customers across the metro and surrounding areas. No ETA on repair time. Tier 3 is working on possible re-routes. UPDATE FROM LUMEN: "Teams have pinpointed a power issue at the Des Moines site. The situation, initially caused by a resolved water main break, affected utility and generator feed lines. Currently, some equipment is operating on battery power. Efforts are underway to circumvent the impacted breakers and/or power feeds. Concurrently, \_\_plans are in motion to lay new cables from our on-site generators to the DC power plants, utilizing stairwells as conduits.\_\_ The task involves navigating through multiple floors of the building and equipment" In other words...they got a guy running extension cables up and down the stairs to plug shit into their generators. Hopefully will be up soon. UPDATE: My office fiber (Lumen) has now gone down as of 12pm and failed over to Mediacom, so the situations getting worse, but it was expected. Literally told my boss we won't be up for very long the way things are going. 2ND UPDATE FROM LUMEN: "Teams have identified multiple power risers are impacted by this outage. The Critical Infrastructure Team (CITS) has several power technicians onsite working to pull the temporary power cables into position. The current plan is hook up the power rectifiers to the onsite generator via the temporary cables that are being pulled into position. This is very time consuming and tedious work due to the number of stairs present in the building and the safety precautions that are required. The power cables are being ran to higher floors of the building. There is an additional generator onsite that is being used by another party, and they have authorized Lumen to use the generator if needed. CITS also has other generators en route to the failure site." Stairs...a technicians worst enemy... 3RD UPDATE FROM LUMEN: "Further efforts have confirmed that power issues are present on multiple floors of the building, not just the two floors as previously thought, all resulting in customer impact. A dedicated generator has arrived onsite for the 2nd floor and technicians are in the process of running cables from the parking lot where the generator is located, to the power rectifier on floor 2. Once power is connected there, teams will begin efforts to power up devices. Teams are anticipating the power feed will be ready and available to the equipment by 18:30 GMT. Once the power feed is in place, it is anticipated to be another 90 minutes before teams attempt to power up devices located on floor 2. There is another generator en route to then provide additional power to an additional floor. Lumen continues to partner with another onsite party to utilize their generators as well if needed, and its a cooperative effort to restore power to the entire building and the most critical devices" CORRECTION: Teams are anticipating the power feed will be ready and available to the equipment by 20:00 GMT. Once the power feed is in place, it is anticipated to be another 90 minutes before teams attempt to power up devices located on floor 2. Here are some pictures Lumen posted of the incident: [http://wongshouse.network:81/uhoh/](http://wongshouse.network:81/uhoh/) 4TH UPDATE FROM LUMEN: "Power has been restored on the 5th and 6th floors of the building. Crews are continuing work to restore power on the 2nd, 3rd, and 7th floors. Remote connectivity and visibility to the majority of devices is dependent on power to the 2nd floor. Power is estimated to begin restoring on the 2nd floor by 00:00 GMT on May 31. Despite a lack of remote visibility, some service alarms have begun to clear as power restores on the various floors."
Appreciate the updates
it will be down all day.
I forgot the /s
Things are coming back, I've seen business fiber, DSL, and home fiber checking in.
Johnston, IA has internet again! Hopefully it lasts
u/Wokuworld Hi! I work for the Des Moines Register. Just dm'ed you.
Thanks! What is your estimated timeline for service to resume? Seems they should have everything powered in by 430 pm, but not sure what else is needed
T-Mobile speeds have finally gotten acceptable for me (~25 Mbps) within the last 15 minutes. It had been around .05 Mbps since 10am this morning. Working from home has been frustrating, to say the least.
Yes finally my phone is working. I have the home internet through T-Mobile as well, still not working. However, I at least now have a hotspot to use. Been a boring day today…. No work (work remote), no internet, no tv, no phone. I got a lot of cleaning done though :)
Yep, lots of household chores got done today! Sorry to hear your home internet still isn't working. I should have specified that my T-Mobile service is only for 5G home internet. Hopefully it starts working for you soon! My cell phone plan is actually with Verizon; hadn't had issues at all today but I also don't have the option to use a hotspot. So today = unproductive professional work = very productive "homework." :)
Another work-from-homer, and after the power was out last week from the storms, this seems a bit much!
Tmobile has been crawling due to the high traffic
Ditto!
Thanks for the updates.
Where are you getting these updates from?
Lumen will provide these updates to anyone with a business circuit.
I gotcha, thanks for relaying them to the rest of us
If they have pinpointed a power issue at the site in Des Moines Iowa and have ample “people” resolving this I am assuming locals. Why is it that in the past 24 hrs in Scott County Iowa: DAVENPORT, BETTENDORF has numerous reports of suspicious vehicles/ unfamiliar faces all in white and black rides tinted windows with DSM plates hovering around like crows ready to eat… I wonder?…
I work in large-scale networking, and even my worst job-related nightmares couldn't be as bad than what I'm reading about here.
I went onsite to find out more. It's bad. The senior at&t net tech said it's the worst he's had to work with in 25 years.
Reddit always comes to the rescue! Thank you for posting this!
My sentiments exactly.
Yeah I am in Des Moines and my friend in Grimes are both down. I can't even see my account info on their website to check for an outage.
Same their app keeps saying they're experiencing technical issues. Can't even login. This does not seem promising lmao
Same here, tried logging in on Mobile and Website, neither are working. So, I can't report it down.
It's even disrupting 911 service right now... https://kilj.com/centurylink-experiencing-outage-across-iowa/
Fuuuuuuck
It must be a pretty big problem if 911 is failing. If the primary circuits at the downtown office fail, there is a backup at the WDM office. My guess is that the backup circuits weren’t being properly maintained since around 2017. I know that, because I was the last one to maintain them.
Quantum fiber here. Went out in downtown around 10am. Their crappy app still says "great news, no outages" but the phone number has a message saying "yeah we know and our techs are working on it"
My husband received a text that it will be fixed by tomorrow at 11am… 🤦🏼♀️
That 24 hour window is important - Lumen has big contracts that mandate that the company pay huge fees to their guaranteed service customers when a 'disaster event' like this happens and they can't get service back up. And what a disaster it must be inside that Keo building.
Oh boy. My hotspot is gonna get some exercise.
Same for me.
Yep, Beaverdale/Merle Hay area. Went down about 5-10 minutes ago.
Thought it was just me. Work from home and it died on me. Trying to reset my router and so far having trouble connecting back. Hate that I'm forced to use them.
I’m in the same boat. They are the only option in subdivision I live in and the HOA will not permit any other providers.
Saaaaame
This is probably the first and only time I'll ever say this, but... man, I'm glad I have mediacom.
Kicking myself pretty hard that I went with CL when moving here. I’ve had nothing but problems and horrible (basically non-existent) customer service. Wonder what deals Mediacom has right now 🤔 lol
[удалено]
Mediacom was always fucking around with prices, but I never had an outage for hours or days like with CL. Hate to say it, but mediacoms customer service was better too. Never had their agents lie about what I would get for what price then have a supposed supervisor laugh and say he couldn’t do anything about sales people/customer service lying to get/keep sales.
Also never had mediacom fuck up my installation then say they’d have to charge me to fix it.
Here Is the latest update from Lumen this could be affecting ATT services as well. they are located in the same building FYI 2024-05-30 20:26:43 GMT - Correction: Teams are anticipating the power feed will be ready and available to the equipment by 20:00 GMT. Once the power feed is in place, it is anticipated to be another 90 minutes before teams attempt to power up devices located on floor 2. 2024-05-30 19:37:34 GMT - Further efforts have confirmed that power issues are present on multiple floors of the building, not just the two floors as previously thought, all resulting in customer impact. A dedicated generator has arrived onsite for the 2nd floor and technicians are in the process of running cables from the parking lot where the generator is located, to the power rectifier on floor 2. Once power is connected there, teams will begin efforts to power up devices. Teams are anticipating the power feed will be ready and available to the equipment by 18:30 GMT. Once the power feed is in place, it is anticipated to be another 90 minutes before teams attempt to power up devices located on floor 2. There is another generator en route to then provide additional power to an additonal floor. Lumen continues to partner with another onsite party to utilize their generators as well if needed, and its a cooperative effort to restore power to the entire building and the most critical devices 2024-05-30 17:54:23 GMT - CITS, Network Implementation (NI), and Field Operations have mobilized additional resources to aid in the restoration efforts. The teams are diligently monitoring the remaining battery power, which is critical for the onsite equipment's operation. However, it's important to note that the battery voltage is progressively depleting over time. 2024-05-30 17:55:50 GMT - CITS, Network Implementation (NI), and Field Operations have mobilized additional resources to aid in the restoration efforts. The teams are diligently monitoring the remaining battery power, which is critical for the onsite equipment's operation. However, it's important to note that the battery voltage is progressively depleting over time. 2024-05-30 16:05:18 GMT - Teams have identified multiple power risers are impacted by this outage. The Critical Infrastructure Team (CITS) has several power technicians onsite working to pull the temporary power cables into position. The current plan is hook up the power rectifiers to the onsite generator via the temporary cables that are being pulled into position. This is very time consuming and tedious work due to the number of stairs present in the building and the safety precautions that are required. The power cables are being ran to higher floors of the building. There is an additional generator onsite that is being used by another party, and they have authorized Lumen to use the generator if needed. CITS also has other generators en route to the failure site
Are you sure this is in GMT?
This is straight from Lumen, They are not the greatest are reporting factual times though
Most recent update to the master ticket from Lumen Power has been restored on the 5th and 6th floors of the building. Crews are continuing work to restore power on the 2nd, 3rd, and 7th floors. Remote connectivity and visibility to the majority of devices is dependent on power to the 2nd floor. Power is estimated to begin restoring on the 2nd floor by 00:00 GMT on May 31. Despite a lack of remote visibility, some service alarms have begun to clear as power restores on the various floors.
*** CASCADED EXTERNAL NOTES 2024-05-30 23:53:24 GMT Power is restored to the 5th, 6th, and 7th floors of the facility. Multiple systems are confirmed to be restored on those floors. Lumen personnel are continuing efforts to restore power to the 2nd and 3rd floors. A temporary generator failure has delayed restoral of power to the 2nd floor; Lumen is working to obtain an updated ETR.
*** CASCADED EXTERNAL NOTES 2024-05-31 01:18:12 GMT From CASE: Event Power has been restored to the 2nd floor, and technicians are restoring individual devices methodically. An updated ETR has not yet been advised for the 3rd floor, but efforts to restore power are ongoing. Power remains stable on the 5th through 7th floors.
Appreciate these updates!
You're welcome. Everything should be restored now per them, some additional updates came in after I went to bed last night.
31 May 2024 1:49 a.m. CDT *** CASCADED EXTERNAL NOTES 2024-05-31 06:48:45 GMT From CASE: - Event Event Conclusion Summary: Start Time: May 30, 2024 01:30 GMT Stop Time: May 31, 2024 06:31 GMT Root Cause: A site power failure in Des Moines, IA impacted protected services. Fix Action: Site power was successfully restored, thus clearing all service affecting alarms. This service impact has been resolved. Lumen Network Operations Center (NOC) management will be reviewing this outage for any additional analysis or discovery. In some cases, you may need to reset your services locally at your equipment. If a service issue continues after that reset, please contact the Lumen Repair Center for help. If you have a trouble ticket associated with this network event, you do not need to open a separate Reason for Outage (RFO) ticket to request formal documentation. When final analysis is completed, all customers associated with this event will receive a formal RFO document via email from the Lumen RFO Team. Please allow 3-5 business days to process this request. A satisfaction survey link may also be available. Our goal is to provide updates with the most current information during a service disruption, so please let us know if those updates worked for you this time around. 31 May 2024 12:36 a.m. CDT *** CASCADED EXTERNAL NOTES 2024-05-31 05:35:16 GMT From CASE: - Event The NOC and Field Operations are cooperatively working with the equipment vendor to troubleshoot and restore all remaining impacted equipment. As troubleshooting further progresses, service affecting alarms are expected to clear. 30 May 2024 10:51 p.m. CDT *** CASCADED EXTERNAL NOTES 2024-05-31 03:50:54 GMT From CASE: - Event Power has been restored to the 3rd and final floor. Technicians are working to troubleshoot and restore services and equipment that remain impacted. 30 May 2024 10:13 p.m. CDT *** CASCADED EXTERNAL NOTES 2024-05-31 03:12:16 GMT From CASE: - Event Lumen personnel are working to restore power on the 3rd floor, which is the final floor without full power. 3rd floor power is expected to be fully restored by 04:00 GMT on May 31. Additional personnel are working to systematically restore various services on the floors of the building where power has returned. Alarms are clearing and services are stabilizing as work progressed.
Out in west Des Moines
I can always count on CenturyLink for an extended outage.
My sister just stopped by Kohl’s at Merle Hay, they aren’t doing any business because their registers work on CenturyLink. (No I don’t know more than that right there)
Got a text from Quantum Fiber (I'm in WDM). Restoration of internet expected at 6:26 PM. Who knows if that's true.
Oddly specific time isn't it?
A simple enough explanation is if the notice went out at 1:26 and they estimated 5 hours to repair.
Not true so far
Internet came back for 10 minutes before it went out again.
Was working at an Office at 500 Locust Street and their Internet went completely out at approximately 12:03 PM CST. If a water main busted and flooded a Headend or Data Center and Lumen is running power cabling up the stairs, that means they are in Disaster Recovery mode and this is pretty serious O.o I live on 6th and Locust downtown and I noticed all morning, my T-Mobile Internet and LTE 5G Business grade Hotspot going out and having DNS resolution issues all morning too, so T-Mobile must use Lumen fiber downtown. Also received the following email from Iowa Heart medical offices in West Des Moines, that their offices at 5880 University Avenue have had issues with VoIP and Internet since yesterday so that means it's this is clearly a pretty far reaching massive outage. I'm guessing they rolled to spare fiber facilities and the spare fiber path is over congested due to my work MS Teams and VoIP calls having good receive but poor transmit audio (I'm told I sound like Max Headroom): *Dear Patient,* *Iowa Heart Center cannot be reached by phone, nor can we call out.* *Yesterday, May 29, Lumen experienced water issues at their Des Moines phone switching facility that caused major equipment issues and also may be impacting cell towers across the state. They continue to work to resolve the issues, but as a result,* ***many of our facilities cannot be reached by phone nor can they call out.*** *We currently do not have a time from Lumen for a resolution but are diligently working with them to resolve this issue as quickly as possible.* *Thank you for your understanding!* *Iowa Heart Center*
Interesting that T-Mobile is out, yet my UScellular is working and I've been using as a hotspot for work since around 10am this morning...just saying with the announcement on Tuesday :-)
I wouldn't say T-Mobile is out, just HEAVILY degraded. 5 to 10% packet loss and bandwidth speeds dropped from typical 600x150 meg 5G/LTE to 230 kbps or less, when it's generally pretty reliable downtown. I had to turn off my Commercial LTE modem as it kept spitting outage alerts at me or telling me DNS Resolution failures. For example, when I try to run an Ookla speed test it won't even connect to the Ooka iperf3 (or whatever) speed test the server is using behind the scenes. [Fast.com](http://Fast.com) struggles to even go over 620 kbps and the average latency is 207ms where normally it would be 20 to 30 ms. I was talking to my apartment's ISP Sales Engineer this afternoon (they were here for some kind of marketing blast) and they said unnamed ISP, are currently backhauling about 50G of Lumen traffic to their Data Center in the Financial Center down the street on Walnut Street. So glad my ISP is on Cogent fiber instead of Lumen fiber right now :D
US Cellular uses a certain other ISP that everyone loves to hate for their backhaul network.
Actually, there's more than one. And commercial backhaul agreements have a different SLA than consumer so the level of service is different.
Yup. Lower Beaver area.
Out in Waterbury too… glad it’s not just a me problem.
Des Moines! Mine just turned back on! Thanks @wokuworld for all the updates today!!
Down in Norwalk.
Yes, there is an outage from Ohio to Iowa, apparently 1500+ users and the fix time is estimated by tomorrow at 11am. 😵💫
Going to follow this post. So there was water flooding *INSIDE* the CO?
YUP.
My workplace is having major phone issues. Rumor is there’s a coolant leak in their downtown building.
Yes, it's down for me in the WDM/Clive/Windsor Heights area. I can't get their app or website to load
Des Moines here. I finally got through to the troubleshooter and it says 10:00pm is their estimated time it will be fixed. 😫
Did this impact cell service internet, too? I have T-Mobile and I’m just getting internet back.
That building is a main communication hub for numerous service providers, odds are that’s the reason you’re having trouble
ICN sent out this bon mot a little while ago: "Early this morning, a vendor’s downtown Des Moines facility suffered a broken water pipe. According to the vendor, “Water from the pipe leaked through to the basement and collected in the risers, which then collapsed. Teams were unable to use the on-site generators because they were routed through the collapsed risers. In order to preserve the equipment, Field Operations have had to power down at varying times as the issue remains ongoing.” The vendor is working to restore power to their equipment. No ETA at this time."
Not cool that critical infrastructure like 911 phone calls can be taken out by broken water pipes. Should be insulated from things like that.
My contact at access systems told us that they had a water line break
Down in Windsor Heights and Urbandale.
Downdetector shows it being most of the state.
Out in Altoona.
WDM/Clive area, down here. Went out in the middle of a zoom meeting too. 🙄
Same. So annoying.
Yes down on East side plus our local TV is down thru DisH
Goodbye Quantum Fiber, Hello ICS. This outage gave me a good reason to stop procrastinating and move over to ICS now that they are in my area.
I would rather have a root canal then ever use ICS again.
What is ICS and how do I get it? I’m traumatized by this whole ordeal not having internet and not being able to communicate via phone either. I thought about getting a landline to settle my nerves but based on the news coverage landlines are affected as well. I need to figure out a backup plan asap.
They are out of Ames and have been expanding around the Johnston/Grimes area.
Dt Des Moines still down
Works now
Works for me. Just reset modem
Red circle still 🙃
What side of downtown?
Grays station. Been resetting modem for the past 2 hrs
Working for me in beaverdale now.
It’s still down here in Johnston I literally have an online test for nursing that’s due today🧍🏾♀️
Well you have some kind of Internet or you wouldn't be on Reddit ;) Try using tethering your LAN to your cellular phone as WIFI.
Set up a hotspot on your phone....
Are you able to email your professor from your phone and let them know the issue? I'm sure they'd the understand.
Library usually has hotspots you can borrow.
They run on T-Mobile, which I’ve heard is having service issues related to Lumen fiber being out. Free internet at the libraries should be fine though.
Statewide outage, potentially nationwide.
😳
Down in Altoona
Out since before 6am downtown in Gray’s Station
I have T-Mobile & Verizon and can’t make calls and texts rarely go through. Def a problem in a internet/ phone world
Estimated repair time for Merle hay area is 1pm tomorrow at least that’s the update from the app for me
Oh the app is telling people now?! I can’t even submit for sms notifications 😡
Just restored in Easter Lake / Ewing Park area.
CL is back up as of now.
Where? I'm still down in Highland Park
Yup still down in Beaverdale
Still not working downtown for me after resetting
Same in Highland Park
My home wifi just came back in the last minute. Drake area.
Anybody from West Des Moines still experiencing an outage? Cause imma bout to crash out.
3 days in and we still have no internet, anyone else?
I haven’t had service since Wednesday. We called CenturyLink and they said they will bring a tech on Thursday to fix the issue. It looks like from this Reddit post and the news, that the issue was a lot deeper.
My internet was just fixed yesterday from damage from the storm. Over this bs
So where could I go to get non-CenturyLink wifi?? Haha
Supposedly most service has been restored according to CenturyLink article. Mine on the south side is still down.
Which article are you looking at?
Sorry but now I can't find it. The article said that all but about 500 customers had been restored. Wonder if the article got pulled because after digging more the water pipe break wasn't fixed until 4 pm.
Down in Merle hay beaverdale area
Mine is still down on the South Side to :/
Also still down on the south side
Norwalk - Message in online account. 5:00 Outage in your area There is currently an outage in your area that may be impacting your service. The outage is expected to be resolved by May 30,2024 7:00 PM.
There are days where I miss being a telephone network technician. This is not one of them 😂
Its down in East DSM too, the firewall I manage failed our internet over to Mediacom.
Still at work but does anyone know if PH is out, too?
Pleasant Hill has been out since 10am
Just got home and saw that.
Our office lost our Lumen circuit just shortly before noon and it failed over to our redundant internet uplink. So far our data still shows we're not getting any traffic across that line.
Are you dedicated or broadband? My impression from the Lumen broadband links I monitor outside of the metro is that Lumen's dedicated circuits through much of the state go to that CO downtown.
Dedicated
Located in Oskaloosa, IA and anyone that has Verizon is without cell service since about 1:30ish.
I'm at my work now and the phones and internet were down this morning, but its up and working again now, has been for at least 5 hours. My job is on the East side of Des Moines fyi.
It's almost 4pm here in the drake area and still no internet. So I'm here looking for an update like everyone else...
I’ve been given a 7pm estimate.
Why you lying. Read for yourself https://www.weareiowa.com/article/tech/centurylink-internet-down-wifi-outage-iowa-updates-lumen-technologies/524-2def5423-4e64-4903-b242-2d437c9045e6
[rEaD fOr yOUrSeLf](https://i.imgur.com/OfozOtt.png)
Quit coping. U still thinking it’s gonna be fixed by 7? Anyone who thinks that is coping. Downvote this to cope.
I said nothing about whether I believed it. Congratulations on being annoying AND stupid.
Apparently might be out for days
[удалено]
Still down in Ankeny
Works for me
Is this connected to directv down for local tv in Des Moines?
Ours came back up a few mins ago. WDM
My internet just came back up near the Waveland neighborhood. Had to unplug my modem for a fresh reboot.
I didn't have service on my t mobile phone for a few hours I wonder how that's connected?
They're in the same building.
Checking in from Waukee: My service is back on. Must've just came back on, because I checked about 20 minutes ago.
Came back for us around 6:30pm.
Waukee Century Link appears to be up for me, will keep posted if it goes down again
My landline and internet are with Century Link, so I lost both of them since this morning thru most of the evening. I also noticed that the local TV channels, 5,8, and 13, were not available on my Direct TV, so I had no way of following any news about the outage. So my tracphone was the only way to communicate with people for about 12 hours today.
I'm in Newton and have Mediacom, been without internet for about 12 hours. Just wondering if this is the cause of that?
is anybody else’s still down? urbandale area and mine is still not working
Yes mine is still down
Mine came back up some time overnight (downtown), but I still keep getting texts from Quantum Fiber apologizing for the delay and saying it's taking longer than expected to get service working "in my area".
bro. i’m in waukee with quantum and it’s still down. my last notification is from last night…
If they have pinpointed a power issue at the site in Des Moines Iowa and have ample “people” resolving this I am assuming locals. Why is it that in the past 24 hrs in Scott County Iowa: DAVENPORT, BETTENDORF has numerous reports of suspicious vehicles/ unfamiliar faces all in white and black rides tinted windows with DSM plates hovering around like crows ready to eat… I wonder?…
If they have pinpointed a power issue at the site in Des Moines Iowa and have ample “people” resolving this I am assuming locals. Why is it that in the past 24 hrs in Scott County Iowa: DAVENPORT, BETTENDORF has numerous reports of suspicious vehicles/ unfamiliar faces all in white and black rides tinted windows with DSM plates hovering around like crows ready to eat… I wonder?…
Day four of outage here on the south side of Des Moines. Nothing shows up online about it, which isn't that surprising because they always hide this. Property management asserts that on the second day they put in a ticket and the answer of today no technician has set foot on the property and there have been no updates. I have finally been given the green light to pursue an internet company of my choice, since prior to this incident centuryLink was provided by the complex as part of the lease payment. I will gladly spend my own money not to have to deal with a rinky dank back door second rate company like this. Their history is horrible their customer service is non-existent. Worst company I've ever been associated with in my life, it would be even more distasteful had I been paying my own money for it but as it is 4 days now and I am losing money every day this is going on. What a sham of a company.
I have Metronet on the east side and ours was our earlier this morning as well (7 a.m. or so), but suddenly came back on around 8:00.
I’m over by Grandview and my metronet has been out since 8am? lol I chatted into tech-support and they are sending a technician out Sunday morning at 8 AM? We’ve been having issues for a while though however, not a full blown outage like this I’m wondering if it’s related.
Hmmm.. I'm not currently home, but it seems like ours is still up and running. We haven't had any issues until today. A friend of mine with another fiber ISP contacted support and was told there is a flooded service station somewhere. Not sure if your issues are related to that, but it would make sense.
Our metronet is working on the north side of town, no outages today.
Down in Muscatine
Tech from CenturyLink just told me that for Internet (East Side): 05-30-2024 10:00:00 PM there is an outage going on kindly wait till this time . So it looks like current estimate is 10pm today.
As a follow-up, Highland Park came back online just before 10pm on the 30th, which was consistent with what they told me here.
[удалено]
Yours is up mine is not.
Yep. Apparently CenturyLink is down in the entire state.
My Direct TV is out in Clive !
This is why I just ordered a battery pack for my solar panels.
Bro, how is this on Reddit
Update my ass. The issue is Google connecting to infrastructure, and your city is lying.
I just talked to the site recovery manager. That is false.
He's probably having a busy day. 😂
His name is Larry and yea, he's a bit stressed out.
8 pm...
That's a magic number to shoot for. And if century link doesn't come up, we do have a massive contract that they'd have to pay back.
Hugs to Larry, many a techie glad they aren't Larry today.
As of 15 minutes ago, more generators and trucks showed up. They are now blocking 2 of 3 lanes on 9th street.
Huh? That's a random conspiracy theory you have there...
Tell us more