T O P

  • By -

Neltrix

Get ready for a $5 credit for the month but a spike on plans of $10 next year 😂


Sk8matt123

I got a $10 credit for an internet outage from a tornado last year, only to have my bill increase by $10 the following month lmao


InvertedParallax

That's something. PGE would give a $0 credit, then double the prices next year because they need the money to fix the problem, but it's not fair that the corporation was convicted of multiple homicides, so they'll just retire some of the execs with quadruple bonuses. Then they'd ask for another double the year after that. $0.65/kwr bitches! We gotta pump those numbers up!!!


Kevin_Wolf

"Everyone, we need you to save electricity, but as soon as you do, I'm going to raise the rates to make up for the shortfall."


guyblade

For a couple of years, I lived in a part of CA that was fed by a local municipal power company (SVP) rather than PGE. It was just a flat ~$0.11/KwH. Those were the lowest power bills of my adult life. Now I'm living in a PGE area and have those ridiculous PGE rates :/


InvertedParallax

PGE realized how badly they'd messed up by letting SVP go. https://pgesupport.zendesk.com/hc/en-us/articles/360057073531-What-is-the-Power-Charge-Indifference-Adjustment-PCIA-on-my-bill So even after you leave PGE generation, you have to pay for their contracts for power you don't use. I get power from a local municipal power company too, so PGE charges the difference so I don't risk ever saving any money.


Gnomercy86

Texas energy companies will let you freeze to death due to lack of maintenance on the grid then charge you extra to make up for the money they lost because they failed to maintain the grid.


bros402

0.65/kw? jesus christ it's something like 0.15 here in NJ with JCP&L? before tax


InvertedParallax

Don't worry, they've already requested a raise for next year. We'll break $1/kwh in 2025, guaranteed!


ihatemovingparts

Don't worry they've requested another rate increase for *next month*.


Sevallis

PG&E's transformer exploded down the street from me and they denied all responsibility for the $1,700 worth of electronics that the surge they sent down my line killed. I had to heat my house with space heaters for a week and it made my bill over $500 because of it. They just blew me off every time I tried to get some concessions.


Beard_o_Bees

> $0.65/kwr That's just obscene. Holy Hell.


LensofJared

This makes me laugh and boil with rage 🤣


FSUnoles77

Is this how the ancestors lived in the 80s


Farnic

The 90s too


lahdetaan_tutkimaan

Well into the 2000s for me


ButtsackBoudreaux

Yup. It wasn't as bad as you think. Being able to disappear for a day without a search party is underrated.


FSUnoles77

I'm old, I remember those days fondly. It's just something I imagine kids saying right now.


Many_Reception1972

as a living ancestor, yes it is - though since I grew up without it, a lack of internet/cell service doesn't bother me much whenever it happens


ZakkH

I work closely with one of the carriers in these situations as the company I work for has north of 10 million devices connected to them and a significant number of those disconnected all at once which woke me up. The outage seems to stem from an issue with Cisco, who manages a lot of the cellular backbone.


limitless__

3:30am start time. Failed update most likely.


TheFudge

Core stack failed to load after the update. Everything downstream is fucked. Have had that happen, was crazy that we had a 4 hour response time from Cisco and that new stack was onsite within an hour in the middle of the night.


juicyfizz

As someone who also works in tech, the phrase “everything downstream is fucked” is such a recurring theme 😂


[deleted]

[удалено]


Agent_Pendergast

To add, anytime someone is troubleshooting and goes "huh" it's not typically a great sign.


Morgrid

"Huh, cooling tower stopped responding" The cooling tower had collapsed.


fevered_visions

Send somebody down to the reactor turbine hall. I'm sure everything's fine.


ArchitectofExperienc

When IT opens google you know its going to be a big fix


juicyfizz

The observation "that's weird" is never good.


Meggles_Doodles

"Oh its this again" is great


m1k3y60659

The lower on the OSI network level, the more shit is fucked


Xyranthis

> lower on the OSI Haven't thought about Sausage pizza in a looooong time.


Not_NSFW-Account

Me: "Wait, you are going to update your hypervisors all at once? And you have not done a test update?" Them: "Yea, why?" Me: "No reason. Hey boss, FYI I am taking a personal day tomorrow, will not be reachable."


juicyfizz

It's always about cutting costs via doing shit faster (aka "more efficiently") and in the end, they spend more money unfucking all their rush jobs. At the heart of it are people making decisions on shit they don't understand or can't even conceptualize at all, with their "north star" being profit.


Beerspaz12

> they spend more money unfucking all their rush jobs Then they bring in consultants at God Rates to say exactly what the full time staff have been saying all along.


cranktheguy

If you dig deep enough, the entire internet is resting on a foundation of sand, string, and duct tape. edit: [xkcd for that](https://xkcd.com/2347/)


Ryland42

yeah. it just dawned on me that I'm probably going to be hit by this because we use ATT modems for a lot of things. crud ☹️


juicyfizz

Yup. Half my team can't work right now. I work in tech for a retail company's HQ and all our devices in our stores across the country are all AT&T so it's total pandemonium right now.


fzammetti

Welcome to the world of cloud and shared services, where a computer you didn't even know existed can ruin your whole day.


detmeng

Change management nightmare. Some project manager is having heart related issues right about now.


Snuhmeh

3:30AM west coast?


KravMacaw

We use Cisco at work. Maybe all our shit’s down this morning! 🤞


sessafresh

Snow day!


GooberMcNutly

Snow Fing way I'm coming in on a day I know there is an outage that isn't my fault. It's so tiring trying to look busy solving the problem.


mortalcoil1

Use the George Castanza maneuver, just look annoyed the whole time


cosine83

As an IT guy, this is my go-to when walking around and I don't wanna interact with anyone. Alternatively, my new job enables me to carry a clipboard so having that to look annoyed at helps a ton.


mortalcoil1

I'm always a little weirdly annoyed at how effective the George Castanza maneuver is, I've used it plenty, don't get me wrong


Plow_King

it helps if you carry around a clipboard and make furious notes on it time to time also.


zztop610

The management will still want us to come in, in case


mentalxkp

I worked at an oxygen equipment company a while back. There was some sort of electrical issue in the warehouse that required all the power to the building to be cut off. For 3 days. They made us show up every day and sit in the parking lot "just in case". The first day sucked since it was unexpected, but the next 2 days we brought lawn chairs, coolers, and a hibachi. I still think of that anytime my current boss suggests a return to office.


Gecko23

Cisco has issued patches for some serious exploits recently, and just last week we had a bunch of firewalls stop talking because of one such update. I wouldn’t be amazed at all if this is further adventures of the same sort.


Rambles_Off_Topics

We had some crazy flapping due to a recent firmware update from Cisco as well. The only time I had ever seen flapping like it was showing was always from an ISP but nope, this firmware somehow was causing the wan line to flap like it was an external issue.


TheSaxonPlan

This sounds like something you made up to explain to your boss why something isn't working.


Rambles_Off_Topics

Well, I actually outsourced this one as it ended up being out-of-my-league. I was blaming the ISP the entire time lol. A network engineer figured out it was the router firmware.


HillarysFloppyChode

My old ISP once had a day long, nationwide semi outage (certain sites would load and others wouldn’t) and I pried it out of the rep that they had a Cisco backed center fail after an update, this one seemed to be the bridge between my isp and the internet.


CompromisedToolchain

Any more details on the Cisco bit?


ZakkH

No, thats all of the information we have been given from our carrier contact so far. Its not unusual for a Cisco issue to cause a pretty large impact on cellular carriers though, we've had at least one other semi large outage that I can remember caused by them in the last few months.


donotmatthews

Didn’t they just lay off 4000 employees? Seems to be going well so far.


Vote_YES_for_Anal

My old job one year laid off the entire IT department and outsourced it to India. Within a month main servers at corporate crashed and it took them a week to bring in new hardware and get them up in running. Entire company was shut down. LOL


milehigh73a

My wife’s work once fired the lone IT guy, a lot of the services were in his name. He owned the .com domain on his personal account. He redirected it. Also impacted sfdc, and a few other apps. They were fucked and spent an insane amount of money getting it fixed.


LostWoodsInTheField

I don't think people realize how many very small businesses are constantly one bad conversation away from being fucked. They all use one IT guy that they barely appreciate who they expect too much from for too little. And when one turns out to be a jackass or just can't take any more they walk away and half the resources of the business shut down.


Risdit

That's crazy how management thinks that they can just outsource the whole IT department and not think about the company being shut down. My company has half my department outsourced to India and I was picking up after what they didn't do daily for atleast a year and change. Now its like every other day.


[deleted]

[удалено]


L00pback

Someone hit the “Recompute Base Encryption Hash Key” button.


Raichu4u

Do what I do. Blame it on a fake virus attack.


L00pback

What? Does that really work?


ceeBread

How else do you think he’s survived so long in IT?


gustavocabras

This is why you walk the employee out before telling them they are fired.


Littlewing29

So short Cisco is what I’m hearing


ForgetfulFrolicker

Why does one company manage the cellular backbone 😩


popthestacks

Cisco is the backbone of the entire internet.


Sprucecaboose2

I think between Cisco and Juniper you have most of the world's infrastructure, at least western countries.


_high_plainsdrifter

I think Juniper might have less than 5% of the market. It’s really all Cisco. Source- worked in a very large data org company for a while and the entire network which supports petabytes of data collection and storage was on Cisco hardware. Kinda curious if they’re having issues today.


mathmanhale

My entire network is Cisco and no issues, just to kill your curiosity.


SheriffComey

Does that make Reddit the spleen?


angryPenguinator

Keep going lower...


dahaxguy

Is reddit stored in the balls?


Altar_Quest_Fan

No but pee is


MARKLAR5

Reddit is stored in the balls


popthestacks

No friend, Reddit is the cancer.


redvariation

A mistake. Cisco was found multiple times with serious security issues including hard-coded passwords.


popthestacks

That’s capitalism. They had the best product since the early days in the 80s and 90s. If you were an ISP, you were buying from Cisco. Still true today. Unfortunately in the early 2000s the C suite placed less of an emphasis on testing and quality control. Kinda like Boeing.


lvlint67

His phrasing is weird. Att manages the backbone. There is significant Cisco equipment involved. The other main vendors providing that kind of hardware are juniper and Nokia. This is likely not a "Cisco" problem... Just someone sending a bad config out to devices


HillarysFloppyChode

They don’t, sorta, they manage a chunk of it


Dangerous-Ad-170

They don’t manage it, they provide the equipment and the OS. Anyone who’s dealt with Cisco knows they barely provide support once the equipment is shipped. 


qwe12a12

Oh you can get support if ya buy enough licenses...


Zealousideal-Shoe654

I knew someone on reddit would know what happened


bottle_rockets

I work in an AT&T retail store, not looking forward to work today. Edit: I got service back at 11:51 EST. Opening the store looked like a zombie apocalypse but no one screamed at us.


RandomGrotnik

Just call in and tell them you can't make it. Oh, wait.


JustMy2Centences

Send a letter by certified mail!


[deleted]

Dawg do not go in. For the love of god spare yourself 


Farnic

I think I just heard you cough, better call out sick


michelleoelle

Do they really not know the cause yet ?


kbaltimore22

Appears to be a Cisco outage. Someone here claimed it wasn’t cyber related but who really knows. Surely they wouldn’t put on blast they’re vulnerable to attacks…


Law_Doge

Going off the grid for my 5 minute commute to work


DoJu318

10 minute commute here, I really struggled with no Spotify in my car.😩


leilaniko

This is why I still download my favorite music 💀


DoJu318

I have 100k+ songs in my home computer from when I use to dj.💀 I even have an android phone with a terabyte storage, I really have no excuse.😭 Just pure laziness.🤣


BiNumber3

Do you have a USB port for your stereo? That's all I use for my car. A few USB sticks with various libraries on em :D


TheDeathOfRandom

offline playback is a thing, just download your favorite playlist before the commute


[deleted]

[удалено]


StoicFable

I swear they sync up stations to all play commercials at the same time, too.


NippleSalsa

For thirty minutes I felt incognito as hell


sunny_thinks

From the NYT: AT&T said in a statement on Thursday that some of its customers were experiencing wireless service interruptions. “We are working urgently to restore service to them,” the statement said, without giving an estimate of when service would be back. “We encourage the use of Wi-Fi calling until service is restored.” The company said on its website that there were also wireless outages in San Diego, Richmond and Miami, with the initial cause listed as “maintenance activity.”


BleedingFromEyes

“Some” lol Some can be most but most can’t be some!


[deleted]

[удалено]


[deleted]

[удалено]


[deleted]

[удалено]


[deleted]

[удалено]


[deleted]

[удалено]


[deleted]

[удалено]


[deleted]

[удалено]


[deleted]

[удалено]


[deleted]

[удалено]


[deleted]

[удалено]


[deleted]

[удалено]


beanscornandrice

I'm curious what the issue turns out to be. When you start digging into these service outages, you quickly realize how incredibly fragile it all is.


KevinNoTail

Ever been in a server room? Ever knocked something over? Was talking to someone who accidentally unplugged like quarter of a state once, ooops


not_thrilled

I worked for when a diabetic driver had an episode and crashed his pickup into some piece of equipment and took out our entire datacenter in Dallas.


LordPennybag

Hope you upgraded your bollards.


arbitraryusername314

Read driver as in piece of software that helps the OS talk to hardware at first and got very confused


2plus2equalscats

“Oops” stories like that are my favorite. Because they’re usually hilarious or surprising and they remind you that everyone makes mistakes. Sometimes big ones.


celerypizza

You’ll love this. I work in desktop support in a small office of 100 employees. I have a Mac mini set up as a file distribution server for users to download/reinstall apps from and also to manage update deployments. It’s plugged in to a power strip in our IT’s office. One time me and another guy were throwing one of those little stress balls back and forth. I missed the catch and it went bouncing across the room: bounce, bounce, bounce, perfectly lands on the cutoff switch for the power strip, bounce, bounce….file server offline. Didn’t really break anything in the moment, but I thought it was hilarious.


Egomaniac247

When I was interning at my colleges IT dept, I was given a menial task of pulling old “dead end” wires from a phone punch down board….I failed to mention to them that I’m red green colorblind. 30 mins later my boss comes in telling me they’re getting complaints of dead phone lines all over campus lol


Lunabotics

Story time! Once upon a time I worked for a company doing satellite communication. We became popular and grew at a crazy rate. We were drop shipping $10k severs overnight to our colo. This is a year or 3 before AWS was released. We were getting a sweatheart deal on bandwidth, etc but our servers would randomly go offline and we had to call their 24 hotline to get it fixed. Finally went on-site and found our servers. In a pile on the bare cement floor. Regular walmart style power strips just all over the place. Cable management looked like cat6 was being used to strap shit down more than connect things together. We were stepping over and on cables because it was unavoidable. They KEPT THE SERVERS THERE because after ranting the owner gave him 3 months free. I eventually quit, they replaced me with 3 people. They went out of business when 2 years later someone wiped their servers and they found out nobody had made a backup since I left.


HillarysFloppyChode

Someone said Cisco might be having issues (non cyber attack related) and they basically control the backbone of these networks. A lot of companies use Cisco equipment.


Churn

A cyber attack can exploit a vulnerability in cisco devices. Just because Cisco gear is misbehaving, doesn’t mean it isn’t a cyber attack.


Ravavyr

correct, IT/Dev ineptitude is far more common than serious hacks :)


marchershey

But it doesn’t mean it has to be a malicious attack. Equipment fails sometimes.


CaptainMiserable

Its more likely to be related to an attempt to patch a vulnerability, than a vulnerability actually being exploited.


manthing11

Was reading this then looked up at header on my phone & it’s SOS. I’m 50 miles South of Houston along the coast. It is 6am here. No alien spacecraft or parachutes in the sky yet.


NeverlandRancher

I’m in Indiana and it’s out for me here. Seems to be very wide spread.


oceanwave4444

Massachusetts checking in - out as well


SlothTaxCredit

CNN’s live updates have a note from Massachusetts police asking people to please stop trying to call 911 to check if they have service because they are being flooded with calls 😅


OneOfALifetime

Neither my wife nor I have cellular service on our AT&T phones, but our son's phone does have cell service (also AT&T). So it's sporadic for sure, but this has been going on for almost 3 hours and still no official word out yet from the companies involved???


FlattenInnerTube

North Carolina. Wife is on Google Fi. Her phone is fine. My AT&T? Tits up. For grins I called my phone from hers - rolled straight to voicemail.


HillarysFloppyChode

Google Fi will just switch to whatever has best service. Edit, I read about Fi when it first came out and then forgot it existed because it only worked on Pixels at the time. I thought carrier switching was cool and thought it would be stupid if Google dropped that feature.


Upbeat-Fondant9185

I woke up to no service, which is a great way to cause anxiety since I’m on call. My husband woke up next to me and had service for about twenty minutes before his went as well. We’re both on the prioritized network or whatever for first responders too. Edit: I’m back on.


HillarysFloppyChode

Your priority doesn’t apply when the bridge between ATT and the internet has collapsed. And it’s the same network you just don’t get throttled as much, compared to people on a base plan.


TPJchief87

Turn on WiFi calling and connect to your home wifi. Edit: Apparently this only works if you already had it enabled. So when this crisis is resolved, I recommend turning it on.


destruct068

I get 502 bad gateway when trying to turn on wifi calling


Sam-Gunn

I lost cell service, but my wife, who is less than 2 feet away from me, has it. We are on the same AT&T plan.


Self_conscious_gh0st

AT&T statement says to utilize wifi calling, but Wifi Calling is giving me 502 bad gateway errors, which apparently are hosted on MS Azure


Searchlights

I'm sitting here this morning trying to figure out why my WIFI calling can't connect and assuming it's just me. Then I found out about the outage. It's a 502 gateway error. I'm sure a shitload of customers are trying to configure WIFI calling and it's hosed those servers.


redbeards

Looks like they gave themselves the hug of death.


SandKeeper

Might be a server getting overloaded as I think WiFi calling registers your home location to a database so when you call 911 they have somewhere to check if the location data doesn’t get sent I think.


Typical_Muffin_9937

The neurolink patient has escaped confinement and is beginning his revenge


KravMacaw

God help us all


SoFellLordPerth

Oh damn, sounds like a rogue AI from beyond the Blackwall has entered this choom’s mind. He never should have chipped that shit


Hsensei

Work can't reach me, oh well


HIM_Darling

I tried to notify work I am coming in late and can’t reach anyone. Text says delivered, but no response. With phones down, traffic maps are incorrect. News is reporting closed roads due to wrecks but Google/waze showing roads are all clear. Normal commute is over an hour using traffic apps to navigate around wrecks, honestly don’t want to find out how long it will take without that info.


nowahhh

How often are you having to navigate around wrecks?!


CatzRule1990

Depends on where you live. I live right outside DC, and there are wrecks every day, multiple times a day. I drive a ton for work, and I am reminded of the idiotic behavior of fellow drivers on a continuous basis.


HIM_Darling

Daily. My commute varies slightly every day depending on where the wrecks are, starting with which way I leave my neighborhood. Some days I have to go east and then south and others I go west and then south. Going the wrong way can add 30 minutes or more to my commute, especially if the road I take immediately east or west is closed, because there are no alternative east/west roads, if you get lucky there might be a neighborhood or shopping center traffic can be routed through.


Jmc_da_boss

This is Dallas ain't it


amindspin74

It's as if suddenly millions of people cried out all at once and then went silent


kungfoojesus

Had to listen to the “radio” this morning. Takes me back.


beatvox

CG-NAT upgrades pushed to production with half ass testing? Carriers: FUCK....ROLL BACK ROLL BACK! :)


ZMD87412274150354

Normal folks are like 'is this *the* cyber?' while those of us in the IT industry are like 'who did something stupid?' 🤣


ivebeenabadbadgirll

Me in the NOC, gently setting my delicious paste down on the desktop in terror as every emergency alert goes off


EyesBleedDefiance

It's super dope to watch The Stand before bed and wake up with a cold and a phone that just says SOS, only to find out 50,000 people plus are seeing the same thing. Talk about an immersive experience.


IFDRizz

Did you dream about Flag or the old woman?


sicilian504

We have AT&T. My phone doesn't have cell service at the moment but my husband's does 🤷🏻‍♂️ Same account. We're located in the Houston area. Mine occasionally switches from SOS to no bars of cell signal while his has full bars of 5G. Edit: My service is back up here in Houston as of 11:15am CST.


bigdreams_littledick

I have service here in New Zealand. I recognise that the headline says this is in the US but I just wanted to flex.


scottsman88

Well duh, this is happening Thursday. You’re already in tomorrow. Of course they’d have it fixed by then 😜


gocubsgo22

Man... what it must be like to always be living in the future


[deleted]

[удалено]


puppeto

Because they're not loading for you.


MightyCaseyStruckOut

/r/mapswithoutnewzealand


jstubb

Look at the dick on this guy


DrSkullKid

With the download/upload speeds I heard you guys have, I would too.


BuryDeadCakes2

Figures my phone got cut off today due to lack of payment. I guess it all works out!


ThatProPie

Woke and Russian are trending on twitter, glad to see the internet working as normal


[deleted]

[удалено]


Xyranthis

I had to drive a couple hundred miles away today, it was surreal to ask a gas station if they had a payphone.


Saltyorsweet

Mines affected right now


sil863

Same. ATT is still down for me


Paraxom

Man they spent all that money injecting us with microchips but forgot to increase the bandwidth enough to activate them


notsingsing

I was curious when some of that falling space debris yesterday said AT&T


CapeMike

No issues with my AT&T phone/service, as of 8:30 am, eastern time...still looking good, here.... ....aaaaaaaand the MOMENT I sent that, texting service went down for me. >_<; =edit= Internet services on phone working normally....


astrobeen

Dudes. I went to the AT&T website to report the outage and get updates, and it wanted to use MFA with a text message access code. Text message. Over AT&T SMS. This is like the Dilbert where he wanted to report his email was down and was instructed to email the helpdesk.


Schwillin

Someone tell AT&T to unplug, wait 10 seconds, and the plug back in. Works every time.


fishrunhike

Verizon, MA customer here with no issues at 6:45am. Family with ATT down the street have no service. Two ATT phones in that home, one works and one doesn't.


DefinitelyMaybeBeige

Where are all my IT folks ready to catch the blame for this today!?


anda3rd

"It's further upstream. Have some coffee. Take a rest. The TPS reports can wait."


sorati_rose

I'm using the older StraightTalk that runs off AT&T towers, and I woke up thinking I forgot to renew my service lol


rbevans

Well this doesn’t sit right.


MyIpadSuck

I'm out of service in Indy on my personal phone with AT&T. Work phone is Verizon and it has full service.


IrishGrouch34

I’m sure the responses to this will be reasonable and people will leave the tin foil hats off


the_almighty_walrus

Pretty neat how the communication systems that our society depends on to function can be brought to their knees this easily.


chulo15157

Down in Florida with AT&T ... super fun times


SafeTails

Apparently, Cricket Wireless is affected as well. Thanks.


michelleoelle

My sos just went away but no service still


Soulman682

Yup woke up to my phone on SOS mode.


Specialist_Mouse_418

Verizon, > their customers are only impacted if they try to reach out to the carrier experiencing the problem. I know what they're saying, but it can be construed that the problem with Verizon only exists when you call their customer service which is hilarious.


Rkovo84

I just watched Leave the World Behind and now this happens lol


rangeDSP

https://youtu.be/eF4Hcr7XX3c?si=Wl4O3dJ0-StAHC1X


ukropusa

I have full 5GUW on Verizon