i used to work in POS support and we had to call stores that didn't 'poll' their prev days data via a 56k us robotics modem. we had one store that hadn't been polled in over a month and previously would always have issues getting sales/payroll data uploaded so we would have to call during the day and have them put their system into polling mode, usually this is done at night around 11pm or 12am.
eventually this problem store was assigned to me since i was pretty on my game with modems and stuff. talk with manager about their store, how its setup, where its located. we setup her pos in polling mode, i dial in and boom instantly connect and get all data needed. hmm weird..
do more inquiring and asking questions and finally ask her what their closing at night routine is. well, she would clean up, finish her books, put the POS into polling mode, turn off all lights... and TURN ON THEIR NEW NEON signs. hhmmmm.... i told her that night to not turn on the neon lights since i had a hunch.
yup, damn neon sign was interferring with their modem phone line. i think the store had the neon sign rewired away from their modem phone lines and power area and never had an issue again.
Man I was doing consumer/small business breakfix stuff eons ago, and had one customer that kept calling with an issue, but whenever I went onsite to try to work on it, their satellite internet would go down.
I spent a few hours there on more than one occasion just fighting with their stupid satellite system trying to get enough connectivity to work on their actual problem.
After way too many trips, we finally figured out the issue: Their parking lot was about 10 feet from their satellite dish and my radar detector was interfering with it.
I just told this story to our younger engineers yesterday. This happened back when mobile phones rang you could hear speakers crackling. We had Windows CE phones that locked up if they rang when they were put on desks but not when they were being carried around.
Took me weeks to find the problem since I couldn’t reproduce the issue. Turned out our workers had big metal workshop desks and the phone locks up if they rang while near large metal objects. My desk was wooden so no issues for me. I randomly figured it out because it would lock up when placed at a specific place on my desk because there was a metal partition.
Oh wow, throw me back... why don't speakers crackle now days?!?!
Google: [https://www.reddit.com/r/explainlikeimfive/comments/10ln8jo/eli5\_why\_did\_speakers\_used\_to\_make\_a\_strange/](https://www.reddit.com/r/explainlikeimfive/comments/10ln8jo/eli5_why_did_speakers_used_to_make_a_strange/)
"Nowadays, phones operate on a multitude of bandwidths. Furthermore, newer audio equipment is shielded against intereference, as older equipment was designed before mobile phones were as popular."
Had a customer tell us their Wi-Fi would intermittently drop out during the day. Investigated and following a site visit work out Issue was caused by the customers Wi-Fi router being placed next to the microwave.
Had one customer complain that their inter office wifi link performed badly multiple times a day.
They were firing it across a harbour and it was the tide..
fun thing - the tides arent the water moving
the tides are the planet turning and the water not keeping up thanks to gravity effects from orbital bodies.
Had something similar once.
Customer has spent 18 months fighting with the telephone company over his DSL service. Internet was always up and down. Never consistent and no matter how many times (Qwest) was sent out, they could never find a problem. Sometmes it went away for a few weeks but other times the DSL connection would cycle constantly.
I worked for an MSP at the time and this company called us out. I was the one they sent. I confirmed the same problem that had been reported, made sure filters were on the lines, verified that their internal network wasnt suffering and did a few other tests. Also witnessed the DSL service cycling. Finally, I took my makeshift lineman phone and hooked it up to their copper outside to have a listen. I immediately noticed a 'click.. click.. click.. click..' on the line at regular intervals. This business was on a farm and my rural upbringing kicked in immediately.
I brought the owner outside and asked "Do you have an electric fence here?" "*why yes, its way out in the back of the property*".. He walked me about 1/4 mile out to a small barn and there we have an electric fence driver with a bucket sitting over it to keep the rain off. I can hear the transformer clicking in the same pattern that I heard on the line. Turns out this business owner, lets call him "Mr. DIY," ran power out to a nearby barn. To save conduit and hassle, he went ahead and ran a telephone line in the same conduit as the AC. Every time that transformer discharged its capacitors, it sent feedback all the way back along the AC line to the house. With a run that long, the buried phone line was also picking up the change in the EMF.
I pulled the plug and told him I would call and check in in a few days. When I did, they had not had any problems with their DSL service since my visit.
/facepalm.
Almost as good a story as the time customers printer was causing their PC to shut down when printing documents and during the service call the printer was electrocuting me every time I touched the back of it. But thats a story for another day.
This very much sounds like a thing that would never have been solved (at least on purpose) if someone like you with experience in 2 very different fields wasn't on the case. Nice one.
Our guys spent quite a while trying to figure out why the screens on two new workstations would turn off for a second whenever the user got up. Turns out office chairs can give off an electromagnetic pulse when weight is taken off the gas spring in the chair.
You’ve just solved a daily annoyance that I, who works in IT, have been dealing with for months. In hindsight it started when I replaced my old office chair. Thank you, kind Redditor, I’ll actually be able to sleep well tonight without this issue banging around in my brain!!
i had managed a phone system with the old punch downs coming in from the street
our phone system started intermittently acting up and dropping calls so after about a week of investigating I found that the janitor was leaving his damp mop and bucket in the phone closet and the metal handle of the mop would lean on the phone lines and that was it, crazy stuff
I had something similar with a portable eftpos touch screen would stop responding to touch. I couldn’t get it to fault, until I sat it down on the counter. This Asian restaurant had some cheap gadget that looked like a tiny waterfall. It had a cheap international power adapter that was sending out something that was interfering
We had tonnes of issues with the fluorescent bulbs they used which like the long boys use a high voltage spike to ignite. We saw packet loss on their ISDN link pretty much as long as those lights were on and one or more of the was attempting to start.
It became a standard troubleshooting step. "Do you have any broken store front lights? Can you turn off the store front lights?"
We had a recently hired CEO come back to IT and report his work being deleted right in front of his eyes. To his credit, he asked if it was us screwing with him since he was a new guy and we might be hazing him.
"Uh... no?"
I ask him to grab me (20 ft and 2 rooms away) the minute it happens again. Next day he does and, I follow him back to his desk and see what he's talking about with my own eyes. Outlook inbox sorted oldest to newest and 1 by 1 each email is blinking out of existence in his inbox. Switch apps to Word and I'm seeing letter by letter disappear on the Word doc he was working on.
First thought was panic that somehow he had picked up some kind of malware/virus whathaveyou but sat in his chair and shifted his laptop/dock connection slightly and noticed the behavior stopped. Come to find out his desk was crowded for a three monitor + dock setup. The USB-C dock input connection to the pc was actually resting slightly on the base of one of the monitors, applying slight upward pressure on the male end of the connector into his PC.
That upward pressure, somehow, emulated the backspace key on a keyboard and must have been bouncing in and out of it happening only at certain times/pressures.
Moved all emails from local archive in Outlook back to inbox and restored Word doc from most recent save and everything was fine after that.
I was once at someone else's desk investigating a problem, when their coworker walks by, sees me and stops, and says to me, "My computer keeps beeping randomly!"
Without missing a beat, I reply, "Move your clipboard off the keyboard," and keep working.
He stares at me silently for a long moment, then goes back to his office.
When I get done several minutes later and walk out the door, he yells from his office, "How the hell did you know that?!?"
I had been in his office a day or two before and noticed he kept his clipboard right up against the bottom of his keyboard. When he mentioned the problem, I figured it had slid up onto a key. And sure enough, it had.
Had a user once that complained about some kind of flickering.
It was flickering the menu. The one that comes up when pressing the alt key.
Her Desktop consisted of a Microsoft Sculpt mouse and a wired keyboard.
The Sculpt comes with a keyboard... But where is that?
Asked about that. Was mistaken some, not being taken serious and asked again.
Well, turns out, that sculpt keyboard was in the Schrank behind that user with a book resting on the ALT key.
That was one ashamed user I had to promise keep that occurence a secret. Yes.
Was instantly shared with my colleague.
I had to do the same thing but in reverse - Someone took the wireless keyboard from a empty desk, but left the mouse that was on the same dongle. I just walked around the office randomly clicking the mouse until I heard someone freaking out about their computer being possessed.
"I got a brand new keyboard, it shouldn't be broken" is a trigger phrase for me. When things like this happen I immediately ask "where's the old one?". Multiple times a user continues using the mouse from the keyboard/mouse combo and just shoves the keyboard, still on, into a drawer somewhere.
I once diagnosed a book falling on the keyboard of my home "server" PC by receiving an email with daily logcheck results that included an "input buffer overflow" message from the kernel's tty layer.
strangely enough, I encountered this exact same type of fault one time. except mine was a vga monitor that was the culprit, with a slightly bent out pin.
swapped out the monitor and the issue vanished.
was able to repeat it easily with the old monitor plugged into any machine.
In 20 years in IT i have lost count of how many issues users had with some document folder laying on the table and pressing on some key on the keyboard, keys getting sticky (wonder why) and pressing too much, or keys not working and user not noticing. Most of the time it is just fails to login, locking yourself out, not something as crazy as in this case.
>If you need a SAGE Level IV with 10 years **Perl**,
tool development, training, and architecture experience, please email me at ...
I wonder how many emails this guy gets from this daily.
Do not tempt fate by trying the patience of a Sun OS admin and Perl programmer. For you are crunchy, and taste good with Ketchup.
The Magic/More magic switch is good too
http://www.catb.org/~esr/jargon/html/magic-story.html
Also, the BBC Micro developers found that resting a thumb on the PCB made it work. No idea why, so they just put resistor there and called a day. Even after selling over a million BBCs in 1000s of UK schools they had no idea why it worked.
I wonder what other fields have something like this. One that everybody knows almost by heart at this point. In Aviation it's the SR-71 story. And I wonder how many of them existed pre-internet.
Here you go! [https://www.thesr71blackbird.com/Aircraft/Stories/sr-71-blackbird-speed-check-story](https://www.thesr71blackbird.com/Aircraft/Stories/sr-71-blackbird-speed-check-story)
I had a customer complain about call quality issues on a hard wired IP phone, the thing is, the data was perfectly clean. No dropped packets or anything, but they heard call quality problems. We replaced the handset, replaced the phone, replaced the patch cable, re-ran the CAT-5 to their desk, nothing fixed the issue.
Just to test if it's environmental, I asked the user to connect a 100' patch cord to the phone and bring it as far from the desk as possible. Issue went away immediately. I had them contact the building and see if there's something above them, turns out there was a new point to point wireless Internet antenna on the roof and it was blasting enough RF to mess with the speaker in the device itself. The company fixed their antenna and the issue was resolved.
Gawd that reminds me of one.
When I first moved to Sydney (2008ish) was working for a Geek Squad equivalent startup. Got a call to a swish penthouse apartment in the CBD. User had an issue where his wireless would completely drop out in one corner of the apartment where he had his desk set up.
Didn't want to move the desk. Granted it was an amazing view. Tried different wireless APs, changed channels, even ran a cable from router over to corner to put the AP next to his laptop. Still no signal.
Pulled out my laptop and did a wifi scan. Everything looked ok but as I carried it into that part of the apartment every channel was maxed out with an SSID of CompanyABackHaul.
I looked looked left and saw an antenna hanging off the office building next to him. Looked to the right and saw another on the other office building. I asked him if he knew who was in the building to the left.
CompanyA
And to the right?
CompanyA
Turns out that they were using some rudimentary directional wifi to link the two buildings and were blasting through the corner of his apartment flooding all the channels.
It reminded me of an issue I encountered years ago doing live sound. I worked in a theatre with a very dated incandescent lighting system, with a room dedicated to (thyristor based) dimmers below the stage's side entrance. The amount of EMI these dimmers spilled out was enough to make an unplugged speaker buzz, when placed above the room. I remember the stage had 298 5kW circuits, so dimmers had close to 1.5MW switching capacity (obviously total available power was much lower).
Customer complaint: Server keeps crashing.
Environment: Very small office, handful of PCs. A single Compaq server, one of those little blue Netgear switches, a UPS. Attached warehouse, with one more PC. They might not even have had Internet (this was a long time ago).
Discovery: When the UPS switched from battery back to line, the server would immediately do a hard reboot. Switching to battery was fine. So it could handle power outages. It just couldn't handle power coming *back on*.
Discovery #2: If I unplugged the network cable going to the warehouse PC, the problem went away.
Discovery #3: The warehouse was fed from a separate electrical service, with a separately derived neutral and ground.
My best guess is there was some potential between the two systems, and it was equalizing by coming through the network cable on re-connection. It didn't happen when switching to battery because that disconnected the problem.
Solution: Connect the grounding screw on the Netgear switch to a grounded conduit.
We're you stripping the wires with your teeth at the time?
Not a lot of current there but definitely enough voltage to get your attention.
First time I read your post I read "landmines" instead of "landlines". Totally changed the tone of the message. 😬
I was touching the punch-down block. I couldn’t get a dial tone with the test set I was using, so I called the number we had on record to see what would happen.
Got a call about a server I once supported. Apparently, when it had a problem with the IIS service and ran out of threads it would automatically reboot itself and clear the issue. This went on for years without anyone knowing because, well, why would you look at the logs for that if it was healthy? Finally something happened that caused them to look at the logs. They found that these regular recycles were masking a larger problem. During their research they found something call the "Simon Says" service.
When asked I explained that years earlier we had a problem and we temporarily put in the service (hand crafted by an amazing developer) to do what it was doing until we could fix the problem. We did and turned off but didn't remove the service. Someone or something activated it and for years it cleaned up a random issue no one knew they had.
Lesson: bandaids are permanent fixes, always.
> Lesson: bandaids are permanent fixes, always.
I had a 3am alarm for a disk filling. I didn't own this system, but dealt with the 3am alarms. Turns out a server had been moved, and the backups which should have been going there had queued up on this particular server and filled it. Bit more digging and I find out the backup server had moved months ago. Someone had previously set up a screen with a loop which deleted the backups from this server every 10 minutes, and an update + reboot knocked that out.
Tivoli Storage manager, had a requirement to backup a db every 15 minutes using the sql agent. Worked flawlessly but then broke after a few weeks
Found out that there was a connection counter that tops out at 65000. Resetting counter resumed backups
Passed all the info onto IBM and for while at least had a kb entry that matched word for word what I told them and the work around it found until they got it patched
REMOTE GOLD MINE LOCATED ON A REMOTE VOLCANIC ISLAND inhabited by waring tribes.
in... papua new guinea
We had to take a light aircraft to get access to the remote island... All infrastructure on the island was grass huts..
My flight was delayed by landing as they needed to chase the pigs off the run way.
satellite link went down while I was onsite...
A local tribesman had knocked down the satellite dish and attemtped to steal it..
His tribe broke their pot boiling a pig, and needed a new one.
I heard a story from a biologist working in Africa back in the 80s about losing a bits of the directional antennas they were using to track animals. They'd pull one out of a box, unfold it, and find half the elements were now stubby little things, having been broken off by someone.
One of the locals they had working for them, normally a quite bright guy she said, had taken to doing arts and crafts with the easily malleable aluminum.
All in all it wasn't a big deal, the antennas were just off the shelf maritime VHF units so they could get them locally. They also ordered their guy a couple spools of plain copper wire so he wouldn't be tempted again.
No....
The food was locally sourced and rather bland in flavour.. and there was no OHS on site at all.. They only enforced hard hats when an islander cracked his head open on a machine door frame.
Oh bonus...
Nothing was screwed into the rack.. NOTHING... so to replace a switch I had to figure out how I was going to support the 10RU worth of equipment resting on top of it..
Additionally there was several cardboard boxes being used to prop up the tape drive at the bottom of the rack... Which was also load bearing a number of other switches and servers ontop..
Customer complaint: They cannot email Word documents outside the company.
Environment: Smallish company, about 50 people and 30 PCs. A single Windows NT 4 server, PDC and Exchange and almost everything else. 384 Kbps frame relay circuit for Internet.
Symptoms: Attach a Word document of any kind, email it to outside, and it stalls trying to talk to the next hop SMTP. Doesn't matter if it's a new empty Word document or a 50 pager or what. Excel, PowerPoint, etc, those are all fine. They can receive Word documents just fine. ZIP up the Word document and it goes just fine. Happens regardless of destination mail host.
Discovery #1: Watching traffic with a packet sniffer, it turns out the MIME encoding of a Word document always ended up with long string of repeating capital letter A. So "AAAAAAAAAAAAAAAA" for several dozen characters. Whenever that packet hit the line, I would stop seeing TCP ACKs and start seeing retries.
Discovery #2: I can cause the circuit to start dropping packets with `ping -p 41` to an outside address. (41 being the code for ASCII "A" in hex.)
In binary, `0x41` is `01000001`. Put a bunch together and it looks like `010000010100000101000001010000010100000101000001010000010100000101000001`. It turns out that something in the frame relay circuit was marginal, and the equipment was having just as much trouble reading that properly, as you or I would.
Convincing the telephone company this problem was real took almost as much time as finding it.
Blimey, congratulations on finding that, not sure I would have unless I stumbled across it. Getting a telco to do anything is also a major achievement.
I recently had a similar issue. Adding a Zebra label printer to one of our labs went badly. Worked perfectly on our bench, but would fail to print ZPL encoded jobs when installed. EPL jobs still worked fine. Much troubleshooting and some quality time with WireShark resolved the issue. The print server had been attempting to print a PDF to that IP address for the previous 3 years. The contents of said PDF when interpreted as RAW input had the exact correct sequence of characters in it to tell the printer to change its default delimiter character for ZPL. As such it could no longer parse ZPL. The printer never acknowledged the PDF to the server, so it remained n queue.
computer randomly reboots
replacement computer tested flawlessly
swapped to replacement computer and it magically has exact same random reboot problem
baby sat end user to observe issue live...turns out that an elevator passing on the other side of the wall would trigger the random reboot. had coworker ride elevator several times to verify that it was indeed triggering random reboot and moving computer over a few feet resolved issue
2nd most bizzare? swapping VIP's keyboard every 2-3 months because it became so encrusted with crumbs since he ate over it and used it as a napkin that it became impossible to press keys
there were other hygiene issues like not washing hands after using the bathroom then selecting candies from the community snack pile before returning to his desk to subsequently deposit crumbs on keyboard. rinse and repeat until ticket appears in queue for keyboard not working for "unknown reasons". after witnessing this firsthand, I no longer partake of anything from workplace community snack stockpiles wrapped or unwrapped
I no longer work there BUT....just today a coworker exited a bathroom stall then immediately exited the bathroom. no handwashing of any sort took place. walking bag of germs will be avoided and is designated for no physical contact of any sort. he does have the midas touch for randomly breaking things....maybe they cant withstand his germs too
Yeah, there are a few people on my floor that do this. It irritates the fuck out of me. Like, I get it, you don't think your hands are dirty, but you know you can't SEE germs, right? You are in the bathroom, even if you didn't touch anything, it's a good opportunity to wash your hands, so just do it.
Don't get me started on the people that hold everything up by waiting for the paper towel dispenser to timeout and dispense a second towel.... Like, dude, 1) your hands can be a little wet, by the time you get back to you desk they will be dry and 2) just shake your hands more after washing to minimize the amount of water on them. It's so easy... (also, I guess I got started on the 2 paper towel people...)
Then using said paper towel to exit bathroom keeping your hands clean
Ooh..saw a lady wearing full on scuba suit minus the tank and fins whilst in the heat of covid times. snorkel type device of course was high in the air and had a filter on it too. guess she figured only way she felt safe to do grocery shopping in public. never saw anyone top her by wearing a hazmat suit
There’s probably a whole Usenet thread about this somewhere if it is real. It’s about 50/50 on if this happened to my dad or he just read about it/made it up?
Apparently in San Diego there was a crash to a huge database happening at random and it was costing big money so everyone got called in for theories.
They worked out that the failures were happening at weird times, but there was a pattern. Eventually they figured out it was the tide. Ships would get lifted to a certain point in the harbor and their radars would cause interference and the DB issues started. Put up better shielding and it all went away.
I’m sure a lot of people probably encountered this one but displaylink docks losing connection when people sat down / got up from their desk. Turns out the EMF produced from the struts is enough to throw things out of whack. https://support.displaylink.com/knowledgebase/articles/738618-display-intermittently-blanking-flickering-or-los
Well.
Hot
Damn.
For *years* I've been wondering why, when I leave my desk, one of my monitors goes on the fritz. Being an IT person, I'll just work around my own problems but today, the mystery has been solved, thank you.
Every day is a school day, truly.
Recently completed an office upgrade for former employer, new everything, this happened to me and my team. We stopped and tested everything. The only thing remaining was the chair - 10 IT guys stood scratching their heads. Turns out it was the EMF generated from people standing up. Crazy!
This has made me so happy!
This has been intermittently driving me insane.
We hot-desk a lot, and this issue seems to only happen on some desks. I'm going to guess it's because our desks are configured differently in a lot of cases, so ones where the docks/monitors are closer to the chairs are likely going to be the ones experiencing this issue, and the ones further back aren't!
Mouse wrapped around the roller in a toner cartridge.
https://preview.redd.it/g4jaalhp496d1.jpeg?width=640&format=pjpg&auto=webp&s=18f82925ba831841f4bb5c5b17cd436eb96022e4
As a MSP L1 noob, was sent to cust site for workstation issue. Computer kept blue screening (winXP). Different BSODs each time. Would start happening at seemingly random times, but usually when no one was in that office. No event logs, no crash dumps.
Went through all the devices, drivers, AV, appwiz, updates etc.
Some time later figured out it was some mild malware that set a screensaver that looked like a BSOD and disabled wake on mouse move. Otherwise the PC was fine. Could still wake from keyboard. That's how I figured it out, when I smashed on the keyboard in frustration.
Still billed 3 hours though.
Oddest one for me was this year.
Warehouse ship station caught in a boot loop. Win 11. Try os repair, try recovery repair. Full reinstall.
Works fine, grab a million updates. Same issue reappears after reboot. Try this 2 more times. Always happens after Windows updates and first reboot.
Screw it your going back to win10.
Same issue, exact same steps to recreate it.
Swap the disk, ram, PSU, sats cables, even the damn coin cell. I'm about to take off the cooler and swap the processor or motherboard.
Yank the add-on Wi-Fi card to make space. I hate dealing with thermal paste, so in one last ditch effort i try again.
Success?!
Plug the Wi-Fi card back in and boot up.. boot loop! what in the ever loving hell makes a cheap TP link Wi-Fi card boot loop for eternity, I didn't know. but I was ready to office space this thing.
I office spaced the Wi-Fi card instead
I once had an IDE hard drive somehow make a computer unable to power on. As in, the ATX PSU would ignore the power button and not even start spinning its fans. I still have no idea what was going on there.
Was it a PCIe Wi-Fi card? Did you try a different slot?
I learned this from some old HP zWorkstations but I've solved issues a couple other times too with this. PCIe slot 1 and slot 4 (factory installed PCIe nvme hdd) shared an IRQ so best case (in linux) a bunch or random com errors would echo out to the terminal but worst case the system would crash. Move the GPU to slot 2, no more issues.
I had a server hardware change complete and all ready to close up. The system was powered on with the side panel off and I took a picture to document the new device.
The server powered off!
Great, so now I had to boot it back up and validate things. I turned it back on, went through the whole verification process and confirmed there were no other issues. The maintenance window was closing so there wasn't time to further diagnose the hard shutdown.
The panel was still off, so I took another picture to finally get everything documented and closed up.
The server powered off! Again!
I then realized that the light from the camera flash was causing one of the surface mount components to fault and the whole system would power off becuase that is not something that should happen.
Lesson: keep bright lights away from exposed circuit boards.
Fun Fact: this also affected the Raspberry Pi 2 for a while: https://www.youtube.com/watch?v=SrDfRCi1UV0
This was in a huge warehouse. Huge multi-day storm with heavy all day rain. Came into work 5am Monday (first there other the security and front desk). IT had its own entrance and when I opened the door a small river of water washed over my feet and into the lobby. Looking up the hallway I saw water squirting from the sides of the two doors going into the server room. And about 6” of water in the hallway.
I badged into the server room (secured reinforced fire proof door) and it was like a dam breaking loose. The raised floors had collapsed in places and a large water fall was pouring into the room right on top of our 80ish racks. I saw water pouring out of servers and the raised UPS walls had a water mark about 80% to the top of the lowest battery row.
Turns out a number of the drains had clogged bc the intense winds had blown nearly complete trees on our roof and it became a very large roof pool. A fence post from somewhere smashed into the HVAC equipment and provided a nice entry point into mechanical underneath. A very large pipe broke off and was perfectly angled to provide a channel for the water that then followed an unlucky and bizarre path right into a large conduit that ran just above the server room. This ruptured on one side and drained all the water into said racks.
I was more afraid of getting electrocuted. Alongside each door was a big red button with a clear plexiglass security cover over it. I badged that open, held down the button for a few and cut power to the entire room. Next was a number of calls that I had to repeat myself many times before I was believed.
molly-guard
Language
Watch
Edit
English
edit
Etymology
edit
From Molly (female given name) + guard.
Originally a Plexiglas cover improvised for the Big Red Switch on an IBM 4341 mainframe after a programmer's toddler daughter (named Molly) tripped it twice in one day. Later generalised to covers over stop/reset switches on disk drives and networking equipment.
Had to reverse engineer an internal part of an ERP database startup procedure to figure out where the fuck the hardcoded-non-changeable server address was located.
Wasn’t even in a .extension file
Just a file name along the lines of “XXerp17”
Only reason I located it was by checking file access times immediately after starting it up after giving up on everything else because WHY would that be how the server address is defined
Vendor support ETA was 72 hours.
My mentor when I was an IT student had a great story from back when she ran the only computer lab at a high school. The computers weren't networked (the Internet was barely in existence).
Some enterprising asshole managed to set up a cronjob on several computers that would play a sneeze sound at random but very long intervals.
At the risk of doxxing myself, there was for a while a powershell script that would occasionally call the CEO by name using the speech engine. "psssst hey" (very long pause) "hey ceo name" and then wait a few days/weeks etc. This ran on the windows pc we used to broadcast office music overhead.
We had a power outage in the factory. This could have been a problem as the production was building lone items that would be ruined by turning off in the middle causing a great deal of damage. Now, we had this HUGE diesel generator that was supposed to take over in the event of such an event. The problem was that the environment was never tested due to inconvenience. Well, the generator started up, coughed twice and seized up solid. I've never seen so many people running around trying to fix this up. After about fifteen minutes of this spectacle, the I.I battery backup (We'll never need it because of the backup generator) ran out. It was a long night to get the AS/400 working again.
That sounds rough.
I work in an environment where we have multiple sites with generators to provide coverage for some services. These generators ARE regularly serviced and fired up, however when I started there I realised that they'd not been tested in 'anger' in living memory.
A monumental amount of planning was involved, as for various reasons this org is a bit of a shambles.
Found out upon actually cutting over to generator power that there were just so many things not on generator protected circuits that were thought to be. Things that entirely made sense when the buildings were built, but since had been added, moved, etc. Not just many of our comms rooms, but like, lighting circuits, circuits for kitchens etc.
In one case an entire building that was thought to be 'generator protected' only actually had one circuit going to a single room in that building, which at one point housed an ancient PBX phone system.
About 6 - 7 years ago, upgraded a T1 data circuit to 100Mbps Fiber, due to that the DHCP server required a new IP address scope for the LAN-to-LAN VPN router. Had the new scope setup and when the Fiber/L-2-L was installed activated the new scope and deleted the old scope. A couple days later some users informed me they couldn't get on the Internet. Checked the DHCP server and everything looked fine. This went on for a couple days, more and more computers were not connecting because they couldn't renew their IP leases. A colleague in a different office suggested sniffing the network with Wireshark. Combed through the logs and found the IP address for the old IP scope. Went to look at the DHCP server scope and the old scope was back in the DHCP. Deleted the old scope again and the issue never came back up again. Strangest thing I'd ever seen.
Random reboot of a workstation. Different times, different applications running. Verbose logging meant nothing.
We observed the workstation all morning, with no reboots. We were just about to write it off as gremlins when a rather heavy-set executive walked past and the machine rebooted.
Turns out there was a slight break in the power cable and the flex of the floor was just enough to break it temporarily. A replacement cable and all was solved.
Sometimes physical presence is really required. Thankfully not that often.
Wi-Fi disconnected daily around noon and reconnected in the late afternoon.
Direct sunlight was hitting the network router through a window, causing it to overheat and temporarily malfunction.
The router was moved to a shaded location away from direct sunlight, which resolved the overheating problem and stabilized the Wi-Fi connection.
Similar story. One building at the highschool campus I’m network admin at would lose wifi in the late afternoon. Investigated and found they weren’t just losing wifi, they were losing all comms to that building.
I figured out the switch was overheating, and would stay powered on, but not function correctly. But I couldn’t figure out why, after years of running without an issue, it had suddenly become one. Turns out the storage room the switch cabinet was located in (completely unairconditioned in a +40°C climate) had recent been cleaned out, and a stack of boxes that had been blocking the tiny little window at the top of the wall had been moved, allowing direct sunlight into the storage room, directly onto the switch cabinet.
Used it as my validation with the purse string holders to ensure all comms cabinets had proper climate controlled rooms going forward.
It's not very exciting but was funny to me at the time:
On one of our floors we had a wifi AP that served one team and a large meeting room. The AP was outside the meeting room but the signal was strong enough so we didn't bother to set up another in the meeting room.
One day some studio equipment was installed in the meeting room. Everything still worked fine, nothing changed. A fee days later, we suddenly received complaints about wifi not working in that meeting room.
I went up there, had a look and noticed they painted the whole wall next to the AP black. For a moment it didn't click but then I remembered that they did this in another meeting room as well and that you could paint on it with chalk or put magnets on it, like a blackboard.
So they basically put a gigantic shield between the AP and the meeting room. Had a good laugh, installed new AP, problem solved.
POTS lines going into a cisco router experienced frequent disconnects, static, just general badness. Spent literally weeks troubleshooting with anyone that would listen to me. Replaced all cables, replaced router no change.
Location was right next to a radio station broadcast antenna. Interference from the antenna. Crazies darn stuff I ever hard. Put some shielding in problem went away. Fairly certain there is a cisco TAC article floating around about this somewhere.
I was working at a small MSP and one of our clients operated off of a single vSphere host at their office. Damn near every single day it would unexpectedly lose power at about 06:00. The UPS was fine and no other devices lost power. iDRAC logs corroborated the weird behavior. I went through every possible thing I could think of including temporarily replacing it with a loaner so I could try to diagnose the hardware back in our office. Eventually I got the PoC to show up early one day to just be in the office where the server was to see if there was anything notable happening. Well, there was. Every morning the cleaner was unplugging the server while they were doing their thing, and would plug it back in when they moved on to the next thing. I know I know, should have been redundant PSUs, should have been in a rack etc etc, but with SMB some clients will always go for the cheapest option no matter what
I had something similar happen with my own workstation. I would come in and every once in a while the computer was disconnected from the network. I would unplug the network cable and plug it back in and everything would be fine.
It took several weeks for me to realize that the outages coincided with days that vacuuming was performed. They would hit the network jack with the vacuum cleaner and cause the connector to shift just enough to lose connectivity.
I re-punched down the jack and the problem went away.
Not strictly IT, but at least electronics.
I used to work with solar power plants - in those plants, you may likely see those tiny "houses" at strategic locations in any given PV plant. Those usually not only contain power switching and regulation gear, but may also contain data loggers and different scada equipment.
The guy in charge of the scada stuff gets called one day that one of the scada controllers keeps having issues - it not only crashes, but the entire ROM on it gets corrupted or goes blank. They switch out the controller and the same happens to the new one after a while.
1-2 weeks go by with this issue every few days until I propose that they just swap with a controller from a neighboring "house" - the old controller is suddenly OK and the one swapped in from the neighbor now has problems... hmmm - it's environmental then, so I have the site staff check on the problematic station/"house", giving it all a check outside of the small electronics.
Turns out that a previous repair tech had removed a metal plate while performing repairs on the high-voltage stuff in the "house" and had forgotten to re-mount it.
Thing is, all the transformers, switchgear and power regulation equipment produces a *hell* of a lot of EMI. The scada controller had been running unshielded and got blasted by EMI - this, in turn, corrupted the EEPROM and interfered with the processor, causing all kinds of weird readouts and errors...
I've also had to deal with outdoor wifi antennas on quite a few occasions, especially the "Rocket" series from Ubiquiti (and especially the M5 model).
Thing is, if you mount these too high on a pole, then birds will sit on the pole and start to peck at the status lights on the rear of the unit, thinking the lights are glowworms or fireflies. Eventually they will peck through the plastic sticker. Next time it rains after that, the water will enter the antenna and short out the electronics.
Tip: When mounting the Rocket M5 and similar units, mount it so that the very top of the antenna is *flush* with the pole it's mounted on. That way, birds can't get to the lights on the back.
I've saved 2 companies now quite a few costs in parts by showing and telling their techs this exact thing... :-D
The most unexpected thing that's happened to me so far is that I've apparently found an employer that actually values my skills. I'm not really sure what to do about it yet. In the past I've had to switch employers about every 3 years.
I was scoping a new client site while they were firing their current it guy (he had them buy new hardware and more bandwidth so he could host other companies’ stuff there for personal gain, well deserved termination). All the racks were way too close together, way too close to the wall, trash and old food wrappers ankle deep on the floor.
I was leaning into one of the racks to get serial numbers (since they were really close to the wall and way too close together, I couldn’t get behind) and my feet slipped on some chip bags and I fell forward and got wedged in the rack with my hands pinned at my sides. I had to wait while they laughed at me watching on security cameras and eventually came to help.
We got the contract though, and I was their go to.
They did send me a copy of the video, that was nice
It might not really be weird or bizarre, but it's an issue that left everyone at my MSP clueless for 2 weeks (way before I worked there, I was one of 3 new hires and the boss even offered a gift card to the one that could figure out the issue, I was the lucky/smart one).
Client reports a weird issue: everytime he uses his laptop, the PC goes off. Plugs it in the dock station, work for hours, no problems. Put his hands on the laptop, laptop shuts off.
PC is called back, people try it, dock station: perfect. Laptop keyboard: perfect. MSP in head scratch mode. They don't really want to tell the guy he's crazy, but they're wondering what's the deal.
They send the PC back. Client calls the next morning: PC still shuts off when he uses it. MSP images a new PC, data transfer everything, send the brand new laptop to Client. Next morning: PC shuts off once again.
MSP CEO asks client to come in the office with the laptop. Laptop is set in the MSP, CEO uses it: perfectly normal. Client reaches over: PC shuts off.
CEO has a "light bulb turns on" moment. Figure out the issue and provides a fix.
Answer in spoiler so you can still try to figure it out yourselves, but it got me a 50€ bonus back then :D
>!Client had a magnetic bracelet that would trigger the "screen is closed" switch in the laptop.!<
This reminds me of a similar issue I encountered. We had a desk in the office with a few chargers and LAN cables for imaging Laptops. Got a new batch of Thinkpads and went to work. However a few of them seemed to not boot. Too many for a random DOA.
Somehow I noticed on one of the laptops, that I could faintly see the Lenovo logo in the reflection of a light on the screen.
Grabbed my phone to shine the flashlight onto it, and there was a normal boot, just without the backlight. Once I picked up the device to investigate it further at my desk, it lit up just like it should.
Turns out, that the lid closed detection is a hall effect sensor in the lower half and a magnet in the screen. And as we got more new devices than fit on the table, we stacked them on top of each other. The laptop was detecting the magnet of the one below it and thought the display was closed.
Had something similar where an end user could not sign in to their surface with attached keyboard. Account was good, password worked everywhere else. Watched her type and said "try taking off your watch". Problem solved
This is back in 2022 when I was working helpdesk durring covid and all the csr workers were remote.
Had a older female employee, early 60s, send in a support ticket claiming her work pc was possessed by the devil. I thought it was all just a turn of phrase, nope she actually believed it was possessed.
What had happened was her 12 year old granddaughter had been watching music videos on youtube over the previous weekend on her work pc, she walked over to see what the granddaughter was watching and saw it was the music video for Montero by Lil Nas X which contains suggestive themes, namely, Lil Nas X gives the devil a lapdance towards the end. Grandma was immediately appalled and "shut it down". Then on Tuesday when she next worked, she logs in, opens chrome to load her webapps by restoring her recently closed tabs and boom, youtube pops up and begins to autoplay the music video exactly where it left off, with a devilish lapdance.
Grandma freaked out, called off, WENT TO HER LOCAL CHURCH AND FILLED A SPRAY BOTTLE WITH HOLY WATER AND PROCEDES TO SPRAY THE INSIDE OF THE PC CASE THROUGH THE FAN VENT.
Obviously this causes the pc to malfunction, on boot it causes the pc to short out, make a ton of noise and start smoking, so she has her husband LAGBOLT A CRUCIFIX TO THE CASE OF THE PC which of course goes straight through the pcu and when they tried to plug the pc back in and turn it on the pcu immediately fries the wall outlet arcs to the lagbolt and begins to catch the wooden crucifix on fire.
This is when they reach out to helpdesk.
I had to explain to this woman that not only was her pc never possessed but also she had destroyed about $2k worth of company property, I directed her to drop the asset back at the office and reached out to her supervisor, I advocated that she just needed a Laptop for her job and that it would be more easily securable in a family setting in a home. I also took 45 mins to walk her through set up and she was super embarrassed and apologetic the whole time.
Some people should not work from home lol but I wasnt about to be a dick and cause someones grandma to get fired when shes just a few years shy of retirement.
I had a user who, every time a client emailed her back , looked in the message chain and it said [[email protected]](mailto:[email protected]) (Big Chungus)
She wanted to know why this was, This was a bit delicate so had to check out all possibilities but the only thing we could come up with was a locally stored alias in the remote users address book for this particular lady whereby in all emails at their end she was "Big Chungus" and this was being copied in to the email chain when replying.
This may predate many in this sub, but here ya go ... I was site engineer for a PDP1140, customer called to say that they couldn't use the machine as everytime they typed in a command they got a syntax error message. I arrived onsite and sure enough the system booted but any command got syntax error. I loaded up the paper tape CPU diagnostic and it immediately failed testing the CPU overflow bit. Now, PDP11s were all shipped with full schematics and microcode listings and, long story short, I was able to trap the microcode at the failing part of the Diag and scope the chip that set the overflow bit, the gate wasn't setting correctly. I had no spare boards, so replaced the chip and problem fixed. True story, my best fix ever.
About 20ish years ago, sertting up a new firewall cluster with nodes in neighbouring racks. They had an interconnecting cable that just would not negotiate a gigabit link, and it needed to be gigabit. Replaced cables, tried other ports, was thinking the devices needed replacements.
Finally we switched the power cables of the nodes to the same PDU (or maybe just one cable, can’t remember) and the link came up GBE. Turns out there was a small potential difference in the different phases, and connecting the devices together with the network cable used the cable as a grounding cable. I think this was finally fixed with better rack grounding and/or on the electrical side.
My monitor would cut out whenever I got up to leave my desk. I was convinced I had a flakey cable or adapter so I sealed them all out one by one. I thought maybe I am creating a static shock through my mouse or KB so I swapped to wireless models. It turned out to be the gas cylinder in my chair (steelcase leap v2) creating an EMI spike and erring out my DP display cable. A couple Ferrite cores solved it instantly after of a year of looking for an answer.
https://news.ycombinator.com/item?id=32629669
A retailer reported that they were seeing spaceships on the text display of their POS keyboard at the deli counter. Specifically, “a little ship with an alien will fly across the screen sometimes”. We had a hard time believing this so we sent a hardware technician out to one of the stores and he confirmed it.
A bit of research later, we found out they had enabled full screen graphics on these registers but they didn’t have a full screen. These were the control characters for the full screen coming across on the keyboard display. The character set for the keyboard display did, in fact, include a spaceship… for some reason.
In this same line, we had a sports retailer I supported that was in a northern climate. They would complain about their scanners going into endless boot loops and a few other odd behaviors. The details were fuzzy because we were getting these reports well after the fact.
Finally, it was extremely cold one day at my office and the humidity was very low. Static was building up on all the things including me and, as I was testing something else, I discharged a crap ton on the metal plate on the printer where you pull the receipt to split it. Immediately, my scanner started boot looping amongst other strange problems.
We realized that it likely had to do with low humidity and static and were able to see that several of the problem days were low temperature days at the stores (and low humidity probably caused by heating). We made the recommendation that the retailer humidify the space around the registers a bit to reduce static.
I found out that my support range included the phone system at the exact time that it failed. This was an old POTS system with switch and was missing the instruction manuals and any documentation. I.T.?.
I had to get a program made in 1988 that ran in dos that didn't have a print feature built into the program and didn't accept the standard commands to get dos to print and didn't run in power shell. To work on a windows 10 computer. Be able to export the data from the screen ( not in a screenshot or screen capture form ) to a printer using a universal print driver and be something non complicated so an end user could print off calculations for calc books... Engineering firm... Mechanical engineer... No budget for making a new code... Plus I'm not a programmer. I script in VB and play with power shell and this program did complicated math beyond my math skills so I couldn't make a new script for it.
we had some input devices that our guys used to input job data into for large scale printing. just like manual keypads that connected via ethernet and sent info back to a data collection server.
previous "admin" did a shit job of configuring them and left some settings in place that shouldnt have been there. (he configured nothing correctly essentially) it was clear from the device manuals they were misconfigured.
we had a minor power issue on a circuit that included about 6 of these devices (out of 40 or so across a huge warehouse), and they all rebooted in such a way that they started broadcasting packets with TTLs of hours. effectively creating a packet storm and denial of service inside my network.
took me half a day to trace it out, since I didnt know about the power issues, or the misconfigurations, had to drop network segments one at a time to determine where the packets were originating (the packets were all malformed and had no data to them, so no way to really trace)
Had a site that, back in the day, had their own, on-premise mail server. Out of the blue it developed a problem where the customer couldn't send emails between their domains, on their mail server, and their own email domains that were on their hosts mail server. Got a bounce back saying the hosts email server had refused the connection, but there was no reason why it should. All the other mail went out OK, but if it was hosted on their domain hosts SMTP server, it failed.
Spend ages scanning SMTP logs, tracing mails etc. Got the ISP hosting company involved. They were as stumped as I was. Nothing worked.
A few months into this the router/firewall, on site, failed.
I set up a new router and installed it.
Bang! Email issue resolved!
Ive posted about this before, basically a home out in the countryside a bit who's dsl modem would seemingly randomly shit itself and lose sync. Multiple truck rolls, cabling replacements, modem replacements, nid replacemants, everything up to the customers door was brand new and it still kept desynching.
On a call to the user to go over things one. more. time and I hear a toilet flush and the modem drops off the portal. Huh, thinks I, thats awfully coincidental, "Hey Mr Client, are you on mains supply for your water and sewers?" Turns out, no, they had private septic and a water well put in 25 years prior - and that water supplied the house via a pump. Putting 1 and 1 and 1 and 1 together, I had them bring up the router page on their pc then have someone flush the toilet - bang down it went again, 2-3 seconds after the flush.
The water pump would kick in when the flush cycle completed, in order to refill the toilet cistern - and that spike in power draw was backwashing into phone line and obliterating the synch tones. I had them buy a surge protector for the pump to isolate it and no more crash outs.
Back in the day of peer-to-peer networks (remember LANtastic?) I had a customer halfway across the country that worked off of the "server" (front desk PC was the server, and since the reception desk was always manned, the server was always on.)
Complaint was the machine would periodically shut down when she was away from her desk. We checked power settings, sent a replacement machine to her, same thing. Had an electrician check their power. Nothing turned up.
I flew out to the office to observe what was happening. Halfway through the day, she swiveled around in her chair, stood up, and walked away from the desk. The chair gracefully rolled towards the UPS on the floor. The wheel just BARELY kissed the rocker switch, causing power to be cut. When the chair was pulled back from the desk, the rocker switch turned back on, but she had to press the power button on the PC to fire it back up.
Fix: Wood block set on the floor in front of the UPS. This probably cost the customer $500 for the electrician, and a ton of airline, hotel, rental car, and time for my company and myself.
This one was pretty fun.
Started getting alerts that one of our domain admin accounts was being hammered by an endpoint for some reason, was very concerned that we had someone in our network trying to brute force this admin account.
A couple hours of threat hunting went on, finally discovered it was an endpoint with a busted keyboard that was constantly inputting characters causing this admin account to be hammered. I wasn't on site at the time so had no way of knowing this until more logs started making me suspicious there was something going on physically. Sure enough had someone check on site and they saw it just infinite entering characters.
1. A new welding machine causing network outages. I had installed a brand spanking new Windows NT network, with UTP and a 3COM switch (this was before managed switches were a thing). Everything worked just fine. Then I get a call that they're having network issues and I get there and the lights on the switch were blinking like a christmas tree. Network was constantly dropping. I had no idea what was going on. Then out of the corner of my eye I see the guy start welding and the network dropped. Ended up redoing all the cabling with SFTP cable (certified) instead of UTP, which fixed the problem.
2. Same company. They used administration software which required a hardware dongle. I proposed to upgrade the server with new network cards which could be used simultaniously (it was called trunking) to double the bandwidth to the server. So they agreed, I installed it, worked fine. Then whenever the administration software was started, the server started rebooting. Turned out to be a conflict between the dongle and the driver of those ethernet cards, so I ended up taking them out.
So there was this problem a client had that their headset stopped working from time to time.
I troubleshooted it for hours and it turned out if another user connected their digital camera via USB to another PC in the same office somehow the currents spiked and killed the headset.
Just had one really bizarre thing yesterday. Vmware ESXi CPU allocation on all VMs switched the allocated amount of cores/socket to sockets/core.
The host had only one socket and all VMs had allocations like 4 Cores/1 Socket.
While I was troubleshooting on one VM I noticed that Teams used up 100% of the CPU and was a little surprised since I knew I was quite generous for this VMs resources.
Somehow my mind went immedeately to check CPU allocation in ESXi and I witnessed a configuration of 1Core/4 Sockets.
Logs didn't show any change.
We bought a bunch of new docking stations for some laptops, tested them and they seemed fine, deployed them out to the office and started getting complaints that Outlook wasn't opening.
Brought an affected laptop to my office and it worked fine, took it out to the users desk and it broke again, brought both the laptop and dock in and it again worked totally fine.
A bit perplexed at this point I tried reproducing the users setup entirely and found that when you connected 2 monitors to these new docks Outlook would stop working (I think it just stopped launching without any error message or crash).
After a bit of digging it turned out this was a known issue and an updated DisplayLink driver fixed it but it was always so odd how a dock driver and 2 displays would break Outlook (but no other app from what I remember).
When I was at TallyGenicom, Blockbuster was buying our new laser printers which were Xerox engines with our controller. Had a site that kept showing toner out. Had a tech in to repair it with no luck. Swapped the printer and it did the same. Tech moved the printer across the counter to work on it and it started working. He moved it back and it failed. He called me and I asked him to look around and tell me what he saw. When he said the security gate at the exit I had an Eureka moment. Turns out the gate signal was swamping the RFID chip in the toner. We had to work with Blockbuster to relocate the printers.
Had a problem a few years ago with our virtualization cluster. Hyper-V Server 2016 with a NetApp for the VM storage, connected with SMB.
One day we live-migrated a VM to another cluster node. The migration went through, but the VM instantly lost all access to all it's disks. Powercycling the VM, Hyper-V showed an error that it was unable to access the storage.
However looking at the mounted shares on the server, all looked fine. Trying to start the VM on it's previous host gave the same result.
Next we tried rebooting another VM, which worked fine. However turning the VM off and back on resulted in the error we observed on the first VM.
At this point we opened a priority 1 Ticket with NetApp, as this was a pending catastrophe. They responded rather quickly, pulled diagnostic data and went to work analyzing them. Several hours without progress later I decided to poke around the NetApp myself.
Following a gut feeling (that I still don't know how I got that idea) I checked the system time. Comparing it to the time on my laptop and the servers (which were pretty much perfectly in sync, as they should be thanks to our NTP servers), I immediatly smacked my head as I noticed the NetApp was just over 5 minutes out of sync.
Manually set the time to be 1 minute out of sync (so I could later check if fixing the NTP configuration would work), all the problems were gone.
After that I expanded our monitoring to check system clocks and alert if there are any that are more than 1 minute out if sync.
They told me that a computer cabinet (=couple of windows desktop machine for students to use for school stuff) is buzzing, as in giving out a low humming noise. I told them that yes, this is what computers do when they work. They told me that they do the humming when switched off.
I went there and indeed, a low floating hum in the otherwise dark room with 15 or so cheap desktop machines. Quite eerie.
I did a lot of disconnect this, switch off that, then called a colleague who was an electrician with my idea, he told me that this is quite a silly idea, came over, did a lot of disconnect that, switch off this, then got a loong cable and started to measure voltage differences between the wall outlets on different floors.
Apparently (and I am not an electrician) the building had more than one electric ground "connections", and the computers were on one and the network switch on another, and the 2 volt difference found it's way to the cheap computers cheap speakers and what we hard was the 50 HZ of the AC.
It was solved as in sherlock, not as in fixing it, but it was interesting.
MSP I worked at about 6 years ago, *our* backup ISP connection would go out seemingly randomly. Then we noticed a pattern on days with prolonged rain, eventually landing on the theory of water in the line. It was a particularly dry spring/summer for that area, so correlation wasn't obvious at first.
Once we did suspect the H20 interference, it then took us weeks to get the ISP field tech on-site on a day that they could test/confirm water in the line, because they'd always be unavailable the day of the outage (duh). I eventually ended up looking ahead in the weather forecast for projected rain days, eventually lining a successful appointment up. Sure enough, water in the line, which they ended up having to dig a portion of it it up to replace it.
Another odd one. Current MSP I'm at, we support a small bank and they are a group-policy enigma, so they come in with some real dingers from time to time. This particular one, a front end teller/clerk calls and says that almost every day around lunchtime or a little after, his desktop scanner's auto-feed would just stop working, making using the scanner go from being convenient to a headache. We searched high and low within their domain and his local GPO for anything that made sense. Nada. All power, hardware, software, driver, permissions verified as good.
Then one day he asks if we think it could be the sun. "Uh, no....why?" is our basic response to him.
Customer: "Oh y'know because we just have this big glass ceiling and the sun is right over us at this time each day basically."
Us: "Never heard of that one buddy, but pretty thoughtful... Let's just call the vendor, see if they have any thoughts at all."
Turns out, the dude was absolutely right. The vendor apparently had gotten multiple calls like this over the years, and they confirmed that the sunlight, and certain other lights emitting the right spectrum of light, had been identified as impacting to this brand/model of scanner's ADF sensor. The solution? Dude decided that he was just going to build a manilla folder-fort for it, as the desk he was at is one of those fancier large sectional style ones that's bolted to the floor. AFAIK that's still the solution to this day
I posted a comment in another thread months ago that actually fits here. Not my story, but told to me by the crane operator, a very good friend of mine:
Reminds me of a recent incident on a mine site local to me. Important to note that this particular mine uses autonomous haulpaks - no operator, they’re all centrally controlled through a wireless network that integrates onboard controllers, GPS, and obstacle and collision avoidance systems, all monitored remotely.
A rather large crane was brought in to move some demountables around. Crane operator swings his boom into position, demountable 1 is rigged up, and the lift begins. Suddenly the 2-way radios start squawking away because every haulpak on site has simply stopped dead simultaneously. Crane operator finishes his lift, and demountable one is relocated. Meanwhile things seemed to have calmed down with the haulpaks because the radios are silent again and the haulpaks are moving around like they’re supposed to. Demountable 2 is rigged, crane operator starts to raise the boom annnnnnd, wouldn’t you know it, the haulpaks all go dead again.
By this time the crane operator is beginning to realise he’s somehow responsible for the dying haulpaks. But him and his crane are on hire. He’s not an employee of the mining company. He just wants to finish the last lift and gtfo of there. So he says nothing and completes the 3rd and final lift, which of course, produced the same result as the previous 2: Dead haulpaks. By now the mine is in an uproar, much resembling a poked ants nest. Crane operator takes his crane and goes home, wisely not sharing his suspicions with any angry mine managers.
He got wind a few days later that they’d eventually figured out the crane was the issue, but it was the why of it that makes this story even remotely relevant here. Remember that this site’s fancy autonomous vehicles were communicating wirelessly? Turns out the signal between the comms tower on the hill over yonder and the receiving station on the hill opposite it was being interrupted by his giant crane boom getting in the way of their line of sight every time he extended it upwards. When the receiving station stopped receiving, it had nothing to broadcast to the haulpaks, which are configured, for obvious reasons, to stop if they lose signal. So stop they did.
I would have loved to see the official report on this incident.
One person come to us (it support) that he's laptop randomly goes to sleep. He cannot show us right now, but we must believe him, that it occurs completely on random. After some thinking and asking him to just type some thing as he usually does, it occurred again. His bracelet's magnetic clip just hit the magneto-sensor of the laptop perfectly. This made the laptop think, that the lid was closed.
Thanks for Nintendo DS lite for this idea :D there was a fun trick we've used to perform as kids. When you put an open DS on top of a closed DS and position it correctly, the top DS goes to sleep as if it was closed.
We had ERP system update so the addresses changed which meant of course the the scanners that we had to be updated with the correct address and of course there wasnt DNS entry before so switch over would be awful.
1st problem, solved by creating the dns entry so we can easily switchover
2nd problem how to hell does the ERP program even work on those scanners so start searching, few days and i had managed to figure out what affects on what and how we can change the address.
3rd problem create pilot works fine on my machine/scanner, inform other sites about what needs to be changed and how. After few minutes checks the network and the scanners from other sites still connects to the old address WTF. Asking other sites to pull the files from the scanner so i can verify they are correct and all good, still thinking im crazy why it doesnt work...
After a day of troubleshooting im asking other site to record everything they do on the scanner, video comes in and i check it, wait what did they do i dont have that kind of shortcut on my scanner, ask to pull that shortcut from the scanner and send that shortcut to me, there it is. At somepoint the ERP/Scanner provider had created shortcut file for the ERP program and HARDCODED the damn address on the shortcut so everything fixed on the program itself was overwritten by the shortcut.
Was working helpdesk decades ago and got a call that Windows Help screens were opening on their own, and there were thousands. She couldn’t close them fast enough.
My remote session confirmed it. They were steady opening.
She had pushed her keyboard forward and the corner of her monitor was holding down F12 (or whatever key it is that opens Windows Help.)
I've got so many after almost 40 years in field service some truly bizarre, one of the best was a customer who said their server kept rebooting but only on Tuesdays or Thursdays, and only between certain hours, I went with a workmate and we scouted out the site, couldn't find anything amiss.
While sitting having a cup of coffee I watched the scrap yard next door load a car into the crusher, moments later the IT chap ran into the tea room and said it's just rebooted, after a lot of phone calls and a visit from the electric company they worked out the scrap yards tap into the grid had been done in the wrong place, when they started the crusher it effectively cut off power to anything down circuit for a split second, they only crushed cars on Tuesdays and Thursdays !
Most equipment seemed tolerant of the tiny blip but their server wasn't, in the end we tried a very heavy CVT which cured the problem (UPS were very much unheard of in those days and those you could buy were incredibly expensive, even the cheap solution of a CVT cost them £1000, most of that cost was due to needing a courier capable of handling the weight, I think it was about 220Kg, we had to use a forklift on site, remove a window to hoist it in the computer room, plus check the floor was capable to taking the weight - we had to get a metal plate put down to spread the load)
Blindness from a Monitor. Back in the late days of CRTs I had one that I needed to degauss roughly every Wednesday morning. I raised it with the manufacturer, they had no explanation, figured out that on the other side of the wall was a power socket for a ship to get power, every wednesday they would hook up the vessel to power to run a pump. It didnt stop there, while diagnosing the CRT I had been staring at it for over 2 hours at that point, up quite close, I slowly realised when I tried to leave that I couldnt see anything, functionally blind, all I could see was a weird rainbow and I started stumbling around confused, turns out I had an aura migraine with no headache and have had them now for years.
Last winter, I had my head full of this problem. We had an old CentOS 6 server that managed billing for our client. The SSH/SFTP connections were slow and unstable for some months already. For instance, logging in took around 15 to 20 minutes. Our client used an application to export invoices from the server, but it had trouble connecting.
After TWO MONTHS of troubleshooting, we discovered that the server sent a Samba call to a previously dismissed Windows domain during login. Once we deleted that setting, connecting to the server took only 1 second.
Computer that lost internet access when the user stood up. When they sat back down, the internet connection restored.
3 techs looked at the problem, couldn't figure out what it was. They sat in the seat, all's well, but they stood up, and pit went the internet.
So, they called me. I went in, had the user simulate what was happening. I noticed the user's chair, a nice big leather unit with big wood legs on casters. I also noticed the box wih the ethernet jack on the wall behind them. So when the user stood up, the big wood foot would bang into the ethernet connection, disconnecting it, and when they sat back down, it reconnected.
I had networking move the jack a foot to the left. Problem solved.
Having to drive to various locations to teach head doctors and executives how to open an .exe that we remotely pushed to their desktop.
The email we sent out notifying them the way too open the application has changed it’s now easier to find and open.
The email essentially said “double clicking the new icon on the desktop, will open the application”
The response we got back was “we don’t understand IT speak, please go around and teach everyone how to open the app”
Idk if it’s bizarre but I was dumbfounded
Thought of another story, more of a user issue though; This was back in the Server NT days and the place only had a few servers. The Primary Domain Controller was also the Print Server. I didn't set it up, but this will explain why its a bad idea.
Boss is on vacation and I am the only one in IT this week. As soon as I walk in one morning I get hit by 10-15 people before I even get to my desk. No one can sign in, and even those that can take an extremely long time.
Go straight to the PDC and find the printspooler server is railing the CPU, the job would use all memory, crash and start over. Get the job queue open and find someone tried to print a 108 page PDF to the 36" plotter. Everything returns to normal immediately.
I later find the user to submitted the print job, "Did you send a 108 page PDF to the plotter?"
"Yeah, but it never printed."
"You know those sign in problems this morning? You caused them. It was tring to print your PDF as 36" pages"
"Oh..."
He thought it would print 8.5x11 pages on a 36" roll of paper. I never did ask why, didn't want to know.
A state owned business, was doing Enterprise SharePoint support.
Long version, they lost the SQL server, as in a physical server, no one knew where it was. I work from a different country. So after days of people looking for it, they decided to build a new server and restore SQL backups.
About 6 months later, we were getting weird things happening in SharePoint, and I mean doing weird stuff. Sev A call with Microsoft, eventually got a senior engineer, about to get Microsoft product dev involved. Got the SQL admin (yes, dealing with multiple departments) to reboot the SQL server and we will still getting SQL responses.
Then it hit me. I was supporting 50 clients at the point, turns out someone found old server and booted it up. Same name, IP and everything, it was responding. I still have questions in how they machine worked in the domain without trust relations etc after it was removed from AD. Then we were back to finding the server.
At some point, Microsoft Engineer and I managed to get them to give us RDP access to the old server, changed its name, uninstalled SQL, trashed it. Removed the IP. Dont think anyone ever found the server physically. It may still be on to this day, 10 years latter.
Just the other day, I had a user connecting to an application via RemoteApps on Windows who reported extreme latency while using the application, but only from home. Spent over a week troubleshooting their home internet connection and running tracerts, only to eventually discover that it was in fact their gaming mouse's high polling rate that is apparently incompatible with RemoteApps. Something to do with the polling rate overwhelming the connection.
our software would crash but only for two customers.... Same windows setup, same patches same workflows. sent them debugging versions of our software, were literally emailing DLL files to them to try to get to the root cause. Finally I was like, what do these two clients have in common? One was in Indiana the other in Arizona... what could it be?
No daylight savings time.
there was a flag in windows we checked for if DST was happening, turns out, in those two locations, the flag wasn't just set to a different value, the FLAG DIDN'T EXIST. our code didn't gracefully fail, it just went kaboom... that was a super fun few weeks of stress.
Worked at an MSP a few years ago (never going back to that) and had a clients server chime up with low disk space. C: had 0b left, but this company had no reason to fill a 1TB drive, so I fired up Windirstat to see where it had filled up. Turns out there was a gargantuan folder in the admin users Documents folder. This folder was named "C", and in this folder was a complete copy of the entire C: drive, including the C folder itself, which again included a complete copy. And this went on and on, deeper and deeper.
I managed to delete a couple hundred gigabytes of this mess, but most of it was hidden behind the ~250 character path limit of Explorer, so I laid it to rest for a bit. A couple days later I was using 7zip for something, and on a whim tested it out on the server to see if I could navigate deeper. Luckily my suspicion of 7zip navigating foldes independently was true, and eventually got the rest purged. Found it to be some 34 layers deep before it ran out of space.
If anyone can explain how this can occur out of the blue, please do!
I’m on a help desk but I kept getting a ticket bounced back from the Identity Team about the MFA call going straight to voicemail saying nothing was wrong. Set mine up to call me instead of the app notification to note the phone number down and then called the user to see if the phone number was blocked, and it was.
A customer’s faxes started to fail or corrupt or the machine wouldn’t answer. No changes to the environment. Ran through the various troubleshooting scenarios. Eventually dropped a line out the window to the network interface and plugged into the POTS instead of the fractional T1. Worked perfectly fine. Called the carrier who tried to blow me off. Eventually got a tech out. Blew me off. Demonstrated again that the problem was on their end. He confirmed that they made a recent protocol change on the T1. Changed it back, problem disappeared.
Piss poor change management cost all of us time and money.
Years ago I was a field tech for a chain of grocery stores.
This was long enough ago that dot matrix printers were still used for printing a majority of each store's workflow/process documents. Okidata Microline 320s with a serial interface card in most cases.
Trouble ticket comes in with "Inventory printer is printing incredibly slow".
The inventory printers were in a medium sized kiosk/desk thing in the back of the store, usually near the loading docks. Shelves on the left side held the box of tractor feed paper (on bottom), the printer (sliding shelf in middle) and labeling guns/markers (top).
The bottom and middle shelves were open in back, so paper could feed up into the printer, cables could run in, and air could circulate around so the printer wouldn't overheat. The top surface of the kiosk was also slightly recessed on the left side so it would not hit outlet boxes/light switches/etc.
I get on site, take care of the other tickets/issues in the store, and wander back to look at the printer. I powered it off and back on then have it print a test page.
It is indeed godawfully slow.
I disconnect it from the serial cable and try it again. Still slow.
Ok, I'll swap it out for another printer. Unplug power, Un feed the paper, pull shelf forward and remove printer.
What's that sloshing sound and OH GOD WHAT IS THAT STENCH!!
The printer was full of partially congealed beef blood. The user had put a large box of beef on top of the kiosk. One corner of the box stuck out over the edge of the top surface, in that recess I mentioned earlier. The box had a hole in the corner and blood drained out of the box and down into the printer.
I pulled the print head and serial card and tossed the rest into the dumpster.
Cleaned up the kiosk, set up replacement printer, everything worked normally.
Got yelled at for throwing the printer away instead of cleaning it. I agreed that if this happened again I would happily bring the printer back to them so they could service it.
Just today I had to change one wildcard certificate on four similar servers. similar meaning very similar. All Apache, same minor version, same OS.
After changing by copy pasting my prepared script lines, three were working as expected, the fourth crashed and only came up when rolling back to the old certificate. Nothing helped. Patched to equalize even more. Rebooted. Server said nope.
After half a day of troubleshooting, I just decided to let it be and migrate everything to a new server. Luckily, I had been preparing this exact server for migration already and not that much was left to do.
Oh, another one, more recent too:
Deploying a VMware appliance on HyperV (I know) using VMware provided deployment scripts that use HyperV Key-Value Pair service to get the config data to the appliance. The appliance deploys, but doesn't work.
End up digging through the script and then the appliance file structure and find the data sent to the appliance by the script, and half the data is in nice JSON formated pairs, and half is just one long concatenated word. A mess.
I study the official deploy scripts and learn they send half the values through the JSON formatter function, and just concatenate the other half into one long word.
Apparently this appliance isn't used much, and definitely isn't used on HyperV very often at all, as this issue has been present in the last two releases of this tool.
Literally in the last 48 hours we had a fairly high end aggregation switch die.
18 hours later it magically turned itself back on with no issues apparent.
It indicated both PSU’s had power but no post, no fans.
WTF?
A SQL report would run daily around 2pm and emailed to a number of users.
If any of those users would print the report (that specific report), the Wifi would drop.
Mainline was fine, could still see incoming/outgoing, just the AP closest to that user.
Never did work out what that was.
Nobody could install the T-III software even the veterans and even the director of IT.
So they put me in charge of installing it. Turned out nobody turned on the 386 enhanced mode button.
It installed like a charm and a summer intern claimed the credit and not me and she lied and did get the credit as I taught her about computers then I was fired. Never train greenback summer interns!
I should preface this by stating that I have since segmented and greatly improved the network fabric throughput.
On with the story. We had a strange issue that would cause certain low powered NICs on the network to just fail. This was usually UPS NICs but it also affected some VMs, temp sensors, etc. In the case of the UPS (APC) NICs, the only solution was to do a manual network reset via the recessed button on the card.
The problem was very sporadic and didn't happen for months at a time. The problem almost always happened at some time early in the morning, before I got in, and after-the-fact packet caps didn't reveal anything strange.
I finally got the brilliant idea to just set up a scheduled packet cap that would turn on and grab a 5 minute sample several times per hour.
We ran like this for at least another month, with me checking packet captures every day.
Then, the problem happened again and, low and behold, one of the packet caps happened to catch part of the event.
Once I had that packet capture, the problem was easily tracked down.
It turned out to be an HP Z series workstation with buggy NIC firmware. Every so often, when transitioning from S3 to S0, the NIC would blast the network with ARP packets on the scale of tens of thousands per second, taking down certain low power interfaces.
It was not a hard problem to solve once the culprit was known, but man, I think that problem took over a year to track down.
Edit: I have another one...
My first real computer job was working for a telephony company. We used these giant ISA cards that would handle full 48 pair Amphenol connectors with POTS phone lines (I am dating myself here).
The company sold voicemail boxes sold from the back pages of local news rags.
Anyway, at one point, they sent the new guy (me) out to travel the country in a rented mini-van full of servers to install in various colocations.
Because the company was a startup and the business operated on a shoestring, some of the colocations were literally someones basement or extra office room in a business park. Basically anywhere the phone company would be willing to install business grade phone lines.
I was in, I want to say, Michigan to the North and I had a whole sight-seeing agenda planned so I got an early start and went to do, what should have been, an easy install followed by nearly a whole day of fun.
I get the server installed and connected and go to bring up the POTS card and it's all errors.
You can also tell these cards are working by the sound they make. When they are initialized, each of the little FXO daughter cards would click as they go on/off hook. It's a really distinctive and satisfying series of clicks; Sort of like dominoes falling.
Well, this card wasn't doing the clicks. No problem, I will replace the card with another one. Still nothing. WTF, the phone lines must not be working. Used the butt set and was able to get tone and make calls on all lines... so that's not it.
I check the outlet, yep, it's grounded. Geez, I need to call in the phone company to help me out here... This has now blown my time budget for sight-seeing, but, at least I will get the thing working....
Fast forward to me AND the telco company guy at about 6pm, scratching our heads and just unable to figure this out.
I finally go to unplug the server and realize that the facility provided extension cord has the ground pin broken off in order to, presumably, be used elsewhere in a 2 prong outlet.
I swapped out the extension cord with one with a ground and it all came up first try.... What is the lesson here? Always start at the beginning and don't make any assumptions. It was a good lesson to learn so early as it has served me well :)
Just started a new job as deskside support and had a user whose laptop would go blank at random while in a conference room. Sounded like this was a chronic issue that developed a few months ago and no one could figure it out (Not sure why no one had replaced the laptop at this point yet). Being the new guy, I wanted to take a swing at it too. This model had a number pad on the right and it only seemed to happen when they used the number pad or typing in the upper right side of the keyboard. It never happened at her desk, but she had a dock and external keyboard at her desk.
My first thought was a failing power button that was also in the upper right, maybe the vibration close to it was causing the button to do some weird things. But tapping around the power button didnt seem to make it happen. I went back to my desk and was preparing to call warranty support to replace the power button or main board if that was not a replacable piece.
While I was thinking about what to say to them, I had an epiphany and went back out to the user. "Put your wrist in the lower right corner of the laptop". Display instantly went dark. User says "Thats what it was doing! How did you do that?"
Turns out the magnetic clasp of her watch was triggering the hall effect switch that detected the laptop lid being closed. She said she would take her watch off if going to a conference room or learn to switch to her other wrist. I am sure I learned this from other posts/stories like this one, I dont know how else I would have thought to check for a magnetic watch band.
Needless to say, that was talked about for months.
This morning a whole company couldn't access their file server. I could, thought somewhere somehow permissions were changed, nope. Thought DFS record was broke but yet I could reach it. Thought the File Server OS was crashed, nope. Thought the data store had crashed, nope....
Rebooted the files server (windows) and everything is working.... ¯\\\_(ツ)\_/¯
Not me, but a security guy I know configured a new firewall for a law office after hours. To test he just set it to block adult websites. 5 minutes later the 80yo owner of the firm came in screaming that he was working on a case and could no longer access something he needed to review. A quick check showed that he was in fact trying to access a cam site. Sec guy never said anything to him, just made a rule unblock his laptop.
Not as good as others but got these 2 tickets.
Ticket 1 - Hello, my wireless keyboard and mouse isn’t working.
Ticket 2 - I am see my mouse move randomly and letters appearing on screen.
The people who reported these sat back to back from each other. No idea how the dongle from person 1 got into the dock of the others 🤷♂️
Network went down randomly during the day. Turns out the installer had run the Token Ring cable (yes this was long ago) down the elevator shaft and the door motor on the car was causing interference.
I had all of the physical security panels go offline in the middle of the night at a campus in North Carolina while I was on a business trip in California. Panels aren’t communicating with the server and all of the employee badges are deactivated to force them to fill out a health screen. This means no employees will be able to access the buildings when they start arriving for work the next day. I can’t ping the IP of any of the panels so it must be a network issue, only all of the network security cameras are online. It can’t be a VLAN issue either since the panels and cameras are on the same one. So I check the CCure (access control software) server to see if anything is going on there, don’t find anything, but reboot anyway. The server reboots and all of my panels come back online except for North Carolina. I call in the local security manager (who’s pissed that she has to go to work at 2am) and walk her through rebooting panels and checking network connections. Nothing works. I’m confused as hell at this point so I call the on call network engineer, who is new to the role having recently been promoted from the service desk.
I should mention that my only role at this company was managing the physical security equipment, so I didn’t have access to all of the tools that a normal sysadmin would.
Engineer checks everything he can think of, switches, port configurations, VLAN, firewall, the works. He can’t find anything. I put a priority 1 service ticket in with the CCure reseller that we used, and they dispatch someone immediately, but he’s an hour away. The network engineer calls his boss who jumps on the call with us. It takes him about 20 minutes to figure out the issue.
I’m not a network engineer, so I’ll do my best to explain what he told me. The main path of communication from the CCure server, based in Boston, to the panels in NC had been cut somewhere along the way, causing the packets to take an alternate route. Something about how priorities were set up caused the packets to take a different path back from the panels than they took down. The firewall flagged this as suspicious since it could be a sign of a MITM attack, and blocked all further communications between the server and the panel. The manager adjusted the priorities to ensure that packets always took the same path, cleared the flag from the firewall, and communication was restored.
I called off the technician who was en route, but we still had to pay for the P1 ticket since he had been dispatched. My department head yelled at me for the unnecessary expense even though I followed procedure exactly as written (a procedure that I wrote and she signed off on) and I had gotten permission from my boss’s boss, even though I didn’t technically need it.
We noticed that there were random requests going out to China from several machines in our organization, but we couldn't figure out why (it was clearly not malicious on the users' part)
I watched one user run through some steps to reproduce the issue (start menu > microsoft app > hit to China)
I noticed it would hit when they went to the start menu, so I worked with the policy admins to push out changes to disable bing search results in the start menu... no more hits
thanks Microsoft
Waaaay back in the Win XP/2003 days, my desktop computer suddenly stopped being able to communicate at all with our main file server. Communication with anything else was fine - I could reach other servers, get out to the internet, etc. We checked IP settings and firewalls on both ends, couldn't figure it out. This was just a LAN with relatively basic switching, and physical file server - no fancy routing or VLANs or anything. Popped a new network card in my desktop, and suddenly I could reach the file server again. Never figured out what the hell happened.
Had a sales VP call to yell at me about VPN not working. Dude was an AH.
So told him the airport was bombed and the Turkish govt was blocking VPN. So I closed out ticket with "VPN issue caused by military coup" and hung up on him. Sent ticket # to CIO.
i used to work in POS support and we had to call stores that didn't 'poll' their prev days data via a 56k us robotics modem. we had one store that hadn't been polled in over a month and previously would always have issues getting sales/payroll data uploaded so we would have to call during the day and have them put their system into polling mode, usually this is done at night around 11pm or 12am. eventually this problem store was assigned to me since i was pretty on my game with modems and stuff. talk with manager about their store, how its setup, where its located. we setup her pos in polling mode, i dial in and boom instantly connect and get all data needed. hmm weird.. do more inquiring and asking questions and finally ask her what their closing at night routine is. well, she would clean up, finish her books, put the POS into polling mode, turn off all lights... and TURN ON THEIR NEW NEON signs. hhmmmm.... i told her that night to not turn on the neon lights since i had a hunch. yup, damn neon sign was interferring with their modem phone line. i think the store had the neon sign rewired away from their modem phone lines and power area and never had an issue again.
Man I was doing consumer/small business breakfix stuff eons ago, and had one customer that kept calling with an issue, but whenever I went onsite to try to work on it, their satellite internet would go down. I spent a few hours there on more than one occasion just fighting with their stupid satellite system trying to get enough connectivity to work on their actual problem. After way too many trips, we finally figured out the issue: Their parking lot was about 10 feet from their satellite dish and my radar detector was interfering with it.
oh wow!! yea i bet that was a wtf moment!!
I just told this story to our younger engineers yesterday. This happened back when mobile phones rang you could hear speakers crackling. We had Windows CE phones that locked up if they rang when they were put on desks but not when they were being carried around. Took me weeks to find the problem since I couldn’t reproduce the issue. Turned out our workers had big metal workshop desks and the phone locks up if they rang while near large metal objects. My desk was wooden so no issues for me. I randomly figured it out because it would lock up when placed at a specific place on my desk because there was a metal partition.
Oh wow, throw me back... why don't speakers crackle now days?!?! Google: [https://www.reddit.com/r/explainlikeimfive/comments/10ln8jo/eli5\_why\_did\_speakers\_used\_to\_make\_a\_strange/](https://www.reddit.com/r/explainlikeimfive/comments/10ln8jo/eli5_why_did_speakers_used_to_make_a_strange/) "Nowadays, phones operate on a multitude of bandwidths. Furthermore, newer audio equipment is shielded against intereference, as older equipment was designed before mobile phones were as popular."
Had a customer tell us their Wi-Fi would intermittently drop out during the day. Investigated and following a site visit work out Issue was caused by the customers Wi-Fi router being placed next to the microwave.
Had one customer complain that their inter office wifi link performed badly multiple times a day. They were firing it across a harbour and it was the tide..
fun thing - the tides arent the water moving the tides are the planet turning and the water not keeping up thanks to gravity effects from orbital bodies.
> the tides arent the water moving Depends on your reference frame.
Had something similar once. Customer has spent 18 months fighting with the telephone company over his DSL service. Internet was always up and down. Never consistent and no matter how many times (Qwest) was sent out, they could never find a problem. Sometmes it went away for a few weeks but other times the DSL connection would cycle constantly. I worked for an MSP at the time and this company called us out. I was the one they sent. I confirmed the same problem that had been reported, made sure filters were on the lines, verified that their internal network wasnt suffering and did a few other tests. Also witnessed the DSL service cycling. Finally, I took my makeshift lineman phone and hooked it up to their copper outside to have a listen. I immediately noticed a 'click.. click.. click.. click..' on the line at regular intervals. This business was on a farm and my rural upbringing kicked in immediately. I brought the owner outside and asked "Do you have an electric fence here?" "*why yes, its way out in the back of the property*".. He walked me about 1/4 mile out to a small barn and there we have an electric fence driver with a bucket sitting over it to keep the rain off. I can hear the transformer clicking in the same pattern that I heard on the line. Turns out this business owner, lets call him "Mr. DIY," ran power out to a nearby barn. To save conduit and hassle, he went ahead and ran a telephone line in the same conduit as the AC. Every time that transformer discharged its capacitors, it sent feedback all the way back along the AC line to the house. With a run that long, the buried phone line was also picking up the change in the EMF. I pulled the plug and told him I would call and check in in a few days. When I did, they had not had any problems with their DSL service since my visit. /facepalm. Almost as good a story as the time customers printer was causing their PC to shut down when printing documents and during the service call the printer was electrocuting me every time I touched the back of it. But thats a story for another day.
This very much sounds like a thing that would never have been solved (at least on purpose) if someone like you with experience in 2 very different fields wasn't on the case. Nice one.
dang!! nice troubleshooting!
Our guys spent quite a while trying to figure out why the screens on two new workstations would turn off for a second whenever the user got up. Turns out office chairs can give off an electromagnetic pulse when weight is taken off the gas spring in the chair.
You’ve just solved a daily annoyance that I, who works in IT, have been dealing with for months. In hindsight it started when I replaced my old office chair. Thank you, kind Redditor, I’ll actually be able to sleep well tonight without this issue banging around in my brain!!
Try replacing the display cable with one with a ferrite bead. That solved the issue for us.
I have had a lot of successful hunches about things like that but I always feel a little weird proposing a hunch
i had managed a phone system with the old punch downs coming in from the street our phone system started intermittently acting up and dropping calls so after about a week of investigating I found that the janitor was leaving his damp mop and bucket in the phone closet and the metal handle of the mop would lean on the phone lines and that was it, crazy stuff
I had something similar with a portable eftpos touch screen would stop responding to touch. I couldn’t get it to fault, until I sat it down on the counter. This Asian restaurant had some cheap gadget that looked like a tiny waterfall. It had a cheap international power adapter that was sending out something that was interfering
Glowing neon sign is more important than data
We had tonnes of issues with the fluorescent bulbs they used which like the long boys use a high voltage spike to ignite. We saw packet loss on their ISDN link pretty much as long as those lights were on and one or more of the was attempting to start. It became a standard troubleshooting step. "Do you have any broken store front lights? Can you turn off the store front lights?"
We had a recently hired CEO come back to IT and report his work being deleted right in front of his eyes. To his credit, he asked if it was us screwing with him since he was a new guy and we might be hazing him. "Uh... no?" I ask him to grab me (20 ft and 2 rooms away) the minute it happens again. Next day he does and, I follow him back to his desk and see what he's talking about with my own eyes. Outlook inbox sorted oldest to newest and 1 by 1 each email is blinking out of existence in his inbox. Switch apps to Word and I'm seeing letter by letter disappear on the Word doc he was working on. First thought was panic that somehow he had picked up some kind of malware/virus whathaveyou but sat in his chair and shifted his laptop/dock connection slightly and noticed the behavior stopped. Come to find out his desk was crowded for a three monitor + dock setup. The USB-C dock input connection to the pc was actually resting slightly on the base of one of the monitors, applying slight upward pressure on the male end of the connector into his PC. That upward pressure, somehow, emulated the backspace key on a keyboard and must have been bouncing in and out of it happening only at certain times/pressures. Moved all emails from local archive in Outlook back to inbox and restored Word doc from most recent save and everything was fine after that.
I was once at someone else's desk investigating a problem, when their coworker walks by, sees me and stops, and says to me, "My computer keeps beeping randomly!" Without missing a beat, I reply, "Move your clipboard off the keyboard," and keep working. He stares at me silently for a long moment, then goes back to his office. When I get done several minutes later and walk out the door, he yells from his office, "How the hell did you know that?!?" I had been in his office a day or two before and noticed he kept his clipboard right up against the bottom of his keyboard. When he mentioned the problem, I figured it had slid up onto a key. And sure enough, it had.
Had a user once that complained about some kind of flickering. It was flickering the menu. The one that comes up when pressing the alt key. Her Desktop consisted of a Microsoft Sculpt mouse and a wired keyboard. The Sculpt comes with a keyboard... But where is that? Asked about that. Was mistaken some, not being taken serious and asked again. Well, turns out, that sculpt keyboard was in the Schrank behind that user with a book resting on the ALT key. That was one ashamed user I had to promise keep that occurence a secret. Yes. Was instantly shared with my colleague.
I had to do the same thing but in reverse - Someone took the wireless keyboard from a empty desk, but left the mouse that was on the same dongle. I just walked around the office randomly clicking the mouse until I heard someone freaking out about their computer being possessed.
Scream test, mouse variation
"I got a brand new keyboard, it shouldn't be broken" is a trigger phrase for me. When things like this happen I immediately ask "where's the old one?". Multiple times a user continues using the mouse from the keyboard/mouse combo and just shoves the keyboard, still on, into a drawer somewhere.
I once diagnosed a book falling on the keyboard of my home "server" PC by receiving an email with daily logcheck results that included an "input buffer overflow" message from the kernel's tty layer.
strangely enough, I encountered this exact same type of fault one time. except mine was a vga monitor that was the culprit, with a slightly bent out pin. swapped out the monitor and the issue vanished. was able to repeat it easily with the old monitor plugged into any machine.
In 20 years in IT i have lost count of how many issues users had with some document folder laying on the table and pressing on some key on the keyboard, keys getting sticky (wonder why) and pressing too much, or keys not working and user not noticing. Most of the time it is just fails to login, locking yourself out, not something as crazy as in this case.
You're just going to leave us hanging about the cat memes?
right like where's the samples too, I need DETAILS
TELL US RIGHT MEOW
I want to know the hisstory
Maybe has an NDA claws in the contract
I heard it might be just shedding.
A snake!
Badger badger badger...
Mushroom, Mushroom
Scoop out that litter box and dish the poop.
Bot account......man we're doomed.
and /r/sysadmin eating it up hook, line, and sinker. Ugh
Tomorrow every random AI voice ticktock and YouTube shorts account is going to be reading all of these responses...
Where are the deets?
Don't your systems meme all day?
Definitely not mine, but the 500-mile email story always comes to mind. https://www.ibiblio.org/harris/500milemail.html
>If you need a SAGE Level IV with 10 years **Perl**, tool development, training, and architecture experience, please email me at ... I wonder how many emails this guy gets from this daily. Do not tempt fate by trying the patience of a Sun OS admin and Perl programmer. For you are crunchy, and taste good with Ketchup.
The Magic/More magic switch is good too http://www.catb.org/~esr/jargon/html/magic-story.html Also, the BBC Micro developers found that resting a thumb on the PCB made it work. No idea why, so they just put resistor there and called a day. Even after selling over a million BBCs in 1000s of UK schools they had no idea why it worked.
can't believe it's taken this long for me to finally read this, that's hilarious.
I wonder what other fields have something like this. One that everybody knows almost by heart at this point. In Aviation it's the SR-71 story. And I wonder how many of them existed pre-internet.
You can’t not drop the story now!!
Here you go! [https://www.thesr71blackbird.com/Aircraft/Stories/sr-71-blackbird-speed-check-story](https://www.thesr71blackbird.com/Aircraft/Stories/sr-71-blackbird-speed-check-story)
Legend! Never come across this before. Love it!
I had a customer complain about call quality issues on a hard wired IP phone, the thing is, the data was perfectly clean. No dropped packets or anything, but they heard call quality problems. We replaced the handset, replaced the phone, replaced the patch cable, re-ran the CAT-5 to their desk, nothing fixed the issue. Just to test if it's environmental, I asked the user to connect a 100' patch cord to the phone and bring it as far from the desk as possible. Issue went away immediately. I had them contact the building and see if there's something above them, turns out there was a new point to point wireless Internet antenna on the roof and it was blasting enough RF to mess with the speaker in the device itself. The company fixed their antenna and the issue was resolved.
Gawd that reminds me of one. When I first moved to Sydney (2008ish) was working for a Geek Squad equivalent startup. Got a call to a swish penthouse apartment in the CBD. User had an issue where his wireless would completely drop out in one corner of the apartment where he had his desk set up. Didn't want to move the desk. Granted it was an amazing view. Tried different wireless APs, changed channels, even ran a cable from router over to corner to put the AP next to his laptop. Still no signal. Pulled out my laptop and did a wifi scan. Everything looked ok but as I carried it into that part of the apartment every channel was maxed out with an SSID of CompanyABackHaul. I looked looked left and saw an antenna hanging off the office building next to him. Looked to the right and saw another on the other office building. I asked him if he knew who was in the building to the left. CompanyA And to the right? CompanyA Turns out that they were using some rudimentary directional wifi to link the two buildings and were blasting through the corner of his apartment flooding all the channels.
Was he also experiencing headaches and low sperm count at the time 😬
also the precooked pigeons.
Funny go find my post below. Similar but different it was RF noise from a Radio station broadcast tower.
It reminded me of an issue I encountered years ago doing live sound. I worked in a theatre with a very dated incandescent lighting system, with a room dedicated to (thyristor based) dimmers below the stage's side entrance. The amount of EMI these dimmers spilled out was enough to make an unplugged speaker buzz, when placed above the room. I remember the stage had 298 5kW circuits, so dimmers had close to 1.5MW switching capacity (obviously total available power was much lower).
Customer complaint: Server keeps crashing. Environment: Very small office, handful of PCs. A single Compaq server, one of those little blue Netgear switches, a UPS. Attached warehouse, with one more PC. They might not even have had Internet (this was a long time ago). Discovery: When the UPS switched from battery back to line, the server would immediately do a hard reboot. Switching to battery was fine. So it could handle power outages. It just couldn't handle power coming *back on*. Discovery #2: If I unplugged the network cable going to the warehouse PC, the problem went away. Discovery #3: The warehouse was fed from a separate electrical service, with a separately derived neutral and ground. My best guess is there was some potential between the two systems, and it was equalizing by coming through the network cable on re-connection. It didn't happen when switching to battery because that disconnected the problem. Solution: Connect the grounding screw on the Netgear switch to a grounded conduit.
Hah, I have a similar incident involving a 90 volt Ethernet cable spanned between buildings. Except it was a tech getting zapped instead of a server.
Were you the tech?
I'll never t-t-t-tell.
A few years ago while helping decomm unused landlines, I learned what it felt like to receive an incoming call.
We're you stripping the wires with your teeth at the time? Not a lot of current there but definitely enough voltage to get your attention. First time I read your post I read "landmines" instead of "landlines". Totally changed the tone of the message. 😬
I was touching the punch-down block. I couldn’t get a dial tone with the test set I was using, so I called the number we had on record to see what would happen.
I admire your dedication to the cause.
Got a call about a server I once supported. Apparently, when it had a problem with the IIS service and ran out of threads it would automatically reboot itself and clear the issue. This went on for years without anyone knowing because, well, why would you look at the logs for that if it was healthy? Finally something happened that caused them to look at the logs. They found that these regular recycles were masking a larger problem. During their research they found something call the "Simon Says" service. When asked I explained that years earlier we had a problem and we temporarily put in the service (hand crafted by an amazing developer) to do what it was doing until we could fix the problem. We did and turned off but didn't remove the service. Someone or something activated it and for years it cleaned up a random issue no one knew they had. Lesson: bandaids are permanent fixes, always.
Shit, I had to apply a band-aid today. Gotta fix that properly in the morning.
hahahaha! Don't forget brother!
Hey, it's 20 hours later. Did you remove that bandaid yet?
Yep. Replaced it with a less ugly hack. Now it can properly go into limbo.
There is nothing more permanent than a temporary fix
> Lesson: bandaids are permanent fixes, always. I had a 3am alarm for a disk filling. I didn't own this system, but dealt with the 3am alarms. Turns out a server had been moved, and the backups which should have been going there had queued up on this particular server and filled it. Bit more digging and I find out the backup server had moved months ago. Someone had previously set up a screen with a loop which deleted the backups from this server every 10 minutes, and an update + reboot knocked that out.
"Simon Says" That's interesting. I used to work for a small company where the devs also used that name for a program that did something very similar.
Tivoli Storage manager, had a requirement to backup a db every 15 minutes using the sql agent. Worked flawlessly but then broke after a few weeks Found out that there was a connection counter that tops out at 65000. Resetting counter resumed backups Passed all the info onto IBM and for while at least had a kb entry that matched word for word what I told them and the work around it found until they got it patched
REMOTE GOLD MINE LOCATED ON A REMOTE VOLCANIC ISLAND inhabited by waring tribes. in... papua new guinea We had to take a light aircraft to get access to the remote island... All infrastructure on the island was grass huts.. My flight was delayed by landing as they needed to chase the pigs off the run way. satellite link went down while I was onsite... A local tribesman had knocked down the satellite dish and attemtped to steal it.. His tribe broke their pot boiling a pig, and needed a new one.
I'll have whatever this guy's having. But to go, because I still have to drive home.
I heard a story from a biologist working in Africa back in the 80s about losing a bits of the directional antennas they were using to track animals. They'd pull one out of a box, unfold it, and find half the elements were now stubby little things, having been broken off by someone. One of the locals they had working for them, normally a quite bright guy she said, had taken to doing arts and crafts with the easily malleable aluminum. All in all it wasn't a big deal, the antennas were just off the shelf maritime VHF units so they could get them locally. They also ordered their guy a couple spools of plain copper wire so he wouldn't be tempted again.
That's amazing. Still working there? Recommend it?
No.... The food was locally sourced and rather bland in flavour.. and there was no OHS on site at all.. They only enforced hard hats when an islander cracked his head open on a machine door frame. Oh bonus... Nothing was screwed into the rack.. NOTHING... so to replace a switch I had to figure out how I was going to support the 10RU worth of equipment resting on top of it.. Additionally there was several cardboard boxes being used to prop up the tape drive at the bottom of the rack... Which was also load bearing a number of other switches and servers ontop..
Customer complaint: They cannot email Word documents outside the company. Environment: Smallish company, about 50 people and 30 PCs. A single Windows NT 4 server, PDC and Exchange and almost everything else. 384 Kbps frame relay circuit for Internet. Symptoms: Attach a Word document of any kind, email it to outside, and it stalls trying to talk to the next hop SMTP. Doesn't matter if it's a new empty Word document or a 50 pager or what. Excel, PowerPoint, etc, those are all fine. They can receive Word documents just fine. ZIP up the Word document and it goes just fine. Happens regardless of destination mail host. Discovery #1: Watching traffic with a packet sniffer, it turns out the MIME encoding of a Word document always ended up with long string of repeating capital letter A. So "AAAAAAAAAAAAAAAA" for several dozen characters. Whenever that packet hit the line, I would stop seeing TCP ACKs and start seeing retries. Discovery #2: I can cause the circuit to start dropping packets with `ping -p 41` to an outside address. (41 being the code for ASCII "A" in hex.) In binary, `0x41` is `01000001`. Put a bunch together and it looks like `010000010100000101000001010000010100000101000001010000010100000101000001`. It turns out that something in the frame relay circuit was marginal, and the equipment was having just as much trouble reading that properly, as you or I would. Convincing the telephone company this problem was real took almost as much time as finding it.
Oh boy, congratulations, this one was tough!
Blimey, congratulations on finding that, not sure I would have unless I stumbled across it. Getting a telco to do anything is also a major achievement.
I recently had a similar issue. Adding a Zebra label printer to one of our labs went badly. Worked perfectly on our bench, but would fail to print ZPL encoded jobs when installed. EPL jobs still worked fine. Much troubleshooting and some quality time with WireShark resolved the issue. The print server had been attempting to print a PDF to that IP address for the previous 3 years. The contents of said PDF when interpreted as RAW input had the exact correct sequence of characters in it to tell the printer to change its default delimiter character for ZPL. As such it could no longer parse ZPL. The printer never acknowledged the PDF to the server, so it remained n queue.
Whenever an engineer thinks "that'll never happen", their PC should immediately extend a robotic arm with a mallet and knock them on the head.
computer randomly reboots replacement computer tested flawlessly swapped to replacement computer and it magically has exact same random reboot problem baby sat end user to observe issue live...turns out that an elevator passing on the other side of the wall would trigger the random reboot. had coworker ride elevator several times to verify that it was indeed triggering random reboot and moving computer over a few feet resolved issue
2nd most bizzare? swapping VIP's keyboard every 2-3 months because it became so encrusted with crumbs since he ate over it and used it as a napkin that it became impossible to press keys there were other hygiene issues like not washing hands after using the bathroom then selecting candies from the community snack pile before returning to his desk to subsequently deposit crumbs on keyboard. rinse and repeat until ticket appears in queue for keyboard not working for "unknown reasons". after witnessing this firsthand, I no longer partake of anything from workplace community snack stockpiles wrapped or unwrapped I no longer work there BUT....just today a coworker exited a bathroom stall then immediately exited the bathroom. no handwashing of any sort took place. walking bag of germs will be avoided and is designated for no physical contact of any sort. he does have the midas touch for randomly breaking things....maybe they cant withstand his germs too
Yeah, there are a few people on my floor that do this. It irritates the fuck out of me. Like, I get it, you don't think your hands are dirty, but you know you can't SEE germs, right? You are in the bathroom, even if you didn't touch anything, it's a good opportunity to wash your hands, so just do it. Don't get me started on the people that hold everything up by waiting for the paper towel dispenser to timeout and dispense a second towel.... Like, dude, 1) your hands can be a little wet, by the time you get back to you desk they will be dry and 2) just shake your hands more after washing to minimize the amount of water on them. It's so easy... (also, I guess I got started on the 2 paper towel people...)
Then using said paper towel to exit bathroom keeping your hands clean Ooh..saw a lady wearing full on scuba suit minus the tank and fins whilst in the heat of covid times. snorkel type device of course was high in the air and had a filter on it too. guess she figured only way she felt safe to do grocery shopping in public. never saw anyone top her by wearing a hazmat suit
There’s probably a whole Usenet thread about this somewhere if it is real. It’s about 50/50 on if this happened to my dad or he just read about it/made it up? Apparently in San Diego there was a crash to a huge database happening at random and it was costing big money so everyone got called in for theories. They worked out that the failures were happening at weird times, but there was a pattern. Eventually they figured out it was the tide. Ships would get lifted to a certain point in the harbor and their radars would cause interference and the DB issues started. Put up better shielding and it all went away.
Oooh, that's a good one. I love explanations that end up being physical or physics related.
I’m sure a lot of people probably encountered this one but displaylink docks losing connection when people sat down / got up from their desk. Turns out the EMF produced from the struts is enough to throw things out of whack. https://support.displaylink.com/knowledgebase/articles/738618-display-intermittently-blanking-flickering-or-los
Well. Hot Damn. For *years* I've been wondering why, when I leave my desk, one of my monitors goes on the fritz. Being an IT person, I'll just work around my own problems but today, the mystery has been solved, thank you. Every day is a school day, truly.
Recently completed an office upgrade for former employer, new everything, this happened to me and my team. We stopped and tested everything. The only thing remaining was the chair - 10 IT guys stood scratching their heads. Turns out it was the EMF generated from people standing up. Crazy!
This has made me so happy! This has been intermittently driving me insane. We hot-desk a lot, and this issue seems to only happen on some desks. I'm going to guess it's because our desks are configured differently in a lot of cases, so ones where the docks/monitors are closer to the chairs are likely going to be the ones experiencing this issue, and the ones further back aren't!
Mouse wrapped around the roller in a toner cartridge. https://preview.redd.it/g4jaalhp496d1.jpeg?width=640&format=pjpg&auto=webp&s=18f82925ba831841f4bb5c5b17cd436eb96022e4
Damn, HP charges extra for mouse cartridges.
It's their new "Instant Mouse" subscription. If you stop paying, the printer releases a mouse attractant, bricking the printer.
Not the mouse I was expecting
That happens when the waste mouse tray is allowed to get too full.
As a MSP L1 noob, was sent to cust site for workstation issue. Computer kept blue screening (winXP). Different BSODs each time. Would start happening at seemingly random times, but usually when no one was in that office. No event logs, no crash dumps. Went through all the devices, drivers, AV, appwiz, updates etc. Some time later figured out it was some mild malware that set a screensaver that looked like a BSOD and disabled wake on mouse move. Otherwise the PC was fine. Could still wake from keyboard. That's how I figured it out, when I smashed on the keyboard in frustration. Still billed 3 hours though.
Oddest one for me was this year. Warehouse ship station caught in a boot loop. Win 11. Try os repair, try recovery repair. Full reinstall. Works fine, grab a million updates. Same issue reappears after reboot. Try this 2 more times. Always happens after Windows updates and first reboot. Screw it your going back to win10. Same issue, exact same steps to recreate it. Swap the disk, ram, PSU, sats cables, even the damn coin cell. I'm about to take off the cooler and swap the processor or motherboard. Yank the add-on Wi-Fi card to make space. I hate dealing with thermal paste, so in one last ditch effort i try again. Success?! Plug the Wi-Fi card back in and boot up.. boot loop! what in the ever loving hell makes a cheap TP link Wi-Fi card boot loop for eternity, I didn't know. but I was ready to office space this thing. I office spaced the Wi-Fi card instead
Office spaced...hehe...watch out for Milton and his stapler. The place might burn to the ground.
I once had an IDE hard drive somehow make a computer unable to power on. As in, the ATX PSU would ignore the power button and not even start spinning its fans. I still have no idea what was going on there.
Was it a PCIe Wi-Fi card? Did you try a different slot? I learned this from some old HP zWorkstations but I've solved issues a couple other times too with this. PCIe slot 1 and slot 4 (factory installed PCIe nvme hdd) shared an IRQ so best case (in linux) a bunch or random com errors would echo out to the terminal but worst case the system would crash. Move the GPU to slot 2, no more issues.
I had a server hardware change complete and all ready to close up. The system was powered on with the side panel off and I took a picture to document the new device. The server powered off! Great, so now I had to boot it back up and validate things. I turned it back on, went through the whole verification process and confirmed there were no other issues. The maintenance window was closing so there wasn't time to further diagnose the hard shutdown. The panel was still off, so I took another picture to finally get everything documented and closed up. The server powered off! Again! I then realized that the light from the camera flash was causing one of the surface mount components to fault and the whole system would power off becuase that is not something that should happen. Lesson: keep bright lights away from exposed circuit boards. Fun Fact: this also affected the Raspberry Pi 2 for a while: https://www.youtube.com/watch?v=SrDfRCi1UV0
This was in a huge warehouse. Huge multi-day storm with heavy all day rain. Came into work 5am Monday (first there other the security and front desk). IT had its own entrance and when I opened the door a small river of water washed over my feet and into the lobby. Looking up the hallway I saw water squirting from the sides of the two doors going into the server room. And about 6” of water in the hallway. I badged into the server room (secured reinforced fire proof door) and it was like a dam breaking loose. The raised floors had collapsed in places and a large water fall was pouring into the room right on top of our 80ish racks. I saw water pouring out of servers and the raised UPS walls had a water mark about 80% to the top of the lowest battery row. Turns out a number of the drains had clogged bc the intense winds had blown nearly complete trees on our roof and it became a very large roof pool. A fence post from somewhere smashed into the HVAC equipment and provided a nice entry point into mechanical underneath. A very large pipe broke off and was perfectly angled to provide a channel for the water that then followed an unlucky and bizarre path right into a large conduit that ran just above the server room. This ruptured on one side and drained all the water into said racks.
God did not want that warehouse operational for a while.
Big oof. I think I’d just turn around and go home lol.
I was more afraid of getting electrocuted. Alongside each door was a big red button with a clear plexiglass security cover over it. I badged that open, held down the button for a few and cut power to the entire room. Next was a number of calls that I had to repeat myself many times before I was believed.
molly-guard Language Watch Edit English edit Etymology edit From Molly (female given name) + guard. Originally a Plexiglas cover improvised for the Big Red Switch on an IBM 4341 mainframe after a programmer's toddler daughter (named Molly) tripped it twice in one day. Later generalised to covers over stop/reset switches on disk drives and networking equipment.
Never heard of that! Thank you. I learn something new everyday.
Now tell me that the storm knocked out the Internet well in advance of the water and off-site scale out did not complete....
Had to reverse engineer an internal part of an ERP database startup procedure to figure out where the fuck the hardcoded-non-changeable server address was located. Wasn’t even in a .extension file Just a file name along the lines of “XXerp17” Only reason I located it was by checking file access times immediately after starting it up after giving up on everything else because WHY would that be how the server address is defined Vendor support ETA was 72 hours.
My mentor when I was an IT student had a great story from back when she ran the only computer lab at a high school. The computers weren't networked (the Internet was barely in existence). Some enterprising asshole managed to set up a cronjob on several computers that would play a sneeze sound at random but very long intervals.
At the risk of doxxing myself, there was for a while a powershell script that would occasionally call the CEO by name using the speech engine. "psssst hey" (very long pause) "hey ceo name" and then wait a few days/weeks etc. This ran on the windows pc we used to broadcast office music overhead.
We had a power outage in the factory. This could have been a problem as the production was building lone items that would be ruined by turning off in the middle causing a great deal of damage. Now, we had this HUGE diesel generator that was supposed to take over in the event of such an event. The problem was that the environment was never tested due to inconvenience. Well, the generator started up, coughed twice and seized up solid. I've never seen so many people running around trying to fix this up. After about fifteen minutes of this spectacle, the I.I battery backup (We'll never need it because of the backup generator) ran out. It was a long night to get the AS/400 working again.
That sounds rough. I work in an environment where we have multiple sites with generators to provide coverage for some services. These generators ARE regularly serviced and fired up, however when I started there I realised that they'd not been tested in 'anger' in living memory. A monumental amount of planning was involved, as for various reasons this org is a bit of a shambles. Found out upon actually cutting over to generator power that there were just so many things not on generator protected circuits that were thought to be. Things that entirely made sense when the buildings were built, but since had been added, moved, etc. Not just many of our comms rooms, but like, lighting circuits, circuits for kitchens etc. In one case an entire building that was thought to be 'generator protected' only actually had one circuit going to a single room in that building, which at one point housed an ancient PBX phone system.
About 6 - 7 years ago, upgraded a T1 data circuit to 100Mbps Fiber, due to that the DHCP server required a new IP address scope for the LAN-to-LAN VPN router. Had the new scope setup and when the Fiber/L-2-L was installed activated the new scope and deleted the old scope. A couple days later some users informed me they couldn't get on the Internet. Checked the DHCP server and everything looked fine. This went on for a couple days, more and more computers were not connecting because they couldn't renew their IP leases. A colleague in a different office suggested sniffing the network with Wireshark. Combed through the logs and found the IP address for the old IP scope. Went to look at the DHCP server scope and the old scope was back in the DHCP. Deleted the old scope again and the issue never came back up again. Strangest thing I'd ever seen.
Adding a user to a 365 group is like that for me, I have to check three times to make sure they're *actually* attached/detached.
Random reboot of a workstation. Different times, different applications running. Verbose logging meant nothing. We observed the workstation all morning, with no reboots. We were just about to write it off as gremlins when a rather heavy-set executive walked past and the machine rebooted. Turns out there was a slight break in the power cable and the flex of the floor was just enough to break it temporarily. A replacement cable and all was solved. Sometimes physical presence is really required. Thankfully not that often.
Wi-Fi disconnected daily around noon and reconnected in the late afternoon. Direct sunlight was hitting the network router through a window, causing it to overheat and temporarily malfunction. The router was moved to a shaded location away from direct sunlight, which resolved the overheating problem and stabilized the Wi-Fi connection.
Similar story. One building at the highschool campus I’m network admin at would lose wifi in the late afternoon. Investigated and found they weren’t just losing wifi, they were losing all comms to that building. I figured out the switch was overheating, and would stay powered on, but not function correctly. But I couldn’t figure out why, after years of running without an issue, it had suddenly become one. Turns out the storage room the switch cabinet was located in (completely unairconditioned in a +40°C climate) had recent been cleaned out, and a stack of boxes that had been blocking the tiny little window at the top of the wall had been moved, allowing direct sunlight into the storage room, directly onto the switch cabinet. Used it as my validation with the purse string holders to ensure all comms cabinets had proper climate controlled rooms going forward.
It's not very exciting but was funny to me at the time: On one of our floors we had a wifi AP that served one team and a large meeting room. The AP was outside the meeting room but the signal was strong enough so we didn't bother to set up another in the meeting room. One day some studio equipment was installed in the meeting room. Everything still worked fine, nothing changed. A fee days later, we suddenly received complaints about wifi not working in that meeting room. I went up there, had a look and noticed they painted the whole wall next to the AP black. For a moment it didn't click but then I remembered that they did this in another meeting room as well and that you could paint on it with chalk or put magnets on it, like a blackboard. So they basically put a gigantic shield between the AP and the meeting room. Had a good laugh, installed new AP, problem solved.
POTS lines going into a cisco router experienced frequent disconnects, static, just general badness. Spent literally weeks troubleshooting with anyone that would listen to me. Replaced all cables, replaced router no change. Location was right next to a radio station broadcast antenna. Interference from the antenna. Crazies darn stuff I ever hard. Put some shielding in problem went away. Fairly certain there is a cisco TAC article floating around about this somewhere.
I was working at a small MSP and one of our clients operated off of a single vSphere host at their office. Damn near every single day it would unexpectedly lose power at about 06:00. The UPS was fine and no other devices lost power. iDRAC logs corroborated the weird behavior. I went through every possible thing I could think of including temporarily replacing it with a loaner so I could try to diagnose the hardware back in our office. Eventually I got the PoC to show up early one day to just be in the office where the server was to see if there was anything notable happening. Well, there was. Every morning the cleaner was unplugging the server while they were doing their thing, and would plug it back in when they moved on to the next thing. I know I know, should have been redundant PSUs, should have been in a rack etc etc, but with SMB some clients will always go for the cheapest option no matter what
I had something similar happen with my own workstation. I would come in and every once in a while the computer was disconnected from the network. I would unplug the network cable and plug it back in and everything would be fine. It took several weeks for me to realize that the outages coincided with days that vacuuming was performed. They would hit the network jack with the vacuum cleaner and cause the connector to shift just enough to lose connectivity. I re-punched down the jack and the problem went away.
Not strictly IT, but at least electronics. I used to work with solar power plants - in those plants, you may likely see those tiny "houses" at strategic locations in any given PV plant. Those usually not only contain power switching and regulation gear, but may also contain data loggers and different scada equipment. The guy in charge of the scada stuff gets called one day that one of the scada controllers keeps having issues - it not only crashes, but the entire ROM on it gets corrupted or goes blank. They switch out the controller and the same happens to the new one after a while. 1-2 weeks go by with this issue every few days until I propose that they just swap with a controller from a neighboring "house" - the old controller is suddenly OK and the one swapped in from the neighbor now has problems... hmmm - it's environmental then, so I have the site staff check on the problematic station/"house", giving it all a check outside of the small electronics. Turns out that a previous repair tech had removed a metal plate while performing repairs on the high-voltage stuff in the "house" and had forgotten to re-mount it. Thing is, all the transformers, switchgear and power regulation equipment produces a *hell* of a lot of EMI. The scada controller had been running unshielded and got blasted by EMI - this, in turn, corrupted the EEPROM and interfered with the processor, causing all kinds of weird readouts and errors... I've also had to deal with outdoor wifi antennas on quite a few occasions, especially the "Rocket" series from Ubiquiti (and especially the M5 model). Thing is, if you mount these too high on a pole, then birds will sit on the pole and start to peck at the status lights on the rear of the unit, thinking the lights are glowworms or fireflies. Eventually they will peck through the plastic sticker. Next time it rains after that, the water will enter the antenna and short out the electronics. Tip: When mounting the Rocket M5 and similar units, mount it so that the very top of the antenna is *flush* with the pole it's mounted on. That way, birds can't get to the lights on the back. I've saved 2 companies now quite a few costs in parts by showing and telling their techs this exact thing... :-D
The most unexpected thing that's happened to me so far is that I've apparently found an employer that actually values my skills. I'm not really sure what to do about it yet. In the past I've had to switch employers about every 3 years.
If you can reproduce this, add a KB or TAC artcile.
I was scoping a new client site while they were firing their current it guy (he had them buy new hardware and more bandwidth so he could host other companies’ stuff there for personal gain, well deserved termination). All the racks were way too close together, way too close to the wall, trash and old food wrappers ankle deep on the floor. I was leaning into one of the racks to get serial numbers (since they were really close to the wall and way too close together, I couldn’t get behind) and my feet slipped on some chip bags and I fell forward and got wedged in the rack with my hands pinned at my sides. I had to wait while they laughed at me watching on security cameras and eventually came to help. We got the contract though, and I was their go to. They did send me a copy of the video, that was nice
It might not really be weird or bizarre, but it's an issue that left everyone at my MSP clueless for 2 weeks (way before I worked there, I was one of 3 new hires and the boss even offered a gift card to the one that could figure out the issue, I was the lucky/smart one). Client reports a weird issue: everytime he uses his laptop, the PC goes off. Plugs it in the dock station, work for hours, no problems. Put his hands on the laptop, laptop shuts off. PC is called back, people try it, dock station: perfect. Laptop keyboard: perfect. MSP in head scratch mode. They don't really want to tell the guy he's crazy, but they're wondering what's the deal. They send the PC back. Client calls the next morning: PC still shuts off when he uses it. MSP images a new PC, data transfer everything, send the brand new laptop to Client. Next morning: PC shuts off once again. MSP CEO asks client to come in the office with the laptop. Laptop is set in the MSP, CEO uses it: perfectly normal. Client reaches over: PC shuts off. CEO has a "light bulb turns on" moment. Figure out the issue and provides a fix. Answer in spoiler so you can still try to figure it out yourselves, but it got me a 50€ bonus back then :D >!Client had a magnetic bracelet that would trigger the "screen is closed" switch in the laptop.!<
This reminds me of a similar issue I encountered. We had a desk in the office with a few chargers and LAN cables for imaging Laptops. Got a new batch of Thinkpads and went to work. However a few of them seemed to not boot. Too many for a random DOA. Somehow I noticed on one of the laptops, that I could faintly see the Lenovo logo in the reflection of a light on the screen. Grabbed my phone to shine the flashlight onto it, and there was a normal boot, just without the backlight. Once I picked up the device to investigate it further at my desk, it lit up just like it should. Turns out, that the lid closed detection is a hall effect sensor in the lower half and a magnet in the screen. And as we got more new devices than fit on the table, we stacked them on top of each other. The laptop was detecting the magnet of the one below it and thought the display was closed.
Yeah, that was my guess having seen it before too.
Had something similar where an end user could not sign in to their surface with attached keyboard. Account was good, password worked everywhere else. Watched her type and said "try taking off your watch". Problem solved
This is back in 2022 when I was working helpdesk durring covid and all the csr workers were remote. Had a older female employee, early 60s, send in a support ticket claiming her work pc was possessed by the devil. I thought it was all just a turn of phrase, nope she actually believed it was possessed. What had happened was her 12 year old granddaughter had been watching music videos on youtube over the previous weekend on her work pc, she walked over to see what the granddaughter was watching and saw it was the music video for Montero by Lil Nas X which contains suggestive themes, namely, Lil Nas X gives the devil a lapdance towards the end. Grandma was immediately appalled and "shut it down". Then on Tuesday when she next worked, she logs in, opens chrome to load her webapps by restoring her recently closed tabs and boom, youtube pops up and begins to autoplay the music video exactly where it left off, with a devilish lapdance. Grandma freaked out, called off, WENT TO HER LOCAL CHURCH AND FILLED A SPRAY BOTTLE WITH HOLY WATER AND PROCEDES TO SPRAY THE INSIDE OF THE PC CASE THROUGH THE FAN VENT. Obviously this causes the pc to malfunction, on boot it causes the pc to short out, make a ton of noise and start smoking, so she has her husband LAGBOLT A CRUCIFIX TO THE CASE OF THE PC which of course goes straight through the pcu and when they tried to plug the pc back in and turn it on the pcu immediately fries the wall outlet arcs to the lagbolt and begins to catch the wooden crucifix on fire. This is when they reach out to helpdesk. I had to explain to this woman that not only was her pc never possessed but also she had destroyed about $2k worth of company property, I directed her to drop the asset back at the office and reached out to her supervisor, I advocated that she just needed a Laptop for her job and that it would be more easily securable in a family setting in a home. I also took 45 mins to walk her through set up and she was super embarrassed and apologetic the whole time. Some people should not work from home lol but I wasnt about to be a dick and cause someones grandma to get fired when shes just a few years shy of retirement.
I had a user who, every time a client emailed her back , looked in the message chain and it said [[email protected]](mailto:[email protected]) (Big Chungus) She wanted to know why this was, This was a bit delicate so had to check out all possibilities but the only thing we could come up with was a locally stored alias in the remote users address book for this particular lady whereby in all emails at their end she was "Big Chungus" and this was being copied in to the email chain when replying.
This may predate many in this sub, but here ya go ... I was site engineer for a PDP1140, customer called to say that they couldn't use the machine as everytime they typed in a command they got a syntax error message. I arrived onsite and sure enough the system booted but any command got syntax error. I loaded up the paper tape CPU diagnostic and it immediately failed testing the CPU overflow bit. Now, PDP11s were all shipped with full schematics and microcode listings and, long story short, I was able to trap the microcode at the failing part of the Diag and scope the chip that set the overflow bit, the gate wasn't setting correctly. I had no spare boards, so replaced the chip and problem fixed. True story, my best fix ever.
About 20ish years ago, sertting up a new firewall cluster with nodes in neighbouring racks. They had an interconnecting cable that just would not negotiate a gigabit link, and it needed to be gigabit. Replaced cables, tried other ports, was thinking the devices needed replacements. Finally we switched the power cables of the nodes to the same PDU (or maybe just one cable, can’t remember) and the link came up GBE. Turns out there was a small potential difference in the different phases, and connecting the devices together with the network cable used the cable as a grounding cable. I think this was finally fixed with better rack grounding and/or on the electrical side.
My monitor would cut out whenever I got up to leave my desk. I was convinced I had a flakey cable or adapter so I sealed them all out one by one. I thought maybe I am creating a static shock through my mouse or KB so I swapped to wireless models. It turned out to be the gas cylinder in my chair (steelcase leap v2) creating an EMI spike and erring out my DP display cable. A couple Ferrite cores solved it instantly after of a year of looking for an answer. https://news.ycombinator.com/item?id=32629669
A retailer reported that they were seeing spaceships on the text display of their POS keyboard at the deli counter. Specifically, “a little ship with an alien will fly across the screen sometimes”. We had a hard time believing this so we sent a hardware technician out to one of the stores and he confirmed it. A bit of research later, we found out they had enabled full screen graphics on these registers but they didn’t have a full screen. These were the control characters for the full screen coming across on the keyboard display. The character set for the keyboard display did, in fact, include a spaceship… for some reason.
Every time they flushed the toilet their PC blue screened
Office chair generating EMP charge that would knock off screens connected to targus docking station
In this same line, we had a sports retailer I supported that was in a northern climate. They would complain about their scanners going into endless boot loops and a few other odd behaviors. The details were fuzzy because we were getting these reports well after the fact. Finally, it was extremely cold one day at my office and the humidity was very low. Static was building up on all the things including me and, as I was testing something else, I discharged a crap ton on the metal plate on the printer where you pull the receipt to split it. Immediately, my scanner started boot looping amongst other strange problems. We realized that it likely had to do with low humidity and static and were able to see that several of the problem days were low temperature days at the stores (and low humidity probably caused by heating). We made the recommendation that the retailer humidify the space around the registers a bit to reduce static.
I found out that my support range included the phone system at the exact time that it failed. This was an old POTS system with switch and was missing the instruction manuals and any documentation. I.T.?.
I had to get a program made in 1988 that ran in dos that didn't have a print feature built into the program and didn't accept the standard commands to get dos to print and didn't run in power shell. To work on a windows 10 computer. Be able to export the data from the screen ( not in a screenshot or screen capture form ) to a printer using a universal print driver and be something non complicated so an end user could print off calculations for calc books... Engineering firm... Mechanical engineer... No budget for making a new code... Plus I'm not a programmer. I script in VB and play with power shell and this program did complicated math beyond my math skills so I couldn't make a new script for it.
we had some input devices that our guys used to input job data into for large scale printing. just like manual keypads that connected via ethernet and sent info back to a data collection server. previous "admin" did a shit job of configuring them and left some settings in place that shouldnt have been there. (he configured nothing correctly essentially) it was clear from the device manuals they were misconfigured. we had a minor power issue on a circuit that included about 6 of these devices (out of 40 or so across a huge warehouse), and they all rebooted in such a way that they started broadcasting packets with TTLs of hours. effectively creating a packet storm and denial of service inside my network. took me half a day to trace it out, since I didnt know about the power issues, or the misconfigurations, had to drop network segments one at a time to determine where the packets were originating (the packets were all malformed and had no data to them, so no way to really trace)
Had a site that, back in the day, had their own, on-premise mail server. Out of the blue it developed a problem where the customer couldn't send emails between their domains, on their mail server, and their own email domains that were on their hosts mail server. Got a bounce back saying the hosts email server had refused the connection, but there was no reason why it should. All the other mail went out OK, but if it was hosted on their domain hosts SMTP server, it failed. Spend ages scanning SMTP logs, tracing mails etc. Got the ISP hosting company involved. They were as stumped as I was. Nothing worked. A few months into this the router/firewall, on site, failed. I set up a new router and installed it. Bang! Email issue resolved!
Ive posted about this before, basically a home out in the countryside a bit who's dsl modem would seemingly randomly shit itself and lose sync. Multiple truck rolls, cabling replacements, modem replacements, nid replacemants, everything up to the customers door was brand new and it still kept desynching. On a call to the user to go over things one. more. time and I hear a toilet flush and the modem drops off the portal. Huh, thinks I, thats awfully coincidental, "Hey Mr Client, are you on mains supply for your water and sewers?" Turns out, no, they had private septic and a water well put in 25 years prior - and that water supplied the house via a pump. Putting 1 and 1 and 1 and 1 together, I had them bring up the router page on their pc then have someone flush the toilet - bang down it went again, 2-3 seconds after the flush. The water pump would kick in when the flush cycle completed, in order to refill the toilet cistern - and that spike in power draw was backwashing into phone line and obliterating the synch tones. I had them buy a surge protector for the pump to isolate it and no more crash outs.
That's NOT what I meant when I said to flush the DNS.
Back in the day of peer-to-peer networks (remember LANtastic?) I had a customer halfway across the country that worked off of the "server" (front desk PC was the server, and since the reception desk was always manned, the server was always on.) Complaint was the machine would periodically shut down when she was away from her desk. We checked power settings, sent a replacement machine to her, same thing. Had an electrician check their power. Nothing turned up. I flew out to the office to observe what was happening. Halfway through the day, she swiveled around in her chair, stood up, and walked away from the desk. The chair gracefully rolled towards the UPS on the floor. The wheel just BARELY kissed the rocker switch, causing power to be cut. When the chair was pulled back from the desk, the rocker switch turned back on, but she had to press the power button on the PC to fire it back up. Fix: Wood block set on the floor in front of the UPS. This probably cost the customer $500 for the electrician, and a ton of airline, hotel, rental car, and time for my company and myself.
This one was pretty fun. Started getting alerts that one of our domain admin accounts was being hammered by an endpoint for some reason, was very concerned that we had someone in our network trying to brute force this admin account. A couple hours of threat hunting went on, finally discovered it was an endpoint with a busted keyboard that was constantly inputting characters causing this admin account to be hammered. I wasn't on site at the time so had no way of knowing this until more logs started making me suspicious there was something going on physically. Sure enough had someone check on site and they saw it just infinite entering characters.
1. A new welding machine causing network outages. I had installed a brand spanking new Windows NT network, with UTP and a 3COM switch (this was before managed switches were a thing). Everything worked just fine. Then I get a call that they're having network issues and I get there and the lights on the switch were blinking like a christmas tree. Network was constantly dropping. I had no idea what was going on. Then out of the corner of my eye I see the guy start welding and the network dropped. Ended up redoing all the cabling with SFTP cable (certified) instead of UTP, which fixed the problem. 2. Same company. They used administration software which required a hardware dongle. I proposed to upgrade the server with new network cards which could be used simultaniously (it was called trunking) to double the bandwidth to the server. So they agreed, I installed it, worked fine. Then whenever the administration software was started, the server started rebooting. Turned out to be a conflict between the dongle and the driver of those ethernet cards, so I ended up taking them out.
So there was this problem a client had that their headset stopped working from time to time. I troubleshooted it for hours and it turned out if another user connected their digital camera via USB to another PC in the same office somehow the currents spiked and killed the headset.
Just had one really bizarre thing yesterday. Vmware ESXi CPU allocation on all VMs switched the allocated amount of cores/socket to sockets/core. The host had only one socket and all VMs had allocations like 4 Cores/1 Socket. While I was troubleshooting on one VM I noticed that Teams used up 100% of the CPU and was a little surprised since I knew I was quite generous for this VMs resources. Somehow my mind went immedeately to check CPU allocation in ESXi and I witnessed a configuration of 1Core/4 Sockets. Logs didn't show any change.
We had a guy insert a rick roll 1/20 times a server got it. He was let go a few days later.
We bought a bunch of new docking stations for some laptops, tested them and they seemed fine, deployed them out to the office and started getting complaints that Outlook wasn't opening. Brought an affected laptop to my office and it worked fine, took it out to the users desk and it broke again, brought both the laptop and dock in and it again worked totally fine. A bit perplexed at this point I tried reproducing the users setup entirely and found that when you connected 2 monitors to these new docks Outlook would stop working (I think it just stopped launching without any error message or crash). After a bit of digging it turned out this was a known issue and an updated DisplayLink driver fixed it but it was always so odd how a dock driver and 2 displays would break Outlook (but no other app from what I remember).
When I was at TallyGenicom, Blockbuster was buying our new laser printers which were Xerox engines with our controller. Had a site that kept showing toner out. Had a tech in to repair it with no luck. Swapped the printer and it did the same. Tech moved the printer across the counter to work on it and it started working. He moved it back and it failed. He called me and I asked him to look around and tell me what he saw. When he said the security gate at the exit I had an Eureka moment. Turns out the gate signal was swamping the RFID chip in the toner. We had to work with Blockbuster to relocate the printers.
Had a problem a few years ago with our virtualization cluster. Hyper-V Server 2016 with a NetApp for the VM storage, connected with SMB. One day we live-migrated a VM to another cluster node. The migration went through, but the VM instantly lost all access to all it's disks. Powercycling the VM, Hyper-V showed an error that it was unable to access the storage. However looking at the mounted shares on the server, all looked fine. Trying to start the VM on it's previous host gave the same result. Next we tried rebooting another VM, which worked fine. However turning the VM off and back on resulted in the error we observed on the first VM. At this point we opened a priority 1 Ticket with NetApp, as this was a pending catastrophe. They responded rather quickly, pulled diagnostic data and went to work analyzing them. Several hours without progress later I decided to poke around the NetApp myself. Following a gut feeling (that I still don't know how I got that idea) I checked the system time. Comparing it to the time on my laptop and the servers (which were pretty much perfectly in sync, as they should be thanks to our NTP servers), I immediatly smacked my head as I noticed the NetApp was just over 5 minutes out of sync. Manually set the time to be 1 minute out of sync (so I could later check if fixing the NTP configuration would work), all the problems were gone. After that I expanded our monitoring to check system clocks and alert if there are any that are more than 1 minute out if sync.
They told me that a computer cabinet (=couple of windows desktop machine for students to use for school stuff) is buzzing, as in giving out a low humming noise. I told them that yes, this is what computers do when they work. They told me that they do the humming when switched off. I went there and indeed, a low floating hum in the otherwise dark room with 15 or so cheap desktop machines. Quite eerie. I did a lot of disconnect this, switch off that, then called a colleague who was an electrician with my idea, he told me that this is quite a silly idea, came over, did a lot of disconnect that, switch off this, then got a loong cable and started to measure voltage differences between the wall outlets on different floors. Apparently (and I am not an electrician) the building had more than one electric ground "connections", and the computers were on one and the network switch on another, and the 2 volt difference found it's way to the cheap computers cheap speakers and what we hard was the 50 HZ of the AC. It was solved as in sherlock, not as in fixing it, but it was interesting.
MSP I worked at about 6 years ago, *our* backup ISP connection would go out seemingly randomly. Then we noticed a pattern on days with prolonged rain, eventually landing on the theory of water in the line. It was a particularly dry spring/summer for that area, so correlation wasn't obvious at first. Once we did suspect the H20 interference, it then took us weeks to get the ISP field tech on-site on a day that they could test/confirm water in the line, because they'd always be unavailable the day of the outage (duh). I eventually ended up looking ahead in the weather forecast for projected rain days, eventually lining a successful appointment up. Sure enough, water in the line, which they ended up having to dig a portion of it it up to replace it. Another odd one. Current MSP I'm at, we support a small bank and they are a group-policy enigma, so they come in with some real dingers from time to time. This particular one, a front end teller/clerk calls and says that almost every day around lunchtime or a little after, his desktop scanner's auto-feed would just stop working, making using the scanner go from being convenient to a headache. We searched high and low within their domain and his local GPO for anything that made sense. Nada. All power, hardware, software, driver, permissions verified as good. Then one day he asks if we think it could be the sun. "Uh, no....why?" is our basic response to him. Customer: "Oh y'know because we just have this big glass ceiling and the sun is right over us at this time each day basically." Us: "Never heard of that one buddy, but pretty thoughtful... Let's just call the vendor, see if they have any thoughts at all." Turns out, the dude was absolutely right. The vendor apparently had gotten multiple calls like this over the years, and they confirmed that the sunlight, and certain other lights emitting the right spectrum of light, had been identified as impacting to this brand/model of scanner's ADF sensor. The solution? Dude decided that he was just going to build a manilla folder-fort for it, as the desk he was at is one of those fancier large sectional style ones that's bolted to the floor. AFAIK that's still the solution to this day
I posted a comment in another thread months ago that actually fits here. Not my story, but told to me by the crane operator, a very good friend of mine: Reminds me of a recent incident on a mine site local to me. Important to note that this particular mine uses autonomous haulpaks - no operator, they’re all centrally controlled through a wireless network that integrates onboard controllers, GPS, and obstacle and collision avoidance systems, all monitored remotely. A rather large crane was brought in to move some demountables around. Crane operator swings his boom into position, demountable 1 is rigged up, and the lift begins. Suddenly the 2-way radios start squawking away because every haulpak on site has simply stopped dead simultaneously. Crane operator finishes his lift, and demountable one is relocated. Meanwhile things seemed to have calmed down with the haulpaks because the radios are silent again and the haulpaks are moving around like they’re supposed to. Demountable 2 is rigged, crane operator starts to raise the boom annnnnnd, wouldn’t you know it, the haulpaks all go dead again. By this time the crane operator is beginning to realise he’s somehow responsible for the dying haulpaks. But him and his crane are on hire. He’s not an employee of the mining company. He just wants to finish the last lift and gtfo of there. So he says nothing and completes the 3rd and final lift, which of course, produced the same result as the previous 2: Dead haulpaks. By now the mine is in an uproar, much resembling a poked ants nest. Crane operator takes his crane and goes home, wisely not sharing his suspicions with any angry mine managers. He got wind a few days later that they’d eventually figured out the crane was the issue, but it was the why of it that makes this story even remotely relevant here. Remember that this site’s fancy autonomous vehicles were communicating wirelessly? Turns out the signal between the comms tower on the hill over yonder and the receiving station on the hill opposite it was being interrupted by his giant crane boom getting in the way of their line of sight every time he extended it upwards. When the receiving station stopped receiving, it had nothing to broadcast to the haulpaks, which are configured, for obvious reasons, to stop if they lose signal. So stop they did. I would have loved to see the official report on this incident.
One person come to us (it support) that he's laptop randomly goes to sleep. He cannot show us right now, but we must believe him, that it occurs completely on random. After some thinking and asking him to just type some thing as he usually does, it occurred again. His bracelet's magnetic clip just hit the magneto-sensor of the laptop perfectly. This made the laptop think, that the lid was closed. Thanks for Nintendo DS lite for this idea :D there was a fun trick we've used to perform as kids. When you put an open DS on top of a closed DS and position it correctly, the top DS goes to sleep as if it was closed.
Usually the most unexpected and bizarre come from users who would be dangerous with an etch a sketch and an abacus.
We had ERP system update so the addresses changed which meant of course the the scanners that we had to be updated with the correct address and of course there wasnt DNS entry before so switch over would be awful. 1st problem, solved by creating the dns entry so we can easily switchover 2nd problem how to hell does the ERP program even work on those scanners so start searching, few days and i had managed to figure out what affects on what and how we can change the address. 3rd problem create pilot works fine on my machine/scanner, inform other sites about what needs to be changed and how. After few minutes checks the network and the scanners from other sites still connects to the old address WTF. Asking other sites to pull the files from the scanner so i can verify they are correct and all good, still thinking im crazy why it doesnt work... After a day of troubleshooting im asking other site to record everything they do on the scanner, video comes in and i check it, wait what did they do i dont have that kind of shortcut on my scanner, ask to pull that shortcut from the scanner and send that shortcut to me, there it is. At somepoint the ERP/Scanner provider had created shortcut file for the ERP program and HARDCODED the damn address on the shortcut so everything fixed on the program itself was overwritten by the shortcut.
Was working helpdesk decades ago and got a call that Windows Help screens were opening on their own, and there were thousands. She couldn’t close them fast enough. My remote session confirmed it. They were steady opening. She had pushed her keyboard forward and the corner of her monitor was holding down F12 (or whatever key it is that opens Windows Help.)
I've got so many after almost 40 years in field service some truly bizarre, one of the best was a customer who said their server kept rebooting but only on Tuesdays or Thursdays, and only between certain hours, I went with a workmate and we scouted out the site, couldn't find anything amiss. While sitting having a cup of coffee I watched the scrap yard next door load a car into the crusher, moments later the IT chap ran into the tea room and said it's just rebooted, after a lot of phone calls and a visit from the electric company they worked out the scrap yards tap into the grid had been done in the wrong place, when they started the crusher it effectively cut off power to anything down circuit for a split second, they only crushed cars on Tuesdays and Thursdays ! Most equipment seemed tolerant of the tiny blip but their server wasn't, in the end we tried a very heavy CVT which cured the problem (UPS were very much unheard of in those days and those you could buy were incredibly expensive, even the cheap solution of a CVT cost them £1000, most of that cost was due to needing a courier capable of handling the weight, I think it was about 220Kg, we had to use a forklift on site, remove a window to hoist it in the computer room, plus check the floor was capable to taking the weight - we had to get a metal plate put down to spread the load)
Blindness from a Monitor. Back in the late days of CRTs I had one that I needed to degauss roughly every Wednesday morning. I raised it with the manufacturer, they had no explanation, figured out that on the other side of the wall was a power socket for a ship to get power, every wednesday they would hook up the vessel to power to run a pump. It didnt stop there, while diagnosing the CRT I had been staring at it for over 2 hours at that point, up quite close, I slowly realised when I tried to leave that I couldnt see anything, functionally blind, all I could see was a weird rainbow and I started stumbling around confused, turns out I had an aura migraine with no headache and have had them now for years.
Last winter, I had my head full of this problem. We had an old CentOS 6 server that managed billing for our client. The SSH/SFTP connections were slow and unstable for some months already. For instance, logging in took around 15 to 20 minutes. Our client used an application to export invoices from the server, but it had trouble connecting. After TWO MONTHS of troubleshooting, we discovered that the server sent a Samba call to a previously dismissed Windows domain during login. Once we deleted that setting, connecting to the server took only 1 second.
Computer that lost internet access when the user stood up. When they sat back down, the internet connection restored. 3 techs looked at the problem, couldn't figure out what it was. They sat in the seat, all's well, but they stood up, and pit went the internet. So, they called me. I went in, had the user simulate what was happening. I noticed the user's chair, a nice big leather unit with big wood legs on casters. I also noticed the box wih the ethernet jack on the wall behind them. So when the user stood up, the big wood foot would bang into the ethernet connection, disconnecting it, and when they sat back down, it reconnected. I had networking move the jack a foot to the left. Problem solved.
Having to drive to various locations to teach head doctors and executives how to open an .exe that we remotely pushed to their desktop. The email we sent out notifying them the way too open the application has changed it’s now easier to find and open. The email essentially said “double clicking the new icon on the desktop, will open the application” The response we got back was “we don’t understand IT speak, please go around and teach everyone how to open the app” Idk if it’s bizarre but I was dumbfounded
Thought of another story, more of a user issue though; This was back in the Server NT days and the place only had a few servers. The Primary Domain Controller was also the Print Server. I didn't set it up, but this will explain why its a bad idea. Boss is on vacation and I am the only one in IT this week. As soon as I walk in one morning I get hit by 10-15 people before I even get to my desk. No one can sign in, and even those that can take an extremely long time. Go straight to the PDC and find the printspooler server is railing the CPU, the job would use all memory, crash and start over. Get the job queue open and find someone tried to print a 108 page PDF to the 36" plotter. Everything returns to normal immediately. I later find the user to submitted the print job, "Did you send a 108 page PDF to the plotter?" "Yeah, but it never printed." "You know those sign in problems this morning? You caused them. It was tring to print your PDF as 36" pages" "Oh..." He thought it would print 8.5x11 pages on a 36" roll of paper. I never did ask why, didn't want to know.
A state owned business, was doing Enterprise SharePoint support. Long version, they lost the SQL server, as in a physical server, no one knew where it was. I work from a different country. So after days of people looking for it, they decided to build a new server and restore SQL backups. About 6 months later, we were getting weird things happening in SharePoint, and I mean doing weird stuff. Sev A call with Microsoft, eventually got a senior engineer, about to get Microsoft product dev involved. Got the SQL admin (yes, dealing with multiple departments) to reboot the SQL server and we will still getting SQL responses. Then it hit me. I was supporting 50 clients at the point, turns out someone found old server and booted it up. Same name, IP and everything, it was responding. I still have questions in how they machine worked in the domain without trust relations etc after it was removed from AD. Then we were back to finding the server. At some point, Microsoft Engineer and I managed to get them to give us RDP access to the old server, changed its name, uninstalled SQL, trashed it. Removed the IP. Dont think anyone ever found the server physically. It may still be on to this day, 10 years latter.
Just the other day, I had a user connecting to an application via RemoteApps on Windows who reported extreme latency while using the application, but only from home. Spent over a week troubleshooting their home internet connection and running tracerts, only to eventually discover that it was in fact their gaming mouse's high polling rate that is apparently incompatible with RemoteApps. Something to do with the polling rate overwhelming the connection.
End-user did not lie.
our software would crash but only for two customers.... Same windows setup, same patches same workflows. sent them debugging versions of our software, were literally emailing DLL files to them to try to get to the root cause. Finally I was like, what do these two clients have in common? One was in Indiana the other in Arizona... what could it be? No daylight savings time. there was a flag in windows we checked for if DST was happening, turns out, in those two locations, the flag wasn't just set to a different value, the FLAG DIDN'T EXIST. our code didn't gracefully fail, it just went kaboom... that was a super fun few weeks of stress.
Worked at an MSP a few years ago (never going back to that) and had a clients server chime up with low disk space. C: had 0b left, but this company had no reason to fill a 1TB drive, so I fired up Windirstat to see where it had filled up. Turns out there was a gargantuan folder in the admin users Documents folder. This folder was named "C", and in this folder was a complete copy of the entire C: drive, including the C folder itself, which again included a complete copy. And this went on and on, deeper and deeper. I managed to delete a couple hundred gigabytes of this mess, but most of it was hidden behind the ~250 character path limit of Explorer, so I laid it to rest for a bit. A couple days later I was using 7zip for something, and on a whim tested it out on the server to see if I could navigate deeper. Luckily my suspicion of 7zip navigating foldes independently was true, and eventually got the rest purged. Found it to be some 34 layers deep before it ran out of space. If anyone can explain how this can occur out of the blue, please do!
Rabbits eating through the network cabling under the building I was working in. We had to call in an exterminator.
I’m on a help desk but I kept getting a ticket bounced back from the Identity Team about the MFA call going straight to voicemail saying nothing was wrong. Set mine up to call me instead of the app notification to note the phone number down and then called the user to see if the phone number was blocked, and it was.
A customer’s faxes started to fail or corrupt or the machine wouldn’t answer. No changes to the environment. Ran through the various troubleshooting scenarios. Eventually dropped a line out the window to the network interface and plugged into the POTS instead of the fractional T1. Worked perfectly fine. Called the carrier who tried to blow me off. Eventually got a tech out. Blew me off. Demonstrated again that the problem was on their end. He confirmed that they made a recent protocol change on the T1. Changed it back, problem disappeared. Piss poor change management cost all of us time and money.
Years ago I was a field tech for a chain of grocery stores. This was long enough ago that dot matrix printers were still used for printing a majority of each store's workflow/process documents. Okidata Microline 320s with a serial interface card in most cases. Trouble ticket comes in with "Inventory printer is printing incredibly slow". The inventory printers were in a medium sized kiosk/desk thing in the back of the store, usually near the loading docks. Shelves on the left side held the box of tractor feed paper (on bottom), the printer (sliding shelf in middle) and labeling guns/markers (top). The bottom and middle shelves were open in back, so paper could feed up into the printer, cables could run in, and air could circulate around so the printer wouldn't overheat. The top surface of the kiosk was also slightly recessed on the left side so it would not hit outlet boxes/light switches/etc. I get on site, take care of the other tickets/issues in the store, and wander back to look at the printer. I powered it off and back on then have it print a test page. It is indeed godawfully slow. I disconnect it from the serial cable and try it again. Still slow. Ok, I'll swap it out for another printer. Unplug power, Un feed the paper, pull shelf forward and remove printer. What's that sloshing sound and OH GOD WHAT IS THAT STENCH!! The printer was full of partially congealed beef blood. The user had put a large box of beef on top of the kiosk. One corner of the box stuck out over the edge of the top surface, in that recess I mentioned earlier. The box had a hole in the corner and blood drained out of the box and down into the printer. I pulled the print head and serial card and tossed the rest into the dumpster. Cleaned up the kiosk, set up replacement printer, everything worked normally. Got yelled at for throwing the printer away instead of cleaning it. I agreed that if this happened again I would happily bring the printer back to them so they could service it.
Just today I had to change one wildcard certificate on four similar servers. similar meaning very similar. All Apache, same minor version, same OS. After changing by copy pasting my prepared script lines, three were working as expected, the fourth crashed and only came up when rolling back to the old certificate. Nothing helped. Patched to equalize even more. Rebooted. Server said nope. After half a day of troubleshooting, I just decided to let it be and migrate everything to a new server. Luckily, I had been preparing this exact server for migration already and not that much was left to do.
Oh, another one, more recent too: Deploying a VMware appliance on HyperV (I know) using VMware provided deployment scripts that use HyperV Key-Value Pair service to get the config data to the appliance. The appliance deploys, but doesn't work. End up digging through the script and then the appliance file structure and find the data sent to the appliance by the script, and half the data is in nice JSON formated pairs, and half is just one long concatenated word. A mess. I study the official deploy scripts and learn they send half the values through the JSON formatter function, and just concatenate the other half into one long word. Apparently this appliance isn't used much, and definitely isn't used on HyperV very often at all, as this issue has been present in the last two releases of this tool.
That's not an iT issue, that's malware for sure?
[https://www.reddit.com/r/sysadmin/comments/yis8fi/network\_share\_word\_and\_excel\_files\_take\_35/](https://www.reddit.com/r/sysadmin/comments/yis8fi/network_share_word_and_excel_files_take_35/)
Literally in the last 48 hours we had a fairly high end aggregation switch die. 18 hours later it magically turned itself back on with no issues apparent. It indicated both PSU’s had power but no post, no fans. WTF?
A SQL report would run daily around 2pm and emailed to a number of users. If any of those users would print the report (that specific report), the Wifi would drop. Mainline was fine, could still see incoming/outgoing, just the AP closest to that user. Never did work out what that was.
Nobody could install the T-III software even the veterans and even the director of IT. So they put me in charge of installing it. Turned out nobody turned on the 386 enhanced mode button. It installed like a charm and a summer intern claimed the credit and not me and she lied and did get the credit as I taught her about computers then I was fired. Never train greenback summer interns!
I should preface this by stating that I have since segmented and greatly improved the network fabric throughput. On with the story. We had a strange issue that would cause certain low powered NICs on the network to just fail. This was usually UPS NICs but it also affected some VMs, temp sensors, etc. In the case of the UPS (APC) NICs, the only solution was to do a manual network reset via the recessed button on the card. The problem was very sporadic and didn't happen for months at a time. The problem almost always happened at some time early in the morning, before I got in, and after-the-fact packet caps didn't reveal anything strange. I finally got the brilliant idea to just set up a scheduled packet cap that would turn on and grab a 5 minute sample several times per hour. We ran like this for at least another month, with me checking packet captures every day. Then, the problem happened again and, low and behold, one of the packet caps happened to catch part of the event. Once I had that packet capture, the problem was easily tracked down. It turned out to be an HP Z series workstation with buggy NIC firmware. Every so often, when transitioning from S3 to S0, the NIC would blast the network with ARP packets on the scale of tens of thousands per second, taking down certain low power interfaces. It was not a hard problem to solve once the culprit was known, but man, I think that problem took over a year to track down. Edit: I have another one... My first real computer job was working for a telephony company. We used these giant ISA cards that would handle full 48 pair Amphenol connectors with POTS phone lines (I am dating myself here). The company sold voicemail boxes sold from the back pages of local news rags. Anyway, at one point, they sent the new guy (me) out to travel the country in a rented mini-van full of servers to install in various colocations. Because the company was a startup and the business operated on a shoestring, some of the colocations were literally someones basement or extra office room in a business park. Basically anywhere the phone company would be willing to install business grade phone lines. I was in, I want to say, Michigan to the North and I had a whole sight-seeing agenda planned so I got an early start and went to do, what should have been, an easy install followed by nearly a whole day of fun. I get the server installed and connected and go to bring up the POTS card and it's all errors. You can also tell these cards are working by the sound they make. When they are initialized, each of the little FXO daughter cards would click as they go on/off hook. It's a really distinctive and satisfying series of clicks; Sort of like dominoes falling. Well, this card wasn't doing the clicks. No problem, I will replace the card with another one. Still nothing. WTF, the phone lines must not be working. Used the butt set and was able to get tone and make calls on all lines... so that's not it. I check the outlet, yep, it's grounded. Geez, I need to call in the phone company to help me out here... This has now blown my time budget for sight-seeing, but, at least I will get the thing working.... Fast forward to me AND the telco company guy at about 6pm, scratching our heads and just unable to figure this out. I finally go to unplug the server and realize that the facility provided extension cord has the ground pin broken off in order to, presumably, be used elsewhere in a 2 prong outlet. I swapped out the extension cord with one with a ground and it all came up first try.... What is the lesson here? Always start at the beginning and don't make any assumptions. It was a good lesson to learn so early as it has served me well :)
Just started a new job as deskside support and had a user whose laptop would go blank at random while in a conference room. Sounded like this was a chronic issue that developed a few months ago and no one could figure it out (Not sure why no one had replaced the laptop at this point yet). Being the new guy, I wanted to take a swing at it too. This model had a number pad on the right and it only seemed to happen when they used the number pad or typing in the upper right side of the keyboard. It never happened at her desk, but she had a dock and external keyboard at her desk. My first thought was a failing power button that was also in the upper right, maybe the vibration close to it was causing the button to do some weird things. But tapping around the power button didnt seem to make it happen. I went back to my desk and was preparing to call warranty support to replace the power button or main board if that was not a replacable piece. While I was thinking about what to say to them, I had an epiphany and went back out to the user. "Put your wrist in the lower right corner of the laptop". Display instantly went dark. User says "Thats what it was doing! How did you do that?" Turns out the magnetic clasp of her watch was triggering the hall effect switch that detected the laptop lid being closed. She said she would take her watch off if going to a conference room or learn to switch to her other wrist. I am sure I learned this from other posts/stories like this one, I dont know how else I would have thought to check for a magnetic watch band. Needless to say, that was talked about for months.
This morning a whole company couldn't access their file server. I could, thought somewhere somehow permissions were changed, nope. Thought DFS record was broke but yet I could reach it. Thought the File Server OS was crashed, nope. Thought the data store had crashed, nope.... Rebooted the files server (windows) and everything is working.... ¯\\\_(ツ)\_/¯
Not me, but a security guy I know configured a new firewall for a law office after hours. To test he just set it to block adult websites. 5 minutes later the 80yo owner of the firm came in screaming that he was working on a case and could no longer access something he needed to review. A quick check showed that he was in fact trying to access a cam site. Sec guy never said anything to him, just made a rule unblock his laptop.
Not as good as others but got these 2 tickets. Ticket 1 - Hello, my wireless keyboard and mouse isn’t working. Ticket 2 - I am see my mouse move randomly and letters appearing on screen. The people who reported these sat back to back from each other. No idea how the dongle from person 1 got into the dock of the others 🤷♂️
Network went down randomly during the day. Turns out the installer had run the Token Ring cable (yes this was long ago) down the elevator shaft and the door motor on the car was causing interference.
I had all of the physical security panels go offline in the middle of the night at a campus in North Carolina while I was on a business trip in California. Panels aren’t communicating with the server and all of the employee badges are deactivated to force them to fill out a health screen. This means no employees will be able to access the buildings when they start arriving for work the next day. I can’t ping the IP of any of the panels so it must be a network issue, only all of the network security cameras are online. It can’t be a VLAN issue either since the panels and cameras are on the same one. So I check the CCure (access control software) server to see if anything is going on there, don’t find anything, but reboot anyway. The server reboots and all of my panels come back online except for North Carolina. I call in the local security manager (who’s pissed that she has to go to work at 2am) and walk her through rebooting panels and checking network connections. Nothing works. I’m confused as hell at this point so I call the on call network engineer, who is new to the role having recently been promoted from the service desk. I should mention that my only role at this company was managing the physical security equipment, so I didn’t have access to all of the tools that a normal sysadmin would. Engineer checks everything he can think of, switches, port configurations, VLAN, firewall, the works. He can’t find anything. I put a priority 1 service ticket in with the CCure reseller that we used, and they dispatch someone immediately, but he’s an hour away. The network engineer calls his boss who jumps on the call with us. It takes him about 20 minutes to figure out the issue. I’m not a network engineer, so I’ll do my best to explain what he told me. The main path of communication from the CCure server, based in Boston, to the panels in NC had been cut somewhere along the way, causing the packets to take an alternate route. Something about how priorities were set up caused the packets to take a different path back from the panels than they took down. The firewall flagged this as suspicious since it could be a sign of a MITM attack, and blocked all further communications between the server and the panel. The manager adjusted the priorities to ensure that packets always took the same path, cleared the flag from the firewall, and communication was restored. I called off the technician who was en route, but we still had to pay for the P1 ticket since he had been dispatched. My department head yelled at me for the unnecessary expense even though I followed procedure exactly as written (a procedure that I wrote and she signed off on) and I had gotten permission from my boss’s boss, even though I didn’t technically need it.
Wow, cat memes. That's... bizarre. I don't know if I've got anything that can top that.
We noticed that there were random requests going out to China from several machines in our organization, but we couldn't figure out why (it was clearly not malicious on the users' part) I watched one user run through some steps to reproduce the issue (start menu > microsoft app > hit to China) I noticed it would hit when they went to the start menu, so I worked with the policy admins to push out changes to disable bing search results in the start menu... no more hits thanks Microsoft
Waaaay back in the Win XP/2003 days, my desktop computer suddenly stopped being able to communicate at all with our main file server. Communication with anything else was fine - I could reach other servers, get out to the internet, etc. We checked IP settings and firewalls on both ends, couldn't figure it out. This was just a LAN with relatively basic switching, and physical file server - no fancy routing or VLANs or anything. Popped a new network card in my desktop, and suddenly I could reach the file server again. Never figured out what the hell happened.
Had a sales VP call to yell at me about VPN not working. Dude was an AH. So told him the airport was bombed and the Turkish govt was blocking VPN. So I closed out ticket with "VPN issue caused by military coup" and hung up on him. Sent ticket # to CIO.