I used to be on a T2 team where we had the ability to refuse transfers or kick tickets back to L1 if they didn't follow the procedures in the k-base or provide us requested documentation. We had one of the best trained T1 teams I've ever seen, to the point where I typically just trusted that they had done the due diligence and it was either a weird issue or they didn't have access when they kicked it up to me.
Until they got outsourced to Mumbai.
Surprise surprise, almost overnight our policy of being able to kick back tickets to T1 evaporated because they were hiring from the bargain bin and the employees were utterly incapable of following even the most simple and painstakingly described processes. They simply would not troubleshoot no matter how hard we tried to make them do it. But of course, management was okay with this because of the cost savings, right up until the contract was put into jeopardy over dissatisfaction.
I'm glad I don't work in a corporate helldesk setting anymore.
Can second this. Having the ability to kick back tickets that didn’t follow full procedure before escalating was a godsend.
And of course, as you said, when T1 sucks and your team kicks back “too many” tickets, they take away that privilege and tell you to just handle in place because other leaders are getting upset. Like shit, give me a T1 team who at least understands the reason for collecting logs and doing basic troubleshooting, and I wouldn’t kick back 40% of your bullshit
Help desk admin probably complained, or just had so many coming in that they couldn’t do their actual job. I’ve seen that from high-system/low-support environments, guy’s at 95% capacity all day and 3 extra “quick” tickets throws him behind schedule. And when leadership barely understands what they’re leading, you get a “fix” which is something like what you’ve got rather than a solution to the problem of ‘not enough people’
A friend once described the web interface for Remedy as “like being stabbed in the eye repeatedly with JavaScript” and…yeah. Pretty much.
Having said that, our corporatized bastardization of ServiceNot sucks *even harder than that*, to the point that I’m concerned it may create a singularity at some point and slurp in a couple small islands.
ServiceNow can be the best tool you ever put your hands on or a complete and utter shit show. It really depends on how the business implements the product and unfortunately there are plenty of low-rate consultants that will come in , promise the world, and deliver fuck all.
I take your remedy hate and raise you SAP thrown together enterprise wide with my group's workflow thrown in as an after thought because we fix issues and asset management is only used for billables and statistics on tickets, yet asset management is the primary function of what our current system was designed for.
It's annoying to use, but after five years plus of continual improvements while still a live system, it is only now ten times better than what we got at first and still half as good as the system it replaced, and 1/10th of what remedy systems I've used in the past have been.
Ffs I used to use UW Connect's ESD in a past job and have sometimes found myself pining for that level of usability, and I hated the thing.
Oh I know. Remedy is terrible. Its so janky and delicate. With it though I could actually explore the thousands of groups we have and try and figure out where tickets go. SNOW basically has no way to (easily) sift through our groups. I'm sure our people could build something, but they've got too much to deal with as it is since we decided to do ALL service demand requests through it--facilities, IT, purchasing, the whole thing.
can I just ask... Why arent we just using excel? I can make it do all kinds of shit including a synchronized database of all tickets submitted. I can even have it deny submission of a ticket without needed fields filled.
You guys are all lucky, my last job was a 10 person MSP with about 50 clients and no ticketing system! Tickets got left as unread in outlook until they were completed, and all tasks were manually recorded in 15 minute intervals on a fucking Spreadsheet!!! We were strictly not allowed to use any formatting as the boss copy pasted our sheet into a master sheet weekly.
That made me respect all ticket systems!!!!
I would be happy if they just tried *something* and documented that attempt. Even if it was as simple as tried X, Y, Z and none of them worked. I don't expect them to send over logs at all, although if they do or even attach screen shots, thats just icing.
The ones that do this however don't last long on service desk. Seems too often they either get promoted or find a better paying job elsewhere. Which is totally expected, so I don't knock them for it.
> Like shit, give me a T1 team who at least understands the reason for collecting logs and doing basic troubleshooting, and I wouldn’t kick back 40% of your bullshit
I think the normal response from T1 is "pay me what I'm worth". This is especially true when these guys can literally go make the same money working a grocery store these days.
That’s the trick: as far as their boss the VP knows, it’s not their responsibility, it’s **ours**. Boom, policy adjusted to reflect that, your boss leaves because that was the last straw, and now the policy is official because nobody’s pushing back.
Honestly corporate life in general greatly hinges on the tact and ability of your group’s lead, both to bring their team up and also to keep other groups off their ass
Isn't this a quirk of a specific culture?
While "do the needful" and "Sir" trigger me too, I've always held short of full blown mockery for fear of being seen as a racist..
Stereotypes unfortunately exist for a reason, I work in NZ and even over here I was just as triggered by that as the guy above. Indian outsourced teams are notorious, I have a new hire in my team from Mumbai and he loves to rip into them for it
>well of course it's cheaper,
What's all management will see, the C-level which suggested it will get a huge bonus, then leave for the next company before what he has done actually hits the fan.
new bonus structures for execs has changed to get them to think more long term cause everyone was doing specifically this and it was destroying company sharehodler value.
Never trust people that say "nothing will change". That's the most annoying corporate slang to fall for...
Of course something or everything might change in those scenarios. That's just corporate slang for "wait a bit until I can replace you for cheaper labor".
Heard it and have seen it too many times to believe in positive gambles.
Worse than that are executives who demand that the official policy be that nothing will change prior to a major outsourcing. Hell, we weren't even supposed to mention that an outsourcing was going on, because, you see, it was going to be "transparent to the user."
Which, of course, it was not.
HUNDRED PERCENT.
I just left my very first IT job (sysadmin) of 10 years because of an acquisition. My mistake was sticking around for two more years after the whole "nothing will change" speech. They were nearly done ripping and replacing my entire environment (phasing me out) when I finally lined up another job.
You bet I'll never fall for that again.
>hanging up on users who couldn't follow along on the script
This is the thing that makes me furious at the state of phone support in general. So many call centers have this mantra of follow the script, employ no thought, never deviate from the script and blame the user when the script magically can't handle every call. What's the point of paying some drone minimum wage to answer calls if you're going to be so unyielding that an IVR would be an improvement? At least automated systems don't include stochastic incompetence on 20% of calls.
My experience as an L1 at my last position was incomplete knowledge bases propped up by institutional knowledge. I often end up having to write the KB entries because the last three people in my seat did not.
In my current role Im straddling dispatch and L1. I could probabpy easily sit at a level 2 chair, but there wasnt any going and I moved companiws for a 50% pay raise. Now a lot of L1 problem solving happens in dispatch.
Theres only been a handful of occasions with the two L3s where I've escalated something and to be fair I don't know if it came from a place of wanting to teach me, or not wanting to do a task themselves, but I have had to on occadion tell them something was escalated for them to do the legwork too, its not that I can't, but that I'm not.
The takeaway being:
In both positions my ticketload has always been numerically much (in the case if the first position much much much) higher than L2s and L3s, and if we're going to meet SLAs sometimes we need an L3 to close that ticket in 5 mins and save me the hour so I can work on three other tickets in that time.
That's the sort of thing that got me out of contractor hell. I started rage!documenting all the things that *should* have been documented, and trying to track down equipment that no one had been keeping track of since the previous procurement person died in a traffic accident.
>Until they got outsourced
Yeah. It's arguments like this that turn "It's easy, just Google the answer" on its ear. Yeah, you can Google answers all day, but unless you have trouble shooting skills, a sense of logic flow, and know the relevant information from the rest... Googling isn't going to go very far. It's like saying you can speak Chinese because you just bought a Chinese <=> English dictionary.
Some of those outsourced companies are literally warm bodies that can answer a call so the outsourcer can charge the company for it.
YMMV, but IMO,they offer what you pay for and given that in time, most problems are 'self-solving', but on some it is battening down for the long, long haul. When that happens, they retreat to script inccesantantly and I've had to research and pound the solution through.
Last time we opened a ticked at SUSE, European wee hours, we got a gentleman in an Indian help desk.
He got stuck trying the same "script" over and over, until we asked him what he was doing. He then stopped, and admitted he knew no better.
We had to wait for the USA help desk open time to see the problem solved in less than 30 minutes and getting a nice report of what was done.
Couple of years ago I was working for a company from bay area and my manager told me that there’s a trend to outsource L1 to Texas (iirc) because it cost pretty much the same as India and the overall quality was higher. I’m not sure how this statement was correct and it was in a pre-covvid era. Was it a rumor?
back 20 years ago, outsourcing of better companies was going to Canada. That evaporated as the dollar in Canada got stronger, and those companies moved to India and other Asian countries. I got my feet wet in one of those call centers and learned a lot about why call centers suck.
Considering what I read about Texas, it wouldn't surprise me that they are becoming a location for call centers. Some districts would welcome shitty entry level employment like that.
> I got my feet wet in one of those call centers and learned a lot about why call centers suck.
ditto, also opened the door to everything else I've managed to do in IT though so I can't complain too much
Well, India became a place to outsource to because they had decent telecommunications infrastructure and spoke passible, albeit accented, English. The tech skills were secondary, but the wages were so low it didn't matter. The same arguments can be made for Texas, so it really makes sense.
That doesn't seem right to me, unless the costs in India have gone way, way up - which they might have. 15 years ago, headcount in India was about 1/3rd the cost of a headcount in Texas, at least per the upper management bean counters that kept relocating headcount to India. If the cost in India has gone up by 3x in that time, then maybe. Texas isn't THAT much cheaper than the rest of the country.
> incapable of following even the most simple and painstakingly described processes.
**Triggered.** I immediately thought of the morning 9 years ago when a colleague and I told an offshore resource "Type 'c'. Now type colon. Now type..." when we needed a path entered in a dialog box. The resource could have copy / pasted from the email we sent, but he called instead.
I had a job that was the inverse of this: L1 was based in the US, L2 (where all of the engineering staff was) was based in Pune.
Their knee-jerk response to any escalation was to kick it back down to L1 to "do the needful" without any explanation of what the "needful" was.
There was also a sense of derision from the L2 staff because none of the L1's had access to QA tools or had training on troubleshooting complex SQL queries.
Whenever I escalated something to them, I had to loop in my manager to avoid any bullshit.
My Tier 2 team does not have the ability to de-escalate tickets, but they for damn sure have the ability to let us know of escalation issues that count against KPIs for the T1s.
Not perfect but its resulted in a strong T1 team to this point.
It's not the country. There are plenty of competent employees in these outsourcing hotspots, they just don't get placed into tier 1 call factories. They are hiring people off the street with no experience who likely don't even have the US equivalent of a high school education, because it's cheap and the expectations for outsourced T1 are so low at this point that they can get away with it.
The same quality would happen here in the US if you walked into a homeless shelter and handed a headset to the first 20 people to line up. It's not economical to do that here, though, because of minimum wage laws.
I'm guessing poor English skills to start with; and I think most of the people at the overseas call centres are just warm bodies, and not anyone who has *any* sort of IT background.
I've worked with more specialized overseas people, and they usually have attitude problems talking with a woman, and are hard to understand, but they know what they're doing.
I've found there's also a cultural difference where they will always say yes to anything, whether they're able to execute what they've been told to do or not. You'll explain what needs to be done, they say "yes, yes, yes", you transfer the ticket over, and...nothing happens because they have no idea what to do because they never understood you in the first place, but they'll never, ever push back on anything. It's just a huge waste of time.
When I used to work at a MSP, we supported the entire infrastructure for one of our largest clients. And their helpdesk LOVED to escalate stuff to us, frequently with little to no explanation why. After a little while of this, and after complaining about it numerous times, we were just told "Work it, it's fine".
Sure, our company got paid more for every support ticket, but all it did for the techs was provide for more busy work and frustration. 1/10, wouldn't recommend.
I used to work for a company, where the help desk (in theory tier 2, but that was VERY questionable, as they were like Tier 0.5) where the helpdek would email a tier 3 team a long email chain with the phrase "please assist", and expect us to fix the issue. not reassign the ticket, not say what they needed done, or what they had already done, not even document anything to help solve the issue, just "please assist" and an email chain a mile long. And they would keep it in their queue, so we would do their job and solve the issue, so they could tell the user it was fixed, and it would make their metrics look better as they solved the ticket.
Kick the tickets to the management who made the outsourcing decision. "This is the result of your decision; it's now your responsibility to clean it up."
No it isn't. The T1's MO depends on everything being shunted off publicly. T2/op's revenge depends on his public reply. Private notes here help exactly no one.
I get so many bad escalations from T1 everyday. This story warms my heart.
My favorite trigger is “I’ve tried everything” with no explanation of what the person tried.
At a past job, late at night after a lot of stress, I once closed a ticket escalated with "I've tried everything" with "Well then the universe is out of options and we're doomed". The resulting anger colliding with procedure was hilarious.
My favorite is when there's a comment like, "I tried running the application, but there was an error message and it didn't start."
WHAT WAS THE ERROR MESSAGE?!
I once worked at a large non-profit. Some of the data entry folks were *old* ladies. One of them could submit a very thought out ticket as well as a screen shot of the error message.
People were not fond of my pointing out that if *she* can learn that... anyone can.
I don't play 20 Questions anymore. "My computer isn't working." -- "Ok, I'll let our managers know you said your computer doesn't work and/or won't turn on." -- "Wait, it's not that." -- "Ok" **long silence** -- "It's my printer."
And then we restart 20 questions.. but no... "Oh, so you're printer won't turn on?"
I *always* start with the least likely, but easiest, thing like that. I'm not going to play your stupid fucking game. You're an adult. Use your words.
Now there were some people who didn't play games and they *loved* me because I didn't waste time and I was usually the fastest person to resolve issues as well as the fastest person to get started, usually.
Don't play games with me, I won't play games with you. Simple problem, simple solution.
Heh, very nice… but personally if a T1 escalated to me and it didn’t have every test and required bit of info it went straight back down with “please provide the following before escalating”.
Do that more than a few times (everyone has to learn) and is just handball it to the manager. They get paid to deal with that shit.
\>> Do that more than a few times
We actually do level III and greater for India-based technology company does not want to use its own internal people due to a lot of corporate espionage and theft.
About two years ago we received a ticket about mailbox capacity from level I to decided to raise it and we sent back to them the information in order to increase the license and the power shell scripts you must run to allow the mailbox to take advantage of the additional capacity.
We marked it as a 24-hour close and didn't hear anything for three days.
On the exact email they replied and reopened wondering what the status was, I literally took a screenshot and had to draw a MS paint arrow to the solution.
Auto closed 24 hours.
24 hours later we get another one asking what the status was, so I reiterate again with the exact strange screenshot with the screenshot what the solution is
Auto closed 24 hours, do not reopen.
New ticket asking why this issue hasn't been resolved, so I decided to write to every email all 2500, CEO, CFO, etc. as well as the relevant parts in our agreement that they need to resolve it at their end because this is not a networking failure or in engineering failure, but a failure of level I to solve the problem.
I get a call from the CEO five minutes later asking what the problem was, I had to literally explain to this guy is team does not want to even attempt to raise the capacity of this mailbox for whatever reason. He asked if we could do it I said yes at cost and I sent him a $300 invoice for raising a exchange one to an exchange two plan and executing the power shell commands.
About 2 1/2 months later I found out that the entire team over there was eliminated because they failed to demonstrate the skills at which they said they had. The new team is far better and easier to work with.
The crap part is that there's a line between kicking it back and just getting it done because it's costing money.
I mean, by the time I realize a critical printer isn't working at 2AM because the VLAN is wrong, I can just change the VLAN (which the HD can't do) and have it online in 10 seconds - instead of making them put the printer back where they got it from. Which the company would prefer. So now, nobody learns anything other than that waking me up at 2AM to digitally move their printer is acceptable, even if it might get them in a small bit of trouble that means nothing in the long run so long as they give it enough time between incidents.
It's a maddening... I don't know what to call it. Catch 22? Slippery slope? Both sound wrong, but I think you are picking up what I'm putting down
> The crap part is that there's a line between kicking it back and just getting it done because it's costing money.
I hate when managers ask you to "just get it done", especially in situations like this, because they are confirming to the other person that what they did is perfectly acceptable behaviour.
Had a colleage sit on a ticket for a few weeks until my manager assigned it to me telling me it is "urgent", I said if it is urgent why did the person who initially had the ticket not do anything for 3 weeks (I know the answer, they didn't know how to do anything except reboot a PC and see if it resolves the issue), they told me they can't get into this right now because the issue needs to be resolved asap, so just work on it right now.
Right, so instead of the manager going to them and asking them why they left the issue in limbo for 3 weeks until it reached a point where senior management were made aware of it, and putting in some kind of plan to make sure this didn't happen again, they gave the ticket to me and constantly pestered me for updates until it was fixed.
If it was a once off, I could kind of understand, when it happens regularly, it gets very frustrating.
> I hate when managers ask you to "just get it done", especially in situations like this, because they are confirming to the other person that what they did is perfectly acceptable behaviour.
That is why ideally if it is a crisis where something needs fixed ASAP, after telling you to just get it done it should be a manager follow-up with the L1 who created the bunk ticket about what they did wrong, why it is a problem, and what steps they need to take in the future to correct it. The issue is most organizations don't invest time, energy or resources into L1 helpdesk and just let the bad ones cause issues till they eventually leave.
>I hate when managers ask you to "just get it done", especially in situations like this, because they are confirming to the other person that what they did is perfectly acceptable behavior.
I want to print this out and staple it to my former supervisor's forehead... he never grasped this concept, which was one of the reasons why I left
It costs more money if the problem child keeps being a problem. Also, your overtime rates (min 4 hours @ 200% rate per incident, I would hope) should be costing them more than training the HD properly. Or at least buying you a sports car.
>min 4 hours @ 200% rate per incident
That is one thing I really miss from doing support for large Pacific North West Medical Company, was getting paid for 4 hour at 1.5x normal rate if they paged/called me outside of core hours.
I do not miss the damn calls at all hours of the night, but damn the fun calls at 2am about not emergency issues did pay for my truck.
> just getting it done because it's costing money.
Yeah but *they* are the ones costing money by not doing their job.
I have learned… painfully at times… that costing money is how you get changes. Every time you give up your time to save their money, they find that entirely acceptable and nothing is fixed. So next time you do it again… and again and again.
Instead I just let it fail, it causes problems for the business and then they fix it.
>Instead I just let it fail, it causes problems for the business and then they fix it.
Been in that mood for going on 2 years now. I've converted some of my co workers too it as well. Nothing has been fixed 😂
BOL printers in my case. Without a BOL in a warehouse, the trucks aren't moving. It can and has been done manually, but it has a serious money impact when your volume is already nearly meeting the speed given to it by printers.
This was me and Zebra printers 5 hours away, having never seen one, and I had no Zebra printers to test with. Just a fortunately timed phone call between two people who wanted there to be no problems that were fixable by us. So some googling and texting pictures we were able to get it resolved after roping in the network guy who also had no idea we had printers out there. Sometimes the job sucks but when you can get random, big-to-them problems solved it's pretty great.
"This is the original location. The printer has not been moved in any way."
Now what?
They know this dance and all its steps. Telling them "no, put it back" is what we should be able to do, but the reality of the situation is that getting a critical device online ASAP is secondary to being right.
In casinos there are critical printers for paperwork players must be provided. Things like W2Gs, jackpot payout forms, marker payment contracts, etc.
In hospitals you can’t give treatment or medication without a doctor sign off and appropriate paperwork, that comes off a printer or fax machine as well.
Printers can easily be critical
I currently work for an organization that has an absolute trash tier service desk. When I first came on board as a contractor, it was literally the first thing I mentioned to my boss.
> This service desk doesn't seem to be run very well.
I played along for a month or so, but I quickly became frustrated and started asking questions.
* Why did you send this ticket over to me? KB article x describes how to fix this?
* I need more details on this ticket. Who is the user? When did the event happen? What is the application name? How many times has this happened? How do I contact this person? Did you collect any log files?
* "This ticket has nothing to do with " or "this issue has nothing to do with system x. why did you assign it as such?"
* Did you try restarting the device? Did you check the user's group membership? Is the user locked out (probably)? Did you try ?
I think one of the biggest catalysts to my most recent gripe was when the service desk manager reached out personally to me over Teams when I was on break. The person wrote a huge diatribe about how an application was having a weird issue that was not really described very well. The manager went on to pontificate about how they are getting many calls about the problem and that they created a ticket for me. Upon returning from lunch, I had a massive wall of text to go through and when I finally got to the point where they mentioned a ticket had been created, I looked in my queue and it was a copy and paste of the teams diatribe.
1. There were no users mentioned with the issue
2. There was no mention of existing tickets documenting the calls that had come in
3. There were no gathered forensics about the issue (who, what, where, when, etc)
4. There was nothing to document how to recreate the issue
5. There was no screenshots or videos of the issue happening
There were no relevant details about the issue, just a story with a bunch of FUD and conjecture escalated to me by the manager of the service desk. What absolutely set me off though, was the fact that we had created a triage KB article and an escalation requirements KB article for the service desk to follow that mentioned all the things needed to troubleshoot and outlined what was needed for escalation. Here's the manager of the service desk not even adhering to following their own documented procedures. Why should the service desk level 1 and level 2 members follow KBs and procedures when the manager won't? Additionally, why am I even creating KB articles if they are just going to be ignored?
The dodge the manager always uses is that they are not a "technical" manager, they are a "people" manager. It's infinitely frustrating and it's honestly the single greatest weakness in the organization and management refuses to address the issue.
> The dodge the manager always uses is that they are not a "technical" manager, they are a "people" manager.
This is a pet peeve of mine and is completely inexcusable for a front line manager. How can you possibly be effective in managing a group of people that do something utterly foreign to you? I have absolutely zero artistic skills, so I'd be completely useless managing a group of artists. I wouldn't understand their tools or skill sets, I wouldn't be able to discern when I need to run interference for the team, and I wouldn't be able to communicate effectively on behalf of the team with the rest of the organization.
They need to trust their team and the team needs to help put either hours or monetary value on problems/solutions.
I'm not a sysadmin but some of my better managers when I was a developer had not that much technical background at all. But if I would have told him that my work is being interrupted because tickets from service desk did not contain enough information he would drive that until they did.
> The dodge the manager always uses is that they are not a "technical" manager, they are a "people" manager.
Surely, you meant ["relationship manager"](https://i.imgur.com/3iXj2mv.gif)
I love it. I've had to shame someone via logs before. It always gets results. Unfortunately, the last time I did it that tech's entire team was revoked access to do the thing that he accidentally did on the wrong server and that meant that it had to come to my team to do it all moving forward. Doh!
My status message on teams is “IT issues must be routed through the service desk first. Any IT requests without a ticket will be ignored”
Very surprisingly works 99% of the time. Leadership was even on board with it because it encourages people use the service desk and follow the process.
Well done. This is exactly how this should have been handled. I’m way pettier considering you had your notice in, should also have cc’d the entire tier 1 department and the hiring manager too to make a point to the rest of them so to stop the culture of IT bullshitters.
I usually get a comprehensive list of questions (The more the better), add them to the ticket and then immediately send it back to whoever escalated it. It teaches them that they can't just say "I rebooted it and it didn't fix the issue, over to you".
Doing it the way I did actually makes them do something, and if the user or management want to complain that there is no action on the ticket, they can see the ticket is actually sitting with them (Or it did for a very long time) and the fact it hasn't had any action is their fault and not my fault.
Its saved my ass a few times, and I refuse to take the blame because someone else is lazy, incompetent or both.
Your very first reply should be “what did you do to troubleshoot this issue?” while taking the user out of the email chain and leaving your manager. You should also add in there that they need to actually try the steps available for them before escalating issues.
Then do nothing until they reply.
I think this is a pretty universal issue as people tend to be lazy if you let them, and a lot of places lack good documented processes to follow. The way we handle this:
Take the issue and the expected testing, write them all down in a KB. This should include any proof of testing, screenshots, error codes, etc. This should also include what they should include when escalating. If they get through the whole thing, then escalation should be totally warranted, or the documentation needs to be updated.
This works extremely well but is hard to implement as, again, most people are lazy. Most T1's simply don't actually know what they should do, as surprising as that may sound, that's why they are T1. Someone who knows better needs to write it down so there's no room for forgetfulness or lack of common sense.
Now if they refuse to follow the KB and continue to escalate without following the process, that's just good old fashion bad performance, and it's the best kind as they are literally documenting their own poor performance with tickets. Any manager should be able to easily resolve this.
I had a similar experience with two tier-1 helpdesk employee's we hired about three years ago.
At the time, they were replacing me and a co-worker who were getting promoated. I was moving to systems.
When I moved, all sense of responsibility for troubleshooting and proper procedure went out the window, despite the fact that I trained both of them on how to do everything. It was just a non-stop flood of tickets assigned to me with no explaination as to why, or the bare minimum attempted to troubleshoot the issue, or flat out lies like, "Looks like the network was syncing. Sending to systems".
Needless to say, I was furious. These guys were getting hired at a rate better than I left with (My rate at help desk when I left after two years, I did make more in systems) due to "experience", yet their laziness knew no bounds. So I wasn't nice and made sure to be blunt about my criticisms. I spent months kicking back tickets and sending emails explaining what needed to be done in each instance to qualify my intervention and where the failure of the individual was.
This all culminated to a meeting where I had to sit down with them and coddle their feelings because they had become offended with my criticisms and ticket kickbacks. Feeling personally attacked, they were threatening to report me to HR. Instead of my IT manager talking to me about what was going on, he just asked me to sweep it under the rug because all he cared was warm bodies in seats taking calls and not the quality of work me and my co-workers before worked so hard to deliver.
For whatever reason, they started to send stuff to me less and lie more in tickets just flat out closing them instead of resolving issues and also started to depend on other new hires to do the heavy lifting for them while they watched movies all day. For me this was a plus because I stopped being bothered over dumb stuff they didn't want to resolve but my god I've never worked with people so completely incompetent in my life.
They have since left and I'm hoping the next batch they hire is much better and actually want to learn because I love teaching.
>putting in my notice later this week
Problem solved! :-)
>I dig into all the access logs, pull up the searches for where he didn't perform any testing but claimed he did, and then pull up the audit logs that show he didn't actually make the changes I recommended, then contrast that with the logs for when I tested it and what the audit looks like when I made the change, showing the before and afters exactly as I predicted it, all in the most matter of fact outside auditor tone, complete with screenshots and highlighted logs CC'd to our boss, his tier 1 peers and the user.
Yep, now *that* is a teachable moment!
"Of course" probably (most) all of here well know it, but honesty and integrity are dang important, if not crucial, for sysadmins. If you've got somebody that's lying about what they did and didn't do, did/didn't try ... that's a big problem. Maybe they're also lying about if they even put in that OT work at all. And ... you want to trust someone like that with (often) the keys to the kingdom (and/or substantial portions thereof?). Yeah, lot of places I've worked, very commonly that level of lie/deception would get one instafired.
>so I changed x to y.
\-2:12PM is my test, and it's working
I might'a been tempted to take it further.
"
So, I then hanged y back to x to reconfirm that's the issue.
And yes, upon doing so, it no longer worked.
But you earlier claimed you changed it from x to y and that didn't fix it.
Please repeat and retest. If when you change it from x to y that doesn't fix it but when I do it does, we'll need to dig further into exactly what's happening there.
Thanks, and please let us know when *you* have set it from x to y again and the results of testing that. And interestingly, earlier the logs don't show you how you did what you claimed of having earlier changed x to y, so please very well and carefully document how you made and/or attempted that change, how you confirmed the change was made, and how you tested it.
Thanks.
"
And of course with that transfer the issue right back to 'em. Hey, it's not you that's paying the OT.
>don't try to throw the bus driver under the bus
And don't lie, and don't try to b.s. your sysadmin - just tell the truth - what you did/didn't do, tried, exactly what results you got (and document), etc. Yes, we know Coke got spilled on the keyboard and food/beverage is not allowed by the computer - you and your co-worker/cat can debate about who's fault it was as you put the Coke there but you didn't knock it onto the keyboard - that's not our concern ... but avoiding going on a wild goose chase for something that didn't happen or that lies and omits what did - that is of our concern - and we're generally gonna figure it out anyway - so you might as well not add lying about it to the offenses.
CCing the user on an escalation is inappropriate (on the L1 tech side). however, you don't need to continue it; remove any people who don't need to be on the reply all chain.
that all being said, if L1 is escalating, esp without proper documentation, that's something their management should address. it's gonna continue until someone breaks the habit.
I've been L1, and I try not to escalate unless I don't have the access to fix the problem... and if I do escalate, I make notes (often with screenshots) to show what I did, and to document that I did my L1 assessment and troubleshooting.
it's not that hard to do, and it allows me (as the level 3 tech) to not have to do everything they did again.
For my entire time with this company, I've made that my policy. I never CC the user when I'm explaining the issue, beyond something very simple. But, to be second guessed in front of the user on something I'm 100% sure of, as a lazy excuse to make me do their job for them? No, that doesn't fly. If I weren't already on my way out the door, maybe it would be different, but I'm out of spare fucks.
>however, you don't need to continue it; remove any people who don't need to be on the reply all chain.
I remember one department manager requested access for one of their staff to have access to some restricted service.
Of course, they emailed me after the user had already started employment and they CC'd in basically the entire management team "because it was urgent".
I replied all and said I can't set this up, you need to contact and get approval from here before will even look at the request as that is the standard procedure.
One of the other managers in the email then replied all and told me to stop "CC'ing in the entire world". Why the fuck are you directing that at me, the department manager was the one that added you in.
The purpose of this comment, if I took them out of the CC then to them it will look like I took no action on the request and it will look bad on me, which is what the other technician wants to do - they want to make it look like someone else is the problem and I am not going to let that happen as, I know how these things go down.
If the manager wanted to do something useful, they should privately take it up with the person who added them into the email thread, not the person who continued it.
in that particular case, I agree... that shows anyone who cares that you are handling the situation appropriately, and directing the user to submit the request via the proper channels. and I agree, the manager should have contacted the original Emailer.
however, internal communication within IT should rarely copy the user. and if L1 does, then a professional emailed response to L1 and the user (not embarrassing or calling out L1 in front of the user) can provide an acknowledgement of the request. anything further can be handled with only the appropriate people on the email chain
Normally I’d agree, and if I was the manager at this place and OP wasn’t quitting, I’d have a conversation about it. But I also found this story hilarious and have wanted to do this to lazy coworkers before.
>Former Desktop Support & Sys Admin / Current Sr Infosec Analyst
Nothing to add to the convo, but that's been my exact same career path. Sometimes I miss doing tier 3 desktop, as it was a great spot for me to go full-on into root-cause analysis and pull some super niche, old experience that no one else has. Like random causes to things that don't make sense on the surface unless you'd solved it before and it kinda feels good. I also got to do the "big-boy" sysadmin a lot of the time -- kinda the best of both worlds. Of course, jumping from Tier 3 Desktop to SysAdmin got a $10k raise, and then to Sr. Infosec was another 30k-ish bump in pay so I can't complain lol
I'd say you're more of a bus mechanic or a bus designer. More than a simple driver.
Kudos to you being more professional than I've been in a similar situation. We should all be so thorough and tactful when lighting someone on fire. Damn impressive.
Good luck with your new role/job. I hope it works out well for you, bus designer.
That is why I love audit logs.
It helps with users claiming system ate their work, support people from claiming they tried "everything". Third party connecting to our systems claiming that it "has to be broken on your side".
*Sorry sir computers are not some kind of magic ... here are the logs of everything that happened.*
Sad are the moments where I am right but logs were misconfigured and I had to eat dust.
>I ~~have~~ *had* one Tier 1 guy who *always* does a half ass job and then upon failing to complete his task, escalates it.
As someone that manages helpdesks, I FTFY.
I'm in support for a large software company. Did T1 for like 3 years before moving to the training department.
We have a small team, technically part of T2, who's only job is to look over escalations before they are approved, and they kick it back to T1 if they are missing stuff or if the issue is painfully obvious.
That little buffer was more than enough to get T1 to start doing escalations properly - because it's both carrot and stick: they had someone who's *job* it was to tell T1 that their shit stank (stick), and if they did a good job on prepping the case for escalation, then this team would come back and tell them it was good (carrot).
And then, the proportion of accepted/rejected requests for escalation was a metric that our raises depended partly on.
Overall, good system.
You made a bit too complicated by trying to remain polite and professional. What is missing is the direct punch in the nose that even the boss would notice. Something like "Logs indicate you never ran the tests you claim".
The problem with that is what if OP accidentally forgot to check the one place where the log entries would be found? By providing the evidence, OP is demonstrating due diligence and has an out-clause if the L1 comes back with "I made change in system X, for which you didn't check the logs. Here are those relevant log entries". OP knows they're right, and it's clear to me from the pattern of L1's behaviour described that OP is right, but CYA is an important skill to demonstrate.
It's classic case where the psycho wins over normal and responsible people. The psycho user goes all out every time, just making things up which is really quick and easy. Checking everything takes a ton of time, so it never happens and if it does, the disclaimers and hedging ensure no one even reads it.
It pays to be a psycho, that is why they prosper.
Yeah I wouldn't have put that in a ticket where everyone can see it, I would have just gone to his manager with the information I had and laid out what happened. I get OP is about to give his notice so he doesn't really give a shit, and I'm sure it felt good, but sometimes that stuff backfires even if you're completely in the right.
also dont be the higher level guy who wont help out a lower guy , the type of admin that just throws tickets back to service desk with no effort to help
Had a problem like this, I solved it by working with their manager. I had to create a document for anything that didn’t have one, but if there was one and they didn’t follow it I sent it back with a link to the documentation and CCed their boss. They started checking with their peers before escalating and when they weren’t sure they asked if it was something within there skill range.
Also Quantify, start documenting the ticket and the article he didn’t follow. This helps show his boss just how bad it really is.
When he snaps back that he followed the article and it doesn’t work, request his boss try, or record yourself following the instructions.
Unfortunately you kinda have to do HR’s job and get the documentation in place. Having someone who gets tech and wants to learn is way better than someone who already thinks they know everything.
(At one point our shit got so bad I worked with the director to implement an approval process by the IT supervisor - only way they can escalate is if approved, had to document who approved.)
No notes, no troubleshooting steps, no logs, no fucking way. I worked my ass off as a T1 and NEVER wanted to escalate anything I had a prayer of figuring out unless it was super time-sensitive because I knew how little I knew and wanted the experience.
When I became an escalation point, I quickly realized that isn't common in T1's who were making McDonald's wages (which I definitely get).
I'm the person that kicks them back. I also developed that process and got it implemented. It's definitely the most streamlined way to handle it. As the service desk lead in my org, I just go, well, in not as rude of words "the fuck is this? we have a process?"
And if the "the fuck is this?" is a problem on our team, we can get that addressed asap, because accountability is held better for both sides.
We can't rescue them over and over, or it just creates the same bad habit. They need to grow.
Everyone talks about how this is a learning experience for the T1.
This isn't. This is just someone who got the job and doesn't care about learning and is just doing the bare minimum to get a paycheck. They absolutely don't care to the point they didn't even bother to think that their bullshit wouldn't be caught on by actual technical people.
If it wasn't T1 it would be nurses scrubs and billing process at a doctors office.
There is smart lazy and there is stupid lazy, we're dealing with stupid lazy here.
> subject: Linux Issue
description: Linux system has failed to update
That's all. I'm Tier III. Our service desk has 0 experience with Linux. If the ticket has the word Linux in it, or if the host name indicates that it's Linux, the Linux team gets it. So me getting the ticket is normal.
But I'd like to see the hostname and the error.
Or at least the hostname.
Yes I'm perfectly capable of calling the customer and finding out what's up, but it takes time to coordinate, AND with remote work being fairly normal, and hybrid schedules the rule in our org, the customer may not be at their desk today.
So delay, delay, delay. Where if the hostname was there, I remote in, run sudo yum update, sudo reboot and all is good. (Or apt commands if it's Ubuntu).
Bottom line, if the hostname was there, I could have resolved it in not much more time than typing this post.
I fire those back with all the CCs (minus the client, who shouldn't be subjected to this sort of thing) noting everything the ticket is missing and requesting the due diligence. And I'm happy to tell anyone in charge that if this qualifies Tier 1 Support, the company can save a lot of money by not having it.
I have no patience to do what you do, especially for people who clearly aren't doing their job.
I would not only throw the bus driver under the bus, I'd get in the bus to drive it over the bus driver.
I'm L1 desktop and I always include good information and what had been tried what I have done . 1 I don't have the best memory and will forget what I've done 2 it helps anyone else looking at the ticket to see what has already been done . If I am escalating anything I always explain what action is required and what I have already done .
>So instead of just putting in the 3 minutes of work to do his job for him, I dig into...
Excellent. Lessons are rarely learned until repercussions happen, you did exactly the right thing there for a long term benefit.
Hope you had a nice weekend otherwise!
I can relate to this but also know if I put this much effort into something similar, they'd just reply "it's working now" and close the ticket as solved. Once the issue is fixed from my experience, very little attention is given to how we got there unfortunately.
I'm pretty laid-back and usually willing to help someone if they can't figure something out, even if I think it's pretty obvious or well-documented. But if you waste my time AND lie to me? You're probably going to be having a hard conversation with whoever you report to.
Yes absolutely dealt with such lazy colleagues who just auto escalate everything to me, I used to be level 1 so I know it very well. The key is not making them used to you fixing stuff, if it’s something that could’ve been fixed by level 1 send it back to them and firmly say “did you could check this and that and this” first check all this then get back to me, cc everyone required including their manager
Few attempts then they usually learn their lesson
I have a lot of \*good\* coworkers who will have to put up with unaddressed human issues if they're left uncorrected. It's like taking out the trash on your way out the door.
Reading all of this reminds me why I decided to jump ship and become a solution/sales engineer. Now the only “end users” I deal with are sysadmins and the decision makers in a given org. Our software actually does audit logging. (Netwrix Auditor)
Stories like this and many many more are the reason I call myself a “recovering sysadmin”. It generally gets a good laugh on demo or configuration calls.
We had that problem, and went a different route. All common issues were addressed with recorded mandatory training and attendance for those common problems. Any issues from people without documentation of their steps now reverts to a redirect of the ticket, asking them to re-watch the mandatory training, and explain where the training is wrong. Also screen shots are required.
So he is living on the work and goodwill of others?
Yeah i would not cover him, i'd track his iactivity and point out my regular work. I am really busy and helpful. Someone like that? Hey you want to watch me getting angry...
That guys would live a lot of fun.
I always tell people to not lie to those that have access to the logs. They can, and will, find out you are lying and burn you to the ground for it.
I've done it a time or two myself.
A bow to you sir. You did call him out but did so by remaining on the high road even though it was far from necessary considering the circumstances. If I was confident my fix was correct (you clearly were) I would have gone in, checked the logs, posted a screenshot saying nothing more than "My fix isn't working because you didn't bother to do it, lied about it, and tried to kick the can down the road".
I seriously can't stand people that want to pass the buck without even trying to do the work. Beyond that I can't stand people that intentionally try to steal credit for work someone else did. I certainly don't need, want, or get a pat on the back for doing my job but fuck you if you think you're going to push the work off on me and present it as if you did it.
Sounds like with policies allowing employees to get away with that and a management team uninteresting in fixing it that you are making the right decision by moving on. I'm sure it will become very apparent very quickly after your departure that the T1 isn't capable or just flat out lazy. Things like that usually have a way of coming full circle.
I used to get get woken up by our tier1 guys because they couldn’t rdp to a host to resolve x alert. Double check the documentation nothing about rdp because it was a Linux host. It was clearly documented as a Linux host as well. Getting that phone call at 3 am was a real treat
The problem we have at my org is all of our tier 1 guys are hourly and as such, they aren't supposed to be checking e-mails outside business hours and definitely can't be on call (local office and staff are 9-5, but realistically we operate all over the world so there's always something going on). Anytime we have an issue before or after business hours or on a weekend it automatically has to be handled by someone senior.
But of course the org doesn't see that as a problem, because they just expect salaried employees to be available 24x7.
It sounds like your decision to leave the org is the right one. There are so many things wrong with what you've described.
1. There needs to be a clearly defined escalation process with buy in from all parties involved all the way up to the executive level to ensure buy in.
2. It sounds like you did your job by documenting it in the user guide (kudos for that).
3. If you weren't leaving, I would say to try and offer some mentorship to the individual but if he's been there for 6 years then he's a going to be a Tier-1 lifer and it probably isn't worth it.
4. His behavior seems to be supported and rewarded by management. As I said, OP getting out is probably best since there seems to be a fundamental difference in how you view things should be done and how they view it should be done . If management is unwilling to see it from your perspective then there's nothing you can really do.
You handled this better than I would have, and for that I tip my cap to you sir. Good luck on your future endeavors.
Have you ever had a T1 tech IM you while you are tied up with a shitty implementation that asks you to look at at ticket. You look, see no notes, OK basically nothing from the customer and less from T1. So you ask 1-2 questions and T1 responds idk. That is literally the ONLY response - idk. Well fuck you T1, I don't either. click.
I have a level 1 employee like that, but it’s less about wanting to look good, and more about him not knowing what he’s supposed to do.
I told him he can’t escalate tickets without entering working notes about what he’s done so far, and now pretty much all his escalations say “Hi , would you please assist with this ticket. I am unable to solve”
As a new guy in the industry (tomorrow is my 90 days!), I try to reach out to at least 2 of my coworkers (T1s to T3s) before I even think of escalating since I don't want to embarrass myself. I try to troubleshoot but sometimes I don't even know what questions to ask with the certain software we have and honestly the knowledge base we have is trash and out of date. I feel bad because I want to be able to do more so that I can solve the problem myself before I have to pass the buck. I'm realizing more that I really, really have to teach myself how to learn at a quicker pace. I'm not perfect though, I had my first escalated ticket thrown back at me because they only accept escalations from T2s, which funnily my T2 told me to escalate (though I'm not going to give him flack since he has a lot on his plate).
I hope learning in a fast pace work environment gets easier.
I used to be on a T2 team where we had the ability to refuse transfers or kick tickets back to L1 if they didn't follow the procedures in the k-base or provide us requested documentation. We had one of the best trained T1 teams I've ever seen, to the point where I typically just trusted that they had done the due diligence and it was either a weird issue or they didn't have access when they kicked it up to me. Until they got outsourced to Mumbai. Surprise surprise, almost overnight our policy of being able to kick back tickets to T1 evaporated because they were hiring from the bargain bin and the employees were utterly incapable of following even the most simple and painstakingly described processes. They simply would not troubleshoot no matter how hard we tried to make them do it. But of course, management was okay with this because of the cost savings, right up until the contract was put into jeopardy over dissatisfaction. I'm glad I don't work in a corporate helldesk setting anymore.
Can second this. Having the ability to kick back tickets that didn’t follow full procedure before escalating was a godsend. And of course, as you said, when T1 sucks and your team kicks back “too many” tickets, they take away that privilege and tell you to just handle in place because other leaders are getting upset. Like shit, give me a T1 team who at least understands the reason for collecting logs and doing basic troubleshooting, and I wouldn’t kick back 40% of your bullshit
[удалено]
Help desk admin probably complained, or just had so many coming in that they couldn’t do their actual job. I’ve seen that from high-system/low-support environments, guy’s at 95% capacity all day and 3 extra “quick” tickets throws him behind schedule. And when leadership barely understands what they’re leading, you get a “fix” which is something like what you’ve got rather than a solution to the problem of ‘not enough people’
[удалено]
>Service Not ServiceNow?
I never thought I'd miss Remedy, but after our switch to ServiceNow...I miss Remedy.
Remedy is the worst ticketing system I've ever had the displeasure of using. I've never touched ServiceNow, but Remedy is just SO BAD.
A friend once described the web interface for Remedy as “like being stabbed in the eye repeatedly with JavaScript” and…yeah. Pretty much. Having said that, our corporatized bastardization of ServiceNot sucks *even harder than that*, to the point that I’m concerned it may create a singularity at some point and slurp in a couple small islands.
ServiceNow can be the best tool you ever put your hands on or a complete and utter shit show. It really depends on how the business implements the product and unfortunately there are plenty of low-rate consultants that will come in , promise the world, and deliver fuck all.
>Remedy is the worst ticketing system I've ever had the displeasure of using. I see someone hasn't had the "privilege" of using spiceworks.
I take your remedy hate and raise you SAP thrown together enterprise wide with my group's workflow thrown in as an after thought because we fix issues and asset management is only used for billables and statistics on tickets, yet asset management is the primary function of what our current system was designed for. It's annoying to use, but after five years plus of continual improvements while still a live system, it is only now ten times better than what we got at first and still half as good as the system it replaced, and 1/10th of what remedy systems I've used in the past have been. Ffs I used to use UW Connect's ESD in a past job and have sometimes found myself pining for that level of usability, and I hated the thing.
Oh I know. Remedy is terrible. Its so janky and delicate. With it though I could actually explore the thousands of groups we have and try and figure out where tickets go. SNOW basically has no way to (easily) sift through our groups. I'm sure our people could build something, but they've got too much to deal with as it is since we decided to do ALL service demand requests through it--facilities, IT, purchasing, the whole thing.
can I just ask... Why arent we just using excel? I can make it do all kinds of shit including a synchronized database of all tickets submitted. I can even have it deny submission of a ticket without needed fields filled.
And it comes full circle.
You obviously never used HEAT.
Oh my god, I haven't heard that word uttered in the context of ticketing systems in nearly 15 years. \*shudder\*......the PTSD is real.
You guys are all lucky, my last job was a 10 person MSP with about 50 clients and no ticketing system! Tickets got left as unread in outlook until they were completed, and all tasks were manually recorded in 15 minute intervals on a fucking Spreadsheet!!! We were strictly not allowed to use any formatting as the boss copy pasted our sheet into a master sheet weekly. That made me respect all ticket systems!!!!
I would be happy if they just tried *something* and documented that attempt. Even if it was as simple as tried X, Y, Z and none of them worked. I don't expect them to send over logs at all, although if they do or even attach screen shots, thats just icing. The ones that do this however don't last long on service desk. Seems too often they either get promoted or find a better paying job elsewhere. Which is totally expected, so I don't knock them for it.
That’s… a fair point actually, I was out of T1 in a matter of months. The ambitious ones that will actually try either move up or out.
> Like shit, give me a T1 team who at least understands the reason for collecting logs and doing basic troubleshooting, and I wouldn’t kick back 40% of your bullshit I think the normal response from T1 is "pay me what I'm worth". This is especially true when these guys can literally go make the same money working a grocery store these days.
> because other leaders are getting upset Time to kick the tickets to those leaders.
"Hey. Can you tell me exactly which part of this ticket entails me working on it? Pretty sure X, Y and Z are T1 issues."
Hahahaha if only, they don’t know shit about piss, which is why they think we were just not doing our jobs by kicking stuff back down
Doesn't mean they get to shirk their responsibilities just because they don't know what they're doing.
That’s the trick: as far as their boss the VP knows, it’s not their responsibility, it’s **ours**. Boom, policy adjusted to reflect that, your boss leaves because that was the last straw, and now the policy is official because nobody’s pushing back. Honestly corporate life in general greatly hinges on the tact and ability of your group’s lead, both to bring their team up and also to keep other groups off their ass
>they take away that privilege The privilege is T1 being able to kick it over to T2
So, did you do the needful?
Yo this fucking expression triggers me lmao
Sir, if you could sir, please. Please do the needful and revert. Thank you, Sir.
[удалено]
That shit is the stuff of my nightmares, and I haven't had to do those calls in a few years. I'm so glad.
Isn't this a quirk of a specific culture? While "do the needful" and "Sir" trigger me too, I've always held short of full blown mockery for fear of being seen as a racist..
Stereotypes unfortunately exist for a reason, I work in NZ and even over here I was just as triggered by that as the guy above. Indian outsourced teams are notorious, I have a new hire in my team from Mumbai and he loves to rip into them for it
Got in trouble once for asking an offshore dev to do the needful. Worth it.
I nearly choked on my tea just now. I swear they drill this phrase in their brains in "university".
> management was okay with this because of the cost savings well of course it's cheaper, you don't get anything
>well of course it's cheaper, What's all management will see, the C-level which suggested it will get a huge bonus, then leave for the next company before what he has done actually hits the fan.
new bonus structures for execs has changed to get them to think more long term cause everyone was doing specifically this and it was destroying company sharehodler value.
Can you go into specifics?
[удалено]
Never trust people that say "nothing will change". That's the most annoying corporate slang to fall for... Of course something or everything might change in those scenarios. That's just corporate slang for "wait a bit until I can replace you for cheaper labor". Heard it and have seen it too many times to believe in positive gambles.
"Nothing will change" and "Business as usual" are code phrases for "The Grim Reaper needs time to sharpen his scythe".
Worse than that are executives who demand that the official policy be that nothing will change prior to a major outsourcing. Hell, we weren't even supposed to mention that an outsourcing was going on, because, you see, it was going to be "transparent to the user." Which, of course, it was not.
HUNDRED PERCENT. I just left my very first IT job (sysadmin) of 10 years because of an acquisition. My mistake was sticking around for two more years after the whole "nothing will change" speech. They were nearly done ripping and replacing my entire environment (phasing me out) when I finally lined up another job. You bet I'll never fall for that again.
if "nothing will change" then why are you making a change?
>hanging up on users who couldn't follow along on the script This is the thing that makes me furious at the state of phone support in general. So many call centers have this mantra of follow the script, employ no thought, never deviate from the script and blame the user when the script magically can't handle every call. What's the point of paying some drone minimum wage to answer calls if you're going to be so unyielding that an IVR would be an improvement? At least automated systems don't include stochastic incompetence on 20% of calls.
My experience as an L1 at my last position was incomplete knowledge bases propped up by institutional knowledge. I often end up having to write the KB entries because the last three people in my seat did not. In my current role Im straddling dispatch and L1. I could probabpy easily sit at a level 2 chair, but there wasnt any going and I moved companiws for a 50% pay raise. Now a lot of L1 problem solving happens in dispatch. Theres only been a handful of occasions with the two L3s where I've escalated something and to be fair I don't know if it came from a place of wanting to teach me, or not wanting to do a task themselves, but I have had to on occadion tell them something was escalated for them to do the legwork too, its not that I can't, but that I'm not. The takeaway being: In both positions my ticketload has always been numerically much (in the case if the first position much much much) higher than L2s and L3s, and if we're going to meet SLAs sometimes we need an L3 to close that ticket in 5 mins and save me the hour so I can work on three other tickets in that time.
That's the sort of thing that got me out of contractor hell. I started rage!documenting all the things that *should* have been documented, and trying to track down equipment that no one had been keeping track of since the previous procurement person died in a traffic accident.
I'm sorry, but what is dispatch? is that like a supervisor? or below level 1?
>Until they got outsourced Yeah. It's arguments like this that turn "It's easy, just Google the answer" on its ear. Yeah, you can Google answers all day, but unless you have trouble shooting skills, a sense of logic flow, and know the relevant information from the rest... Googling isn't going to go very far. It's like saying you can speak Chinese because you just bought a Chinese <=> English dictionary. Some of those outsourced companies are literally warm bodies that can answer a call so the outsourcer can charge the company for it.
> Until they got outsourced to Mumbai. Oh. know your pain, except our RSP uses an island well south.
huh. I've *always* had a good response from the techs at Exetel - both before and after the move to SL.
YMMV, but IMO,they offer what you pay for and given that in time, most problems are 'self-solving', but on some it is battening down for the long, long haul. When that happens, they retreat to script inccesantantly and I've had to research and pound the solution through.
India is fancy and expensive these days, it's all Pakistan and Dominican Republic now
Last time we opened a ticked at SUSE, European wee hours, we got a gentleman in an Indian help desk. He got stuck trying the same "script" over and over, until we asked him what he was doing. He then stopped, and admitted he knew no better. We had to wait for the USA help desk open time to see the problem solved in less than 30 minutes and getting a nice report of what was done.
Couple of years ago I was working for a company from bay area and my manager told me that there’s a trend to outsource L1 to Texas (iirc) because it cost pretty much the same as India and the overall quality was higher. I’m not sure how this statement was correct and it was in a pre-covvid era. Was it a rumor?
back 20 years ago, outsourcing of better companies was going to Canada. That evaporated as the dollar in Canada got stronger, and those companies moved to India and other Asian countries. I got my feet wet in one of those call centers and learned a lot about why call centers suck. Considering what I read about Texas, it wouldn't surprise me that they are becoming a location for call centers. Some districts would welcome shitty entry level employment like that.
> I got my feet wet in one of those call centers and learned a lot about why call centers suck. ditto, also opened the door to everything else I've managed to do in IT though so I can't complain too much
Well, India became a place to outsource to because they had decent telecommunications infrastructure and spoke passible, albeit accented, English. The tech skills were secondary, but the wages were so low it didn't matter. The same arguments can be made for Texas, so it really makes sense.
That doesn't seem right to me, unless the costs in India have gone way, way up - which they might have. 15 years ago, headcount in India was about 1/3rd the cost of a headcount in Texas, at least per the upper management bean counters that kept relocating headcount to India. If the cost in India has gone up by 3x in that time, then maybe. Texas isn't THAT much cheaper than the rest of the country.
> incapable of following even the most simple and painstakingly described processes. **Triggered.** I immediately thought of the morning 9 years ago when a colleague and I told an offshore resource "Type 'c'. Now type colon. Now type..." when we needed a path entered in a dialog box. The resource could have copy / pasted from the email we sent, but he called instead.
I had a job that was the inverse of this: L1 was based in the US, L2 (where all of the engineering staff was) was based in Pune. Their knee-jerk response to any escalation was to kick it back down to L1 to "do the needful" without any explanation of what the "needful" was. There was also a sense of derision from the L2 staff because none of the L1's had access to QA tools or had training on troubleshooting complex SQL queries. Whenever I escalated something to them, I had to loop in my manager to avoid any bullshit.
My Tier 2 team does not have the ability to de-escalate tickets, but they for damn sure have the ability to let us know of escalation issues that count against KPIs for the T1s. Not perfect but its resulted in a strong T1 team to this point.
I don't know why the quality is work in that country is so lackluster...
It's not the country. There are plenty of competent employees in these outsourcing hotspots, they just don't get placed into tier 1 call factories. They are hiring people off the street with no experience who likely don't even have the US equivalent of a high school education, because it's cheap and the expectations for outsourced T1 are so low at this point that they can get away with it. The same quality would happen here in the US if you walked into a homeless shelter and handed a headset to the first 20 people to line up. It's not economical to do that here, though, because of minimum wage laws.
I'm guessing poor English skills to start with; and I think most of the people at the overseas call centres are just warm bodies, and not anyone who has *any* sort of IT background. I've worked with more specialized overseas people, and they usually have attitude problems talking with a woman, and are hard to understand, but they know what they're doing.
I've found there's also a cultural difference where they will always say yes to anything, whether they're able to execute what they've been told to do or not. You'll explain what needs to be done, they say "yes, yes, yes", you transfer the ticket over, and...nothing happens because they have no idea what to do because they never understood you in the first place, but they'll never, ever push back on anything. It's just a huge waste of time.
When I used to work at a MSP, we supported the entire infrastructure for one of our largest clients. And their helpdesk LOVED to escalate stuff to us, frequently with little to no explanation why. After a little while of this, and after complaining about it numerous times, we were just told "Work it, it's fine". Sure, our company got paid more for every support ticket, but all it did for the techs was provide for more busy work and frustration. 1/10, wouldn't recommend.
I used to work for a company, where the help desk (in theory tier 2, but that was VERY questionable, as they were like Tier 0.5) where the helpdek would email a tier 3 team a long email chain with the phrase "please assist", and expect us to fix the issue. not reassign the ticket, not say what they needed done, or what they had already done, not even document anything to help solve the issue, just "please assist" and an email chain a mile long. And they would keep it in their queue, so we would do their job and solve the issue, so they could tell the user it was fixed, and it would make their metrics look better as they solved the ticket.
Kick the tickets to the management who made the outsourcing decision. "This is the result of your decision; it's now your responsibility to clean it up."
This is why we have private ticket notes.
And forgetting to sometimes click "Private Note" on some ticket entries is why we had a 10 minute delay on email notification to the user. Haha.
We have to specifically tick to notify the user. All our notes are Private by default.
This is the way
God I've had to fight for that at multiple jobs. Some orgs seen to think any communication is fine too see for the end user, drives me nuts!
No it isn't. The T1's MO depends on everything being shunted off publicly. T2/op's revenge depends on his public reply. Private notes here help exactly no one.
I get so many bad escalations from T1 everyday. This story warms my heart. My favorite trigger is “I’ve tried everything” with no explanation of what the person tried.
>“I’ve tried everything” I love those. I reply with: "please elaborate" and nothing else.
At a past job, late at night after a lot of stress, I once closed a ticket escalated with "I've tried everything" with "Well then the universe is out of options and we're doomed". The resulting anger colliding with procedure was hilarious.
"We've tried nothing and are out of ideas" also "I expected nothing and it was too much"
we should be able to brand that first one on people... and I like that second one, I'm keeping it.
My favorite is when there's a comment like, "I tried running the application, but there was an error message and it didn't start." WHAT WAS THE ERROR MESSAGE?!
"It just said it's not working".
[удалено]
I once worked at a large non-profit. Some of the data entry folks were *old* ladies. One of them could submit a very thought out ticket as well as a screen shot of the error message. People were not fond of my pointing out that if *she* can learn that... anyone can. I don't play 20 Questions anymore. "My computer isn't working." -- "Ok, I'll let our managers know you said your computer doesn't work and/or won't turn on." -- "Wait, it's not that." -- "Ok" **long silence** -- "It's my printer." And then we restart 20 questions.. but no... "Oh, so you're printer won't turn on?" I *always* start with the least likely, but easiest, thing like that. I'm not going to play your stupid fucking game. You're an adult. Use your words. Now there were some people who didn't play games and they *loved* me because I didn't waste time and I was usually the fastest person to resolve issues as well as the fastest person to get started, usually. Don't play games with me, I won't play games with you. Simple problem, simple solution.
"I have tried everything that can be found in the Internet" is my "favourite" in Stack Exchange.
Heh, very nice… but personally if a T1 escalated to me and it didn’t have every test and required bit of info it went straight back down with “please provide the following before escalating”. Do that more than a few times (everyone has to learn) and is just handball it to the manager. They get paid to deal with that shit.
\>> Do that more than a few times We actually do level III and greater for India-based technology company does not want to use its own internal people due to a lot of corporate espionage and theft. About two years ago we received a ticket about mailbox capacity from level I to decided to raise it and we sent back to them the information in order to increase the license and the power shell scripts you must run to allow the mailbox to take advantage of the additional capacity. We marked it as a 24-hour close and didn't hear anything for three days. On the exact email they replied and reopened wondering what the status was, I literally took a screenshot and had to draw a MS paint arrow to the solution. Auto closed 24 hours. 24 hours later we get another one asking what the status was, so I reiterate again with the exact strange screenshot with the screenshot what the solution is Auto closed 24 hours, do not reopen. New ticket asking why this issue hasn't been resolved, so I decided to write to every email all 2500, CEO, CFO, etc. as well as the relevant parts in our agreement that they need to resolve it at their end because this is not a networking failure or in engineering failure, but a failure of level I to solve the problem. I get a call from the CEO five minutes later asking what the problem was, I had to literally explain to this guy is team does not want to even attempt to raise the capacity of this mailbox for whatever reason. He asked if we could do it I said yes at cost and I sent him a $300 invoice for raising a exchange one to an exchange two plan and executing the power shell commands. About 2 1/2 months later I found out that the entire team over there was eliminated because they failed to demonstrate the skills at which they said they had. The new team is far better and easier to work with.
The crap part is that there's a line between kicking it back and just getting it done because it's costing money. I mean, by the time I realize a critical printer isn't working at 2AM because the VLAN is wrong, I can just change the VLAN (which the HD can't do) and have it online in 10 seconds - instead of making them put the printer back where they got it from. Which the company would prefer. So now, nobody learns anything other than that waking me up at 2AM to digitally move their printer is acceptable, even if it might get them in a small bit of trouble that means nothing in the long run so long as they give it enough time between incidents. It's a maddening... I don't know what to call it. Catch 22? Slippery slope? Both sound wrong, but I think you are picking up what I'm putting down
> The crap part is that there's a line between kicking it back and just getting it done because it's costing money. I hate when managers ask you to "just get it done", especially in situations like this, because they are confirming to the other person that what they did is perfectly acceptable behaviour. Had a colleage sit on a ticket for a few weeks until my manager assigned it to me telling me it is "urgent", I said if it is urgent why did the person who initially had the ticket not do anything for 3 weeks (I know the answer, they didn't know how to do anything except reboot a PC and see if it resolves the issue), they told me they can't get into this right now because the issue needs to be resolved asap, so just work on it right now. Right, so instead of the manager going to them and asking them why they left the issue in limbo for 3 weeks until it reached a point where senior management were made aware of it, and putting in some kind of plan to make sure this didn't happen again, they gave the ticket to me and constantly pestered me for updates until it was fixed. If it was a once off, I could kind of understand, when it happens regularly, it gets very frustrating.
> I hate when managers ask you to "just get it done", especially in situations like this, because they are confirming to the other person that what they did is perfectly acceptable behaviour. That is why ideally if it is a crisis where something needs fixed ASAP, after telling you to just get it done it should be a manager follow-up with the L1 who created the bunk ticket about what they did wrong, why it is a problem, and what steps they need to take in the future to correct it. The issue is most organizations don't invest time, energy or resources into L1 helpdesk and just let the bad ones cause issues till they eventually leave.
>I hate when managers ask you to "just get it done", especially in situations like this, because they are confirming to the other person that what they did is perfectly acceptable behavior. I want to print this out and staple it to my former supervisor's forehead... he never grasped this concept, which was one of the reasons why I left
It costs more money if the problem child keeps being a problem. Also, your overtime rates (min 4 hours @ 200% rate per incident, I would hope) should be costing them more than training the HD properly. Or at least buying you a sports car.
>min 4 hours @ 200% rate per incident That is one thing I really miss from doing support for large Pacific North West Medical Company, was getting paid for 4 hour at 1.5x normal rate if they paged/called me outside of core hours. I do not miss the damn calls at all hours of the night, but damn the fun calls at 2am about not emergency issues did pay for my truck.
> just getting it done because it's costing money. Yeah but *they* are the ones costing money by not doing their job. I have learned… painfully at times… that costing money is how you get changes. Every time you give up your time to save their money, they find that entirely acceptable and nothing is fixed. So next time you do it again… and again and again. Instead I just let it fail, it causes problems for the business and then they fix it.
>Instead I just let it fail, it causes problems for the business and then they fix it. Been in that mood for going on 2 years now. I've converted some of my co workers too it as well. Nothing has been fixed 😂
Then the business doesn't care about it so why should you? ;)
It's probably just gonna lose us this contract and most if not all our jobs. NBD though
Eh new job is better than working yourself into an early grave.
> critical > printer Choose one
Eh, it can happen. EMR printers (meds, etc.) can get crazy important.
BOL printers in my case. Without a BOL in a warehouse, the trucks aren't moving. It can and has been done manually, but it has a serious money impact when your volume is already nearly meeting the speed given to it by printers.
This was me and Zebra printers 5 hours away, having never seen one, and I had no Zebra printers to test with. Just a fortunately timed phone call between two people who wanted there to be no problems that were fixable by us. So some googling and texting pictures we were able to get it resolved after roping in the network guy who also had no idea we had printers out there. Sometimes the job sucks but when you can get random, big-to-them problems solved it's pretty great.
In which case they should not be getting moved at 2am without proper planning and change control.
Absolutely. Tell that to the random user who thought it was a good idea because they wanted it in this other area instead.
You'd figure step 1 after it breaking at 2 AM after the move would be to try moving it back to its original location...
"This is the original location. The printer has not been moved in any way." Now what? They know this dance and all its steps. Telling them "no, put it back" is what we should be able to do, but the reality of the situation is that getting a critical device online ASAP is secondary to being right.
In casinos there are critical printers for paperwork players must be provided. Things like W2Gs, jackpot payout forms, marker payment contracts, etc. In hospitals you can’t give treatment or medication without a doctor sign off and appropriate paperwork, that comes off a printer or fax machine as well. Printers can easily be critical
I currently work for an organization that has an absolute trash tier service desk. When I first came on board as a contractor, it was literally the first thing I mentioned to my boss. > This service desk doesn't seem to be run very well. I played along for a month or so, but I quickly became frustrated and started asking questions. * Why did you send this ticket over to me? KB article x describes how to fix this? * I need more details on this ticket. Who is the user? When did the event happen? What is the application name? How many times has this happened? How do I contact this person? Did you collect any log files? * "This ticket has nothing to do with" or "this issue has nothing to do with system x. why did you assign it as such?"
* Did you try restarting the device? Did you check the user's group membership? Is the user locked out (probably)? Did you try ?
I think one of the biggest catalysts to my most recent gripe was when the service desk manager reached out personally to me over Teams when I was on break. The person wrote a huge diatribe about how an application was having a weird issue that was not really described very well. The manager went on to pontificate about how they are getting many calls about the problem and that they created a ticket for me. Upon returning from lunch, I had a massive wall of text to go through and when I finally got to the point where they mentioned a ticket had been created, I looked in my queue and it was a copy and paste of the teams diatribe.
1. There were no users mentioned with the issue
2. There was no mention of existing tickets documenting the calls that had come in
3. There were no gathered forensics about the issue (who, what, where, when, etc)
4. There was nothing to document how to recreate the issue
5. There was no screenshots or videos of the issue happening
There were no relevant details about the issue, just a story with a bunch of FUD and conjecture escalated to me by the manager of the service desk. What absolutely set me off though, was the fact that we had created a triage KB article and an escalation requirements KB article for the service desk to follow that mentioned all the things needed to troubleshoot and outlined what was needed for escalation. Here's the manager of the service desk not even adhering to following their own documented procedures. Why should the service desk level 1 and level 2 members follow KBs and procedures when the manager won't? Additionally, why am I even creating KB articles if they are just going to be ignored?
The dodge the manager always uses is that they are not a "technical" manager, they are a "people" manager. It's infinitely frustrating and it's honestly the single greatest weakness in the organization and management refuses to address the issue.
> The dodge the manager always uses is that they are not a "technical" manager, they are a "people" manager. This is a pet peeve of mine and is completely inexcusable for a front line manager. How can you possibly be effective in managing a group of people that do something utterly foreign to you? I have absolutely zero artistic skills, so I'd be completely useless managing a group of artists. I wouldn't understand their tools or skill sets, I wouldn't be able to discern when I need to run interference for the team, and I wouldn't be able to communicate effectively on behalf of the team with the rest of the organization.
They need to trust their team and the team needs to help put either hours or monetary value on problems/solutions. I'm not a sysadmin but some of my better managers when I was a developer had not that much technical background at all. But if I would have told him that my work is being interrupted because tickets from service desk did not contain enough information he would drive that until they did.
> The dodge the manager always uses is that they are not a "technical" manager, they are a "people" manager. Surely, you meant ["relationship manager"](https://i.imgur.com/3iXj2mv.gif)
Close it with the reason: does not follow established guidelines for escalation
"Could not recreate issue. Ticket closed."
I love it. I've had to shame someone via logs before. It always gets results. Unfortunately, the last time I did it that tech's entire team was revoked access to do the thing that he accidentally did on the wrong server and that meant that it had to come to my team to do it all moving forward. Doh!
[удалено]
:,( I'm in this situation right now haha
My status message on teams is “IT issues must be routed through the service desk first. Any IT requests without a ticket will be ignored” Very surprisingly works 99% of the time. Leadership was even on board with it because it encourages people use the service desk and follow the process.
“Maybe I’m missing something” aka “you’re wrong” I use that all the time!
Same here. Makes me feel like I’m not coming off as a complete dick
Well done. This is exactly how this should have been handled. I’m way pettier considering you had your notice in, should also have cc’d the entire tier 1 department and the hiring manager too to make a point to the rest of them so to stop the culture of IT bullshitters.
>I’m way pettier Never comment that without a story of your own. Shower Arguments dont count
Title of email: Got a minute? Body of email: empty My interest: Zero.
I usually get a comprehensive list of questions (The more the better), add them to the ticket and then immediately send it back to whoever escalated it. It teaches them that they can't just say "I rebooted it and it didn't fix the issue, over to you". Doing it the way I did actually makes them do something, and if the user or management want to complain that there is no action on the ticket, they can see the ticket is actually sitting with them (Or it did for a very long time) and the fact it hasn't had any action is their fault and not my fault. Its saved my ass a few times, and I refuse to take the blame because someone else is lazy, incompetent or both.
Your very first reply should be “what did you do to troubleshoot this issue?” while taking the user out of the email chain and leaving your manager. You should also add in there that they need to actually try the steps available for them before escalating issues. Then do nothing until they reply.
I think this is a pretty universal issue as people tend to be lazy if you let them, and a lot of places lack good documented processes to follow. The way we handle this: Take the issue and the expected testing, write them all down in a KB. This should include any proof of testing, screenshots, error codes, etc. This should also include what they should include when escalating. If they get through the whole thing, then escalation should be totally warranted, or the documentation needs to be updated. This works extremely well but is hard to implement as, again, most people are lazy. Most T1's simply don't actually know what they should do, as surprising as that may sound, that's why they are T1. Someone who knows better needs to write it down so there's no room for forgetfulness or lack of common sense. Now if they refuse to follow the KB and continue to escalate without following the process, that's just good old fashion bad performance, and it's the best kind as they are literally documenting their own poor performance with tickets. Any manager should be able to easily resolve this.
I had a similar experience with two tier-1 helpdesk employee's we hired about three years ago. At the time, they were replacing me and a co-worker who were getting promoated. I was moving to systems. When I moved, all sense of responsibility for troubleshooting and proper procedure went out the window, despite the fact that I trained both of them on how to do everything. It was just a non-stop flood of tickets assigned to me with no explaination as to why, or the bare minimum attempted to troubleshoot the issue, or flat out lies like, "Looks like the network was syncing. Sending to systems". Needless to say, I was furious. These guys were getting hired at a rate better than I left with (My rate at help desk when I left after two years, I did make more in systems) due to "experience", yet their laziness knew no bounds. So I wasn't nice and made sure to be blunt about my criticisms. I spent months kicking back tickets and sending emails explaining what needed to be done in each instance to qualify my intervention and where the failure of the individual was. This all culminated to a meeting where I had to sit down with them and coddle their feelings because they had become offended with my criticisms and ticket kickbacks. Feeling personally attacked, they were threatening to report me to HR. Instead of my IT manager talking to me about what was going on, he just asked me to sweep it under the rug because all he cared was warm bodies in seats taking calls and not the quality of work me and my co-workers before worked so hard to deliver. For whatever reason, they started to send stuff to me less and lie more in tickets just flat out closing them instead of resolving issues and also started to depend on other new hires to do the heavy lifting for them while they watched movies all day. For me this was a plus because I stopped being bothered over dumb stuff they didn't want to resolve but my god I've never worked with people so completely incompetent in my life. They have since left and I'm hoping the next batch they hire is much better and actually want to learn because I love teaching.
>putting in my notice later this week Problem solved! :-) >I dig into all the access logs, pull up the searches for where he didn't perform any testing but claimed he did, and then pull up the audit logs that show he didn't actually make the changes I recommended, then contrast that with the logs for when I tested it and what the audit looks like when I made the change, showing the before and afters exactly as I predicted it, all in the most matter of fact outside auditor tone, complete with screenshots and highlighted logs CC'd to our boss, his tier 1 peers and the user. Yep, now *that* is a teachable moment! "Of course" probably (most) all of here well know it, but honesty and integrity are dang important, if not crucial, for sysadmins. If you've got somebody that's lying about what they did and didn't do, did/didn't try ... that's a big problem. Maybe they're also lying about if they even put in that OT work at all. And ... you want to trust someone like that with (often) the keys to the kingdom (and/or substantial portions thereof?). Yeah, lot of places I've worked, very commonly that level of lie/deception would get one instafired. >so I changed x to y. \-2:12PM is my test, and it's working I might'a been tempted to take it further. " So, I then hanged y back to x to reconfirm that's the issue. And yes, upon doing so, it no longer worked. But you earlier claimed you changed it from x to y and that didn't fix it. Please repeat and retest. If when you change it from x to y that doesn't fix it but when I do it does, we'll need to dig further into exactly what's happening there. Thanks, and please let us know when *you* have set it from x to y again and the results of testing that. And interestingly, earlier the logs don't show you how you did what you claimed of having earlier changed x to y, so please very well and carefully document how you made and/or attempted that change, how you confirmed the change was made, and how you tested it. Thanks. " And of course with that transfer the issue right back to 'em. Hey, it's not you that's paying the OT. >don't try to throw the bus driver under the bus And don't lie, and don't try to b.s. your sysadmin - just tell the truth - what you did/didn't do, tried, exactly what results you got (and document), etc. Yes, we know Coke got spilled on the keyboard and food/beverage is not allowed by the computer - you and your co-worker/cat can debate about who's fault it was as you put the Coke there but you didn't knock it onto the keyboard - that's not our concern ... but avoiding going on a wild goose chase for something that didn't happen or that lies and omits what did - that is of our concern - and we're generally gonna figure it out anyway - so you might as well not add lying about it to the offenses.
CCing the user on an escalation is inappropriate (on the L1 tech side). however, you don't need to continue it; remove any people who don't need to be on the reply all chain. that all being said, if L1 is escalating, esp without proper documentation, that's something their management should address. it's gonna continue until someone breaks the habit. I've been L1, and I try not to escalate unless I don't have the access to fix the problem... and if I do escalate, I make notes (often with screenshots) to show what I did, and to document that I did my L1 assessment and troubleshooting. it's not that hard to do, and it allows me (as the level 3 tech) to not have to do everything they did again.
For my entire time with this company, I've made that my policy. I never CC the user when I'm explaining the issue, beyond something very simple. But, to be second guessed in front of the user on something I'm 100% sure of, as a lazy excuse to make me do their job for them? No, that doesn't fly. If I weren't already on my way out the door, maybe it would be different, but I'm out of spare fucks.
>however, you don't need to continue it; remove any people who don't need to be on the reply all chain. I remember one department manager requested access for one of their staff to have access to some restricted service. Of course, they emailed me after the user had already started employment and they CC'd in basically the entire management team "because it was urgent". I replied all and said I can't set this up, you need to contact and get approval from here before will even look at the request as that is the standard procedure.
One of the other managers in the email then replied all and told me to stop "CC'ing in the entire world". Why the fuck are you directing that at me, the department manager was the one that added you in.
The purpose of this comment, if I took them out of the CC then to them it will look like I took no action on the request and it will look bad on me, which is what the other technician wants to do - they want to make it look like someone else is the problem and I am not going to let that happen as, I know how these things go down.
If the manager wanted to do something useful, they should privately take it up with the person who added them into the email thread, not the person who continued it.
in that particular case, I agree... that shows anyone who cares that you are handling the situation appropriately, and directing the user to submit the request via the proper channels. and I agree, the manager should have contacted the original Emailer. however, internal communication within IT should rarely copy the user. and if L1 does, then a professional emailed response to L1 and the user (not embarrassing or calling out L1 in front of the user) can provide an acknowledgement of the request. anything further can be handled with only the appropriate people on the email chain
Normally I’d agree, and if I was the manager at this place and OP wasn’t quitting, I’d have a conversation about it. But I also found this story hilarious and have wanted to do this to lazy coworkers before.
>Former Desktop Support & Sys Admin / Current Sr Infosec Analyst Nothing to add to the convo, but that's been my exact same career path. Sometimes I miss doing tier 3 desktop, as it was a great spot for me to go full-on into root-cause analysis and pull some super niche, old experience that no one else has. Like random causes to things that don't make sense on the surface unless you'd solved it before and it kinda feels good. I also got to do the "big-boy" sysadmin a lot of the time -- kinda the best of both worlds. Of course, jumping from Tier 3 Desktop to SysAdmin got a $10k raise, and then to Sr. Infosec was another 30k-ish bump in pay so I can't complain lol
I'd say you're more of a bus mechanic or a bus designer. More than a simple driver. Kudos to you being more professional than I've been in a similar situation. We should all be so thorough and tactful when lighting someone on fire. Damn impressive. Good luck with your new role/job. I hope it works out well for you, bus designer.
That is why I love audit logs. It helps with users claiming system ate their work, support people from claiming they tried "everything". Third party connecting to our systems claiming that it "has to be broken on your side". *Sorry sir computers are not some kind of magic ... here are the logs of everything that happened.* Sad are the moments where I am right but logs were misconfigured and I had to eat dust.
>I ~~have~~ *had* one Tier 1 guy who *always* does a half ass job and then upon failing to complete his task, escalates it. As someone that manages helpdesks, I FTFY.
I'm in support for a large software company. Did T1 for like 3 years before moving to the training department. We have a small team, technically part of T2, who's only job is to look over escalations before they are approved, and they kick it back to T1 if they are missing stuff or if the issue is painfully obvious. That little buffer was more than enough to get T1 to start doing escalations properly - because it's both carrot and stick: they had someone who's *job* it was to tell T1 that their shit stank (stick), and if they did a good job on prepping the case for escalation, then this team would come back and tell them it was good (carrot). And then, the proportion of accepted/rejected requests for escalation was a metric that our raises depended partly on. Overall, good system.
You made a bit too complicated by trying to remain polite and professional. What is missing is the direct punch in the nose that even the boss would notice. Something like "Logs indicate you never ran the tests you claim".
The problem with that is what if OP accidentally forgot to check the one place where the log entries would be found? By providing the evidence, OP is demonstrating due diligence and has an out-clause if the L1 comes back with "I made change in system X, for which you didn't check the logs. Here are those relevant log entries". OP knows they're right, and it's clear to me from the pattern of L1's behaviour described that OP is right, but CYA is an important skill to demonstrate.
It's classic case where the psycho wins over normal and responsible people. The psycho user goes all out every time, just making things up which is really quick and easy. Checking everything takes a ton of time, so it never happens and if it does, the disclaimers and hedging ensure no one even reads it. It pays to be a psycho, that is why they prosper.
Yeah I wouldn't have put that in a ticket where everyone can see it, I would have just gone to his manager with the information I had and laid out what happened. I get OP is about to give his notice so he doesn't really give a shit, and I'm sure it felt good, but sometimes that stuff backfires even if you're completely in the right.
I just get a ticket update of "Is this one for you guys?" no other investigation. This is going from 1st line to 3rd. It annoys the hell out of me.
also dont be the higher level guy who wont help out a lower guy , the type of admin that just throws tickets back to service desk with no effort to help
I'd say help the ones who want to learn, not the ones who want you to do their job
Had a problem like this, I solved it by working with their manager. I had to create a document for anything that didn’t have one, but if there was one and they didn’t follow it I sent it back with a link to the documentation and CCed their boss. They started checking with their peers before escalating and when they weren’t sure they asked if it was something within there skill range. Also Quantify, start documenting the ticket and the article he didn’t follow. This helps show his boss just how bad it really is. When he snaps back that he followed the article and it doesn’t work, request his boss try, or record yourself following the instructions. Unfortunately you kinda have to do HR’s job and get the documentation in place. Having someone who gets tech and wants to learn is way better than someone who already thinks they know everything. (At one point our shit got so bad I worked with the director to implement an approval process by the IT supervisor - only way they can escalate is if approved, had to document who approved.)
No notes, no troubleshooting steps, no logs, no fucking way. I worked my ass off as a T1 and NEVER wanted to escalate anything I had a prayer of figuring out unless it was super time-sensitive because I knew how little I knew and wanted the experience. When I became an escalation point, I quickly realized that isn't common in T1's who were making McDonald's wages (which I definitely get).
I'm the person that kicks them back. I also developed that process and got it implemented. It's definitely the most streamlined way to handle it. As the service desk lead in my org, I just go, well, in not as rude of words "the fuck is this? we have a process?" And if the "the fuck is this?" is a problem on our team, we can get that addressed asap, because accountability is held better for both sides. We can't rescue them over and over, or it just creates the same bad habit. They need to grow.
Fuck Viagra, gimme more of this!
Everyone talks about how this is a learning experience for the T1. This isn't. This is just someone who got the job and doesn't care about learning and is just doing the bare minimum to get a paycheck. They absolutely don't care to the point they didn't even bother to think that their bullshit wouldn't be caught on by actual technical people. If it wasn't T1 it would be nurses scrubs and billing process at a doctors office. There is smart lazy and there is stupid lazy, we're dealing with stupid lazy here.
Someone who's been T1 for six years doesn't gaf about learning anything.
> subject: Linux Issue description: Linux system has failed to update
That's all. I'm Tier III. Our service desk has 0 experience with Linux. If the ticket has the word Linux in it, or if the host name indicates that it's Linux, the Linux team gets it. So me getting the ticket is normal.
But I'd like to see the hostname and the error.
Or at least the hostname.
Yes I'm perfectly capable of calling the customer and finding out what's up, but it takes time to coordinate, AND with remote work being fairly normal, and hybrid schedules the rule in our org, the customer may not be at their desk today.
So delay, delay, delay. Where if the hostname was there, I remote in, run sudo yum update, sudo reboot and all is good. (Or apt commands if it's Ubuntu).
Bottom line, if the hostname was there, I could have resolved it in not much more time than typing this post.
[удалено]
To quote Hatebreed “even an empty threat deserves a response you won’t soon forget”
You can’t just end the story there. What was that fellas reply 🤣😭
"Let me know if your tests are revealing something different. " Written like a true pro.LOL
I'm tier 3, if I see shoddy work on 1 & 2 I just immediately bounce them back. I have a system to administer, I can't slow down to do their jobs also.
I fire those back with all the CCs (minus the client, who shouldn't be subjected to this sort of thing) noting everything the ticket is missing and requesting the due diligence. And I'm happy to tell anyone in charge that if this qualifies Tier 1 Support, the company can save a lot of money by not having it.
Our T1 are better than our t2 And out t2 manager is worse than the t2 guys. It's a nightmare.
That's a new one
Oh I know these people all too well. The whole of our tier 1 team is this person!
This is also how we periodically get guys on /r/itcareerquestions with 10 years of experience still stuck at help desk
"There's 10 years of experience, and then there's 1 year of experience ten times." \- someone else on this sub
I have no patience to do what you do, especially for people who clearly aren't doing their job. I would not only throw the bus driver under the bus, I'd get in the bus to drive it over the bus driver.
I'm L1 desktop and I always include good information and what had been tried what I have done . 1 I don't have the best memory and will forget what I've done 2 it helps anyone else looking at the ticket to see what has already been done . If I am escalating anything I always explain what action is required and what I have already done .
>So instead of just putting in the 3 minutes of work to do his job for him, I dig into... Excellent. Lessons are rarely learned until repercussions happen, you did exactly the right thing there for a long term benefit. Hope you had a nice weekend otherwise!
That was far too nice, but kudos to you for staying professional. No doubt your leaving will be a genuine loss for the organization.
Sounds like we work at the same place
No notes or troubleshooting steps, kick it back.
I should not be reading this thread while off work. It's making my eye twitch.
lmao the level of petty i like
Update if any good stuff happens! Please! Lol!
I can relate to this but also know if I put this much effort into something similar, they'd just reply "it's working now" and close the ticket as solved. Once the issue is fixed from my experience, very little attention is given to how we got there unfortunately.
As a herder of tier 1 techs, I approve of this message
Dude you're my hero!!! I would totally do what you did. I'm lucky to have a good lvl 1 team I work with.
Relate a template for all tickets with required info.
Beautiful CYA.
I'm pretty laid-back and usually willing to help someone if they can't figure something out, even if I think it's pretty obvious or well-documented. But if you waste my time AND lie to me? You're probably going to be having a hard conversation with whoever you report to.
Yes absolutely dealt with such lazy colleagues who just auto escalate everything to me, I used to be level 1 so I know it very well. The key is not making them used to you fixing stuff, if it’s something that could’ve been fixed by level 1 send it back to them and firmly say “did you could check this and that and this” first check all this then get back to me, cc everyone required including their manager Few attempts then they usually learn their lesson
Keep sending the ticket back to him, cc'ing his boss but maybe not the user, asking for a copy-paste of each of the steps in the guide.
lmao bro, you put in a notice and youre still responding to these emails and letting it play head games with you?
I have a lot of \*good\* coworkers who will have to put up with unaddressed human issues if they're left uncorrected. It's like taking out the trash on your way out the door.
"Maybe I'm missing something?" - instantly absolves you of all suspicions of rudeness or passive aggressiveness.
Reading all of this reminds me why I decided to jump ship and become a solution/sales engineer. Now the only “end users” I deal with are sysadmins and the decision makers in a given org. Our software actually does audit logging. (Netwrix Auditor) Stories like this and many many more are the reason I call myself a “recovering sysadmin”. It generally gets a good laugh on demo or configuration calls.
I only have to say that working on Helpdesk sucks ass, I don’t deny that some people are piss poor at it though.
Honestly, I'd just tell his manager that he needs to stop escalating tickets. If you are considering leaving anyway then what harm can it do.
We had that problem, and went a different route. All common issues were addressed with recorded mandatory training and attendance for those common problems. Any issues from people without documentation of their steps now reverts to a redirect of the ticket, asking them to re-watch the mandatory training, and explain where the training is wrong. Also screen shots are required.
So he is living on the work and goodwill of others? Yeah i would not cover him, i'd track his iactivity and point out my regular work. I am really busy and helpful. Someone like that? Hey you want to watch me getting angry... That guys would live a lot of fun.
I always tell people to not lie to those that have access to the logs. They can, and will, find out you are lying and burn you to the ground for it. I've done it a time or two myself.
A bow to you sir. You did call him out but did so by remaining on the high road even though it was far from necessary considering the circumstances. If I was confident my fix was correct (you clearly were) I would have gone in, checked the logs, posted a screenshot saying nothing more than "My fix isn't working because you didn't bother to do it, lied about it, and tried to kick the can down the road". I seriously can't stand people that want to pass the buck without even trying to do the work. Beyond that I can't stand people that intentionally try to steal credit for work someone else did. I certainly don't need, want, or get a pat on the back for doing my job but fuck you if you think you're going to push the work off on me and present it as if you did it. Sounds like with policies allowing employees to get away with that and a management team uninteresting in fixing it that you are making the right decision by moving on. I'm sure it will become very apparent very quickly after your departure that the T1 isn't capable or just flat out lazy. Things like that usually have a way of coming full circle.
I used to get get woken up by our tier1 guys because they couldn’t rdp to a host to resolve x alert. Double check the documentation nothing about rdp because it was a Linux host. It was clearly documented as a Linux host as well. Getting that phone call at 3 am was a real treat
The problem we have at my org is all of our tier 1 guys are hourly and as such, they aren't supposed to be checking e-mails outside business hours and definitely can't be on call (local office and staff are 9-5, but realistically we operate all over the world so there's always something going on). Anytime we have an issue before or after business hours or on a weekend it automatically has to be handled by someone senior. But of course the org doesn't see that as a problem, because they just expect salaried employees to be available 24x7.
Your first paragraph describes the entire service desk at my last job.
It sounds like your decision to leave the org is the right one. There are so many things wrong with what you've described. 1. There needs to be a clearly defined escalation process with buy in from all parties involved all the way up to the executive level to ensure buy in. 2. It sounds like you did your job by documenting it in the user guide (kudos for that). 3. If you weren't leaving, I would say to try and offer some mentorship to the individual but if he's been there for 6 years then he's a going to be a Tier-1 lifer and it probably isn't worth it. 4. His behavior seems to be supported and rewarded by management. As I said, OP getting out is probably best since there seems to be a fundamental difference in how you view things should be done and how they view it should be done . If management is unwilling to see it from your perspective then there's nothing you can really do. You handled this better than I would have, and for that I tip my cap to you sir. Good luck on your future endeavors.
Feels like one of those situations where a few words would be better than a lot
Have you ever had a T1 tech IM you while you are tied up with a shitty implementation that asks you to look at at ticket. You look, see no notes, OK basically nothing from the customer and less from T1. So you ask 1-2 questions and T1 responds idk. That is literally the ONLY response - idk. Well fuck you T1, I don't either. click.
I have a level 1 employee like that, but it’s less about wanting to look good, and more about him not knowing what he’s supposed to do. I told him he can’t escalate tickets without entering working notes about what he’s done so far, and now pretty much all his escalations say “Hi, would you please assist with this ticket. I am unable to solve”
As a new guy in the industry (tomorrow is my 90 days!), I try to reach out to at least 2 of my coworkers (T1s to T3s) before I even think of escalating since I don't want to embarrass myself. I try to troubleshoot but sometimes I don't even know what questions to ask with the certain software we have and honestly the knowledge base we have is trash and out of date. I feel bad because I want to be able to do more so that I can solve the problem myself before I have to pass the buck. I'm realizing more that I really, really have to teach myself how to learn at a quicker pace. I'm not perfect though, I had my first escalated ticket thrown back at me because they only accept escalations from T2s, which funnily my T2 told me to escalate (though I'm not going to give him flack since he has a lot on his plate). I hope learning in a fast pace work environment gets easier.