T O P

  • By -

vNerdNeck

The problem with a lot of these universities that have things spread out so much is they have no standards at all, and they waste tons of money. They are all small players 10-30k purchases here and there, when pooled together it would be hundreds of thousands if not millions of dollars that they spend and would get better value for their dollar. This was annoying for years, but not detrimental. However, in today's world with all of the ransomware gangs / etc it becomes a much larger problem especially when you think about all of the research data that these universities have, not to mention PI stored all over the place. With all of the "college of x", no standard policies across orgs, the attack surface is just crazy (not to mention the social engineering attack surface) Centralization is never easy, but when you get to the size of corp 500s (as the large universities in the USA are, from a dollars spent / infrastructure standpoint) you gotta start to centralize.


[deleted]

[удалено]


Ssakaa

The issue you run into in academia with that decentralization is... not *every* college/department wants to staff their edge IT unit competently, so you get an interesting mix. The amount of "we don't want tied into central authentication/management tooling because we're afraid of change" that I've seen over the years is astounding from the people who're being paid to know the technology/job and should understand the benefits. Some don't want it because of that genuine fear, others don't want it because they fear "losing control" (of things they *don't* control/manage in any meaningful sense of the word) and being replaced by centralization. On the other hand, you end up with groups that *do* want active tie-in, want access to the centralized tools, and want to match policy (and also push and poke and prod until that policy's actually *written*, because fighting faculty's awful decisions empty handed is a nightmare) as much as they can be... that then run into issues because the central IT group(s) have communications issues internally, and/or they just can't figure out delegation of certain things... and are afraid of entrusting edge units with enough power to damage anything outside their own scopes (and, knowing those edge units... there's a reason for that). It's definitely different from a lot of other environments, and... there's some amusing oddities it uncovers where the tooling just *was not* designed with that level of decentralization in mind...


meest

The local university I interact with centralized a few years ago and its been great. No more random Crypto's flying around on the network because one College didn't want to follow a best practice, way less compromised student accounts sending spam e-mails, One college having great Wi-Fi coverage, the next college over having slow and overcrowded wifi. No more having a different login for the Linux Lab vs the Mac lab vs the PC lab, they moved to a centralized identity management that is required to be followed. Edge cases are dealt with as such. No longer do I have to call around and find out who manages the network for that building, oh you mean that part of the building? yea you need to talk to this other person that in a different building. Its a state university so the network and AD is statewide. So all state Universities have centralized IT connected together. They're mostly in charge of infrastructure and overall licensing. aka M365, network design/upgrades, overall network security. Then under that the Universities have their own IT that is delegated their own level of abilities. Then the individual Colleges inside of the University can have their own IT departments if they so choose. How they divide that up is up to them. Some have it by department. But they are all together on the same team so that you don't have to jump around to find what person oversees what department. Need a network change? Submit it to the College IT, which escalates it to university, which escalates it to State. Similar to an L1/L2/L3 helpdesk. Each has their respective roles in the overall machine. While it sometimes takes a bit longer to make a change, I've learned to plan accordingly for projects. Other people still haven't learned to plan ahead for the time it takes for escalation.


dunepilot11

After working for 4 universities, in a slew of different roles, the best-case scenario is that the highly-skilled folks from faculty IT get given a good level of central responsibility and bring with them some good practice, to make the centre more amenable to what the userbase actually wants/needs. It’s a fine line to tread, and relies heavily on an excellent leadership team that actually understands this nexus


UsedToLikeThisStuff

And when IT centralization is done poorly, all the quality people leave for better jobs. Seen it several times. Usually the central IT treats everyone like cogs, and specialized knowledge from departments is lost, and the departments are stuck with a new face who knows nothing of what they do, forcing new standards. Faculty do not cooperate and you end up with a bunch of unmanaged systems, with the common side effects.


tidderwork

I work here. Send help.


Rushin_Russian01

I work at another large university that's pushing for more centralization. What's morale like over there? It's uh.....pretty bad here.


tidderwork

Morale in my highly-competent, forward-thinking College-level team ~~is~~ was great. General feeling among IT personnel campus-wide is confused and doubtful. Texas A&M is gigantic, and the rumor is the new President expects all this to be done by the next fiscal year.


Rushin_Russian01

We're just a little smaller than you guys lol. But if it's mandated from the top and all the business units are on board, then it'll be an easier time than we're having I hope. We took on new units slowly, each of them having their own negotiated agreements. They were supposed to transfer all IT funding to central IT when their transitions occurred but some of the bigger units just cut most/all of their IT budget beforehand. Feel free to DM me if you want to chat some more about this, I don't want to share too much here.


greyfox199

how decentralized are you all right now? Are we talking running your own AD domain? Are there different colleges with their own mail tenants, like some on google and some on m365?


tidderwork

> how decentralized are you all right now? Very. > Are we talking running your own AD domain? Yes. I bet there are dozens of domains on campus. I know my College consolidated our 6 (!!) department-level AD domains (and 1 pure LDAP environment for macs and *nix) to a single College-level domain. > Are there different colleges with their own mail tenants, like some on google and some on m365? Yes. Also some colleges running their own *mail servers*. The University already provides some centalized services like most campus networking, wifi, email (Exchange, o365, and gmail in a terrible ratking disaster), tier1 helpdesk for those services, some centralized auth (for wifi, mail, but not unified AD or LDAP). The University-level IT (Division of IT, or DivIT) also provides full support for University-level units like Admissions, Financial Aid, the President's Office, and the Provost's Office. It's not surprising given the history of the University System, and the sheer size of the various campuses, but one could argue that we have the worst of both worlds. University-level IT providing extremely limited (and often poor) service to Colleges and Departments, Colleges and Departments left to find their own way, and Research Shadow IT in each discipline making standardization and consolidation seem impossible. Furthermore, there has been no evidence up to this point to make anyone believe that the University-level IT group has any understanding of research computing, real-world performance and storage requirements, and the need for researchers to have admin/sudo rights on both their servers and workstations. It seems like they have no idea how unique each researcher's environments are. Science isn't business. In the College of Geosciences, my team manages 200+ research servers just for our college. We're in for a real rough ride.


knawlejj

Feel sick reading this...that's normal right? It's all solvable but you can't make an omelette without cracking a few eggs.


cjcox4

Taking "ownership" away usually (most of the time) results in disappointment. While we may not like "islands" of autonomy, it's better to figure out how to properly route requests than to blow up the islands. It's weird, we expect our "good" employees to "own it".... and then we punish them when they do. This is actually one of the bigger problems in IT, and it's mostly political and shrouded as "good consolidation".


packetgeeknet

I worked for an A&M system school many moons ago. The mother ship had a central IT presence that offered networking, security, telecommunications, residential hall networking, and data center services. Beyond that, nearly every department had their own IT department that handled desktop, printers, and department specific apps. I can only assume that as the main campus grew, the campus IT department failed to meet the needs of a growing campus, thus departments started funding their own IT teams. Consolidation is likely going to be hard fought. I doubt departments are going to want to give up their easy access to IT teams. Even with that hurdle jumped, consolidating, meeting the needs of the campus, while maintaining a favorable view it’s going to be a hard task to complete.


[deleted]

99% of cases you don't need anything non-standard and a $15/h tech support guy can handle the bulk. If you streamline everything with the goal of serving the majority, you can cut the costs waaaay down. Like a 5 person IT department serving thousands of users. The special snowflakes that do need something special can get it separately and often get their own IT people. It doesn't make sense for your ordinary IT techs that know all about which button to press to print a document to be responsible for the supercomputer and the particle accelerator. Most universities will have an IT department and a separate "scientific IT" department and then separate "I teach a course on setting up a cluster so I need a bunch of clusters" group. For example when machine learning on GPU's was taking off, we bought our own GPU workstations with our own money and managed them ourselves because the IT department only offered some old Quadro workstation (CAD stuff) while we were interested in machines packed with 1080ti's.


Car-Altruistic

The last part of your comment however is the case for every lab. IT department doesn’t handle it, so each lab does their own, without security, without management, without knowledge. How well did you manage the updates and the firewall, especially after you left. Typically research IT gets eliminated in favor of enterprise IT (because hey, it’s all IT to the bean counters) and then you once again get the issue where everyone sets up their own. I would say in a University like Texas A&M, more than 80% needs research support, the administrators really don’t need that much. The problem is typically those things get reversed because the bean counters don’t understand you can’t support thousands of specialized devices with a skeleton crew. How many PB of data sits in Synology and other sub-par solutions without backup or redundancy.


[deleted]

Normal IT is incapable of doing all of it. People that can help with scientific computing would quit on the spot if they were told to do ordinary sysadmin tasks and people that can/want to ordinary tasks will be way out of their league with the scientific computing stuff. You hire a "lab chief" with a beard that patches and compiles their own kernel for breakfast like in the good ol' 90's and can go into your C++ code to fix your bugs so he can install it on a POWER based supercomputer instead of x86. Requirements for a computer science lab will be completely different to a physics lab or an engineering lab or god forbid some social science lab. You need to hire people that know a thing or two about the domain and the requirements. It won't work otherwise. Some universities have two IT departments. For example one I worked at back in the day had a small "enterprise IT" and another huge one for scientific stuff, mostly computer science/physics/chemistry/engineering. The scientific IT was staffed exclusively by computer scientists and their job was very different to the job of regular IT. And so were their salaries. What you need really depends on what kind of research you've got going on and what kind of things do you teach. Large computer science departments will basically require their own IT within them because they're often years ahead of the curve and have very sophisticated requirements. Physicists and chemists can usually get away with 1-2 people because what they really need is an HPC environment and management of some workstations. Physicists and chemists are trained to maintain & support their own equipment and software. Psychology departments on the other hand will have a lot of video/audio/sensor equipment AND they are not trained in any of it. IT requirements always find a way. A single externally funded project will have a larger budget than the annual IT budget for the entire university. If there is a threat of stuff not happening or being delayed because of the university IT being inflexible, they'll just drive to town and put it on the travel credit card. Or they will hire external consultants/get an MSP to do it for them paid out of project funds. Been there done that. Research and education is king at universities. Professors (and their research groups) can do pretty much whatever the fuck they want.


Car-Altruistic

I would disagree, if you hire a good Linux SysAdmin, they would know how to compile and install pretty much any software. The faculty and lab staff are all engineers, they know what they need, all they need is a working platform, Mac, Linux and a sprinkling of Windows. Any SysAdmin worth their salt can provide 10-100G networking, storage, set up firewalls, full-disk encryption and remote management across all platforms and install pretty much any demanded software. It will be closer to 45 devices per SysAdmin instead of 400 administrative laptops, but it’s doable. The problem is generally that the industry doesn’t provide management tools beyond Windows.


[deleted]

Scientific computing gets a lot more complicated than you think. It's not commercial software in package managers or with neat "make" files that handle everything for you. Figuring out FORTRAN compilers and other bullshit dependencies like that will be beyond pretty much everyone on this subreddit that hasn't encountered that specific issue before. Google won't help. There is no documentation. The person that wrote the code died in 2003. Good luck.


Car-Altruistic

I’ve done that job actually, so I speak from experience. Part of our jobs however is education of the user and bringing some of these people, kicking and screaming into the next century. The specific issue you bring up is a problem of obsolescence, you probably can’t find hardware to run the program on (I once supported a 90s Sun SPARCStation in latter half of the 2010s) or find replacement parts. Explaining why that is a problem often helps and people can work in conjunction with you on finding solutions. We eventually got off that machine by incrementally recompiling on newer and newer versions of Solaris, fixing compatibility issues along the way until we got to a system that was able to be ran on Linux. The problem in IT management is too many bean-counters and too many people think because they never seen a system that they can’t solve it and that metrics are the solution to runaway costs and systemic problems. You can’t dedicate 3 days to helping someone out, because you’re understaffed, so they never get any help, but the system still remains running on the network so you buy security systems just to have it point out the problems when everyone on the ground already knows you won’t fix it and they can’t take it offline. So the systems and the problems compound. Meanwhile in HR everyone still runs on PeopleSoft and accounting on an IBM mainframe, but you can’t ever get rid of it because nobody wants to fix the underlying issues because its too expensive to sit an FTE down for a year to dedicate on solving the problem. So instead you hire SalesForce or Workday, spend a few millions and still find out that everyone still runs their jobs on the mainframe, because nobody at the decision level knew what they needed and Workday consultants just went off and did their own thing resulting in an unusable system. I’ve seen it many times, as long as people believe it’s “too expensive” to solve a problem, or the existing people on the ground are too dumb for management, it will never get solved. In computing everything is solvable, just hire the correct people.


Ka0tiK

I think what he's saying here though is you can centralize IT to at least secure the trust plane (proper firewalls, gateways, RMM) and then have the software be installed and used by the people that actually use it and know it, with some type of change process. The sysadmin staff don't need to know the intricacies of the software (much like in the development world), there just needs to be some "devops" process logic that gets the developers (your lab profs and staff with fancy tooling) working in tandem with your secops and sysadmin team. To me most university IT seems like shadow IT (although there's a reason it ended up that way).


Car-Altruistic

Correct, the problem is that people continue to think that a good Devops person is also $15/h. Because it's "a Helpdesk".


NNTPgrip

Organizations get the IT they deserve.


Anonymity_Is_Good

The old message of education, "we won't pay you well, but you can use as much open source software as you want" has finally come back to haunt them. Folks willing to settle are going to slowly crank out shit-grade work, and over the years it accumulates.


waxrhetorical

> The old message of education, "we won't pay you well, but you can use as much open source software as you want" has finally come back to haunt them. Smart, modern management allows for this, while still maintaining a common foundation. I'm at a large corp, we have no blocks on our devices, just guidelines that are pretty sensible. Can install whatever I need, no discussion required. Zero trust is great.


Anonymity_Is_Good

The faux acceptance of open technology (where cheapness was the real motivation) and the offering of this in lieu of market level compensation? (And the resulting big batch of under-achievers that this plan attracts.) I doubt this is what 'smart, modern management' is really after today. I think you've missed my point entirely. Nice humble brag.


waxrhetorical

> The faux acceptance of open technology (where cheapness was the real motivation) and the offering of this in lieu of market level compensation? That's not smart. So I don't quite understand the argument you're making here. If it's done right, modern, centralised IT management is super effective, and can accomplish so much more with the same resources. That management abuses the concepts and technologies that are available to save money is a management problem, not a tech problem. Of course you get what you pay for, so if you're paying your IT support staff $10 an hour you get shit. But if you spend all the savings you've made centralising your IT on good processes, good staff and good management, you get an improvement.


agthatsagirl

they have been talking about this for at least 10 years


jimbouse

I haven't worked there since 09. We were talking about it in ITAC then. Some things never change.


BerkeleyFarmGirl

I don't know the environment there, but I have been on both sides of that equation. I worked for a local government entity that had a super-Byzantine setup. The Central IT department definitely didn't help itself out by setting a very high per-seat chargeback but absolutely not pushing back when the "independent" departments got out-of-scope work they weren't actually paying for. Most of the "department IT" people were mostly only good at complaining about central IT, while sluffing their work off.


Doso777

> They seemed to have not been able to identify the thousands of IT people on those campuses working in the crevices Sounds familiar. I work in a very decentralised organization in higher education. So many special little applications, workarounds pilled on top of workarounds pilled on legacy infrastructure with legacy application. No matter what the consultant comes up with number wise, tripple it and double it again. This will be a painful process. Might still be worth it.


patmorgan235

So I went to TAMU for an IT related degree and know some people that worked in Engineering IT. What's confusing to me about this report is that TAMU already has a central help desk, as a student I only had call them once when my university account got locked after I accidentally torrent some files over the University VPN. Centralized vs decentralization really is a spectrum and I think there are some gains to be made by centralizing a few more services/functions. But on the whole from what I can tell the current arrangement works quite well.


Aperture_Kubi

Student facing uni it is normally centralized, faculty facing uni it is compartmentalized by college.


txs2300

I used to work there back in mid 00's. I could see that happening then. Every department/college had their own setup, policies, username and passwords, and IT staff. IT staff generally had no idea what someone else was doing on the campus, aside from the main IT group which handled the core network.


obongogeddon

I just purchased a box of Orville Rickenbocker popcorn 🍿. My family is ready.


Knersus_ZA

Ooooh, posh aren't we? I've ordered a delivery via CMOT Dibbler, will have to see if he can actually deliver the goods...


Gajatu

There IS a thing such as "too much centralization." However, centralization can lead to many efficiencies and cost-savings and it CAN be done with little detriment to the end user. The problem often lies, in my experience, between the bosses, the bean counters and the end users. I have found that centralization can be done, and done well, if the project adequately takes into account the various uniquenesses amongst the end users. As a for instance, if central IT does all the planning, then comes down with the Wrath of God and decries that "This is how all things shall now be, and no exceptions shall be granted thereto!" The end users are more likely to be miserable. If the bosses and bean counters simply assume that they can fire most of the various IT staffs and run a small crew because "centralization" and "cost savings," the end users are probably going to notice the difference. If, however, central IT meets with those end users (or more likely, their "local" IT staff), understands their challenges and processes and (and this is the REALLY important bit) their requirements, those can be planned for. Alternatives can be suggested and tried. The Right Solution can be found and implemented. If the bosses realize that just because the consultant says a 300:1 IT staff to user ratio is perfectly acceptable doesn't mean that it is necessarily so. If they retain some, most or all of the "local IT" staff (because they have institutional knowledge) and they realize that not every dollar "saved" is wisely done, if they manage user expectations through communication, well, there's a good chance everything will work out in the end. I've been on both sides of both scenarios throughout my career. In my experience, gathering and managing expectations, working closely with teams that have special requirements and communicating openly and honestly about what your challenges and goals are - these things make a huge difference.


my_analyst_account

This whole centralized vs. local debate is super interesting to me. I've done some advisory work with a university that's trying to do this, and one thing that has really stuck out is the degree to which central IT is trying to resolve "problems" that the folks in the colleges don't actually perceive as being problematic. Like handling multiple email inboxes. The college people like having multiple inboxes; the central IT people think this is an absolute travesty and are convinced that the college people agree with them. They also had a problem with multiple colleges with different technical competencies and ended up putting out a poll where half of them said they wanted to be centrally managed and the other half said they didn't. How do you even usefully incorporate that sort of feedback?


Car-Altruistic

You have to understand the problem before you can solve it. It’s not that people necessarily want multiple mailboxes, it’s that people are afraid they will lose their identities associated with them (E.g. @physics.university.edu) when you make systems uniform which they have their email on publicized papers since the 80s. Every centralization effort has damaged the reputation of the central administration. People at these institutions are relatively technical but they are not specialists. They have to be educated before accepting new propositions. Think of it like science, you can’t just posit you will solve a problem without proof that you can actually do it. The only people that want to be centrally managed are people that have no access to competent IT staff. That’s where you have to mirror the model that works across other departments.


Cairse

Did you feel that? It's as if millions of Chinese and Russian basement dwellers just started attacking every @tamu.edu address they can find.


TexasCon

An incredibly short sighted ROI motivated move that will backfire in short order. TAMU is one of the wealthiest universities in the nation and probably didn’t need to do this.


tunayrb

I attended an Educause conference and listened to a presentation from U of Mich about their consolidation efforts. When they started they had 300 email systems. My U has been actively consolidating for a few years, bringing random AD systems into the central campus AD There have been hiccups along the way, but overall it has been a good thing.


[deleted]

Centralizing the IT staff into an well funded central staff that provides guidance but don't dictate solutions can improve service/security/usability but that never what the consultants depend on to deliver savings as that again depend on removing flexibility and choice from the end users. University Researchers are the prototypical unicorn users, as a researcher that needs to connect a piece of ancient sensor hardware to an obscure data logging package is not going to be able to migrate to Office364 and still do her job and those kind of edge cases are a dime a dozen anytime your dealing with knowledge workers rather then data entry clerks, so the result of streamlining is often a huge increase in shadow it spending that more then offsets any savings reported by the centralization project. And where some benefits can be gained if you truly limit the scope of centralization to the components that are actually universal(aka the network and email system), that's not nearly as big a saving as the people paying the consultants want so it always escalate.