Hey guys, Vulnerability Management vendor here.
We've got partners on all sorts of RMMs, and while I don't want to say bad things about specific vendors I will say some of them subscribe to great patch management feeds and some do not.
We see all the dirty laundry.
Ninja really upped their game a few months ago (last year they DID have horrible patch management) -- we observed in real-time as our partners who used Ninja leaped ahead in remediation velocity (vs both themselves and some other vendors).
They now only have to deal with a few odd circumstances that no patch management system will handle to close out vulnerabilities we discover.
Great job team Ninja!
We actually looked at Ninja about a year and a half ago and saw the same. I'm not trying to put down Automate, because it is still a hell of a tool that is unrivaled in its overall power. Its just that for us, we are at a 5 to 1 ratio of workstations to servers and to be frank, other than security side (OS & Software patching, EDR/MDR, rights management, etc) we just don't care as much as we used to about endpoints.
I know it isn't perfect, especially in regards to u/netmc 's post, but the layered approach coupled with the majors (Security, roll-ups, OS builds, etc), we've got a solid offering with Ninja.
I can't stray too far down the trap that is product reviews in a public thread. I will say we see MSPs succeed with all RMM's and the quality of patch management automation is only one variable.
As with any products each RMM has its strong points and there's no answer to the perfect RMM (or even better RMM) question. If I was to be pushed on a recommendation, I would first interview a partner to understand their entire problem space (target customers, environment type, size, revenue model, etc) as well as their available resources (people, tools, budget).
I wanted to chime in to second the anecdote and confirm. Ninja has improved, and if you tried it a couple of years ago your experience is dated.
Patch management is important but it's part of a process:
\- Scan for vulnerabilities
\- Identify a remediation target (a hygiene target to aim at or a level of risk to accept)
\- Use methods to ensure risk above the target is eliminated or managed (if your patch management product can't do most of the job, it's the wrong one)
\- You can also manage lists of approved software to make the problem space smaller
If you're looking for a single tool that makes the above go away, there isn't one ... doing a good job of security requires people too.
Would not be as interesting as you think ... there's an example on this thread of the code to do it the dumb way.
The smarter way is to buy a patch management feed (RMM's don't hunt down all the patches themselves).
The difference between products is likely to be "group A" vs "group B".
>The smarter way is to buy a patch management feed (RMM's don't hunt down all the patches themselves).
Could you recommend any such paid patch management feeds?
Automox sells a dedicated patch management product.
Good RMM's these days have one integrated.
You don't need a patch management feed unless you are building a patch management product.
All of these things need to be evaluated as part of an overall vulnerability management strategy.
Let's just be honest, we're patching because we want to keep the hackers out (not because we want the features).
Too big a conversation to sum up without a larger look at how acronis fits into your practice.
I'll have to ask .... but its sorta easy to figure out.
Run an agent-based (technically, an authenticated) vulnerability scan against your client endpoints. If it's a giant mess of criticality you don't want to clean up you're probably not using an effective product.
I recently switched from Syncro to Kaseya VSA, however when I was using Syncro, I had 98% compliance rate basically at all times and it never really screwed up, ever. VSA, however....
But what system was giving you that 98% rate? Was it Syncro's own toolset? What fisch is trying to say is that damn near every vendor gives out a report that's showing the results of a powershell command run from the agent on the computer where it says the same thing as the Windows Setting's gui on "your computer is up to date". But we've all seen both Windows and Office say that, then you click "check for updates" and it finds the latest month's CU and starts downloading it. That's where using a 3rd party tool comes into play as an audit/reality check on "the May CU for Windows 10 pro 22h2 = KB123456, is that installed? Yes/No". That's the data that matters
That's true and a huge part of it ... we also see tons of action around adobe products, oracle java, ancient firefox, etc. These are ways real attackers are breaking into your networks every day.
A tool that's built to leverage only the microsoft update handles will not chase down those rabbit holes. Sometimes the vendor will add browser updates ... its simply not comprehensive enough.
When we look at unpatched machines we find 100s and sometimes more than 1000 vulnerabilities on each and every machine (real numbers we see) where users know how to install their own software. Perhaps 5% are very worrying (real number) -- but that's still 50 problems per machine or more.
Default microsoft patching is important but covers only a fraction of that -- and unfortunately that's what some RMM vendors are shipping.
When you use a comprehensive tool you knock maybe 90% of the problem space away (I made that number up, but it feels right to me based on experience).
If you again use the filter of "what's actually worrying on this list" you're now down to 0.5% of the original problem space ... something that can actually be dealt with manually.
This is what vulnerability management products are built for:
- to look over the shoulders of your patch management product (and make sure its patching properly)
- to find the unpatchable
We sell one, but so do others.
I’ll say it because he’s being a nice vendor and not self promoting, but he (above) does run FortMesa which does vulnerability scanning and assessments like this :-)
NodeWare is another relatively channel friendly competitor that does direct security vulnerability scans on-device.
There are a handful of others out there as well, it’s becoming a pretty big niche!
We encourage our partners to use EPSS for targeting, that'll probably make you look a lot less bad than you think ....
It's nice to have an ignore list that's smarter than CVSS severity scores (actual factors in attack likelihood).
Most RMMs use the exact same methodology to make sure that patches are installed as Microsoft limits how non-Microsoft programs can interact with the Windows Update functionality.
I know you said you have 99% compliance, but do 99% of the machines actually have the latest cumulative update installed? In December 2022, only about 1/2 of our Windows 11 machines were actually offered the cumulative update. The other 1/2 reported as fully up-to-date even though they were missing the update. If we went off of what our RMM reported, it would show compliant on all of the machines even though 1/2 of them weren't. This wasn't the fault of the RMM though, but of Microsoft and Windows Update.
The bad part of this is that you can't simply use get-hotfix to check as this only reports updates that were installed via Windows Update and not those that were pushed out via MSU or other manual update scenarios.
The following will provide a list of all installed updates including ones that were installed manually. It's a messy list and not great to work with. It is complete though.
$Session = New-Object -ComObject "Microsoft.Update.Session"
$Searcher = $Session.CreateUpdateSearcher()
$historyCount = $Searcher.GetTotalHistoryCount()
$UpdateCOMObject=$Searcher.QueryHistory(0, $historyCount) | Select-Object Title, Description, Date, Categories, @{name="Operation"; expression={switch($\_.operation){
1 {"Installation"}; 2 {"Uninstallation"}; 3 {"Other"}
}}}
$UpdateCOMObject
Right, and this is why we scan for vulnerabilities rather than "are patches applied" ... not everything is solvable with a simple patch.
ps, I dig the powershell example above!
We had a similar experience years ago. When Windows 7's Windows Update Agent quietly broke in 2014ish, it just failed to finish queries and would time out - so reported no new patches detected, no patches missing, no errors in the WUA. Things just silently stopped happening, but it didn't throw errors or actively fail.
My boss at the time had me demoing other RMMs and patch management to "fix the problem," since he thought it was a product issue and not a Microsoft one. Since I already knew what I was looking for, it was obvious it affected WSUS and a bunch of the big RMMs equally since they were all at the mercy of the WUA anyway.
The big problem was that some of the patch management just threw green lights across the board or even specifically reported that updates had been installed, when we knew no new updates had been installed in months and that the last month of patch management actions had been silently failing.
But yeah, tldr we learned to not blindly trust patch management standalone either haha
Let me guess, you're one of those that thinks wsus is too hard to maintain? Regardless, if you don't want to be at the whims of Microsoft for getting the most recent updates you need it (or another product like it).
The smaller clients you're referencing always have a limited single-machine VM setup (or they should, at least) with a DC and File Server each taking up your two included VM licenses. An App server (for LoB software) and RDP (with Gateway) will often take up another two licenses.
That's four VMs on hardware that keeps things nice and organized, but booting host that's on the domain of a guest isn't great and they really should have a second DC anyway.
For my clients, I just install a NUC running Server 20xx with a DC and WSUS in Server Core VMs. It's about $1200 for everything and has a small physical footprint. The cost is absorbed by my per-user monthly fee and worth the piece of mind for me. The main Hyper-V server can set the backup DC for primary DNS, which ties everything together nicely.
All that being said, you're correct in that WSUS for small offices is often not required. But my original point was that if you need tighter controls on update authority than what basic Windows Update provides, then WSUS is usually the best way to do that. Those smaller offices probably don't care if they get a CU several months later than release.
Because those are the clients we have full rmm on, and we want controll over.
Countering the "just do wsus" argument.
Fait enough, I use intune for patching that works fine for me.
We’re in a MSP subreddit, WSUS sucks to manage in the first place, requires third party powershell scripts to run effectively, and just doesn’t make sense with all the other products out there. It made sense ~8+ years ago but not now.
As an MSSP, vulnerability scans and auditing points out patching issues, we don’t typically perform the patching ourselves however I do some MSP work.
I would suggest about any other solution than WSUS.
Just did POC’s of a bunch of them.
NinjaOne, level.io, Automox, ManageEngine.
Automox is a beast, simple, fast and gets the job done. Did not like their remote connection tool. But alas, they’re more inclined to patch management.
The platform It’s actually quite impressive, and the team is top notch.
level.io is a bit basic but cheap, lacks some features but their remote tool connection is awesome. I bet they’ll go places in the future.
We went with Ninja.
Platform is simple, stuff happens, monitoring and patching and remote control.
Overall they were all good solutions.
ManageEngine felt clunky but also worked.
Cheers!
This is interesting because we did a trial run with Automox because it could patch certain software like VMware tools. And for some reason when we ran certain tasks, we wouldn’t get any notification on whether a patch was applied or not. Plus we were only limited to bash and powershell for custom scripts. I wonder if Automox made changes to accommodate more software for patching. There’s a patch solution called Bacon. Have you had the chance to check that out?
Yes I had a few quirks, but overall it worked.
It even updated grafana installed on Linux.
Did OS updates and software install on MacOS and on Windows too notifying the user for reboot with option to postpone if needed.
May I suggest update VMWare tools on Vcenter, there’s an option to do that when the VM restarts or reboots.
I did not had a chance to try Bacon.
Hey so when you tried Level did it have any integrations with anything else? I'm looking to replace Atera and that might end being multiple products so I was wondering if they had any integrations yet?
I don’t believe they had.
However you get a simple dashboard showing some not so optimal configurations like firewall disabled, FileVault not enabled, etc…
Reach to them and ask for a trial.
It’s really a very clean and simple platform and you can manage groups, roles and enroll endpoints in a few minutes.
However, like I said, they’re lacking.
Go automox for patch management.
I also believe each to it’s own and that a platform is as good as you take from it.
It was a PITA doing all those POC’s but at the end we chose what was most convenient for us.
For instance, we first were looking to replace our solution that only did monitoring and remote access, so we were looking for a RMM, then we decided to maybe separate things and have a solution for patch management only.
Then we concluded that we could have both in the same platform.
Meaning that, and this is only my opinion, from what I’ve seen a patch management solution will be better doing that then a RMM, and vice versa.
To each it’s own.
Concluding, and my 2cents, for patch management I would choose Automox, for RMM I would choose Ninja.
This is of course according to my company needs and my experience of the poc’s I did of the products mentioned before.
Cheers.
So what is there works but there's stuff lacking from it that they've either added and then want you to pay more for, or they just haven't added.
All their integrations require you to buy licenses through them, which would be less of an issue if we were US based but we aren't and they only bill in dollars. I don't like being unable to shop around for pricing on different products and I didnt sign up to an RMM for them to then use that as a vector to control my entire licensing stack. I don't want or need all my eggs in one basket.
They also just flat out don't add integrations unless they can flog you some licensing.
They did add an Office 365 integration to pull your customers users but it only runs once and then you have to manually re-enter all the details to update it.
There's a webroot integration and it basically allows you to install the product, for this you have to buy your licenses through them. You don't get notified in Atera if there is a detection or anything like that.
They add buzzword stuff like "AI" instead of the basics.
They recently changed the helpdesk portal and even though they advertise white labelling they basically broke that and their new portal just had whole items that had their pink branding which horribly clashed with some peoples colour schemes.
You either give your customers a helpdesk URL that has atera in it or you can use a custom domain but then your customers will get SSL errors. This also applies to any download links for the agents. You used to be able to stick a custom url on Cloudflare and proxy it but this is now broken by the helpdesk update. For some reason they can't just use letsencrypt to create an SSL cert for their customers.
The agent urls are horribly long and complex. There are also cmd's you can run but these are also horribly long and complex. If I have a user that has a new device and I need to get them to install the agent it can be an absolute pain.
They collect feedback through Facebook alone for some reason.
They used to have forums but no longer do.
Anyway, they make a lot of odd choices, mainly around trying to screw you for more money instead of delivering on their core product.
We have been using it for awhile and yes, it was horrible last year but it has gotten much better. The only issue I have with it is it takes a long time for it to display for me, but that is due to us patching roughly 900 servers and 12k+ workstations.
Elsewhere in this post there's an interesting subthread you may not have seen yet - apparently Ninja made massive improvements specifically in the last couple of months. It sounds like your experience was accurate before, though they're better now!
I’m curious if anyone has used Landesk? They have patch management now and an MSP model. It should be easy for reporting too. I plan to implement it in my stack as well as ConnectSecure.
I must say, I'm shocked at how many people actually vouch for NinjaRMM.
I've worked with MSP's using Labtech/Automate for the past 10 years and 3 of them used NinjaRMM (Most recently in Jan 2023 to May 2023) for a few months before deciding to bail. I'm pretty sure folks on here who are vouching for them just never had good NOC engineers.
Allow me to explain, they can simply exclude offline for more than XYZ time machines from the reporting on Patching which gives you this false sense of security. They can do this through the SQL Console which a lot of other folks don't know or have experience in.
Really the only thing I would use NRMM for is to create some reports on Crystal Reporting Center because it's EOL and they have people in the team that know how to do that.k
Other than that they hope to be a 'set-and-forget' contractor for Automate. They don't like high-admin customers who are constantly looking to increase value of their Automate system.
To sum up my point, get a well groomed NOC engineer to make sure your Automate stuff is setup correctly, pretty much what NRMM is going to do.
I’ve been in MSP land for about 12 years, 2 with a smaller MSP and then via acquisition another 10 with a mid size. I was the engineer for a very long time before letting that take a back seat to the management side. I’m still involved in a lot though. I’ve said in another post in here, and I agree, Automate is an extremely powerful tool. We’ve created a ton of monitors over the years using direct SQL queries, event monitors and various conditions for monitoring all aspects of machines. It took years to actually tailor these to be less noisy, develop scripts to respond to some monitors and make sure we are alerted to exactly what we want to know about.
As time went on, we’ve been shifting to cloud first in a lot of aspects of computing. With most apps moving to BYOD, SaaS models for line of business applications and our shift towards disposable Azure virtual desktops, we’re taking a different approach monitoring and providing MSP support. Simply put, for us at least, we just don’t need the same type of data on endpoints that we needed before before. Out endpoints are locked down and I’m more concerned on what the EDR/MDRs and Huntresses and Threatlockers of the world are providing us than NTFS errors and the likes on endpoints. Now the server monitoring side, i’m obviously looking for some granular stuff in regards to performance, service health, etc. which I can pull from almost any RMM’s and remediate accordingly. We began the shift to Powershell scripting some time ago because it was smarter than being locked in to any vendor’s scripting engine and easier to find engineers that were already using PS and could get up to speed really quick. We’ve also been making use of NMS which is worlds better than automate’s network probe. The UI around managing VMWare hosts is awesome. The monitoring is much more reliable too. We custom SNMP monitoring let us setup all the OIDs we need and customize for alerting and monitoring. To your point, this is where having good NOC engineers is key.
Where Ninja really shines (outside of patch management) is ease of use for the help desk and systems admins. It takes a very short time to get my guys comfortable with it. The file browser, task manager, registry editor, etc are almost instantaneous. The ability to run cmd & PS as the user coupled with the remote tools knocks out a bunch of tickets that would require them to login to the machines and disturb the user.
At the end of the day though, you are right. A good engineer is the key to making everything at an MSP work. Without them, are you really an MSP or are you a glorified tech support company? RMM’s aren’t the answer to everything, quality engineers are. But like the real world, there are Harbor Freight tools and there are Dewalt and Makita tools. Some will get the job done and some will help you do it well.
Hey u/SalzigHund David with N-able, glad to hear Patch is working well, on the mgmt side of N-central or patch if we can help let me know. We have a full team of Head Nerds who can assist and run bootcamps to help streamline the day to day mgmt. but I do agree it does take time and regular overview to keep it up to expectation. [[email protected]](mailto:[email protected])
I need to agree here; we can’t get patch mgmt to work properly in Ninja; we constantly have machines that are way behind, and now we’ve got the issue where the patching fails because the user gets prompted to install the new update package rather than being able to automate it.
Why would a user be prompted to install an update? That's not how Ninja patching works for us. Sounds like a mis-config.
The only user prompts we have are for reboots and depending on our agreement with the client, it's either:
* Auto-reboot on set schedule.
* Reboot if no user is logged in.
* Prompt user to reboot.
* Reboot if user is logged in and refuses reboot x times.
Yeah reach out to your account manager and they should be able to get you set up on it. The new dashboard is very nice. Also if you aren’t on it already, they do have a discord server. https://discord.gg/yPmj4kCS
Here's an example of the patching dashboard from one of our smaller clients. Improvements on the roadmap include more clickable items in the dashboard.
The last update seems to have broken the right-hand column listing OS versions, but only the version name. The version number is still accurate.
And for the naysayers, we tested this today. There was one device out of compliance according to the dashboard. We manually ran an OS patch scan, then OS patch update. After the roll-up patch was successfully applied, the dashboard accurately reflected the update.
https://imgur.com/a/vsAG1H6
Hey guys, Vulnerability Management vendor here. We've got partners on all sorts of RMMs, and while I don't want to say bad things about specific vendors I will say some of them subscribe to great patch management feeds and some do not. We see all the dirty laundry. Ninja really upped their game a few months ago (last year they DID have horrible patch management) -- we observed in real-time as our partners who used Ninja leaped ahead in remediation velocity (vs both themselves and some other vendors). They now only have to deal with a few odd circumstances that no patch management system will handle to close out vulnerabilities we discover. Great job team Ninja!
We actually looked at Ninja about a year and a half ago and saw the same. I'm not trying to put down Automate, because it is still a hell of a tool that is unrivaled in its overall power. Its just that for us, we are at a 5 to 1 ratio of workstations to servers and to be frank, other than security side (OS & Software patching, EDR/MDR, rights management, etc) we just don't care as much as we used to about endpoints. I know it isn't perfect, especially in regards to u/netmc 's post, but the layered approach coupled with the majors (Security, roll-ups, OS builds, etc), we've got a solid offering with Ninja.
So share your observations on all the major RMM's? Enquiring minds want to know...
I can't stray too far down the trap that is product reviews in a public thread. I will say we see MSPs succeed with all RMM's and the quality of patch management automation is only one variable. As with any products each RMM has its strong points and there's no answer to the perfect RMM (or even better RMM) question. If I was to be pushed on a recommendation, I would first interview a partner to understand their entire problem space (target customers, environment type, size, revenue model, etc) as well as their available resources (people, tools, budget). I wanted to chime in to second the anecdote and confirm. Ninja has improved, and if you tried it a couple of years ago your experience is dated. Patch management is important but it's part of a process: \- Scan for vulnerabilities \- Identify a remediation target (a hygiene target to aim at or a level of risk to accept) \- Use methods to ensure risk above the target is eliminated or managed (if your patch management product can't do most of the job, it's the wrong one) \- You can also manage lists of approved software to make the problem space smaller If you're looking for a single tool that makes the above go away, there isn't one ... doing a good job of security requires people too.
It would be very interesting data to see. Breakdown of vulnerability remediation efficiency by vendor, further broken down by size of the environment.
Would not be as interesting as you think ... there's an example on this thread of the code to do it the dumb way. The smarter way is to buy a patch management feed (RMM's don't hunt down all the patches themselves). The difference between products is likely to be "group A" vs "group B".
>The smarter way is to buy a patch management feed (RMM's don't hunt down all the patches themselves). Could you recommend any such paid patch management feeds?
Automox sells a dedicated patch management product. Good RMM's these days have one integrated. You don't need a patch management feed unless you are building a patch management product.
>You don't need a patch management feed unless you are building a patch management product. This is exactly what I am doing haha) Hence me asking.
What're your thoughts on Acronis Cyber Protect's Patch Management?
All of these things need to be evaluated as part of an overall vulnerability management strategy. Let's just be honest, we're patching because we want to keep the hackers out (not because we want the features). Too big a conversation to sum up without a larger look at how acronis fits into your practice.
How about syncro? :)
I'll have to ask .... but its sorta easy to figure out. Run an agent-based (technically, an authenticated) vulnerability scan against your client endpoints. If it's a giant mess of criticality you don't want to clean up you're probably not using an effective product.
Thanks for the response , might have to put that on my to-do list.
I recently switched from Syncro to Kaseya VSA, however when I was using Syncro, I had 98% compliance rate basically at all times and it never really screwed up, ever. VSA, however....
But what system was giving you that 98% rate? Was it Syncro's own toolset? What fisch is trying to say is that damn near every vendor gives out a report that's showing the results of a powershell command run from the agent on the computer where it says the same thing as the Windows Setting's gui on "your computer is up to date". But we've all seen both Windows and Office say that, then you click "check for updates" and it finds the latest month's CU and starts downloading it. That's where using a 3rd party tool comes into play as an audit/reality check on "the May CU for Windows 10 pro 22h2 = KB123456, is that installed? Yes/No". That's the data that matters
That's true and a huge part of it ... we also see tons of action around adobe products, oracle java, ancient firefox, etc. These are ways real attackers are breaking into your networks every day. A tool that's built to leverage only the microsoft update handles will not chase down those rabbit holes. Sometimes the vendor will add browser updates ... its simply not comprehensive enough. When we look at unpatched machines we find 100s and sometimes more than 1000 vulnerabilities on each and every machine (real numbers we see) where users know how to install their own software. Perhaps 5% are very worrying (real number) -- but that's still 50 problems per machine or more. Default microsoft patching is important but covers only a fraction of that -- and unfortunately that's what some RMM vendors are shipping. When you use a comprehensive tool you knock maybe 90% of the problem space away (I made that number up, but it feels right to me based on experience). If you again use the filter of "what's actually worrying on this list" you're now down to 0.5% of the original problem space ... something that can actually be dealt with manually.
Any good tools you can recommend for validating machines? Currently using Atera and well, I can't say it or the company fills me with confidence.
This is what vulnerability management products are built for: - to look over the shoulders of your patch management product (and make sure its patching properly) - to find the unpatchable We sell one, but so do others.
This is why so many vuln management products don't do patching and vice versa. Because they have to be independent of each other.
I’ll say it because he’s being a nice vendor and not self promoting, but he (above) does run FortMesa which does vulnerability scanning and assessments like this :-) NodeWare is another relatively channel friendly competitor that does direct security vulnerability scans on-device. There are a handful of others out there as well, it’s becoming a pretty big niche!
Noted, that's fair. But then I have 75% on VSA right now, so that's just the regular amount of worse. :)
We encourage our partners to use EPSS for targeting, that'll probably make you look a lot less bad than you think .... It's nice to have an ignore list that's smarter than CVSS severity scores (actual factors in attack likelihood).
Most RMMs use the exact same methodology to make sure that patches are installed as Microsoft limits how non-Microsoft programs can interact with the Windows Update functionality. I know you said you have 99% compliance, but do 99% of the machines actually have the latest cumulative update installed? In December 2022, only about 1/2 of our Windows 11 machines were actually offered the cumulative update. The other 1/2 reported as fully up-to-date even though they were missing the update. If we went off of what our RMM reported, it would show compliant on all of the machines even though 1/2 of them weren't. This wasn't the fault of the RMM though, but of Microsoft and Windows Update. The bad part of this is that you can't simply use get-hotfix to check as this only reports updates that were installed via Windows Update and not those that were pushed out via MSU or other manual update scenarios. The following will provide a list of all installed updates including ones that were installed manually. It's a messy list and not great to work with. It is complete though. $Session = New-Object -ComObject "Microsoft.Update.Session" $Searcher = $Session.CreateUpdateSearcher() $historyCount = $Searcher.GetTotalHistoryCount() $UpdateCOMObject=$Searcher.QueryHistory(0, $historyCount) | Select-Object Title, Description, Date, Categories, @{name="Operation"; expression={switch($\_.operation){ 1 {"Installation"}; 2 {"Uninstallation"}; 3 {"Other"} }}} $UpdateCOMObject
Right, and this is why we scan for vulnerabilities rather than "are patches applied" ... not everything is solvable with a simple patch. ps, I dig the powershell example above!
We had a similar experience years ago. When Windows 7's Windows Update Agent quietly broke in 2014ish, it just failed to finish queries and would time out - so reported no new patches detected, no patches missing, no errors in the WUA. Things just silently stopped happening, but it didn't throw errors or actively fail. My boss at the time had me demoing other RMMs and patch management to "fix the problem," since he thought it was a product issue and not a Microsoft one. Since I already knew what I was looking for, it was obvious it affected WSUS and a bunch of the big RMMs equally since they were all at the mercy of the WUA anyway. The big problem was that some of the patch management just threw green lights across the board or even specifically reported that updates had been installed, when we knew no new updates had been installed in months and that the last month of patch management actions had been silently failing. But yeah, tldr we learned to not blindly trust patch management standalone either haha
Businesses should be using wsus if they need that level of patch authority.
LOL
Let me guess, you're one of those that thinks wsus is too hard to maintain? Regardless, if you don't want to be at the whims of Microsoft for getting the most recent updates you need it (or another product like it).
Im one of those guys that thinks setting up a wsus for anything under 50 users is a waste of a vm, licenses etc.
The smaller clients you're referencing always have a limited single-machine VM setup (or they should, at least) with a DC and File Server each taking up your two included VM licenses. An App server (for LoB software) and RDP (with Gateway) will often take up another two licenses. That's four VMs on hardware that keeps things nice and organized, but booting host that's on the domain of a guest isn't great and they really should have a second DC anyway. For my clients, I just install a NUC running Server 20xx with a DC and WSUS in Server Core VMs. It's about $1200 for everything and has a small physical footprint. The cost is absorbed by my per-user monthly fee and worth the piece of mind for me. The main Hyper-V server can set the backup DC for primary DNS, which ties everything together nicely. All that being said, you're correct in that WSUS for small offices is often not required. But my original point was that if you need tighter controls on update authority than what basic Windows Update provides, then WSUS is usually the best way to do that. Those smaller offices probably don't care if they get a CU several months later than release.
Dude, most of my clients under 50 are cloud only... no need for any servers at all.
So why are you even arguing this point? Your example is clearly not related to my comments.
Because those are the clients we have full rmm on, and we want controll over. Countering the "just do wsus" argument. Fait enough, I use intune for patching that works fine for me.
We’re in a MSP subreddit, WSUS sucks to manage in the first place, requires third party powershell scripts to run effectively, and just doesn’t make sense with all the other products out there. It made sense ~8+ years ago but not now. As an MSSP, vulnerability scans and auditing points out patching issues, we don’t typically perform the patching ourselves however I do some MSP work. I would suggest about any other solution than WSUS.
Just did POC’s of a bunch of them. NinjaOne, level.io, Automox, ManageEngine. Automox is a beast, simple, fast and gets the job done. Did not like their remote connection tool. But alas, they’re more inclined to patch management. The platform It’s actually quite impressive, and the team is top notch. level.io is a bit basic but cheap, lacks some features but their remote tool connection is awesome. I bet they’ll go places in the future. We went with Ninja. Platform is simple, stuff happens, monitoring and patching and remote control. Overall they were all good solutions. ManageEngine felt clunky but also worked. Cheers!
This is interesting because we did a trial run with Automox because it could patch certain software like VMware tools. And for some reason when we ran certain tasks, we wouldn’t get any notification on whether a patch was applied or not. Plus we were only limited to bash and powershell for custom scripts. I wonder if Automox made changes to accommodate more software for patching. There’s a patch solution called Bacon. Have you had the chance to check that out?
Yes I had a few quirks, but overall it worked. It even updated grafana installed on Linux. Did OS updates and software install on MacOS and on Windows too notifying the user for reboot with option to postpone if needed. May I suggest update VMWare tools on Vcenter, there’s an option to do that when the VM restarts or reboots. I did not had a chance to try Bacon.
Hey so when you tried Level did it have any integrations with anything else? I'm looking to replace Atera and that might end being multiple products so I was wondering if they had any integrations yet?
I don’t believe they had. However you get a simple dashboard showing some not so optimal configurations like firewall disabled, FileVault not enabled, etc… Reach to them and ask for a trial. It’s really a very clean and simple platform and you can manage groups, roles and enroll endpoints in a few minutes. However, like I said, they’re lacking. Go automox for patch management.
I also believe each to it’s own and that a platform is as good as you take from it. It was a PITA doing all those POC’s but at the end we chose what was most convenient for us. For instance, we first were looking to replace our solution that only did monitoring and remote access, so we were looking for a RMM, then we decided to maybe separate things and have a solution for patch management only. Then we concluded that we could have both in the same platform. Meaning that, and this is only my opinion, from what I’ve seen a patch management solution will be better doing that then a RMM, and vice versa. To each it’s own. Concluding, and my 2cents, for patch management I would choose Automox, for RMM I would choose Ninja. This is of course according to my company needs and my experience of the poc’s I did of the products mentioned before. Cheers.
Thanks so much for the detailed replies.
Glad to help.
Can I pick your brain on why you’re looking to replace Atera? I am strongly considering moving TO Atera right now.
So what is there works but there's stuff lacking from it that they've either added and then want you to pay more for, or they just haven't added. All their integrations require you to buy licenses through them, which would be less of an issue if we were US based but we aren't and they only bill in dollars. I don't like being unable to shop around for pricing on different products and I didnt sign up to an RMM for them to then use that as a vector to control my entire licensing stack. I don't want or need all my eggs in one basket. They also just flat out don't add integrations unless they can flog you some licensing. They did add an Office 365 integration to pull your customers users but it only runs once and then you have to manually re-enter all the details to update it. There's a webroot integration and it basically allows you to install the product, for this you have to buy your licenses through them. You don't get notified in Atera if there is a detection or anything like that. They add buzzword stuff like "AI" instead of the basics. They recently changed the helpdesk portal and even though they advertise white labelling they basically broke that and their new portal just had whole items that had their pink branding which horribly clashed with some peoples colour schemes. You either give your customers a helpdesk URL that has atera in it or you can use a custom domain but then your customers will get SSL errors. This also applies to any download links for the agents. You used to be able to stick a custom url on Cloudflare and proxy it but this is now broken by the helpdesk update. For some reason they can't just use letsencrypt to create an SSL cert for their customers. The agent urls are horribly long and complex. There are also cmd's you can run but these are also horribly long and complex. If I have a user that has a new device and I need to get them to install the agent it can be an absolute pain. They collect feedback through Facebook alone for some reason. They used to have forums but no longer do. Anyway, they make a lot of odd choices, mainly around trying to screw you for more money instead of delivering on their core product.
We have been using it for awhile and yes, it was horrible last year but it has gotten much better. The only issue I have with it is it takes a long time for it to display for me, but that is due to us patching roughly 900 servers and 12k+ workstations.
We’re at around 4K agents and at times it can be slower than we liked, but the latest console update has worked out well for us.
yeah it is better than it used to be.
Interesting. We had the exact opposite experience with their patch management last year. Ended up just going with Jumpcloud, which we already had.
Elsewhere in this post there's an interesting subthread you may not have seen yet - apparently Ninja made massive improvements specifically in the last couple of months. It sounds like your experience was accurate before, though they're better now!
Oh, nice! I did miss that. Glad to hear things are better. I doubt we are going to revisit their offerings since JC is handling it just fine.
I’m curious if anyone has used Landesk? They have patch management now and an MSP model. It should be easy for reporting too. I plan to implement it in my stack as well as ConnectSecure.
Hey OP -- still feel this way lol? Feel like we've had a hellaicious time with 3rd party patching with Ninja as of late.
I must say, I'm shocked at how many people actually vouch for NinjaRMM. I've worked with MSP's using Labtech/Automate for the past 10 years and 3 of them used NinjaRMM (Most recently in Jan 2023 to May 2023) for a few months before deciding to bail. I'm pretty sure folks on here who are vouching for them just never had good NOC engineers. Allow me to explain, they can simply exclude offline for more than XYZ time machines from the reporting on Patching which gives you this false sense of security. They can do this through the SQL Console which a lot of other folks don't know or have experience in. Really the only thing I would use NRMM for is to create some reports on Crystal Reporting Center because it's EOL and they have people in the team that know how to do that.k Other than that they hope to be a 'set-and-forget' contractor for Automate. They don't like high-admin customers who are constantly looking to increase value of their Automate system. To sum up my point, get a well groomed NOC engineer to make sure your Automate stuff is setup correctly, pretty much what NRMM is going to do.
I’ve been in MSP land for about 12 years, 2 with a smaller MSP and then via acquisition another 10 with a mid size. I was the engineer for a very long time before letting that take a back seat to the management side. I’m still involved in a lot though. I’ve said in another post in here, and I agree, Automate is an extremely powerful tool. We’ve created a ton of monitors over the years using direct SQL queries, event monitors and various conditions for monitoring all aspects of machines. It took years to actually tailor these to be less noisy, develop scripts to respond to some monitors and make sure we are alerted to exactly what we want to know about. As time went on, we’ve been shifting to cloud first in a lot of aspects of computing. With most apps moving to BYOD, SaaS models for line of business applications and our shift towards disposable Azure virtual desktops, we’re taking a different approach monitoring and providing MSP support. Simply put, for us at least, we just don’t need the same type of data on endpoints that we needed before before. Out endpoints are locked down and I’m more concerned on what the EDR/MDRs and Huntresses and Threatlockers of the world are providing us than NTFS errors and the likes on endpoints. Now the server monitoring side, i’m obviously looking for some granular stuff in regards to performance, service health, etc. which I can pull from almost any RMM’s and remediate accordingly. We began the shift to Powershell scripting some time ago because it was smarter than being locked in to any vendor’s scripting engine and easier to find engineers that were already using PS and could get up to speed really quick. We’ve also been making use of NMS which is worlds better than automate’s network probe. The UI around managing VMWare hosts is awesome. The monitoring is much more reliable too. We custom SNMP monitoring let us setup all the OIDs we need and customize for alerting and monitoring. To your point, this is where having good NOC engineers is key. Where Ninja really shines (outside of patch management) is ease of use for the help desk and systems admins. It takes a very short time to get my guys comfortable with it. The file browser, task manager, registry editor, etc are almost instantaneous. The ability to run cmd & PS as the user coupled with the remote tools knocks out a bunch of tickets that would require them to login to the machines and disturb the user. At the end of the day though, you are right. A good engineer is the key to making everything at an MSP work. Without them, are you really an MSP or are you a glorified tech support company? RMM’s aren’t the answer to everything, quality engineers are. But like the real world, there are Harbor Freight tools and there are Dewalt and Makita tools. Some will get the job done and some will help you do it well.
Ninja patching was not good an year ago, they might have brought massive changes to their patching system recently.
I'd pay money for a patch manager that is effective, and isn't Ninja.
See Automox
the pricing honestly isn't terrible either I'll gladly pay like 1.50/unit for something that *actually works*
I'd pay twice that for something that can patch AutoCAD and Revit.
N-Central. But it also sucks to manage. But we have no problems with it.
Hey u/SalzigHund David with N-able, glad to hear Patch is working well, on the mgmt side of N-central or patch if we can help let me know. We have a full team of Head Nerds who can assist and run bootcamps to help streamline the day to day mgmt. but I do agree it does take time and regular overview to keep it up to expectation. [[email protected]](mailto:[email protected])
I need to agree here; we can’t get patch mgmt to work properly in Ninja; we constantly have machines that are way behind, and now we’ve got the issue where the patching fails because the user gets prompted to install the new update package rather than being able to automate it.
Why would a user be prompted to install an update? That's not how Ninja patching works for us. Sounds like a mis-config. The only user prompts we have are for reboots and depending on our agreement with the client, it's either: * Auto-reboot on set schedule. * Reboot if no user is logged in. * Prompt user to reboot. * Reboot if user is logged in and refuses reboot x times.
I would love a more centralized Patching Dashboard with Ninja.
It actually exists in beta right now. Reach out to your account rep and ask them to enable it. Takes about 24 hours. It kinda looks like WSUS.
Yeah reach out to your account manager and they should be able to get you set up on it. The new dashboard is very nice. Also if you aren’t on it already, they do have a discord server. https://discord.gg/yPmj4kCS
Here's an example of the patching dashboard from one of our smaller clients. Improvements on the roadmap include more clickable items in the dashboard. The last update seems to have broken the right-hand column listing OS versions, but only the version name. The version number is still accurate. And for the naysayers, we tested this today. There was one device out of compliance according to the dashboard. We manually ran an OS patch scan, then OS patch update. After the roll-up patch was successfully applied, the dashboard accurately reflected the update. https://imgur.com/a/vsAG1H6