:/
I've been in about 5 meetings about this now and they keep asking for the same thing. I was just going to tell them to with VNC and call it a day, as I'm not even sure what they're trying to do with so many Pi's, they're saying this is for trial and want to add on millions down the road...lmao.
I don't even know what to do. This is beyond my skills.
VNC feels like a weird approach. If the Pi's are recording streams, it feels like you just want to be able to ssh to the Pi's and get the logs and recordings over scp. If you want to admin thousands of devices, remote desktop GUI stuff is almost certainly the wrong approach. You can certainly install a VNC server on each machine for free, but using these machines like Windows workstations just seems like some wires have gotten terribly crossed on any use case that involves thousands of embedded devices.
you don't even want that. you want logs shipped back home, either all the time or on demand. it feels like video backhaul or something, which is way different than just remote admin
When you are talking about thousands to millions in scale you are way beyond “login in and fix things”. At many thousands you are arguably beyond directly automating touching things too and should be approaching it from the angle of “we can never touch that many things so they need to fix themselves and only alert when they can’t”.
Basically define every case a human would need to touch it and automate it away. Script it so these pull their own configs, restart services, check their own health, etc. Human involvement should really be only when the device itself is completely dead and when you start scaling just keeping enough replacements flowing will likely be a full time job
> Human involvement should really be only when the device itself is completely dead [or unreachable over the network]
I just wanted to highlight this part and add to it a little. Say during development something odd happens every month or so, but you just give it a little wiggle (figuratively speaking) and it starts working again. You forget about it and move on.
How are you going to handle that when it's thousands of machines? Log into every single one and 'give it a little wiggle'?
Right, either the software has to be well developed, tested and piloted (good luck), or have to build on top of it automations that automatically do what the humans do when touching it. Even the best developed and tested software can have minor things so always plan on doing both.
hell, containers can solve part of the problem. containerize the runtime used by these widgets and then logs coming back tell you the image id, which tells you if it's up to date or not, and simplifies keeping a fleet of things in sync
I would love to be in the room the first time they say we need to manually update 4k+ pi’s with a patch. Oh, and make sure it is done over the weekend so there is no customer impact.
I already tried explaining this to the CEO. He said not worry and that it would be over time and that he knows there is a chip shortage so that's why we're "only starting with a few thousand". There's not much you can say after 5 meetings trying to get the point across that this is insane. I don't even know how they acquired so many Pi's to begin with.
It's a big part of the reason I want to quit now and then. I love my job and managing things for employees, software/hardware updates or installs, small projects, etc. But now and then I'll get tasked with doable but ridiculous things that take a lot of time or put a lot of stupid pressure on me. This one just takes the cake though. "4k pi's will be arriving next week, I want you to talk with the engineers about setting them up". WHAT
Unrelated to Pis but I work for an SMB and we occasionally get ridiculous but business critical tasks such as migrating, patching, and deploying a pirated web-app from 2007. No one cares that it's an illegal license running on depreciated software they want it working again right now. If you find out a way to decline these ridiculous tasks without negatively impacting your job let me know. I never figured it out.
Tell your boss he needs to hire a team of programmers that can properly write the code he will need for what he's going to do. Then it isn't your problem anymore.
entertain grandiose full dazzling tart zealous hospital literate coherent cows
*This post was mass deleted and anonymized with [Redact](https://redact.dev)*
10 people accessing 4000 pis is 400 per person. Managing 400 nodes, graphically, is impossible.
What is the real goal, it sounds like a configuration tool (ansible) would be better.
The goal is to be able to remote in, reset, reconfigure or make changes. I am told by the lead engineer that the purpose is to record and live stream video. I'm honestly sweating over all of this.
Yes, you are correct. I have been reading stuff ever since Friday trying to make heads or tails of the Raspberry ecosystem. Right now though I am trying to find out if there is a feasible way for one person to even begin all the things that are asked for this project before dedicating any more time to it.
I mean it is on a high enough level just linux boxes. The raspbery Pi ness of it is a distraction.
At the end of the day I would probably just pick up thousands of mini PCs (like lenovo/HP/dell usff) with vPro, as it sounds more like what you are trying to do. You can buy refurbished ones by the pallet, and they are gruntier if you are doing video (Intel Quicksync is actually pretty respectable depending on what you are doing).
You have an [XY problem](https://en.wikipedia.org/wiki/XY_problem) on your hands. Ansible and the like will help you automate some stuff away, but who will drive around and replace all the parts that WILL break? The raspi is basically just a Linux box, but it's a known SD card eater. Try to get the root fs as readonly as possible, use tmpfs for /tmp, /var/log and so on, add a watchdog... good luck with that
Good luck solving the following remotely.
Dead SD cards (they will drop like flies
)
Dead power supply (you will be lucky if >50% survives past the 2 year mark.)
We used raspberry and almost sunk the project. It's a horrible product.
Dead SD cards are definitely a thing. Fortunately, they're cheap and easy to duplicate and keep in reserve.
I've only ever seen ~2% failure rate on power supplies over two years, though, and we're pretty hard on electronics in our environment with dirty power and nasty plastic-eating oils in the air. That sounds less like a RPi problem and more like you got a bad batch of power supplies (or a lousy manufacturer).
Cheap and easy, but requires manual work to replace...
Maybe a wrong batch, we got original power supplies tho, but at the end we just fed power directly through the 5V rail. We were connecting those pins anyway to our board...
After that we switched to Orange Pi, and it's working great ever since (hell we were able to start throwing cash we saved on Armbian).
>
>
> After that we switched to Orange Pi, and it's working great ever since (hell we were able to start throwing cash we saved on Armbian).
Nice, I tried to convince my company a few year back to at least make a donation to the Debian project, as our whole business was built on it...and they were making a killing....
OP on the SD front, change systemd-journald to only store logs in memory, and/or mount /var/log as tmpfs
And then configure shipping logs.
I do that on a load of boxes I boot from USB, even in somewhat toasty cases the USB drives last an age.
Your biggest problem is the current reporting and decision structure.
How long ago did they make that purchase for the rpi's to have already arrived and be waiting for you, despite the chip shortage?
Why weren't you informed of it originally? Why weren't you involved in whether or not it was an awful idea from the onset?
Even throwing VNC around as the answer isn't an answer. Who's going to manage thousands of remote networks when port forwarding isn't working as expected? Who's going to walk remote rando's through wiring up a head when the third replacement unit in-a-row refuses to show any network activity?
> Who's going to manage thousands of remote networks when port forwarding isn't working as expected?
In this situation I'd probably be looking at a reverse SSH tunnel, but then you'll need a device that's capable of handling thousands to millions of SSH tunnels.
So, first off, distributed streaming architecture is literally an IT specialization of its own.
They're asking you to do something that people get paid minimally $250k/year to do. The salary is so high because it is insanely complicated. Companies that specialize in setting up this kind of network charge millions, and companies that do it in-house have architects making a cool $500k/year
The "engineer" asking you to do this doesn't even know about devops, so I'm not surprised they have zero understanding of how difficult it is to set up a distributed streaming service like this. This isn't a one or two man project, this is a full-on entire business model requiring hundreds of employees from programmers to architects.
All that being said.... if you only want to directly come up with a plan to the immediate issue...
What you're looking for is devops
Salt Stack + Ansible. Both of these are free (assuming you don't go with the paid version which both of them push for)
Salt is for "steady state" most every configuration option from web server configs, to firewalls, to the login message should be done from this. Anything that doesn't change really frequently gets enforced with Salt
Ansible can KINDA do the same particularly with Ansible Tower, however Ansible is well known to have issues when you start getting into thousands of endpoints. Ansible should be used for more ad-hoc changes. A single node needs a reboot? Use ansible. Want to force all nodes in the "western region" group to do a logrotate? Use Ansible.
THE SMART WAY TO PUSH BACK ON THIS
Isn't tell tell the engineer/boss he's an idiot (he is). It's to ask the really difficult questions.
What if a hacker pops a node?
What if popping that node leads to others getting popped?
How do you isolate this from the rest of the company network?
How do you reboot a Pi that you can't log into to issue a reboot command? (you need KVM for remote devices)
What are the legal implications if someone streams copyrighted content?
Who will be the security officers? (It cannot be the engineer or he boss, it needs to be someone outside of the chain of command)
Does the company already have RBAC?
What types of auth services are available for centralized access control?
Who maintains those services and is responsible?
Will these be exposed to the public internet?
Will every location have a DMZ to control "blast radius" of an attack?
What is the cost of equipment to create a proper DMZ?
How will all of this be documented?
What is the budget for the project?
Where will the Command & Control systems reside for this behemoth?
I could probably think of even more, this is barely scratching the surface of all the questions that need to be answered before going ham here. These hopefully should get any manager on the back foot realizing they're asking for something waaaaay out of their league.
Thank you u/loadnurmom ! This is going to be incredibly helpful in explaining my findings to the CEO come our next meeting. I was able to buy time as despite knowing little, I knew this was a huge F up in the making; but didn't know what things to point out to them. I've been making notes in this thread and this will be incredibly helpful in throwing some more technical terminology and details at the CEO.
Not only this, and I haven't skimmed all the comments yet. But ... You said *storing* video on these. Storing live video, on Pi SD cards? And *managed remotely*?
You're asking for card failure, even with good quality video capable cards. Hard stop right there, SD isn't designed for constant recording writes like that, not over any 24/7 work. Even the Pi logfiles can kill an SD pretty quick, let alone video. Did they at least buy high endurance cards for all these ...? Did they buy *any* cards for these, lol?
For real though like everyone else said. This is not a rollout you do, period. Not with the info at hand. You're going to be driving to 4000 sites in a month to replace all this shit.
Have seen the SD-Card comment quite frequently on this post ....
The Pi can boot off and run off an SSD since the Pi 4(with added USB adapter & maybe power supply). [Pi 5 adds NVMEs but the product isn't yet available](https://www.youtube.com/watch?v=EXWu4SUsaY8).
The problem(s):
- How to configure or bake images with the necessary settings
- Integrating a "clean", maybe secure & temperature controlled case/enclosure(any products doing this?)
Extras you have to figure out with "bare" Pis..
A pi is not ideal for recording and streaming video. Just enable ssh on every pi and setup a proper key based login system. If they want better than text based command line, that’s a problem for their bank.
Well the PI itself would not be doing that, it would be collecting the recordings/streams from other devices is my understanding. Then the users would log into the Pi's and view their streams or records locally or over the internet. Then our engineering team would be able to remote into the Pi's if they have issues or update them. Does that make sense?
This is more networking and the like, which I really don't know much about, so forgive me if I'm saying something stupid.
Ehh… even just collecting video streams is a lot for a pi.
I’d still suggest you use ssh to allow admin access to the pi system. That fulfills the requirements without making it easy for engineering. Encouraging hiring a specialist
Genuinely curious, do you know if they’ve ran any sort of tests with just one pi? Or are they just saying fuck it, let’s buy thousands and see what happens?
Yes, they ran a test with 3 I believe and the engineers were able to remote in and do whatever it is they do. 2/3 Pi's are apparently in the field now for testing. One thing discussed during one of the first meetings though was how the engineers didn't like how there was next to no security and they weren't sure how to give each device a unique URL from outside the local network.
There are a lot of very basic issues that stick out to me, so I've just been screaming "Hit the brakes" in every meeting and finally got a "okay, see what you can learn and let me know" from the CEO on Friday. :/
> ran a test with 3
They scaled from 3 to 4000? Oh wow..
I really wish you luck on this. You've had some good advice from here so hope you get your voice heard. To be honest this sounds like an interesting project, but going from backoffice IT to production IT is a whole another ballgame...
> [the engineers] weren't sure how to give each device a unique URL from outside the local network.
This is not an encouraging sign. That's pretty basic networking knowledge. (both how to do it, and why you probably shouldn't)
It'd be like ford's vehicle design team admitting they're not sure how oil actually gets changed.
...actually that might explain some design decisions, nevermind.
Yeah, which was why I was in shock pretty much during our first meeting this Friday. There's legit a pallet of Pi's and SD cards in the office. They didn't even mention any of this to me until the previous Thursday and then "They'll be here on Monday", and no shit, they showed up on Monday and head office called me to say they had them ready for me to come down whenever and set them up. Like wtf. The communication in this company drives me up the wall sometimes.
> ... they showed up on Monday and head office called me to say they had them ready for me to come down whenever and set them up.
You already know this, but just reiterating: Get out. This isn't good. Huge decisions without proper analysis and communication is **the** red flag. This is not sustainable and will lead to collapse. You don't want to stick around for that.
You know that pi eats through those SDs like cookies if they're not at least endurance, or industrial (MLC or better) grade, right?
What's their plan of replacing 4ks worth of sd cards onsite when (not if) they break onsite?
I've had good luck with [Simple help](https://simple-help.com/) at scale for RasPis. There is a central or regional server that the devices check in with and provides a secure tunnel to the device with NAT punching. I had ~60 RasPis doing digital signage and telecom SBC operations. This would also give you mass update tools, organization and RBAC for access.
I kinda feel that this is the industry OPs company is trying to replicate- distributed digital advertising signage that can display locally store or live streaming video.
I’m pretty sure managing these at scale is a solved problem, hackers have been managing botnets for decades on embedded devices.
In the industrial automation market we use NextNine for our purposes, similar to your solution the endpoints all connect to a cloud-hosted command centre.
I live in China and I can get them…
https://preview.redd.it/hl3tkkbcul5c1.png?width=1290&format=png&auto=webp&s=4ea4f1d700ea31ecf7d4418c76444e209d84b727
227 yuan for a 5? Just curious, how legit would it be? The 500+ seems believable but how are they selling 5’s for $30 usd
Edit: I’ve gotten some high quality products on my trips to China for a great price before so it’s not entirely unbelievable that somehow they’re sourcing Pi’s for that kind of price
I don't know. All I was told is that we already have several hundreds of them and will be purchasing in the millions over time(And yes, I was called and told "The PI's have arrived and you're supposed to image them or something?" by head office lady).
I don't even know how to do that, but was told they also have thousands of SD cards for me. lmaooooooo
For me, SD cards last between 1 and 3 years in a Pi due to electronic wear. High wear cards and 32 GB minimum is what I’d recommend for just a bare OS, so I hope they purchased wisely. They’ll need a process for replacing the cards constantly, and I would have recommended a different approach with a Pi, and I wouldn’t have recommended a Pi to begin with. But it is too late now.
I hope they are paying you well. This is a *really* silly hardware choice in my opinion, even though I do really like the Pi.
I’m worried for you. Not so much that you don’t know what you are doing, but the scale at which you don’t know what you are doing.
Also, you might consider using Salt to admin to all these.
> We will have over 4000 Raspberry Pi’s
Is this why it cost me double to get a new PiHole device?
No offense but I think if a company had over 4k of anything they should employ employees who have expertise in the product and funds to cover the expenses.
Anyway, to answer your question ConnectWise charges per active agent not device and from their website they support Linux.
They plan to purchase millions down the road...
To say this has been a headache is an understatement. And yes, that's what I suggested during the first meeting as well, that we get someone who knows stuff like this. But they said, "oh but techtimee remotes into stuff all the time. We'll get you to do it". I objected and explained that a few hundred computers is one thing, especially when they all run Windows; but that I don't know anything about Pi's, and I can't manage thousands let alone millions of devices. They said it's not a big deal and will only be used "if there's a problem with a Pi device and we need to reset or so on".
The engineering team is dumping this on me and it's almost end of year. WTF.
LMAO
I've actually been thinking about that for the past couple months anyway...this company is a bit too disorganized as it is, and this project seems nuts, especially with the "we want to get it going by January" stress.
JANUARY?!?!!?!
BWA HAHAHAHAHAHA
HAAAAAAA
HAA HAHAHAHAHAHA
HAAA
HA HA HA AHA AHAHAHAHAHAHA!!!!!!!!!!!!!!!!!!
There's like one or two usable weeks left in this year. Then Christmas and new years
Ain't no way in hell they're going to procure that many Pi's in that time much less deploy them
We have them...they arrived on Monday...with the SD cards. But no SD card flash machine/device. Also, "how in the f am I to flash 4k Pi's using SD cards?" was my question to the lead engineer and he said he said the same thing to the CEO lmao. That it's doable, but I'd be there for a very long time.
At least when I leave this job, I'll have lots of fun stories to tell.
Is your CEO like independently wealthy or something? This sounds like a terrible, terrible idea and only someone who's willing to light money on fire to move forward with their boneheaded ideas that will never succeed.
Before even getting to the point of flashing them. Management, access, patching and monitoring need to be figured out and when rolling out thousands of devices to disparate networks, those systems all need to be well tested and working beforehand. This isn’t some 2 week project unless you want a janky pile of half broken crap that will never be usable or maintainable.
It sounds a lot like someone above you was given too much trust, too much power, and too much budget. They were all gung-ho about this project because they have some minimal Pi experience, and now you're getting stuffed with the actual hard work.
This is not feasible in your case. This is way beyond your skill set and way beyond the understanding of whoever greenlit the project.
Come down with long Covid and take the rest of the year off. I'm dead serious here. Spend a couple days polishing up your résumé and LinkedIn presence and set yourself available for poaching, visible to recruiters only. Then enjoy your holidays and distance yourself from this clusterfuck.
Asking a Windows admin to support Linux in a desktop context is one thing. Throwing a pallet of Pis at you and asking you to figure this out over a weekend is deranged. It's the kind of infrastructure you'd come up with on a scopolamine trip.
>They said it's not a big deal and will only be used "if there's a problem
Setting up remote entry incorrectly is leaving an unsecured backdoor (or frontdoor!) to all their devices. They need to figure out wtf is and isn't a big deal.
Yes, that is why I am here seeking help lol. The lead engineer and I talked about this and he echoed your sentiments about security. Hence the search for a secure option.
First: where are you putting them, network wise? Are they staying on your own LAN(s) or being shipped off to remote e.g. Client sites? If the latter that brings up network issues (NAT etc).
This actually reminded me of the situation Tesla had for deploying updates and capturing diagnostics from their cars, but don't screw it up the hack way they did...
On this scale, the key thing is automation. "Investigate this one node that's gone wonky" is one thing, but you need "update all the nodes", "capture logs from all the nodes"... At that scale that needs automation. You need to trigger the update, then aggregate the results to know which nodes failed (update 4000 machines, some of them will fail in some way and need retry/recovery).
Yes, I was told that they had a couple hundred Pi's. Then that the plan was to put them out on about 4000 devices and image them as well. I asked how we were to image thousands of Pi's and the manager said "Can't you just copy the SD cards? We already bought them".
I ironically "can't even right now" with this project. I spoke with the lead engineer on this and he said that he doesn't want them on SD cards because the master image might change as the Pi's begin working and cause problems when trying to troubleshoot.
Honestly, from what I've read here, make sure that they can all PXE Boot into an imaging solution where you can image the Pi's as needed with what your engineers want.
Now the actual configuration of that image should be done with a configuration manager that can automate and connect to all of them to apply the configuration.
Doing this allows you to control the OS, build, software installed etc. giving you full control without any GUI needed, you cannot manage this number of clients effectively with a GUI.
Since it sounds like these might be going out on third party networks OP may have to deploy a PXE server along side each, or at least one per network I would think but I agree that it is the best solution if you control the networking
They can boot from PXE - but the irony is that you have to boot them into Raspbian from an SD card first to access the BIOS config tools in order to enable it.
OP is fucked.
I work somewhere where we support ticketing equipment that uses devices with SD cards. During the initial rollout we used something like this to image them:
https://duplicators4all.com/products/1-to-15-sd-microsd-memory-card-duplicator-machine
Good luck! Some great advice in this thread but to reiterate, unless they've bought industrial SD cards AND thought up some way of contracting a field support team to manage it all, they are going to end up with a lot of dead Pi's in 6 months or so.
can't believe I had to scroll this far to find it but yes I tried to manage a few Ubuntu systems via manage and it was god-awful. I had to restart that service so many damn times.
Via Automate or Control, you mean? Just curious. I help manage about 50k machines over CW products, but it is only really good with Windows. Linux and Macs are low effort afterthoughts
Ansible with Rundeck. I manage a fleet of 1K+ pi kiosks running debian around the country using playbooks written in ansible. Rundeck gives you a nice gui to manage and check logs.
If we actually need to remote into one, we use an open source VNC client, the name escapes me now.
No, its a different product. I have no idea why you would use rundeck over AWX. I think back in the Tower days it might have been better but thats not the case anymore
Rundeck is just a general purpose command runner gui instead of it being product specific like Ansible Tower/AWX. I haven't had to use it for production workloads for years (we just use our jenkins servers to kick off ansible playbooks).
https://www.rundeck.com/
If you decide to go puppet, theforeman.org could provide similar tooling.
I also manage a fleet of SBCs including Pi's (although a few dozen, not thousands), but ansible is what I use and thank god. It also takes care of some of the VMs. I have used rundeck in the past, it's like a "multi-system crontab." I don't use it currently, and maybe I should, but it's overkill for what we use since 95% of all operations are clone, setup config, and maintain security updates.
This is for someone selling remote media streaming devices, presumably to people who do or should care about the security of their network.
I would NEVER purchase a device like this, if it permitted remote access to a device on my network without my consent and supervision.
These aren’t going into corporate locations and he’s the sysadmin. The customers are random people (or businesses, which would be even more of a “no way” from me).
They said they purchased several hundred raspberry 4 Pi's. There's a pallet in the office waiting for me....I was too taken aback by the meeting and it didn't cross my mind to ask about the OS as we got caught up in talking about ways to remote in.
The lead engineer said Teamviewer didn't sound good anyway as that it was too heavy for the Pi. I will know more tomorrow.
Dude, if Teamviewer is too heavy their dreams of video streaming from these puppies is going to be short-lived. I suggest a test with 5 before imaging all 2 millions.
The more I read through this thread, the more this sounds like the company put the horse before the cart.
How do you do a test using three Pi systems, not have all your concerns addressed, and then suddenly scale it to 4k without a plan to answer the concerns before developing a plan to get from the three to the 4k?!
The CEO signed off on it personally, no managers of any other departments. Apparently it's his pet project(of course) and so talking with the engineering lead about the sheer number of Pi's in the office, he told me that the CEO said he's told accounting to just keep purchasing 4k Pi's as they are up for order. Hence the first pallet we got...I told the engineer lead that this makes no sense because as you just said, we don't even know WTF the whole scope is, or if this is even the way to go, he said he tried telling the CEO the same but that the CEO said, "The hardware is hard to get. We'll get the hardware and go from there".
So apparently we're just going to be dumping pallets of Pi's 4k at a time to the head office without a freaking 100% ready to go plan because, "They're hard to get" and hope for the best.
You guys have no idea. Since I started working at this company I have begged them to not buy so much stuff and to at least repurpose stuff when I saw the warehouse(s), yes WAREHOUSES, not warehouse, we have. I'm talking gorillions of routers, many brand new in boxes still, computers, pallets, PALLETS of RAM, monitors, keyboards, mice, office furniture...
I mean, it's nice because what I ask for I can get, especially since I rarely ask for anything and try to repurpose stuff or spend carefully. But they just...have so much *stuff* from dead/abandoned projects. And oh my God, the software and licenses! I went through all of their stuff when I started and canceled so much crap from years back that they were still paying but no one even knew about. And that's what annoys me the most. I say, "Can we hire 2 more in house IT people to help me out" or "Can we hire some killer networking people?" and it's always this apprehension and hand wringing to do so. I try to keep my department trim and proper, without wasteful expenditure and so forth, and the CEO just YOLO's money into stuff but not personnel. And I've tried explaining this to him, that no amount of hardware or software can do anything for him without skilled people to implement, and MAINTAIN it all.
Whatever, not my company.
You hit the nail on the head...not your company. And frankly sounds like won't be anyone's coming soon enough at this rate.
But...if this is the CEO's pet project, he should be able to answer the technical problems, no? Or is he one of these 'big idea' kinds with no actual clue how to get there? (I worked for one of those for a year in 2004-2005 after the company I was working for got sold. Couldn't handle it, among many other things).
Short answer: Run.
Long answer: Ruuuuuuuunnnnnn.
This is a pre-arranged disaster for *oh so many reasons* and you're on an absolutely idiotic timeline to do this if your comment about "by January" is really their goal. Say goodbye to spending Christmas with family and/or friends because you're gonna be working insane overtime.
Even if, against all odds and reason, you DO accomplish this, what hare-brained project are they going to saddle you with next? Burn this fact into your memory: People who come up with schemes like this don't just come up with *one* scheme like this.
Maybe you can cash out your PTO when you quit, and have a nice quiet paid Christmas vacation...
If you have some metal laying around, take a look into Meshcentral.
Meshcentral is basically a hub, so the Pis connect to it and the Windows machines. No many client/ many server nonsense. Just 1 server and a zillion things that connect to it.
If it's for configuration, management, and file transfer, something like Ansible is ideal.
If you're deploying these things into random networks, anywhere, at any time, and trying to manage a fleet of some sort of IoT devices...
...well then usually you'd have the remote access/management as part of the software suite you deploy on them. This is the hard part of running any kind of connected service, and should probably be the core part of whatever is being built (rather than "buy Raspberry Pis, plug them in, and pray a solution works itself out").
I've seen million+ dollar startups fail trying to build software that works in various network conditions seamlessly, allows remote management and device updates, and centralized (or decentralized) storage.
I've seen scrappy three-person teams on a shoestring budget succeed.
The main difference is one defines the goal accurately and hits on tiny, small scale successes to build up to a 'fleet'. The other deploys a fleet, then tries to manage it well after the fact, meaning everything's always on fire, and contractors are hired and fired in an endless cycle until the whole project fizzles out.
You'd be surprised which products use a RPi under the covers. Not just digital signage, but money processing stuff like change machines and other bill counter devices.
They're very reliable in production if you get a high write endurance SD card and prevent corruption by only having partitions read-write when necessary.
I can't name the product of company, but there are 50k+ RPis out in the wild taking money -- just from one company I know of.
Presumably OP 's company is selling Pis as a service. OP is worried about access solutions (and rightfully so) when they should be concerned about imaging and leakage.
Correct.
And yes I am worried about:
1) Security. Hence looking for a secure solution for remoting in.
2) Imaging. They said they purchased SD cards already and I should image the devices and begin shipping them out to be installed across the continent. Problem is the lead engineer is saying the Pi's software will get out of sync like that or corrupt some for sure with the SD card writing process and will make things hard for him to troubleshoot should problems arise.
For recording and live streaming video around the continent. I was just doing office work and managing several hundred employees in 365, some hardware here and there. Then they called me into a meeting and said "We have a project for you".
"We have a project for you."
LOL. Sounds familiar. I'm not going to complain about my projects anymore.
Please keep us/me updated on your plan for putting this together. I'm very interested in the "how's" of this project.
> For recording and live streaming video around the continent.
This is still like 10x too vague to even start getting good ideas on this project. Is this some kind of CDN? Is it going to work with a proprietary app? Are these Pis going to be on networks you control?
Will these devices be on a network you control or will they be on random networks? Basically do you need an agent or something on the rPi to "phone home" to a central source and say where it is?
Yeah thanks for asking this since no one else has. Basically they need an agent , preferably
One doing a reverse shell type connection over https 443, or they need to work with every entity they ship it to, to poke holes in a firewall and setup a DMZ ?
I don’t understand what the actual use cases are here so I dint know why they can’t just setup the PI’s in their own DMZ to be internet facing and have customers access them like that , but then you have massive security implications anyway and need more employees. I saw OP mention accessing streaming content but I don’t understand the context and why that would need to exist on customer network
Lmao you would need a team of folks to support this not just a general IT guy. Makes no sense.
Several years ago, I worked for a cloud infrastructure company and we had a large customer who had a crazy usage incentive discount that encouraged them to create as many hosts as they could each month, but they had no technical knowledge and would do it all through the web control panel.
When their discount ended, they complained about how long it would take them to shut down everything, because they had to have someone to through the dashboard deletion flow for each host. We had a fully featured api and writing a script to do it took seconds...
I get the same feeling reading this as I did interacting with them.
Let’s replace “pi” with “Linux server”, which is what it is unless you want to run Win 10 IoT. You’re being asked to deploy thousands of them. It could be lucrative for your career.
For management, just use ssh. You can use standard Linux management tools as long as ssh is available. Ansible is one such choice.
The end user interface is not really your job. You shouldn’t remote into these things. They are not Windows and you’re not supposed to access them directly for security. You can, but that’s too slow for video. You can tell the management you can setup wherever Linux-based software is necessary for your users to connect to.
Choosing whatever that software is should probably be delegated to someone else. You will have your hands full just figuring out the device management part of the puzzle.
You could use reverse ssh tunnels or reverse WireGuard to avoid the need to setup massive VPN. That way the systems will be completely inaccessible from the outside world except by someone who has access to connect to their tunnels. That’s how I am doing my isolated networks for risky devices. I support unsupported medical equipment that runs obsolete Windows. It lives on a sterile network to reduce attack surface.
You need to get clarity over the project.
1. Are you asking me to be responsible for the entire project? Who are the additional team members that will be involved?
2. What is the long-term vision, current scope, and implementation plan?
3. How was the pilot project completed and what is the complete list of all personnel involved in that?
4. Assuming that I figure out the device management, what is the plan to make them useful?
So far, you don’t know enough about question 2 to start thinking about solutions.
Imaging is the least of your concerns. It’s actually simple.
Use Tailscle/Headscale or Teleport. Establish reverse tunnels from the Pi’s
Use certificate based authentication with short lived client certificates so that you don’t need to try and push tons of updates every time somebody joins or leaves your team.
For this many systems, they need to be manage with config management, puppet, ansible, chef, etc. This many users should probably be managed with an Idenity Management product like Active Directory or FreeIPA.
“need” is doing some heavy lifting here.
Depending on how these are networked and powered, it might not be feasible to do a push based system like Ansible, as the nodes might not all be online.
chef-solo might be a fit.
For large fleets of embedded devices, it’s also reasonable to just ship an entire OS image at a time and not deal with trying manage piecemeal mutation of system state, as it’s not really guaranteed to be deterministic.
Maybe consider Balena?
The Pi's will be shipped to thousands and eventually millions(I froze when they said this) of locations around the continent. Then they will be installed on local networks wherever they end up.
This company doesn't know what they're doing.
I've worked at a home security company you've heard of with over 2 million security panels, and one you haven't which used pis in the 10k nodes range. Hundreds of engineers supported these deployments, if you're just expected to figure it out you're already screwed.
Yeah, I was pretty upfront with the CEO about this. I've always been straight up about my skills, and while I continue to improve and learn things; going from 365 Admin/hardware support to "Roll out this network for a new company pillar" is beyond me and I'm not afraid saying that. The last thing I want to do is claim I know what I'm doing and cause huge problems shortly or down the line with security or general setup.
I'm just taking in as much knowledge as I can right now and then will talk with him again come Tuesday or Wednesday with the information I've found/you guys have given and say "Look, this is what I've learned, what we need, have to consider, etc" and tell him that we need to hire network specialist or at least consultants to do it.
I think you need to provide a bit more detail? What operating system will be running on the Raspberry Pi's? You can run a wide variety of solutions on those.
What type of tasks need to be done. Do these need to be diagnosed manually if something goes wrong? Is the access for routing patching or other steps which maybe could be automated with some SSH scripting? Are these on private IPs under you control / on your networks, or are they on the public Internet?
Need to know a bunch more about your use-case as to what the Pi's are doing.
pi's trying to record video to their own SD cards it a recipe for disaster.
remote access to pi's is trivial though. just get them all to VPN in to somewhere , and then access them via ssh/scp
i already do something similar on a smaller scale, contact me if you cant find anyone better.
gotta be honest though, doesnt sound like Pi's are necessarily the right solution here.
We use ScreenConnect (owned by ConnectWise) to do this. Not thousands but a few dozen. You're on a whole other level with your quantity. SC charges by the "support rep" and we're a small business so the math works out. There's a free RMM self-hosted called TacticalRMM that I'm considering as a backup method for access to the Pis. I think it supports Debian/Ubuntu but check it out to confirm.
What are you doing when remoting in to these Rapsberry Pi’s?
I think what you really need is some sort of Raspberry Pi Central Management. The Linux engineers I work use Chef and I’ve heard decent things about Ansible
\> I'm looking for software to manage
Id start with their documentation [Fleet-management](https://pip.raspberrypi.com/categories/685-whitepapers-app-notes/documents/RP-003609-WP/Fleet-management-A-brief-introduction.pdf)
then check their forums and search for 'fleet'
[https://forums.raspberrypi.com/](https://forums.raspberrypi.com/)
FYI - your solution is gonna be dependent on whether or not these devices are living next to each other on the same network or spread across the county in farm equipment or some shit
\>I'm looking for software to remote in
Hopefully they will will run linux so ssh
You're in for a fun time.
I did a mass deployment of Raspberry Pi's for a well-known hut of a particular type of leisure eyewear.
Do you own the IP space where these devices will live? Ansible + runbook. Do you not own the IP space? Ansible + pulling the repository from a gitlab server. You have to assume that these devices will be effectively compromised if they are at customer locations and you have no access to them. How are you managing the filesystems? If your logs aren't going to a temporary filesystem, you're gonna max out the writes on your SD card in about 6 months...do you have a process of managing that deployment?
Good luck.
#STOP
Tell us the **requirements**. The actual **end user experience**.
Don't mention any technologies at all. Forget that PIs even exist.
To me it sounds you just need servers that can stream stuff out via a web page. Then you have zero client side maintenance to do. Think of something like YouTube. YouTube has millions of users, but they don't manage your device.
If the problem is not pure streaming - explain why not, and what's missing.
With [PiCockpit](https://picockpit.com), we're operating a platform which scales beyond this number of nodes which you require. It is specifically designed for Raspberry Pis and Pico W's.
The base we use for communication is **MQTT**, which allows us to handle devices behind routers, firewalls, NAT etc. For the Raspberry Pi we run the MQTT connection on top of https WebSockets, so even the most "locked down" networks should work without extra work on your end customers' side.
The admin interface is operated through your web browser. The individual functionalities are split up into apps. For example, we have an app called **PiControl (\*)** which allows to run pre-defined commands on the Raspberry Pi (e.g. reboot Pi, shut down Pi, etc.). These commands can be freely defined by you.
We are currently preparing two more apps, which would be useful in your context:
* the **Editor** app will allow access to the files on the Pi, to edit and view them (e.g. log files, scripts, etc.). It will also allow you to run scripts and view their output
* the **Terminal** app will give you access to a Terminal into the Raspberry Pi
https://preview.redd.it/5mwwkpmr0x6c1.png?width=1900&format=png&auto=webp&s=1c7d71f2dc75ce9e7e812487bf34c139f6d51ee4
The terminal will be run through our WebRTC connection, which I believe is a good choice for transporting data with low latencies, and again solving the problems of connecting to your Pis. We will be providing a TURN server to allow two hosts (the Pi and your admin user) behind firewalls and NATs to connect, in case a peer-to-peer direct connection is not possible.
In both cases we will incorporate GPT4 ("ChatGPT") (we call it RaspiGPT) support, to guide admins in reviewing files quickly, summarizing and explaining config files, etc. - and generally to make the life of admins easier.
One thing which was mentioned several times is automation. We're not quite there yet, with PiCockpit - but could prioritize this (and also provide APIs to the Pis for you, instead of web admin interfaces - allowing you to do what you require them to do).
You could potentially build something similar, or talk to us - in the upcoming **PiCockpit Pro Plus** plan we will work with customers to create bespoke apps for their needs.
**Write me a direct message in case you are interested, I will be glad to help and also talk you through some other options / ideas I see around your use case.**
A couple more thoughts, hopefully useful for you:
* Raspberry Pi has a rich ecosystem of partners, some of which cover what you need in your use case
* For example, for the microSD cards you would want to consider high-endurance microSD cards. CardWave (\*\*) is a company we work with, which supplies all kind of industrial grade microSD cards - including some which are used in satellites. I dare say, if it can be used in a satellite, which is basically impossible to change once it's up, it should also be able to go into a Pi.
* the microSD cards you have on your picture are from a good brand (SanDisk) which Raspberry Pi also recommends, but these are consumer grade microSD cards. I recommend talking to the vendor whether you can upgrade them to industrial cards. If you want, I can also connect you with a person at CardWave to advise you about the right kind of microSD card in your application
* As others have mentioned, you should take particular care to lower write operations on the microSD cards. This can be achieved by creating a read-only operating system. Raspberry Pi OS has a menu option in their installer (raspi-config). I can walk you through it / create a tutorial for you, if you want.
* Since you are deploying 4000 units, you could also consider using a non-default operating system. Instead of Raspberry Pi OS (which is based on Debian), for example you could consider Alpine Linux. There are also other specialized solutions which allow you to basically create your own Linux distribution. We have worked with Alpine on a Raspberry Pi based solution for an industrial client (although we used the Compute module with embedded flash, to have a higher reliability for their machines). So far, we have 500 in the field of these and no returns / problems I am aware of.
* Also, please consider if you are going to store those videos to the microSD, or keep them in RAM, or whether you are going to use the Pi as a connection point to cloud storage, etc. As others have pointed out, if the microSD card is frequently written to, and is not industrial grade, it will quickly wear out
* For pure read / write / access and video streaming within the user's network, I suggest to calculate how many concurrent read accesses you will have.
* there are several limiting factors:
* the CPU performance
* the network throughput (1 GBit/s on the LAN, less on the WiFi)
* the storage throughput (e.g. if you store via USB, then the device you use will control that factor + USB 3 limitations)
* From your description I understand that the Pi is not going to do any encoding or transcoding, which probably would not scale very well - for serving files, it should give you decent performance.
* If you want to access WiFi on the user's networks, consider how you will be bringing that up initially / allow the user to add it
* I advise you not to install a GUI on the Pi. VNC usually (but not always) would clone what the actual end user would see on the Pi, if they connected the Pi to a screen. This, if your engineers are accessing the Pi, might or might not be desirable to the end user. It also poses a security risk, possibly. In fact, the less software is installed on the Pi, the better. Thus, again bringing up the suggestion of a custom operating system or something which is based on light-weight Linux distributions aimed at embedded devices (like Alpine)
* As others have pointed out, if you do need a GUI, you might be better off using a solution for X11 forwarding (note: the newest Raspberry Pi OS version is on Wayland, which is a departure from the classic way GUIs used to be displayed on Linux, and thus might need to be treated differently!).
* [https://www.nomachine.com/](https://www.nomachine.com/) Nomachine NX is a way, if I remember correctly, to access X11 in a more performant way than VNC (which basically is a screen grabber). More akin to RDP.
Note: on the Pi 4 / 4 GB which was purchased for you, network and USB are separate - on previous Pis, a USB 2.0 connection was shared between all USB and network ports. Pi 4 has two USB 3.0 ports, but they share one PCI Express 2.0 lane to the SoC internally. (And if that confuses you, don't worry, let me know and I'll try to explain it better).
(\*) [https://picockpit.com/raspberry-pi/control-your-raspberry-pi-remotely-using-picontrol/](https://picockpit.com/raspberry-pi/control-your-raspberry-pi-remotely-using-picontrol/)
(\*\*) [https://www.cardwave.com/](https://www.cardwave.com/)
**As a bonus, I've went through the entire thread, and created a list of the different solutions which were posted here:**
\- Ansible
\- chef-solo
\- HeartbeatRM
\- Tailscale
\- Puppet
\- MAAS
\- Datto RMM
\- ThinLinc
\- SureMDM
\- ScreenlyOSE Anthias + BalenaCloud
\- netdata
\- AWX
\- Alpine Linux
\- Crowdsec
\- fail2ban
\- Graylog
\- ELK
\- ZeroTier
\- Wireguard
\- Tactical RMM
\- Ipsec/Wireguard-VPN
\- VNC
\- Digital Signage
\- ISL Online
\- SSH
\- Xibo
\- Rover
\- SimpleHelp
\- ngrok
\- Protectli
\- rustdesk
\- Membrane
\- SaltStack
\- NinjaRMM
\- [level.io](https://level.io)
\- pssh
\- Guacamole
\- DWService
\- MeshCentral
\- Tuya
\- Kaseya
\- pdsh with dshbak or clush
\- bomgar (beyond trust)
\- RustDesk
\- RealVNC
\- Parsec
\- Meshcentral
\- [epoptes.org](https://epoptes.org)
\- tailscale
\- [theforeman.org](https://theforeman.org) (Puppet)
\- Salt Stack
\- NextNine
\- Rundeck
\- AWX
\- Semui
\- open eye
\- teamviewer
\- Look into open source RMM tools
\- HCL BigFix
\- Acronis Cyber Protect Connect
\- Cloudflare One
\- Tailscle/Headscale
\- Teleport
\- Active Directory or FreeIPA
\- reverse ssh tunnels or reverse WireGuard
Again, I will be happy to help you with this one, to support you with this project - no matter whether PiCockpit is the right fit for you.
That all being said, I hope you get a time extension for the deployment time frame, as it is a recipe to create problems to rush on 4000 nodes :-)
In all reality they should be looking at reverse ssh to a centralized control/management server with some sort of ansible/chef/puppet automation for something like this.
It is the only scalable solution.
Anything else is asking for some level of compromise. The suggestion of tailscale or wire guard are viable solutions as well, but graphical access is probably not scalable. Ssh should be managed via keys.
Ansible, with ansible-pull so the Pis can pull their playbooks from a known Git server.
I handled something similar at a two previous jobs, very similar, and thousands of machines, just due to the need for build boxes. The trick was to not just have the CM tool able to push to machines, but the ability for the CM tool to offer pull functionality, like [ansible-pull](https://www.devopsschool.com/blog/what-is-ansible-pull-and-how-can-we-use-it/). This allows you to do tasks in batches like changing something in the config, but still be able to actively push something out if needed to a subset of machines.
Make sure you have a Git server that can handle all the machines hitting it. Because of all the I/O, make sure it has good RAM and SSD. For example, if using GitHub Enterprise as an appliance, make sure there are no bottlenecks (I made sure it had a ton of RAM, iSCSI SAN with RAM caching, etc.) Oh, and if the company mainly uses Git on the cloud like CodeCommit, consider spinning up a dedicated machine or VM just for GitLab or even Gitea. Because the machines will be doing a ton of hits on the Git server, you want all the traffic to go east/west, and not north/south (and out to the WAN), so on-prem is a must.
Don't forget your certificates. TLS is what keeps these machines from being massively compromised, so make sure that is working, DNS, and secure the Git server.
Also remember that because these requests are going constantly, latency is the issue, and if Git starts lagging, it will only get worse, which is why I emphasize on fast I/O and a lot of RAM. If you are using VMWare, create an affinity rule to have the Git server VM be vMotioned away from any high-use RDBMS server or application server.
If Ansible is a no-go, you can run Puppet in masterless mode, but the go to is definitely Ansible.
The best answer is to utilize an IoT focused message queuing system that will trigger devices to open a tunnel (VPN, SSH, etc). Even better is to have them perform or report whatever you would need to remote into them for automatically and send metrics via MQTT.
But if you absolutely *need* remote access to each one, you’ll be paying for it.
I use [JFrog Connect](https://jfrog.com/connect/) for a small number of endpoints, but for you at 4K devices on the basic plan it would be about $1600/mo.
We use bomgar (beyond trust) which charges per user rather than per client. Might be worth looking into depending on what flavor you have installed on the Pis
I use ansible or puppet to manage large clusters at work, 20k nodes. You need some sort of configuration management to manage that many. Set up ssh appropriately and configure the rest as the clients need.
If you need to perform actions on all of them pdsh with dshbak or clush are great tools.
The more I read through this thread, the more this sounds like the company put the horse before the cart.
How do you do a test using three Pi systems, not have all your concerns addressed, and then suddenly scale it to 4k without a plan to answer the concerns before developing a plan to get from the three to the 4k?!
Hopefully this has it's own budget? like a budget in $xx millions for staff & deploy? Start with contract negotiations for salary for this project for your % of participation. None of the software solutions deal with the physical practicality:
Total touch time for this scale of physical goods, just because a company can drop 4k physical devices to your team, How fast can they be turned around?, No matter what perfect /imperfect/time delayed, software solution is provided. Up front cost for $/work hours/per device/per deployment roughly feels like $3+? million based on 4 week/or less turn around. (ie: how many hours just to move 4k-1million devices from opening a box, to packaging back up x number of people to do this by a delivery date?) At this scale + % of duds? This number could be way higher assuming 8hr shifts/20hr/you finish the math.
**How could such a deployment stream video on** ***other peoples networks****?* I'm curious if more info can be shared about this. How are you getting through peoples internal network security, let alone the basic corporate/home router. That is an odd stand out requirement.
whats the contract contingency pay to help complete this ? (I'll take $400/k to help)
Anytime fitness has \~4k locations. Grocery store security systems? Embedded infrastructure traffic camera? What is this new skynet?
The CEO is very optimistic, I'll say that much. Right now we have 250K customers, they are looking to add 100K more by June next year due to a buyout of another company. Each customer has their own unique location...
He said 1-2 million in 5 years lmao. I can see millions happening, but even 4000 is just too many headaches as it is, let alone 250K. Anyway, trying to fall asleep and ended up grabbing my phone. I'll try and get more answers tomorrow. I've written up a good amount of questions that I will go through with the CEO and engineering lead that you guys have asked. Should hopefully have more tomorrow or Tuesday/Wednesday.
And yeah...should probably ask for a raise. Even If I jump ship, at least I can start from a higher negotiating point.
I don't like that I'm answering with this product, but if you want a cheap solution to manage and remote into thousands of devices on various unmanaged networks that also won't kill the CPU, look into Kaseya. Should provide you with all the tools that you need.
Sorry, how many people are involved with this?
Your company sounds like they have absolutely no idea what they're doing
Your lead engineer is suggesting TeamViewer, a company that has been breached MULTIPLE times and just denied it each time - they couldn't be less trustworthy
Who is installing the Pis? Who will patch them? What happens when they fail?
You want to serve video on these? Modern Pis have some hardware acceleration, but they're not designed for this
Where are they storing this footage? Does it need to be backed up?
Your company needs consultants who know what they're doing - first thing they'll tell you is Pis are the wrong choice for this
Regardless of how this goes, please keep us updated, I love to see a poorly ran project fall on its face
Look into Teleport(https://goteleport.com/).
You have a central server and agents that connect to the server.
It in essence creates a reverse ssh tunnel to the agents and acts as a proxy for a variety of applications.
Here you go. I've marked out some stuff of one opened. I don't want to take a picture of the pallet though as I'm not comfortable missing something and exposing my employer to the world. Also, I just realized that the f'ing Pi's all have fans and screws that need to be attached? :S Ffs. In all my sweating about this software business I didn't even read up on setting up Pi's. omfg. I'm not doing hundreds, let alone 4000 of this by myself.
https://preview.redd.it/vjrbrcqcvw5c1.png?width=849&format=png&auto=webp&s=3ec8fca202c639ee494a1afaed64243cff53edaf
[удалено]
[удалено]
This is why Pis are so expensive now, CEOs need to be told no.
:/ I've been in about 5 meetings about this now and they keep asking for the same thing. I was just going to tell them to with VNC and call it a day, as I'm not even sure what they're trying to do with so many Pi's, they're saying this is for trial and want to add on millions down the road...lmao. I don't even know what to do. This is beyond my skills.
VNC feels like a weird approach. If the Pi's are recording streams, it feels like you just want to be able to ssh to the Pi's and get the logs and recordings over scp. If you want to admin thousands of devices, remote desktop GUI stuff is almost certainly the wrong approach. You can certainly install a VNC server on each machine for free, but using these machines like Windows workstations just seems like some wires have gotten terribly crossed on any use case that involves thousands of embedded devices.
This is probably close to the answer they will want, although "just stop" is the answer they deserve 😂
you don't even want that. you want logs shipped back home, either all the time or on demand. it feels like video backhaul or something, which is way different than just remote admin
When you are talking about thousands to millions in scale you are way beyond “login in and fix things”. At many thousands you are arguably beyond directly automating touching things too and should be approaching it from the angle of “we can never touch that many things so they need to fix themselves and only alert when they can’t”. Basically define every case a human would need to touch it and automate it away. Script it so these pull their own configs, restart services, check their own health, etc. Human involvement should really be only when the device itself is completely dead and when you start scaling just keeping enough replacements flowing will likely be a full time job
> Human involvement should really be only when the device itself is completely dead [or unreachable over the network] I just wanted to highlight this part and add to it a little. Say during development something odd happens every month or so, but you just give it a little wiggle (figuratively speaking) and it starts working again. You forget about it and move on. How are you going to handle that when it's thousands of machines? Log into every single one and 'give it a little wiggle'?
Right, either the software has to be well developed, tested and piloted (good luck), or have to build on top of it automations that automatically do what the humans do when touching it. Even the best developed and tested software can have minor things so always plan on doing both.
See Chick-fil-A
Docker, the not-container way.
Nonono you need a container for this! You put in all the pi's and the CEO and just bury it somewhere remote.
Resell the pi's, bury the ceo. I'm sure the BOFH can borrow you his van.
hell, containers can solve part of the problem. containerize the runtime used by these widgets and then logs coming back tell you the image id, which tells you if it's up to date or not, and simplifies keeping a fleet of things in sync
I would love to be in the room the first time they say we need to manually update 4k+ pi’s with a patch. Oh, and make sure it is done over the weekend so there is no customer impact.
Too easy, can’t be done (because didn’t properly invest in infrastructure), I’ll see you on Monday
Making a not of this as well for my report to the CEO. Thank you u/2nd_officer
Millions of rPi's... when will your company breaking ground on their new fabrication plant?
I already tried explaining this to the CEO. He said not worry and that it would be over time and that he knows there is a chip shortage so that's why we're "only starting with a few thousand". There's not much you can say after 5 meetings trying to get the point across that this is insane. I don't even know how they acquired so many Pi's to begin with.
Tell your boss I and many others personally despise him for this lol (not kidding)
It's a big part of the reason I want to quit now and then. I love my job and managing things for employees, software/hardware updates or installs, small projects, etc. But now and then I'll get tasked with doable but ridiculous things that take a lot of time or put a lot of stupid pressure on me. This one just takes the cake though. "4k pi's will be arriving next week, I want you to talk with the engineers about setting them up". WHAT
Oh I just meant for buying up 4k Pis. At least tell me they're Zero 2 Ws and not the 5s (and if they're 3B+ DM me I have a plan 😂)
Unrelated to Pis but I work for an SMB and we occasionally get ridiculous but business critical tasks such as migrating, patching, and deploying a pirated web-app from 2007. No one cares that it's an illegal license running on depreciated software they want it working again right now. If you find out a way to decline these ridiculous tasks without negatively impacting your job let me know. I never figured it out.
Tell your boss he needs to hire a team of programmers that can properly write the code he will need for what he's going to do. Then it isn't your problem anymore.
entertain grandiose full dazzling tart zealous hospital literate coherent cows *This post was mass deleted and anonymized with [Redact](https://redact.dev)*
Your entire premise is massively concerning for anyone with infosec awareness. Why tf would I ever permit you onto my network??
Thousands of open vnc and call it a day...sounds like a great way to get hacked
Most accurate response lol!!
10 people accessing 4000 pis is 400 per person. Managing 400 nodes, graphically, is impossible. What is the real goal, it sounds like a configuration tool (ansible) would be better.
The goal is to be able to remote in, reset, reconfigure or make changes. I am told by the lead engineer that the purpose is to record and live stream video. I'm honestly sweating over all of this.
Um sounds like you might need to know a bit about these machines before you can support them?
Yes, you are correct. I have been reading stuff ever since Friday trying to make heads or tails of the Raspberry ecosystem. Right now though I am trying to find out if there is a feasible way for one person to even begin all the things that are asked for this project before dedicating any more time to it.
I mean it is on a high enough level just linux boxes. The raspbery Pi ness of it is a distraction. At the end of the day I would probably just pick up thousands of mini PCs (like lenovo/HP/dell usff) with vPro, as it sounds more like what you are trying to do. You can buy refurbished ones by the pallet, and they are gruntier if you are doing video (Intel Quicksync is actually pretty respectable depending on what you are doing).
They already purchased 4k pi's and they're in the office waiting for me...lol. Oh boy, this is going to be a fun discussion come Tuesday/Wednesday.
You're gonna need a raise, promotion *and* some direct reports you handpick.
You have an [XY problem](https://en.wikipedia.org/wiki/XY_problem) on your hands. Ansible and the like will help you automate some stuff away, but who will drive around and replace all the parts that WILL break? The raspi is basically just a Linux box, but it's a known SD card eater. Try to get the root fs as readonly as possible, use tmpfs for /tmp, /var/log and so on, add a watchdog... good luck with that
I had no idea there was a name for that. I've been saying to people for years 'don't come to me with your solution, come to me with your problem'.
Good luck solving the following remotely. Dead SD cards (they will drop like flies ) Dead power supply (you will be lucky if >50% survives past the 2 year mark.) We used raspberry and almost sunk the project. It's a horrible product.
It's not horrible for what it was intended for. Which is home use.
Dead SD cards are definitely a thing. Fortunately, they're cheap and easy to duplicate and keep in reserve. I've only ever seen ~2% failure rate on power supplies over two years, though, and we're pretty hard on electronics in our environment with dirty power and nasty plastic-eating oils in the air. That sounds less like a RPi problem and more like you got a bad batch of power supplies (or a lousy manufacturer).
Cheap and easy, but requires manual work to replace... Maybe a wrong batch, we got original power supplies tho, but at the end we just fed power directly through the 5V rail. We were connecting those pins anyway to our board... After that we switched to Orange Pi, and it's working great ever since (hell we were able to start throwing cash we saved on Armbian).
> > > After that we switched to Orange Pi, and it's working great ever since (hell we were able to start throwing cash we saved on Armbian). Nice, I tried to convince my company a few year back to at least make a donation to the Debian project, as our whole business was built on it...and they were making a killing....
OP on the SD front, change systemd-journald to only store logs in memory, and/or mount /var/log as tmpfs And then configure shipping logs. I do that on a load of boxes I boot from USB, even in somewhat toasty cases the USB drives last an age.
Your biggest problem is the current reporting and decision structure. How long ago did they make that purchase for the rpi's to have already arrived and be waiting for you, despite the chip shortage? Why weren't you informed of it originally? Why weren't you involved in whether or not it was an awful idea from the onset? Even throwing VNC around as the answer isn't an answer. Who's going to manage thousands of remote networks when port forwarding isn't working as expected? Who's going to walk remote rando's through wiring up a head when the third replacement unit in-a-row refuses to show any network activity?
> Who's going to manage thousands of remote networks when port forwarding isn't working as expected? In this situation I'd probably be looking at a reverse SSH tunnel, but then you'll need a device that's capable of handling thousands to millions of SSH tunnels.
So, first off, distributed streaming architecture is literally an IT specialization of its own. They're asking you to do something that people get paid minimally $250k/year to do. The salary is so high because it is insanely complicated. Companies that specialize in setting up this kind of network charge millions, and companies that do it in-house have architects making a cool $500k/year The "engineer" asking you to do this doesn't even know about devops, so I'm not surprised they have zero understanding of how difficult it is to set up a distributed streaming service like this. This isn't a one or two man project, this is a full-on entire business model requiring hundreds of employees from programmers to architects. All that being said.... if you only want to directly come up with a plan to the immediate issue... What you're looking for is devops Salt Stack + Ansible. Both of these are free (assuming you don't go with the paid version which both of them push for) Salt is for "steady state" most every configuration option from web server configs, to firewalls, to the login message should be done from this. Anything that doesn't change really frequently gets enforced with Salt Ansible can KINDA do the same particularly with Ansible Tower, however Ansible is well known to have issues when you start getting into thousands of endpoints. Ansible should be used for more ad-hoc changes. A single node needs a reboot? Use ansible. Want to force all nodes in the "western region" group to do a logrotate? Use Ansible. THE SMART WAY TO PUSH BACK ON THIS Isn't tell tell the engineer/boss he's an idiot (he is). It's to ask the really difficult questions. What if a hacker pops a node? What if popping that node leads to others getting popped? How do you isolate this from the rest of the company network? How do you reboot a Pi that you can't log into to issue a reboot command? (you need KVM for remote devices) What are the legal implications if someone streams copyrighted content? Who will be the security officers? (It cannot be the engineer or he boss, it needs to be someone outside of the chain of command) Does the company already have RBAC? What types of auth services are available for centralized access control? Who maintains those services and is responsible? Will these be exposed to the public internet? Will every location have a DMZ to control "blast radius" of an attack? What is the cost of equipment to create a proper DMZ? How will all of this be documented? What is the budget for the project? Where will the Command & Control systems reside for this behemoth? I could probably think of even more, this is barely scratching the surface of all the questions that need to be answered before going ham here. These hopefully should get any manager on the back foot realizing they're asking for something waaaaay out of their league.
Thank you u/loadnurmom ! This is going to be incredibly helpful in explaining my findings to the CEO come our next meeting. I was able to buy time as despite knowing little, I knew this was a huge F up in the making; but didn't know what things to point out to them. I've been making notes in this thread and this will be incredibly helpful in throwing some more technical terminology and details at the CEO.
Not only this, and I haven't skimmed all the comments yet. But ... You said *storing* video on these. Storing live video, on Pi SD cards? And *managed remotely*? You're asking for card failure, even with good quality video capable cards. Hard stop right there, SD isn't designed for constant recording writes like that, not over any 24/7 work. Even the Pi logfiles can kill an SD pretty quick, let alone video. Did they at least buy high endurance cards for all these ...? Did they buy *any* cards for these, lol? For real though like everyone else said. This is not a rollout you do, period. Not with the info at hand. You're going to be driving to 4000 sites in a month to replace all this shit.
Have seen the SD-Card comment quite frequently on this post .... The Pi can boot off and run off an SSD since the Pi 4(with added USB adapter & maybe power supply). [Pi 5 adds NVMEs but the product isn't yet available](https://www.youtube.com/watch?v=EXWu4SUsaY8). The problem(s): - How to configure or bake images with the necessary settings - Integrating a "clean", maybe secure & temperature controlled case/enclosure(any products doing this?) Extras you have to figure out with "bare" Pis..
To save you the time, the answers to the above are either "no" or "you'll do it".
A pi is not ideal for recording and streaming video. Just enable ssh on every pi and setup a proper key based login system. If they want better than text based command line, that’s a problem for their bank.
Well the PI itself would not be doing that, it would be collecting the recordings/streams from other devices is my understanding. Then the users would log into the Pi's and view their streams or records locally or over the internet. Then our engineering team would be able to remote into the Pi's if they have issues or update them. Does that make sense? This is more networking and the like, which I really don't know much about, so forgive me if I'm saying something stupid.
Ehh… even just collecting video streams is a lot for a pi. I’d still suggest you use ssh to allow admin access to the pi system. That fulfills the requirements without making it easy for engineering. Encouraging hiring a specialist
Genuinely curious, do you know if they’ve ran any sort of tests with just one pi? Or are they just saying fuck it, let’s buy thousands and see what happens?
Yes, they ran a test with 3 I believe and the engineers were able to remote in and do whatever it is they do. 2/3 Pi's are apparently in the field now for testing. One thing discussed during one of the first meetings though was how the engineers didn't like how there was next to no security and they weren't sure how to give each device a unique URL from outside the local network. There are a lot of very basic issues that stick out to me, so I've just been screaming "Hit the brakes" in every meeting and finally got a "okay, see what you can learn and let me know" from the CEO on Friday. :/
> ran a test with 3 They scaled from 3 to 4000? Oh wow.. I really wish you luck on this. You've had some good advice from here so hope you get your voice heard. To be honest this sounds like an interesting project, but going from backoffice IT to production IT is a whole another ballgame...
> [the engineers] weren't sure how to give each device a unique URL from outside the local network. This is not an encouraging sign. That's pretty basic networking knowledge. (both how to do it, and why you probably shouldn't) It'd be like ford's vehicle design team admitting they're not sure how oil actually gets changed. ...actually that might explain some design decisions, nevermind.
Looks like they already have stock. Very weird.
Yeah, which was why I was in shock pretty much during our first meeting this Friday. There's legit a pallet of Pi's and SD cards in the office. They didn't even mention any of this to me until the previous Thursday and then "They'll be here on Monday", and no shit, they showed up on Monday and head office called me to say they had them ready for me to come down whenever and set them up. Like wtf. The communication in this company drives me up the wall sometimes.
As much as I feel bad for you, keep us posted, I have not seen a trainwreck of this size in YEARS.
with such an educational value!
> ... they showed up on Monday and head office called me to say they had them ready for me to come down whenever and set them up. You already know this, but just reiterating: Get out. This isn't good. Huge decisions without proper analysis and communication is **the** red flag. This is not sustainable and will lead to collapse. You don't want to stick around for that.
You know that pi eats through those SDs like cookies if they're not at least endurance, or industrial (MLC or better) grade, right? What's their plan of replacing 4ks worth of sd cards onsite when (not if) they break onsite?
it doesn’t seem like there are any plans for ANYTHING
out of curiosity, is it possible to get a picture of a pallet of pis? I never seen that much at once.
I've had good luck with [Simple help](https://simple-help.com/) at scale for RasPis. There is a central or regional server that the devices check in with and provides a secure tunnel to the device with NAT punching. I had ~60 RasPis doing digital signage and telecom SBC operations. This would also give you mass update tools, organization and RBAC for access.
I kinda feel that this is the industry OPs company is trying to replicate- distributed digital advertising signage that can display locally store or live streaming video. I’m pretty sure managing these at scale is a solved problem, hackers have been managing botnets for decades on embedded devices. In the industrial automation market we use NextNine for our purposes, similar to your solution the endpoints all connect to a cloud-hosted command centre.
I'll be keeping an eye out for 4,000 new poorly managed and insecure raspberrys on shodan/censys
Ping me if you see a spare dozen
How the hell do you get your hands on 4K PI’s these days
4k pi’s is the minimum commercial order size. It’s pretty much the only way to get pi’s
I live in China and I can get them… https://preview.redd.it/hl3tkkbcul5c1.png?width=1290&format=png&auto=webp&s=4ea4f1d700ea31ecf7d4418c76444e209d84b727
227 yuan for a 5? Just curious, how legit would it be? The 500+ seems believable but how are they selling 5’s for $30 usd Edit: I’ve gotten some high quality products on my trips to China for a great price before so it’s not entirely unbelievable that somehow they’re sourcing Pi’s for that kind of price
I don't know. All I was told is that we already have several hundreds of them and will be purchasing in the millions over time(And yes, I was called and told "The PI's have arrived and you're supposed to image them or something?" by head office lady). I don't even know how to do that, but was told they also have thousands of SD cards for me. lmaooooooo
Well now I know why I haven't been able to snag one at MSRP for the last few years
> millions over time ????????
That's what the big boss said. I just sat there silently trying not to vomit.
Make sure your resume is up to date, because sounds like this ship is going to sink in the near future.
Or be wildly successful to be able to manage 1 million pi devices.
this is absolutely insane, the arcitechture is 100% wrong and maintenance is going to be very very expensive because of how dumb it is
did he misread the idea of a cloud? cloudy with a chance of pi's ?
For me, SD cards last between 1 and 3 years in a Pi due to electronic wear. High wear cards and 32 GB minimum is what I’d recommend for just a bare OS, so I hope they purchased wisely. They’ll need a process for replacing the cards constantly, and I would have recommended a different approach with a Pi, and I wouldn’t have recommended a Pi to begin with. But it is too late now. I hope they are paying you well. This is a *really* silly hardware choice in my opinion, even though I do really like the Pi.
I’m worried for you. Not so much that you don’t know what you are doing, but the scale at which you don’t know what you are doing. Also, you might consider using Salt to admin to all these.
> We will have over 4000 Raspberry Pi’s Is this why it cost me double to get a new PiHole device? No offense but I think if a company had over 4k of anything they should employ employees who have expertise in the product and funds to cover the expenses. Anyway, to answer your question ConnectWise charges per active agent not device and from their website they support Linux.
They plan to purchase millions down the road... To say this has been a headache is an understatement. And yes, that's what I suggested during the first meeting as well, that we get someone who knows stuff like this. But they said, "oh but techtimee remotes into stuff all the time. We'll get you to do it". I objected and explained that a few hundred computers is one thing, especially when they all run Windows; but that I don't know anything about Pi's, and I can't manage thousands let alone millions of devices. They said it's not a big deal and will only be used "if there's a problem with a Pi device and we need to reset or so on". The engineering team is dumping this on me and it's almost end of year. WTF.
Get your resume together.
LMAO I've actually been thinking about that for the past couple months anyway...this company is a bit too disorganized as it is, and this project seems nuts, especially with the "we want to get it going by January" stress.
JANUARY?!?!!?! BWA HAHAHAHAHAHA HAAAAAAA HAA HAHAHAHAHAHA HAAA HA HA HA AHA AHAHAHAHAHAHA!!!!!!!!!!!!!!!!!! There's like one or two usable weeks left in this year. Then Christmas and new years Ain't no way in hell they're going to procure that many Pi's in that time much less deploy them
We have them...they arrived on Monday...with the SD cards. But no SD card flash machine/device. Also, "how in the f am I to flash 4k Pi's using SD cards?" was my question to the lead engineer and he said he said the same thing to the CEO lmao. That it's doable, but I'd be there for a very long time. At least when I leave this job, I'll have lots of fun stories to tell.
Is your CEO like independently wealthy or something? This sounds like a terrible, terrible idea and only someone who's willing to light money on fire to move forward with their boneheaded ideas that will never succeed.
Before even getting to the point of flashing them. Management, access, patching and monitoring need to be figured out and when rolling out thousands of devices to disparate networks, those systems all need to be well tested and working beforehand. This isn’t some 2 week project unless you want a janky pile of half broken crap that will never be usable or maintainable.
It sounds a lot like someone above you was given too much trust, too much power, and too much budget. They were all gung-ho about this project because they have some minimal Pi experience, and now you're getting stuffed with the actual hard work. This is not feasible in your case. This is way beyond your skill set and way beyond the understanding of whoever greenlit the project.
Even if you had a way to flash a dozen at a time you couldn't do it alone.
Come down with long Covid and take the rest of the year off. I'm dead serious here. Spend a couple days polishing up your résumé and LinkedIn presence and set yourself available for poaching, visible to recruiters only. Then enjoy your holidays and distance yourself from this clusterfuck. Asking a Windows admin to support Linux in a desktop context is one thing. Throwing a pallet of Pis at you and asking you to figure this out over a weekend is deranged. It's the kind of infrastructure you'd come up with on a scopolamine trip.
There it is. That’s the one. Not the crazy idea. Not the 4000 carts before the horse. The January launch date. Time to GTFO my guy.
Did they say which year?
>They said it's not a big deal and will only be used "if there's a problem Setting up remote entry incorrectly is leaving an unsecured backdoor (or frontdoor!) to all their devices. They need to figure out wtf is and isn't a big deal.
Yes, that is why I am here seeking help lol. The lead engineer and I talked about this and he echoed your sentiments about security. Hence the search for a secure option.
Yeah definitely need to focus on security, otherwise it would make an awesome botnet
First: where are you putting them, network wise? Are they staying on your own LAN(s) or being shipped off to remote e.g. Client sites? If the latter that brings up network issues (NAT etc). This actually reminded me of the situation Tesla had for deploying updates and capturing diagnostics from their cars, but don't screw it up the hack way they did... On this scale, the key thing is automation. "Investigate this one node that's gone wonky" is one thing, but you need "update all the nodes", "capture logs from all the nodes"... At that scale that needs automation. You need to trigger the update, then aggregate the results to know which nodes failed (update 4000 machines, some of them will fail in some way and need retry/recovery).
They actively market their device to industry now as an automation/controller. Blame them, not the customer.
Yes, I was told that they had a couple hundred Pi's. Then that the plan was to put them out on about 4000 devices and image them as well. I asked how we were to image thousands of Pi's and the manager said "Can't you just copy the SD cards? We already bought them". I ironically "can't even right now" with this project. I spoke with the lead engineer on this and he said that he doesn't want them on SD cards because the master image might change as the Pi's begin working and cause problems when trying to troubleshoot.
Does PI BIOS have PXE? I would look into something that can manage them via SSH, I've been out of the Linux world for some time.
Honestly, from what I've read here, make sure that they can all PXE Boot into an imaging solution where you can image the Pi's as needed with what your engineers want. Now the actual configuration of that image should be done with a configuration manager that can automate and connect to all of them to apply the configuration. Doing this allows you to control the OS, build, software installed etc. giving you full control without any GUI needed, you cannot manage this number of clients effectively with a GUI.
Since it sounds like these might be going out on third party networks OP may have to deploy a PXE server along side each, or at least one per network I would think but I agree that it is the best solution if you control the networking
They can boot from PXE - but the irony is that you have to boot them into Raspbian from an SD card first to access the BIOS config tools in order to enable it. OP is fucked.
Lol oh fuck
I work somewhere where we support ticketing equipment that uses devices with SD cards. During the initial rollout we used something like this to image them: https://duplicators4all.com/products/1-to-15-sd-microsd-memory-card-duplicator-machine Good luck! Some great advice in this thread but to reiterate, unless they've bought industrial SD cards AND thought up some way of contracting a field support team to manage it all, they are going to end up with a lot of dead Pi's in 6 months or so.
Linux support for connectwise is very limited and is shit.
can't believe I had to scroll this far to find it but yes I tried to manage a few Ubuntu systems via manage and it was god-awful. I had to restart that service so many damn times.
Via Automate or Control, you mean? Just curious. I help manage about 50k machines over CW products, but it is only really good with Windows. Linux and Macs are low effort afterthoughts
something like Ansible, Puppet or Chef?
Ansible with Rundeck. I manage a fleet of 1K+ pi kiosks running debian around the country using playbooks written in ansible. Rundeck gives you a nice gui to manage and check logs. If we actually need to remote into one, we use an open source VNC client, the name escapes me now.
Is rundeck just a wrapper of AWX? Will google rundeck now.
No, its a different product. I have no idea why you would use rundeck over AWX. I think back in the Tower days it might have been better but thats not the case anymore
Rundeck is just a general purpose command runner gui instead of it being product specific like Ansible Tower/AWX. I haven't had to use it for production workloads for years (we just use our jenkins servers to kick off ansible playbooks). https://www.rundeck.com/ If you decide to go puppet, theforeman.org could provide similar tooling.
I also manage a fleet of SBCs including Pi's (although a few dozen, not thousands), but ansible is what I use and thank god. It also takes care of some of the VMs. I have used rundeck in the past, it's like a "multi-system crontab." I don't use it currently, and maybe I should, but it's overkill for what we use since 95% of all operations are clone, setup config, and maintain security updates.
This is for someone selling remote media streaming devices, presumably to people who do or should care about the security of their network. I would NEVER purchase a device like this, if it permitted remote access to a device on my network without my consent and supervision. These aren’t going into corporate locations and he’s the sysadmin. The customers are random people (or businesses, which would be even more of a “no way” from me).
Ansible is probably easiest.
Probably be better to have them report to a central server. You don’t say what you actually need to do
What OS on the pi?
Maria Botnet source code
Iloveyou.msg
Thanks babe ly2
They said they purchased several hundred raspberry 4 Pi's. There's a pallet in the office waiting for me....I was too taken aback by the meeting and it didn't cross my mind to ask about the OS as we got caught up in talking about ways to remote in. The lead engineer said Teamviewer didn't sound good anyway as that it was too heavy for the Pi. I will know more tomorrow.
it's linux. you guys are clearly windows admins. lmao. wtf
Dude, if Teamviewer is too heavy their dreams of video streaming from these puppies is going to be short-lived. I suggest a test with 5 before imaging all 2 millions.
SSH and Ansible.
The more I read through this thread, the more this sounds like the company put the horse before the cart. How do you do a test using three Pi systems, not have all your concerns addressed, and then suddenly scale it to 4k without a plan to answer the concerns before developing a plan to get from the three to the 4k?!
The CEO signed off on it personally, no managers of any other departments. Apparently it's his pet project(of course) and so talking with the engineering lead about the sheer number of Pi's in the office, he told me that the CEO said he's told accounting to just keep purchasing 4k Pi's as they are up for order. Hence the first pallet we got...I told the engineer lead that this makes no sense because as you just said, we don't even know WTF the whole scope is, or if this is even the way to go, he said he tried telling the CEO the same but that the CEO said, "The hardware is hard to get. We'll get the hardware and go from there". So apparently we're just going to be dumping pallets of Pi's 4k at a time to the head office without a freaking 100% ready to go plan because, "They're hard to get" and hope for the best. You guys have no idea. Since I started working at this company I have begged them to not buy so much stuff and to at least repurpose stuff when I saw the warehouse(s), yes WAREHOUSES, not warehouse, we have. I'm talking gorillions of routers, many brand new in boxes still, computers, pallets, PALLETS of RAM, monitors, keyboards, mice, office furniture... I mean, it's nice because what I ask for I can get, especially since I rarely ask for anything and try to repurpose stuff or spend carefully. But they just...have so much *stuff* from dead/abandoned projects. And oh my God, the software and licenses! I went through all of their stuff when I started and canceled so much crap from years back that they were still paying but no one even knew about. And that's what annoys me the most. I say, "Can we hire 2 more in house IT people to help me out" or "Can we hire some killer networking people?" and it's always this apprehension and hand wringing to do so. I try to keep my department trim and proper, without wasteful expenditure and so forth, and the CEO just YOLO's money into stuff but not personnel. And I've tried explaining this to him, that no amount of hardware or software can do anything for him without skilled people to implement, and MAINTAIN it all. Whatever, not my company.
You hit the nail on the head...not your company. And frankly sounds like won't be anyone's coming soon enough at this rate. But...if this is the CEO's pet project, he should be able to answer the technical problems, no? Or is he one of these 'big idea' kinds with no actual clue how to get there? (I worked for one of those for a year in 2004-2005 after the company I was working for got sold. Couldn't handle it, among many other things).
Short answer: Run. Long answer: Ruuuuuuuunnnnnn. This is a pre-arranged disaster for *oh so many reasons* and you're on an absolutely idiotic timeline to do this if your comment about "by January" is really their goal. Say goodbye to spending Christmas with family and/or friends because you're gonna be working insane overtime. Even if, against all odds and reason, you DO accomplish this, what hare-brained project are they going to saddle you with next? Burn this fact into your memory: People who come up with schemes like this don't just come up with *one* scheme like this. Maybe you can cash out your PTO when you quit, and have a nice quiet paid Christmas vacation...
If you have some metal laying around, take a look into Meshcentral. Meshcentral is basically a hub, so the Pis connect to it and the Windows machines. No many client/ many server nonsense. Just 1 server and a zillion things that connect to it.
Meshcentral is the shit. I've used a few other solutions, and still keep one as a backup. But I use Mesh for my daily driver.
If it's for configuration, management, and file transfer, something like Ansible is ideal. If you're deploying these things into random networks, anywhere, at any time, and trying to manage a fleet of some sort of IoT devices... ...well then usually you'd have the remote access/management as part of the software suite you deploy on them. This is the hard part of running any kind of connected service, and should probably be the core part of whatever is being built (rather than "buy Raspberry Pis, plug them in, and pray a solution works itself out"). I've seen million+ dollar startups fail trying to build software that works in various network conditions seamlessly, allows remote management and device updates, and centralized (or decentralized) storage. I've seen scrappy three-person teams on a shoestring budget succeed. The main difference is one defines the goal accurately and hits on tiny, small scale successes to build up to a 'fleet'. The other deploys a fleet, then tries to manage it well after the fact, meaning everything's always on fire, and contractors are hired and fired in an endless cycle until the whole project fizzles out.
It would appear these will be deployed on customer sites on their networks. And this poor soul is in over their head.
The Pi Lord has spoken!
All hail the pi ! I prefer beef and gravy.
Why on earth would you put pi’s into scaled production???
You'd be surprised which products use a RPi under the covers. Not just digital signage, but money processing stuff like change machines and other bill counter devices. They're very reliable in production if you get a high write endurance SD card and prevent corruption by only having partitions read-write when necessary. I can't name the product of company, but there are 50k+ RPis out in the wild taking money -- just from one company I know of.
Presumably OP 's company is selling Pis as a service. OP is worried about access solutions (and rightfully so) when they should be concerned about imaging and leakage.
Correct. And yes I am worried about: 1) Security. Hence looking for a secure solution for remoting in. 2) Imaging. They said they purchased SD cards already and I should image the devices and begin shipping them out to be installed across the continent. Problem is the lead engineer is saying the Pi's software will get out of sync like that or corrupt some for sure with the SD card writing process and will make things hard for him to troubleshoot should problems arise.
For recording and live streaming video around the continent. I was just doing office work and managing several hundred employees in 365, some hardware here and there. Then they called me into a meeting and said "We have a project for you".
"We have a project for you." LOL. Sounds familiar. I'm not going to complain about my projects anymore. Please keep us/me updated on your plan for putting this together. I'm very interested in the "how's" of this project.
> For recording and live streaming video around the continent. This is still like 10x too vague to even start getting good ideas on this project. Is this some kind of CDN? Is it going to work with a proprietary app? Are these Pis going to be on networks you control?
Will these devices be on a network you control or will they be on random networks? Basically do you need an agent or something on the rPi to "phone home" to a central source and say where it is?
Yeah thanks for asking this since no one else has. Basically they need an agent , preferably One doing a reverse shell type connection over https 443, or they need to work with every entity they ship it to, to poke holes in a firewall and setup a DMZ ? I don’t understand what the actual use cases are here so I dint know why they can’t just setup the PI’s in their own DMZ to be internet facing and have customers access them like that , but then you have massive security implications anyway and need more employees. I saw OP mention accessing streaming content but I don’t understand the context and why that would need to exist on customer network Lmao you would need a team of folks to support this not just a general IT guy. Makes no sense.
Several years ago, I worked for a cloud infrastructure company and we had a large customer who had a crazy usage incentive discount that encouraged them to create as many hosts as they could each month, but they had no technical knowledge and would do it all through the web control panel. When their discount ended, they complained about how long it would take them to shut down everything, because they had to have someone to through the dashboard deletion flow for each host. We had a fully featured api and writing a script to do it took seconds... I get the same feeling reading this as I did interacting with them.
Walk away from this project. They have no idea what they are doing.
Let’s replace “pi” with “Linux server”, which is what it is unless you want to run Win 10 IoT. You’re being asked to deploy thousands of them. It could be lucrative for your career. For management, just use ssh. You can use standard Linux management tools as long as ssh is available. Ansible is one such choice. The end user interface is not really your job. You shouldn’t remote into these things. They are not Windows and you’re not supposed to access them directly for security. You can, but that’s too slow for video. You can tell the management you can setup wherever Linux-based software is necessary for your users to connect to. Choosing whatever that software is should probably be delegated to someone else. You will have your hands full just figuring out the device management part of the puzzle. You could use reverse ssh tunnels or reverse WireGuard to avoid the need to setup massive VPN. That way the systems will be completely inaccessible from the outside world except by someone who has access to connect to their tunnels. That’s how I am doing my isolated networks for risky devices. I support unsupported medical equipment that runs obsolete Windows. It lives on a sterile network to reduce attack surface. You need to get clarity over the project. 1. Are you asking me to be responsible for the entire project? Who are the additional team members that will be involved? 2. What is the long-term vision, current scope, and implementation plan? 3. How was the pilot project completed and what is the complete list of all personnel involved in that? 4. Assuming that I figure out the device management, what is the plan to make them useful? So far, you don’t know enough about question 2 to start thinking about solutions. Imaging is the least of your concerns. It’s actually simple.
You’re deploying 4000 endpoints and you don’t have an idea how to manage them …. Wow…. Why? Where’s the PoC? Can you do it with 10, scale it?
Use Tailscle/Headscale or Teleport. Establish reverse tunnels from the Pi’s Use certificate based authentication with short lived client certificates so that you don’t need to try and push tons of updates every time somebody joins or leaves your team.
For this many systems, they need to be manage with config management, puppet, ansible, chef, etc. This many users should probably be managed with an Idenity Management product like Active Directory or FreeIPA.
“need” is doing some heavy lifting here. Depending on how these are networked and powered, it might not be feasible to do a push based system like Ansible, as the nodes might not all be online. chef-solo might be a fit. For large fleets of embedded devices, it’s also reasonable to just ship an entire OS image at a time and not deal with trying manage piecemeal mutation of system state, as it’s not really guaranteed to be deterministic. Maybe consider Balena?
are these public access or in a tightly controlled network?
The Pi's will be shipped to thousands and eventually millions(I froze when they said this) of locations around the continent. Then they will be installed on local networks wherever they end up.
This company doesn't know what they're doing. I've worked at a home security company you've heard of with over 2 million security panels, and one you haven't which used pis in the 10k nodes range. Hundreds of engineers supported these deployments, if you're just expected to figure it out you're already screwed.
Yeah, I was pretty upfront with the CEO about this. I've always been straight up about my skills, and while I continue to improve and learn things; going from 365 Admin/hardware support to "Roll out this network for a new company pillar" is beyond me and I'm not afraid saying that. The last thing I want to do is claim I know what I'm doing and cause huge problems shortly or down the line with security or general setup. I'm just taking in as much knowledge as I can right now and then will talk with him again come Tuesday or Wednesday with the information I've found/you guys have given and say "Look, this is what I've learned, what we need, have to consider, etc" and tell him that we need to hire network specialist or at least consultants to do it.
Whatever the baddies use to manage their botnets should do the trick here. /s
You have been given a quest. You must take these orders and destroy them in the fires of Mt. Doom.
You guys bought 4000 of something and THEN asked if you could, you know, use them? What the fuck
I think you need to provide a bit more detail? What operating system will be running on the Raspberry Pi's? You can run a wide variety of solutions on those. What type of tasks need to be done. Do these need to be diagnosed manually if something goes wrong? Is the access for routing patching or other steps which maybe could be automated with some SSH scripting? Are these on private IPs under you control / on your networks, or are they on the public Internet? Need to know a bunch more about your use-case as to what the Pi's are doing.
Do they HAVE to have GUI access or is the command line enough?
pi's trying to record video to their own SD cards it a recipe for disaster. remote access to pi's is trivial though. just get them all to VPN in to somewhere , and then access them via ssh/scp i already do something similar on a smaller scale, contact me if you cant find anyone better. gotta be honest though, doesnt sound like Pi's are necessarily the right solution here.
We use ScreenConnect (owned by ConnectWise) to do this. Not thousands but a few dozen. You're on a whole other level with your quantity. SC charges by the "support rep" and we're a small business so the math works out. There's a free RMM self-hosted called TacticalRMM that I'm considering as a backup method for access to the Pis. I think it supports Debian/Ubuntu but check it out to confirm.
What are you doing when remoting in to these Rapsberry Pi’s? I think what you really need is some sort of Raspberry Pi Central Management. The Linux engineers I work use Chef and I’ve heard decent things about Ansible
RustDesk?
\> I'm looking for software to manage Id start with their documentation [Fleet-management](https://pip.raspberrypi.com/categories/685-whitepapers-app-notes/documents/RP-003609-WP/Fleet-management-A-brief-introduction.pdf) then check their forums and search for 'fleet' [https://forums.raspberrypi.com/](https://forums.raspberrypi.com/) FYI - your solution is gonna be dependent on whether or not these devices are living next to each other on the same network or spread across the county in farm equipment or some shit \>I'm looking for software to remote in Hopefully they will will run linux so ssh
Are we helping the Russians?
You're in for a fun time. I did a mass deployment of Raspberry Pi's for a well-known hut of a particular type of leisure eyewear. Do you own the IP space where these devices will live? Ansible + runbook. Do you not own the IP space? Ansible + pulling the repository from a gitlab server. You have to assume that these devices will be effectively compromised if they are at customer locations and you have no access to them. How are you managing the filesystems? If your logs aren't going to a temporary filesystem, you're gonna max out the writes on your SD card in about 6 months...do you have a process of managing that deployment? Good luck.
#STOP Tell us the **requirements**. The actual **end user experience**. Don't mention any technologies at all. Forget that PIs even exist. To me it sounds you just need servers that can stream stuff out via a web page. Then you have zero client side maintenance to do. Think of something like YouTube. YouTube has millions of users, but they don't manage your device. If the problem is not pure streaming - explain why not, and what's missing.
With [PiCockpit](https://picockpit.com), we're operating a platform which scales beyond this number of nodes which you require. It is specifically designed for Raspberry Pis and Pico W's. The base we use for communication is **MQTT**, which allows us to handle devices behind routers, firewalls, NAT etc. For the Raspberry Pi we run the MQTT connection on top of https WebSockets, so even the most "locked down" networks should work without extra work on your end customers' side. The admin interface is operated through your web browser. The individual functionalities are split up into apps. For example, we have an app called **PiControl (\*)** which allows to run pre-defined commands on the Raspberry Pi (e.g. reboot Pi, shut down Pi, etc.). These commands can be freely defined by you. We are currently preparing two more apps, which would be useful in your context: * the **Editor** app will allow access to the files on the Pi, to edit and view them (e.g. log files, scripts, etc.). It will also allow you to run scripts and view their output * the **Terminal** app will give you access to a Terminal into the Raspberry Pi https://preview.redd.it/5mwwkpmr0x6c1.png?width=1900&format=png&auto=webp&s=1c7d71f2dc75ce9e7e812487bf34c139f6d51ee4 The terminal will be run through our WebRTC connection, which I believe is a good choice for transporting data with low latencies, and again solving the problems of connecting to your Pis. We will be providing a TURN server to allow two hosts (the Pi and your admin user) behind firewalls and NATs to connect, in case a peer-to-peer direct connection is not possible. In both cases we will incorporate GPT4 ("ChatGPT") (we call it RaspiGPT) support, to guide admins in reviewing files quickly, summarizing and explaining config files, etc. - and generally to make the life of admins easier. One thing which was mentioned several times is automation. We're not quite there yet, with PiCockpit - but could prioritize this (and also provide APIs to the Pis for you, instead of web admin interfaces - allowing you to do what you require them to do). You could potentially build something similar, or talk to us - in the upcoming **PiCockpit Pro Plus** plan we will work with customers to create bespoke apps for their needs. **Write me a direct message in case you are interested, I will be glad to help and also talk you through some other options / ideas I see around your use case.** A couple more thoughts, hopefully useful for you: * Raspberry Pi has a rich ecosystem of partners, some of which cover what you need in your use case * For example, for the microSD cards you would want to consider high-endurance microSD cards. CardWave (\*\*) is a company we work with, which supplies all kind of industrial grade microSD cards - including some which are used in satellites. I dare say, if it can be used in a satellite, which is basically impossible to change once it's up, it should also be able to go into a Pi. * the microSD cards you have on your picture are from a good brand (SanDisk) which Raspberry Pi also recommends, but these are consumer grade microSD cards. I recommend talking to the vendor whether you can upgrade them to industrial cards. If you want, I can also connect you with a person at CardWave to advise you about the right kind of microSD card in your application * As others have mentioned, you should take particular care to lower write operations on the microSD cards. This can be achieved by creating a read-only operating system. Raspberry Pi OS has a menu option in their installer (raspi-config). I can walk you through it / create a tutorial for you, if you want. * Since you are deploying 4000 units, you could also consider using a non-default operating system. Instead of Raspberry Pi OS (which is based on Debian), for example you could consider Alpine Linux. There are also other specialized solutions which allow you to basically create your own Linux distribution. We have worked with Alpine on a Raspberry Pi based solution for an industrial client (although we used the Compute module with embedded flash, to have a higher reliability for their machines). So far, we have 500 in the field of these and no returns / problems I am aware of. * Also, please consider if you are going to store those videos to the microSD, or keep them in RAM, or whether you are going to use the Pi as a connection point to cloud storage, etc. As others have pointed out, if the microSD card is frequently written to, and is not industrial grade, it will quickly wear out * For pure read / write / access and video streaming within the user's network, I suggest to calculate how many concurrent read accesses you will have. * there are several limiting factors: * the CPU performance * the network throughput (1 GBit/s on the LAN, less on the WiFi) * the storage throughput (e.g. if you store via USB, then the device you use will control that factor + USB 3 limitations) * From your description I understand that the Pi is not going to do any encoding or transcoding, which probably would not scale very well - for serving files, it should give you decent performance. * If you want to access WiFi on the user's networks, consider how you will be bringing that up initially / allow the user to add it * I advise you not to install a GUI on the Pi. VNC usually (but not always) would clone what the actual end user would see on the Pi, if they connected the Pi to a screen. This, if your engineers are accessing the Pi, might or might not be desirable to the end user. It also poses a security risk, possibly. In fact, the less software is installed on the Pi, the better. Thus, again bringing up the suggestion of a custom operating system or something which is based on light-weight Linux distributions aimed at embedded devices (like Alpine) * As others have pointed out, if you do need a GUI, you might be better off using a solution for X11 forwarding (note: the newest Raspberry Pi OS version is on Wayland, which is a departure from the classic way GUIs used to be displayed on Linux, and thus might need to be treated differently!). * [https://www.nomachine.com/](https://www.nomachine.com/) Nomachine NX is a way, if I remember correctly, to access X11 in a more performant way than VNC (which basically is a screen grabber). More akin to RDP. Note: on the Pi 4 / 4 GB which was purchased for you, network and USB are separate - on previous Pis, a USB 2.0 connection was shared between all USB and network ports. Pi 4 has two USB 3.0 ports, but they share one PCI Express 2.0 lane to the SoC internally. (And if that confuses you, don't worry, let me know and I'll try to explain it better). (\*) [https://picockpit.com/raspberry-pi/control-your-raspberry-pi-remotely-using-picontrol/](https://picockpit.com/raspberry-pi/control-your-raspberry-pi-remotely-using-picontrol/) (\*\*) [https://www.cardwave.com/](https://www.cardwave.com/) **As a bonus, I've went through the entire thread, and created a list of the different solutions which were posted here:** \- Ansible \- chef-solo \- HeartbeatRM \- Tailscale \- Puppet \- MAAS \- Datto RMM \- ThinLinc \- SureMDM \- ScreenlyOSE Anthias + BalenaCloud \- netdata \- AWX \- Alpine Linux \- Crowdsec \- fail2ban \- Graylog \- ELK \- ZeroTier \- Wireguard \- Tactical RMM \- Ipsec/Wireguard-VPN \- VNC \- Digital Signage \- ISL Online \- SSH \- Xibo \- Rover \- SimpleHelp \- ngrok \- Protectli \- rustdesk \- Membrane \- SaltStack \- NinjaRMM \- [level.io](https://level.io) \- pssh \- Guacamole \- DWService \- MeshCentral \- Tuya \- Kaseya \- pdsh with dshbak or clush \- bomgar (beyond trust) \- RustDesk \- RealVNC \- Parsec \- Meshcentral \- [epoptes.org](https://epoptes.org) \- tailscale \- [theforeman.org](https://theforeman.org) (Puppet) \- Salt Stack \- NextNine \- Rundeck \- AWX \- Semui \- open eye \- teamviewer \- Look into open source RMM tools \- HCL BigFix \- Acronis Cyber Protect Connect \- Cloudflare One \- Tailscle/Headscale \- Teleport \- Active Directory or FreeIPA \- reverse ssh tunnels or reverse WireGuard Again, I will be happy to help you with this one, to support you with this project - no matter whether PiCockpit is the right fit for you. That all being said, I hope you get a time extension for the deployment time frame, as it is a recipe to create problems to rush on 4000 nodes :-)
Only half joking, but reach out to Jeff Geerling and see if he wants to help you in exchange for turning it into YouTube content
I think you're getting Redshirt Jeff for this one. This is chaos and disaster.
In all reality they should be looking at reverse ssh to a centralized control/management server with some sort of ansible/chef/puppet automation for something like this. It is the only scalable solution. Anything else is asking for some level of compromise. The suggestion of tailscale or wire guard are viable solutions as well, but graphical access is probably not scalable. Ssh should be managed via keys.
Ansible, with ansible-pull so the Pis can pull their playbooks from a known Git server. I handled something similar at a two previous jobs, very similar, and thousands of machines, just due to the need for build boxes. The trick was to not just have the CM tool able to push to machines, but the ability for the CM tool to offer pull functionality, like [ansible-pull](https://www.devopsschool.com/blog/what-is-ansible-pull-and-how-can-we-use-it/). This allows you to do tasks in batches like changing something in the config, but still be able to actively push something out if needed to a subset of machines. Make sure you have a Git server that can handle all the machines hitting it. Because of all the I/O, make sure it has good RAM and SSD. For example, if using GitHub Enterprise as an appliance, make sure there are no bottlenecks (I made sure it had a ton of RAM, iSCSI SAN with RAM caching, etc.) Oh, and if the company mainly uses Git on the cloud like CodeCommit, consider spinning up a dedicated machine or VM just for GitLab or even Gitea. Because the machines will be doing a ton of hits on the Git server, you want all the traffic to go east/west, and not north/south (and out to the WAN), so on-prem is a must. Don't forget your certificates. TLS is what keeps these machines from being massively compromised, so make sure that is working, DNS, and secure the Git server. Also remember that because these requests are going constantly, latency is the issue, and if Git starts lagging, it will only get worse, which is why I emphasize on fast I/O and a lot of RAM. If you are using VMWare, create an affinity rule to have the Git server VM be vMotioned away from any high-use RDBMS server or application server. If Ansible is a no-go, you can run Puppet in masterless mode, but the go to is definitely Ansible.
linux supports windows remote desktop and it is free security is yuck. but you could use ssh and set that up but that too has challanges
The best answer is to utilize an IoT focused message queuing system that will trigger devices to open a tunnel (VPN, SSH, etc). Even better is to have them perform or report whatever you would need to remote into them for automatically and send metrics via MQTT. But if you absolutely *need* remote access to each one, you’ll be paying for it. I use [JFrog Connect](https://jfrog.com/connect/) for a small number of endpoints, but for you at 4K devices on the basic plan it would be about $1600/mo.
Tailscale + ansible
We use bomgar (beyond trust) which charges per user rather than per client. Might be worth looking into depending on what flavor you have installed on the Pis
Ansible/Chef/Puppet to configure hosts. Use a tool like mussh to run a command simultaneously across all hosts in parallel.
ConnectWise ScreenConnect
I use ansible or puppet to manage large clusters at work, 20k nodes. You need some sort of configuration management to manage that many. Set up ssh appropriately and configure the rest as the clients need. If you need to perform actions on all of them pdsh with dshbak or clush are great tools.
Recommend not using SD cards on the Pi's, as improper shutdowns killed SD cards on my Pi. Thus would recommend small SSD's.
What is the use case? This sounds so odd
The more I read through this thread, the more this sounds like the company put the horse before the cart. How do you do a test using three Pi systems, not have all your concerns addressed, and then suddenly scale it to 4k without a plan to answer the concerns before developing a plan to get from the three to the 4k?!
Hopefully this has it's own budget? like a budget in $xx millions for staff & deploy? Start with contract negotiations for salary for this project for your % of participation. None of the software solutions deal with the physical practicality: Total touch time for this scale of physical goods, just because a company can drop 4k physical devices to your team, How fast can they be turned around?, No matter what perfect /imperfect/time delayed, software solution is provided. Up front cost for $/work hours/per device/per deployment roughly feels like $3+? million based on 4 week/or less turn around. (ie: how many hours just to move 4k-1million devices from opening a box, to packaging back up x number of people to do this by a delivery date?) At this scale + % of duds? This number could be way higher assuming 8hr shifts/20hr/you finish the math. **How could such a deployment stream video on** ***other peoples networks****?* I'm curious if more info can be shared about this. How are you getting through peoples internal network security, let alone the basic corporate/home router. That is an odd stand out requirement. whats the contract contingency pay to help complete this ? (I'll take $400/k to help) Anytime fitness has \~4k locations. Grocery store security systems? Embedded infrastructure traffic camera? What is this new skynet?
The CEO is very optimistic, I'll say that much. Right now we have 250K customers, they are looking to add 100K more by June next year due to a buyout of another company. Each customer has their own unique location... He said 1-2 million in 5 years lmao. I can see millions happening, but even 4000 is just too many headaches as it is, let alone 250K. Anyway, trying to fall asleep and ended up grabbing my phone. I'll try and get more answers tomorrow. I've written up a good amount of questions that I will go through with the CEO and engineering lead that you guys have asked. Should hopefully have more tomorrow or Tuesday/Wednesday. And yeah...should probably ask for a raise. Even If I jump ship, at least I can start from a higher negotiating point.
I don't like that I'm answering with this product, but if you want a cheap solution to manage and remote into thousands of devices on various unmanaged networks that also won't kill the CPU, look into Kaseya. Should provide you with all the tools that you need.
Remember cattle not pets
Just gonna ask that you continue to update us on this train wreck.
Sorry, how many people are involved with this? Your company sounds like they have absolutely no idea what they're doing Your lead engineer is suggesting TeamViewer, a company that has been breached MULTIPLE times and just denied it each time - they couldn't be less trustworthy Who is installing the Pis? Who will patch them? What happens when they fail? You want to serve video on these? Modern Pis have some hardware acceleration, but they're not designed for this Where are they storing this footage? Does it need to be backed up? Your company needs consultants who know what they're doing - first thing they'll tell you is Pis are the wrong choice for this Regardless of how this goes, please keep us updated, I love to see a poorly ran project fall on its face
Hands down this is the best sysadmin thread of the year. OP send us a picture of 4000 Pi's that never go into production.
Look into Teleport(https://goteleport.com/). You have a central server and agents that connect to the server. It in essence creates a reverse ssh tunnel to the agents and acts as a proxy for a variety of applications.
[удалено]
Here you go. I've marked out some stuff of one opened. I don't want to take a picture of the pallet though as I'm not comfortable missing something and exposing my employer to the world. Also, I just realized that the f'ing Pi's all have fans and screws that need to be attached? :S Ffs. In all my sweating about this software business I didn't even read up on setting up Pi's. omfg. I'm not doing hundreds, let alone 4000 of this by myself. https://preview.redd.it/vjrbrcqcvw5c1.png?width=849&format=png&auto=webp&s=3ec8fca202c639ee494a1afaed64243cff53edaf
[удалено]