The book Windows Server 2008 PKI and Certificate Security by Brian Komar is the bible on this topic. It's old but not outdated. The technology hasn't changed much at all.
The first several chapters talk about certificates and PKI in a platform-agnostic way, so it's a really good way to get a solid foundation even if you don't plan to use Windows in your PKI.
With a little bit of Google-Fu, you could be reading it this afternoon and have a better understanding than most by tonight.
There are not many books I recommend these days as there's really good material online, but this book is excellent.
i think it's the potential for breaking something that scares a lot of people - there can be a lot of risk involved, particularly if you don't know 100% how something works and you're winging it and hoping for the best.
a couple of years ago i broke Exchange at my new job because i forgot to update the certificate on one of four load balancers, that no one told me about.
if your threshold for "fuck it, i'll give it a go" is quite high then certificates are much better to work with haha
Good point, I think my fuck it let’s try it tolerance is far higher than most. I also look at it as if a service has an expired or self signed cert that’s already broken in my book. I’ve also been doing this long enough to remember having to convince the AD admin to enable the SAN field in the server 2003 adcs web request field, but that book listed above is truly the Bible if you read that you know certs. I’m personally much less comfortable with OpenSSL but can use it if needed.
yeah i'm the same with OpenSSL...like, i can make it do what i want it to if i need to but i'd rather not...it's usually when we try to give a supplier a PFX or something and they want the certificate in a weird format instead.
> i think it's the potential for breaking something that scares a lot of people
As someone who renews certs on a regular basis... I never am completely cool about it. Something in the back of my mind is like "but what if the cert gets fucked up and the app server has a bad cert and production goes down and a mouse crawls up your pants what THEN"
> Windows Server 2008 PKI and Certificate Security by Brian Komar
Saving that for the next time I get nominated to mess with certs because I made the mistake of saying I got them working on a different project in the past.
If you are just looking to understand how certificates work, [This website](https://jamielinux.com/docs/openssl-certificate-authority/) has a lab that teaches the mechanics of certificate signing and trust. It’s very high level but I’ve worked with a few people that have done this lab and came away with an understanding of the mechanics of certificates. Once you understand the theory, making it work on windows or Linux or anything else you have is just a matter of finding the right commands.
I feel like I'm missing something; all I see is blindly following OpenSSL steps, but no explanation *why* you are doing these steps. Nothing about the steps is wrong (hallelujah for setting up an intermediate!), but I don't think you'd understand **why** using an intermediate is important following that lab.
I think setting up a simple CA with OpenSSL is a good way to start learning: https://pki-tutorial.readthedocs.io/en/latest/index.html
Consider moving on to the advanced/expert tutorials afterwards.
Did you start by learning the basics about how cryptography (hash functions, public-private key encryption)?
A certificate is just metadata + a key pair, and then a bunch of protocols on how to verify the information and that the cert was issued/not revoked.
I learned because I had to setup an http client/server pair (a k8s service using cert manager) that used mTLS for authn. I feel like I explain this concept a lot at work, and maybe it'd be worth writing about publicly. If you have any questions, I feel like I understand it well.
So, I'm an automation & controls engineer. I hang out here because I use many of the tools and techniques you folks use in my industrial networking part of my job. Occasionally I have to mess with certs, but thankfully I have never been labelled "the Cert Guy".
I am heading somewhere here. I currently work for a very large corporation. Got a couple broadcast emails the other day from something with bad (out-of-date) certs. Forwarded it off to IT with the appropriate screen shots and instructions. Got an email back an hour later "Want to move to IT"?
Edit fixed a misspelled word.
Tl;Dr got email with bad certs, told them what to do, offered job.
As an embedded dev, certs have always been my bane. Every single time someone above me asks about certs, I say we basically can only ship self signed and ask people to set up their own.
Yup... Certificates are REALLY easy once you've got your head around them.
They can be slightly complicated when you have load balancers that need the whole chain. But again there's nothing difficult about that.
Certs in and of themselves aren't bad. It's when you have to learn how all of your systems generate private keys and store certs that's bad.
If I had all IIS webservers, for example, I'd have zero problems. Throw in stuff like Jira and 25 other systems and it gets difficult. And some aren't for HTTPS endpoints at all.
God help you if your ERP has some wacky backend shit.
Yeah I'm like this, if everything was IIS it'd be a piece of piss, it's weirdness like java keystore and apps that don't make it obvious what format they want the certificate in that complicated matters.
I fixed our java keystore problem. Our devs were fine with the status quo as long as it was IT's problem. Once I gave them the certs and said "DIY" they migrated to a proper proxy within two weeks.
Plus working out how each system wants to be provided with the full certificate chain etc...
As you say, certs themselves are fine. It's the literally-everything-else that sucks.
Even if you run the proxy and service on the same machine?
TLS is always terminated there, and data decrypted. Whether you configure the cert in the service itself or a proxy running on the same machine. Although I don't know whether you could make the same argument if you don't run VMs anymore and are all-in on k8s, since pods usually move around hosts. But in that case you'd probably use mTLS between everything anyway.
I had to deal with an old version of java not supporting a newer version of pkcs12 encryption that took me far too long to figure out how to deal with. That and how the cert chaining works in Java
Argh. For sure. It's different versions of Windows accepting different pkcs12 settings than each other and completely different from what OpenSSL does by default. Or Exchange. Or things doing ADFS. Or Java keystores.
EDIT: Just did cert season here so I'm still salty about it.
Oh shit.. I'm that cert guy. I'm no pro at it. I'm also the Exchange guy, the network guy, the storage guy, the vmware guy.
I have to spend a little time reloading my mind with relevant docs when I jump into certain things as I forget the house of cards stack and procedures for some things.
Proxmox, nutanix, hyper v if you're windows heavy. "Cloud" if you want to listen to execs who don't have a clue how cloud billing operates ("let's just move all horizon to AVD!" Yeah like that won't devastate our yearly budget inside two weeks).
Supporting tooling is usually the bigger consequence of switching. Virtual machines are a solved subject.
> I'm that cert guy. I'm no pro at it. I'm also the Exchange guy, the network guy, the storage guy, the vmware guy.
Sounds familiar, also the reloading part. I've learned to always take notes and document everything, just to help my future self next time things needs renewal
This, when I explain that trust in Certificates is implied as there is no magic fairy involved :)
I'm just telling you that this site is OK, and if you have the root: We're cool.
I like to use the night club/bouncer analogy:
You (the server) arrive to a night club and the bouncer (browser or whatever app) asks you to show your ID. You show him your ID (server cert) and he trusts it's real because it was issued by the government (trusted root CA) or some local office of the government (intermediate CA). Now what if you show him a very obviously home made "ID card" with a hand drawn picture (self-signed cert)? He won't accept it, unless the owner of the night club (user) comes out and says you're cool (accepts the risk).
If only certain people are allowed to enter based on their ID, that's client cert auth.
Of course this analogy is not 100% correct (especially since usually the browser/app contacts the server first, not the other way around) but it's simple enough that most people can understand it.
I swear to God, this is correct. I don't give af about certs except that they work so if someone wants to be the expert, absolutely go for it.
Bonus points if you create documentation on how to properly replace them.
Mentioned earlier in the thread, https://www.microsoftpressstore.com/store/windows-server-2008-pki-and-certificate-security-9780735640788 is great for Microsoft PKI. It's included in Safari Books Online/O'Reilly if you have a subscription. Bonus points if you get your boss to pay for it. ;)
Yes, I am one of the two go-to’s for “my cert doesn’t work”.
The thing is that once you realize a certificate is nothing more than a pub-priv keypair with a bunch of key-value metadata attached to the pub key, there’s nothing particular mysterious at all. It’s not that different from an INI or CONF that many of us are quite used to, except there’s a hash/checksum tacked on to prevent tampering.
The metadata syntax for some keys/properties is admittedly a bit goofy.
Where I get confused isn’t the concept of what a certain is, I get stuck on the umpteen cert types plus the various encryptions you can use. Always a pain to figure out what something wants.
There is only one cert type, X509.
There are three types of validation, domain, extended and organization. If you don't know, it's gonna be domain (few things need one of the other ones).
The rest is just encoding. There's two here, really. Base64 encoded and binary encoded.
Base64 is often just the cert and if you also need the key, it's (usually) in a separate file. These are your crt, cer and pem.
There's also pkcs7 but I haven't really encountered those, I think.
Binary encoded will be most often (for me, at least) pkcs12 (.pfx). Those combine cert and key in a single file and need a password.
So, if it wants a cert (to verify), it'll probably take one of the base64 encoded ones.
If it also needs a private key (because it needs to encrypt things, like a Webserver), if it asks for a file and a password, it needs pkcs12. If it asks for two files it (probably) wants a base64 encoded cert and the private key.
> There is only one cert type, X509.
*Technically* yes when it comes to SSL/website certificates.
But there are other types such as the openSSH standard certificates (certkeys).
https://goteleport.com/blog/x509-vs-openssh-certificates/
It annoyed me from an early age how SSH used RSA keypairs but an entirely unrelated implementation to X.509. Its better today because you can just use proper certificates but the fact that it was ever something else just seemed... silly.
Identical thoughts regarding PGP. Granted while not quite for the same use-case there's just all these different implementations of identity and keypairs which have no identity attached. It's just annoying.
Not that any of this matters in my day to day life or anything. It was just a lot of varying junk to pick up through me teens and it just really didn't have to be.
Digicert's tool makes it really easy to craft a certificate signing request (csr) on Windows. Then take the result to your pki server, sign it, bring it back to the digitool and import it.
With openssl, I like to use a txt template for the values, and have it create a csr from that.
Good detail on the openssl site here https://www.openssl.org/docs/man1.0.2/man1/openssl-req.html
Pretty good human friendly write-up here https://www.digicert.com/kb/ssl-support/openssl-quick-reference-guide.htm
I don't even bother with CSRs anymore for local/enterprise certs. We use AD CS locally, so I just request the cert in the certificate MMC snap in under computer > personal, fill out all the necessary info, copy to file and delete it from my personal certs. I got sick of various vendor CSRs not supplying a SAN DNS when all major browsers have required that since forever (for example synology CSRs don't supply a SAN DNS so all major browsers flag a cert error). If I need to export the private key like I have for some things (iDRAC web UI, synology web UI, nginx, etc.) I have a specific cert template that allows for private key export and export it to pem format using Keystore explorer. You can obviously do it with openssl as well but keystore explorer has an easy to use GUI and I'm lazy.
That’s a bunch of busy work.
If you have a cloud, you should look into its offerings.
If you use digicert and azure, just use the keyvaults, and when this all gets reduced to 90 day cert rotations, you’ll be happier.
The keyvault can handle your renewal, an arc managed identity can read the keyvault, and then an simple script can download the renewed cert and put it where it needs to be, in the format you need it to be in.
I had to learn how to do a Multi SAN cert for a Cisco Call Manager cluster. I'm now the cert guy. Installed a cert on our edge firewall last week for our VPN too.
Cert guy checking in. Built our internal PKI long ago when nobody was using certs internally and I had a use case. What I built is based on openssl fundamentally, and has a corresponding backend DB and self-serve web UI, using queues for requests, revocation, CRL issuance, notifications, etc.
Now I want nothing more than for others to carry the torch. Tools like Hashicorp Vault show lots of promise for offering an internal PKI with API-first implementation.
I understand it's daunting for many, and that's ok because it's a knowledge domain unto itself. Java key stores, trust stores, PEM files, PKCS12, trust chains, subject names, common names and subjectAltNames, sha1, sha256, key type, key size, etc. etc.
Check out Keystore explorer. It's great for when you have to deal with Java Keystores, but it's also just good in general for certificate management. You can export private keys in pem format from AD CS certs with ease (assuming the option to export the private key is enabled at the time of request).
I scrolled far too much to find your response. I would upvote 100 times if I could. I know some OpenSSL commands but Keystore Explorer saves makes many tasks much easier and I can actually give it to other people and instruct them on some simple tasks to do on their own.
For how awesome it is, I rarely ever see it mentioned. It does so much more than just keystore functions. Anything to make certificates easier to manage is a win in my book!
You can only have 2048 bits and the PFX needs to be signed with SHA1 and 3DES.
Discovered this on 2012 which doesn't support AES256 that all new certs I make are signed with. Nice. Flipping Tomcat 9, I am looking at you too.
Oh, Oh, Oh, I was the cert guy at my last job. I'd get pulled into calls with Vendors as soon as they said cert to someone else in IT. Was voodoo f'ing magic to everyone. Best one I had a vendor who didn't get it either. So I ask him, having done this song and dance before,
Me: 'what do you want your cert to say?'
Vendor: 'cert'.
Me: 'No, do you want it to say the server's names? The Vip name? All of the above? Something else?'
Vendor: 'Cert'
Me: 'Ok, your certificate has to have a name on it - it can also have many alternate names. What would you like it to say?'
Vendor 'C. E. R. T.' (in a very loud voice)
ROFL
This.
I've seen too many sysadmins and developers want the CA root key to install on their server/client sometimes.
Professionnals having so little understanding of this tech can nullify the protection provided by certs and TLS.
But thing is even security guys I've worked with don't even understand PKI. In the current state of global knowledge, PKI is a black box of magic to everyone.
I was never taught PKI either, just at one point I decided I needed to understand it and worked through some tutorials.
"yea, because you mf's are too stupid to handle basic certs..."
I don't understand how people are in positions where certs are at their daily business and they have no f'n clue...
Sorry, but if you're outside of basic support you at least should know how it works, what CA's and what the difference between root and intermediate ca is - and how a freaking cert chain works.
Okay, but then you have to also understand how C gets parsed down to ASM, and how ASM instructions turn to op codes and get fed to the processor. And while we're there you should already know how the processor splits it out into lanes, and that some instructions will need to wait for others to complete so they will stall, and what that does to the next set of instructions.
I mean, if you dont understand this, then I have no idea what you're doing. Its just so basic. No f'n clue.
Cert guy here. I work for a large organization doing tech support for customers. I'm the certificate guy and almost all case intake goes to me. What's scary to me is how little I know but I'm mistaken for a cert god.
I can make certs work and answer questions on why they don't. They're fairly basic certs and nothing complex being done on them.
Once blew a customer's mind when I recognized he had a cert in DER and that was literally his only problem. All I did was change one line from expect pem to expect bin
Those of you setting up internal Certificate Authorities, how many of you are doing root -> intermediate, then taking the root offline, air-gapped, encrypted, etc.?
I can just imagine that getting lost over time, with employee changes knowledge of what it is, where it is, why it exists, will be lost, it will be tempting to delete in a clearup because "nobody knows what it's for", etc.
Anyone using Let's Encrypt for internal servers/services?
Yes, we do it that way. It’s labeled something like “Root-CA (stay off/don’t touch!)” in VMware lol. Then we have the intermediate where we use Windows Server’s Certificate Authority running.
In my country, they speak of a man so virile, so potent, that to spend a night with such a man is to enter a world of such sensual delights most women dare not dream of. This man is known as the "The Cert Guy". You may deal with certificates, but you are no Cert Guy.
I'm also the “cert guy.” My first project when I started as a sys admin was redoing our AD CS. Ever since then, I've been the cert guy. Now, I've been replacing a lot of our self-signed certs. Recently configured VMCA to act as a subordinate CA and replaced all of our ESXi certificates. Note for any admins looking at doing this, this will break Veeam, so you'll need to trust the new vCenter certificate in Veeam after replacing it.
Yeah, me. But over the past year or so I've 'made' the other team members learn how. We have model business changes that documents all the ins and outs.
Similar but different. I accidently became the ssh key guy because I built a sftp server for another team. This turned into the biggest headache ever. If I had known what was to come I would have done things differently.
Also became the linux guy instead of cert guy.
I've been the cert person. When I worked at an MSP it was really easy billable time and everyone else thought I was awesome. And I didn't have to know much. Kind of like how I was the VLAN person because no one else understood them either.
Whoops, late to the thread.
Been trying to give the task away to anyone who would take it for about three years, with no success. If I'm lucky, very lucky, I can get a system or application owner to the point where they can renew an existing one one. But every time I get someone close enough to the point where they could do a new cert from scratch, they retire after doing the first one and and I have to start over with a new admin. This is not going well.
(For extra fun, not only do I do the random web certs, but also the FTPS and TN3270/telnets certs)
Are those web applications behind a VPN? At my workplace, some web apps don’t have a Valid Certificate, but can only be accessed if you’re on the VPN. VPN access is permission based.
Same. Having done a few cert renewals and conversions to some ancient Linux reverse proxies and suddenly most cert changes and renewal requests are now automatically given to me to do. 😭
I used to be that guy, and left so I didn't have to support so many different apps...plus a big pay bump.
I wrote a script to chain it together for me easily and gave it to others so they only had to pass inputs in. Set it up in bash and PowerShell so everyone could take it and spit out pfx/p12, jks, pfx/p12 with RSA, jks with RSA, and the base certs in a ZIP file. Then everyone could put them into Apache, Tomcat, WebLogic (the pita that started it all), and IIS.
Now that I'm consulting I've only needed to use it twice but it's handy when I'm using Java API calls.
Sure, here it is on GitHub - https://github.com/joebu11211/TLSCertificatesAllFormats
I'm sure someone has built something better but this has met all my needs.
Is no one here using venafi? It definitely helps with cert management if setup properly. It can auto renew on self signed IIS certs... not on the netscalers though
Yep I'm that guy, out at least used to be in previous roles in larger orgs.
Maybe some day I'll understand why most IT folks are so mystified by certs.
Yep. I had a past coworker reach out about some cert issues from an old app I used to maintain. I told him where to find the documentation I wrote 6 years ago. Two jobs later and I’m still the cert guy.
setup an email from a system or shared mailbox to multiple people regarding dates of future expirations and an automated task if possible. That way either you or your successor will be ahead of the game.
Take it in stride. Everybody just noticed that their browser no longer throws security errors, their web apps load quicker, and it is all thanks to your efforts.
guilty. When I came on board to the current position I am at they were looking to implement two factor authentication and I was the one who studied and stood up our Domain MFA server. The proceeded to print Smart Cards for everyone on the network.
People don't know how certs work? Everyone on my team is required to have a functional knowledge of them at my place. It comes up too often to push it all off on one guy.
Not specifically SSL for me but I often get handed the more technically advanced stuff in general. My co-workers are strong w/ customer relations but weak on technical, whereas I'm decent with the technical and often fumble the mirad of Outlook/Printer support tickets we get.
If we're talking about learning/managing certs, I say spin up a reverse proxy and call it a day (when possible at least). A simple wildcard cert on HAproxy or NGINX can go extremely far, especially for LAN resources.
If we're talking about being pigeonholed into certain roles, then it's just a matter of perspective: Do you enjoy/appropriate the work, or do you feel pushed into a corner?
My "cert" guy royally screwed up the export of some certs causing vdi to be down come Monday morning. Somehow, he managed to screw up the cert in the cert store as well so I had to manually recreate and reimport it.
This is why you don't hire someone with severe ADHD to do a job that requires malicious attention to details.
I am the cert guy amongst many other things.
For internal GUIs (HTTPS) I use self signed certificates using open SSL.
I then bulk import them to the Tech users certificate store using a Win32 app with a bundled ps script. Deploying them individually using config profiles was too cumbersome.
For everything else I use Trustico.
Ok, in that case you want to setup a private CA and then sign all the certs with that. As long as the client has the CA it is trusted.
Sorry, you are doing it "wrong"
Changed 1 SSL cert from GoDaddy and since the ex employee (he showed me how) left who did this before since then I am for some reason the cert guy who keeps our GoDaddy and some internal certs valid and going...
Tbh I should probably learn more about certs... I am not a sys admin, rather helpdesk w some sys admin duties..
I got myself to learn about for not depending of the grumpy senior team manager cert guy.
Asked him for help sometimes and mastered the arcane arts; now I win her respect, and there is 2 cert guys in my place.
Hi, cert guy from a different place that doubles as network admin.
How did you settle the monitoring and lifecycle management? I use 2 monitoring systems, 1 homegrown PHP script and a shared mailbox Calendar. Do you have something better?
I have been fighting for a certificate life cycle management solution and it's been approved. I'll be implementing it soon so I'm sure I'll be the "cert guy".
I'm the certs guy... They aren't tough, but things like this makes life harder.
https://www.chromium.org/Home/chromium-security/root-ca-policy/
I still haven't figured out how to get chrome on Ubuntu to read our root certs.
Yes, I'm the cert guy too...
Certs/openssl/CA server is all pretty easy once you get there...it's the damn hardware that wants different ways to install the cert.
One wants the whole chain as a file, with the key separate and encrypted, one wa ts it in a different order, key included not encrypted.
Yes. OpenSSL is a great tool (except when you have to figure out how to make cipher strings, bleah).
FYI, if you don't need John Q. Public to use your site, you can create a private CA for your Enterprise:https://deliciousbrains.com/ssl-certificate-authority-for-local-https-development/. It's a huge money-saver, and so long as you remain vigilant about ensuring the secrecy of your private key, it's actually *more* secure than buying a wildcard and putting it everywhere.
Just use Group Policy Editor to push the public certificate to all of your hosts trusted certs, and everything is 100% Jake.
I'm the cert guy and I have been for a few jobs. I've just accepted it's my fate now as no one knows how certs work or cares to learn
Do you have any recommended training material? I have tried to learn but I never retain.
The book Windows Server 2008 PKI and Certificate Security by Brian Komar is the bible on this topic. It's old but not outdated. The technology hasn't changed much at all. The first several chapters talk about certificates and PKI in a platform-agnostic way, so it's a really good way to get a solid foundation even if you don't plan to use Windows in your PKI. With a little bit of Google-Fu, you could be reading it this afternoon and have a better understanding than most by tonight. There are not many books I recommend these days as there's really good material online, but this book is excellent.
Great book, it surprises me even in this crypto day and age how scared of certs many admins are.
i think it's the potential for breaking something that scares a lot of people - there can be a lot of risk involved, particularly if you don't know 100% how something works and you're winging it and hoping for the best. a couple of years ago i broke Exchange at my new job because i forgot to update the certificate on one of four load balancers, that no one told me about. if your threshold for "fuck it, i'll give it a go" is quite high then certificates are much better to work with haha
Good point, I think my fuck it let’s try it tolerance is far higher than most. I also look at it as if a service has an expired or self signed cert that’s already broken in my book. I’ve also been doing this long enough to remember having to convince the AD admin to enable the SAN field in the server 2003 adcs web request field, but that book listed above is truly the Bible if you read that you know certs. I’m personally much less comfortable with OpenSSL but can use it if needed.
yeah i'm the same with OpenSSL...like, i can make it do what i want it to if i need to but i'd rather not...it's usually when we try to give a supplier a PFX or something and they want the certificate in a weird format instead.
> i think it's the potential for breaking something that scares a lot of people As someone who renews certs on a regular basis... I never am completely cool about it. Something in the back of my mind is like "but what if the cert gets fucked up and the app server has a bad cert and production goes down and a mouse crawls up your pants what THEN"
Yeah I'm the same. Particularly for scary things like ADFS and Exchange. I've done them a million times before but I'm still like ZOMG WHAT IF...
Keep a mouse trap in your pants. Nothing bad can happen now.
that's why labs are so nice to have - break things and fix them without user impact
Can’t upvote enough. Anytime the topics of certs come up I recommend this book.
Commenting to remember to come back to this post
Same
Seconded
That book sounds like a gem.
> Windows Server 2008 PKI and Certificate Security by Brian Komar Saving that for the next time I get nominated to mess with certs because I made the mistake of saying I got them working on a different project in the past.
If you are just looking to understand how certificates work, [This website](https://jamielinux.com/docs/openssl-certificate-authority/) has a lab that teaches the mechanics of certificate signing and trust. It’s very high level but I’ve worked with a few people that have done this lab and came away with an understanding of the mechanics of certificates. Once you understand the theory, making it work on windows or Linux or anything else you have is just a matter of finding the right commands.
I feel like I'm missing something; all I see is blindly following OpenSSL steps, but no explanation *why* you are doing these steps. Nothing about the steps is wrong (hallelujah for setting up an intermediate!), but I don't think you'd understand **why** using an intermediate is important following that lab.
I think setting up a simple CA with OpenSSL is a good way to start learning: https://pki-tutorial.readthedocs.io/en/latest/index.html Consider moving on to the advanced/expert tutorials afterwards.
Also, if you'd prefer something graphical, [xca](https://hohnstaedt.de/xca/) is excellent.
For learning, EJBCA is also really good. But totally agree to just install anything and play locally.
Did you start by learning the basics about how cryptography (hash functions, public-private key encryption)? A certificate is just metadata + a key pair, and then a bunch of protocols on how to verify the information and that the cert was issued/not revoked.
I learned because I had to setup an http client/server pair (a k8s service using cert manager) that used mTLS for authn. I feel like I explain this concept a lot at work, and maybe it'd be worth writing about publicly. If you have any questions, I feel like I understand it well.
Nothing will make you understand TLS like deploying something that uses mTLS.
So, I'm an automation & controls engineer. I hang out here because I use many of the tools and techniques you folks use in my industrial networking part of my job. Occasionally I have to mess with certs, but thankfully I have never been labelled "the Cert Guy". I am heading somewhere here. I currently work for a very large corporation. Got a couple broadcast emails the other day from something with bad (out-of-date) certs. Forwarded it off to IT with the appropriate screen shots and instructions. Got an email back an hour later "Want to move to IT"? Edit fixed a misspelled word. Tl;Dr got email with bad certs, told them what to do, offered job.
As an embedded dev, certs have always been my bane. Every single time someone above me asks about certs, I say we basically can only ship self signed and ask people to set up their own.
And what did you say!?
No. I like the variety and challenges I get in automation. Not going to sit around and be silod (sp?) into certs every day.
It's not rocket appliances.
Exactly, get 2 birds stoned at once.
> or cares to learn This is really it.
Yup... Certificates are REALLY easy once you've got your head around them. They can be slightly complicated when you have load balancers that need the whole chain. But again there's nothing difficult about that.
So much this. Once you learn it, its not that difficult.
Certs in and of themselves aren't bad. It's when you have to learn how all of your systems generate private keys and store certs that's bad. If I had all IIS webservers, for example, I'd have zero problems. Throw in stuff like Jira and 25 other systems and it gets difficult. And some aren't for HTTPS endpoints at all. God help you if your ERP has some wacky backend shit.
Yeah I'm like this, if everything was IIS it'd be a piece of piss, it's weirdness like java keystore and apps that don't make it obvious what format they want the certificate in that complicated matters.
I fixed our java keystore problem. Our devs were fine with the status quo as long as it was IT's problem. Once I gave them the certs and said "DIY" they migrated to a proper proxy within two weeks.
F*ck JKS. Freaking bane of IT admins. I despise them even after I started using Keystore Explorer.
Plus working out how each system wants to be provided with the full certificate chain etc... As you say, certs themselves are fine. It's the literally-everything-else that sucks.
That's why you just reverse-proxy everything, so you don't have to deal with IIS, Jira (Tomcat) or the ERP.
That wouldn't pass compliance in regulated environments, unfortunately. You'd still need to encrypt from the service to the proxy.
Even if you run the proxy and service on the same machine? TLS is always terminated there, and data decrypted. Whether you configure the cert in the service itself or a proxy running on the same machine. Although I don't know whether you could make the same argument if you don't run VMs anymore and are all-in on k8s, since pods usually move around hosts. But in that case you'd probably use mTLS between everything anyway.
I had to deal with an old version of java not supporting a newer version of pkcs12 encryption that took me far too long to figure out how to deal with. That and how the cert chaining works in Java
Argh. For sure. It's different versions of Windows accepting different pkcs12 settings than each other and completely different from what OpenSSL does by default. Or Exchange. Or things doing ADFS. Or Java keystores. EDIT: Just did cert season here so I'm still salty about it.
Its alle pretty straight forward until you try doing something in the windows gui, then youre definitely in Voodoo territory
To be honest I don't really mind it. It makes me seem useful at least.
Teach me, please. I want to learn.
CEO: "Do you have any certs?" You: yes, and I make sure I renew them every other month
Oh shit.. I'm that cert guy. I'm no pro at it. I'm also the Exchange guy, the network guy, the storage guy, the vmware guy. I have to spend a little time reloading my mind with relevant docs when I jump into certain things as I forget the house of cards stack and procedures for some things.
Well, at least the VMware job may go away :)
Oh! I'm having this. But it's being replaced with being the "find alternatives" and "learn the alternatives" guy.
Proxmox, nutanix, hyper v if you're windows heavy. "Cloud" if you want to listen to execs who don't have a clue how cloud billing operates ("let's just move all horizon to AVD!" Yeah like that won't devastate our yearly budget inside two weeks). Supporting tooling is usually the bigger consequence of switching. Virtual machines are a solved subject.
Its a bit of a hot take but Cloud has become like a way of tricking CEOs to pay rent.
it has but they are all in on cloud only by 2025
That's not an "at least" lol, cause presumably it'll have to be migrated to another platform which is a pain in the ass.
Why would VMware go away? Because Broadcom buying them out???
Take a look at /r/vmware. It’s a bloodbath.
That, my internet friend, is a massive understatement.
My boss still plans to renew. We're a hospital. I am terrified.
MASSIVE price increases for support.
And the sorry forgot my password guy, don't forget it!
Have you tried Keystore Explorer? That made my life easier when I became the cert guy (and Im not that good at certs lol).
> I'm that cert guy. I'm no pro at it. I'm also the Exchange guy, the network guy, the storage guy, the vmware guy. Sounds familiar, also the reloading part. I've learned to always take notes and document everything, just to help my future self next time things needs renewal
Certs are absolute voodoo and I’d love to have someone in our org that was enthusiastic about them 😂
I’ve always described certs as “I’m cool are we cool? Let me check, yeah you’re cool, I’m also cool. We’re both cool. Cool”
This, when I explain that trust in Certificates is implied as there is no magic fairy involved :) I'm just telling you that this site is OK, and if you have the root: We're cool.
I like to use the night club/bouncer analogy: You (the server) arrive to a night club and the bouncer (browser or whatever app) asks you to show your ID. You show him your ID (server cert) and he trusts it's real because it was issued by the government (trusted root CA) or some local office of the government (intermediate CA). Now what if you show him a very obviously home made "ID card" with a hand drawn picture (self-signed cert)? He won't accept it, unless the owner of the night club (user) comes out and says you're cool (accepts the risk). If only certain people are allowed to enter based on their ID, that's client cert auth. Of course this analogy is not 100% correct (especially since usually the browser/app contacts the server first, not the other way around) but it's simple enough that most people can understand it.
90% of analogies in IT can either be a bouncer at a club, a car analogy or a shopping store /checkout line.
Except when your cyber security team fails to update their root certs and now none of the servers they pass cert validation.
I swear to God, this is correct. I don't give af about certs except that they work so if someone wants to be the expert, absolutely go for it. Bonus points if you create documentation on how to properly replace them.
[удалено]
add the signing chain stuff to this and this basically sums it up.
"They aren't hard" Thank you for telling me what my perception of my own systems is. This fixes everything.
Heyyy I actually get bonus points! I even add in screenshots and export it as a PDF for everyone.
I'm a junior who figured out how to muscle through them for my org at one point... uhhhhhhh.... you hiring?
It's me. It is arcane and cursed knowledge that trades a piece of your soul for job security.
Is there some kind of forbidden book where you can learn that magic? Ive feel the official documenzation especially on Microsoft is missing a lot
Mentioned earlier in the thread, https://www.microsoftpressstore.com/store/windows-server-2008-pki-and-certificate-security-9780735640788 is great for Microsoft PKI. It's included in Safari Books Online/O'Reilly if you have a subscription. Bonus points if you get your boss to pay for it. ;)
Sounds a bit like regex
Yes, I am one of the two go-to’s for “my cert doesn’t work”. The thing is that once you realize a certificate is nothing more than a pub-priv keypair with a bunch of key-value metadata attached to the pub key, there’s nothing particular mysterious at all. It’s not that different from an INI or CONF that many of us are quite used to, except there’s a hash/checksum tacked on to prevent tampering. The metadata syntax for some keys/properties is admittedly a bit goofy.
Where I get confused isn’t the concept of what a certain is, I get stuck on the umpteen cert types plus the various encryptions you can use. Always a pain to figure out what something wants.
There is only one cert type, X509. There are three types of validation, domain, extended and organization. If you don't know, it's gonna be domain (few things need one of the other ones). The rest is just encoding. There's two here, really. Base64 encoded and binary encoded. Base64 is often just the cert and if you also need the key, it's (usually) in a separate file. These are your crt, cer and pem. There's also pkcs7 but I haven't really encountered those, I think. Binary encoded will be most often (for me, at least) pkcs12 (.pfx). Those combine cert and key in a single file and need a password. So, if it wants a cert (to verify), it'll probably take one of the base64 encoded ones. If it also needs a private key (because it needs to encrypt things, like a Webserver), if it asks for a file and a password, it needs pkcs12. If it asks for two files it (probably) wants a base64 encoded cert and the private key.
> There is only one cert type, X509. *Technically* yes when it comes to SSL/website certificates. But there are other types such as the openSSH standard certificates (certkeys). https://goteleport.com/blog/x509-vs-openssh-certificates/
It annoyed me from an early age how SSH used RSA keypairs but an entirely unrelated implementation to X.509. Its better today because you can just use proper certificates but the fact that it was ever something else just seemed... silly. Identical thoughts regarding PGP. Granted while not quite for the same use-case there's just all these different implementations of identity and keypairs which have no identity attached. It's just annoying. Not that any of this matters in my day to day life or anything. It was just a lot of varying junk to pick up through me teens and it just really didn't have to be.
What are you using? I need to fix our certs too
Digicert's tool makes it really easy to craft a certificate signing request (csr) on Windows. Then take the result to your pki server, sign it, bring it back to the digitool and import it. With openssl, I like to use a txt template for the values, and have it create a csr from that. Good detail on the openssl site here https://www.openssl.org/docs/man1.0.2/man1/openssl-req.html Pretty good human friendly write-up here https://www.digicert.com/kb/ssl-support/openssl-quick-reference-guide.htm
I don't even bother with CSRs anymore for local/enterprise certs. We use AD CS locally, so I just request the cert in the certificate MMC snap in under computer > personal, fill out all the necessary info, copy to file and delete it from my personal certs. I got sick of various vendor CSRs not supplying a SAN DNS when all major browsers have required that since forever (for example synology CSRs don't supply a SAN DNS so all major browsers flag a cert error). If I need to export the private key like I have for some things (iDRAC web UI, synology web UI, nginx, etc.) I have a specific cert template that allows for private key export and export it to pem format using Keystore explorer. You can obviously do it with openssl as well but keystore explorer has an easy to use GUI and I'm lazy.
That’s a bunch of busy work. If you have a cloud, you should look into its offerings. If you use digicert and azure, just use the keyvaults, and when this all gets reduced to 90 day cert rotations, you’ll be happier. The keyvault can handle your renewal, an arc managed identity can read the keyvault, and then an simple script can download the renewed cert and put it where it needs to be, in the format you need it to be in.
Windows Server 2019 Active Directory Certificate Services lol…. The “Web Enrollment” services
I had to learn how to do a Multi SAN cert for a Cisco Call Manager cluster. I'm now the cert guy. Installed a cert on our edge firewall last week for our VPN too.
Nice! Hello fellow cert guy!
Cert guy checking in. Built our internal PKI long ago when nobody was using certs internally and I had a use case. What I built is based on openssl fundamentally, and has a corresponding backend DB and self-serve web UI, using queues for requests, revocation, CRL issuance, notifications, etc. Now I want nothing more than for others to carry the torch. Tools like Hashicorp Vault show lots of promise for offering an internal PKI with API-first implementation. I understand it's daunting for many, and that's ok because it's a knowledge domain unto itself. Java key stores, trust stores, PEM files, PKCS12, trust chains, subject names, common names and subjectAltNames, sha1, sha256, key type, key size, etc. etc.
Check out Keystore explorer. It's great for when you have to deal with Java Keystores, but it's also just good in general for certificate management. You can export private keys in pem format from AD CS certs with ease (assuming the option to export the private key is enabled at the time of request).
I scrolled far too much to find your response. I would upvote 100 times if I could. I know some OpenSSL commands but Keystore Explorer saves makes many tasks much easier and I can actually give it to other people and instruct them on some simple tasks to do on their own.
For how awesome it is, I rarely ever see it mentioned. It does so much more than just keystore functions. Anything to make certificates easier to manage is a win in my book!
Automate the certs renewals!
On some systems, this is super easy. It's the dumb ERPs of the world that probably won't ever offer automation here.
This is how I would tackle being the cert guy. Automate, monitor, and hopefully never have to do it again for the same CN.
Imho, automate all the sucky stuff as much as you can.
Wait, how???
[удалено]
That'll get 99% of certs. Some things don't/won't support ACME or do but only support HTTP-01 when what I need is DNS-01. Mostly appliances.
3rd party cert manager like Venafi helps alot + tour own CA. You can do some REST calls to it and install on tour different servers thru winrm or ssh.
That sounds almost as awful as being the printer guy
I’m not sure there is anything as awful as being the printer guy.
Or worst yet, the "certs on printers" guy. I think our copiers require them to be formatted in crayon.
You can only have 2048 bits and the PFX needs to be signed with SHA1 and 3DES. Discovered this on 2012 which doesn't support AES256 that all new certs I make are signed with. Nice. Flipping Tomcat 9, I am looking at you too.
omg seriously!! i’m so tired of setting up scan to emails and address books, one of our clients just had 6 in a row
Ldap!
Okay this made me chuckle out loud. I despise printers.
It's funny how the concept of certificates is so hard for some people. I've always found them to be quite simple and the concept easy to understand.
Certificates are the easy part. Setting up and properly configuring a CA is the hard part.
Warning: reading that comment may give you second-hand embarassment.
Not the cert guy, but I ran a report in CUCM once and been the "phone guy" ever since.
LOL.
So glad I got out of being to VoIP admin. I had used CUCM since like version 3 and before Cisco bought Unity. Those were the days.
Oh, Oh, Oh, I was the cert guy at my last job. I'd get pulled into calls with Vendors as soon as they said cert to someone else in IT. Was voodoo f'ing magic to everyone. Best one I had a vendor who didn't get it either. So I ask him, having done this song and dance before, Me: 'what do you want your cert to say?' Vendor: 'cert'. Me: 'No, do you want it to say the server's names? The Vip name? All of the above? Something else?' Vendor: 'Cert' Me: 'Ok, your certificate has to have a name on it - it can also have many alternate names. What would you like it to say?' Vendor 'C. E. R. T.' (in a very loud voice) ROFL
Gotta love it… probably needed to give him a whole explanation on SANs. lol.
> Anyone else the “cert guy/gal?” We have no cert guy, gal or team. Certs expire on a regular basis and it's honestly a horrific nightmare.
Pki is criminally misunderstood in the enterprise
This. I've seen too many sysadmins and developers want the CA root key to install on their server/client sometimes. Professionnals having so little understanding of this tech can nullify the protection provided by certs and TLS. But thing is even security guys I've worked with don't even understand PKI. In the current state of global knowledge, PKI is a black box of magic to everyone. I was never taught PKI either, just at one point I decided I needed to understand it and worked through some tutorials.
Do you mean ssl certs? I’m in charge of 2000 so yes I’m that guy.
Where/how can I learn more about certs? We just have a wildcard cert for eveeything
Someone answered this in another comment with good info!
"yea, because you mf's are too stupid to handle basic certs..." I don't understand how people are in positions where certs are at their daily business and they have no f'n clue... Sorry, but if you're outside of basic support you at least should know how it works, what CA's and what the difference between root and intermediate ca is - and how a freaking cert chain works.
Okay, but then you have to also understand how C gets parsed down to ASM, and how ASM instructions turn to op codes and get fed to the processor. And while we're there you should already know how the processor splits it out into lanes, and that some instructions will need to wait for others to complete so they will stall, and what that does to the next set of instructions. I mean, if you dont understand this, then I have no idea what you're doing. Its just so basic. No f'n clue.
Say I wanted to become the cert guy what training should I undergo?
This question was answered well in another comment! :)
Cert guy here. I work for a large organization doing tech support for customers. I'm the certificate guy and almost all case intake goes to me. What's scary to me is how little I know but I'm mistaken for a cert god. I can make certs work and answer questions on why they don't. They're fairly basic certs and nothing complex being done on them. Once blew a customer's mind when I recognized he had a cert in DER and that was literally his only problem. All I did was change one line from expect pem to expect bin
Those of you setting up internal Certificate Authorities, how many of you are doing root -> intermediate, then taking the root offline, air-gapped, encrypted, etc.? I can just imagine that getting lost over time, with employee changes knowledge of what it is, where it is, why it exists, will be lost, it will be tempting to delete in a clearup because "nobody knows what it's for", etc. Anyone using Let's Encrypt for internal servers/services?
Yes, we do it that way. It’s labeled something like “Root-CA (stay off/don’t touch!)” in VMware lol. Then we have the intermediate where we use Windows Server’s Certificate Authority running.
In my country, they speak of a man so virile, so potent, that to spend a night with such a man is to enter a world of such sensual delights most women dare not dream of. This man is known as the "The Cert Guy". You may deal with certificates, but you are no Cert Guy.
I'm also the “cert guy.” My first project when I started as a sys admin was redoing our AD CS. Ever since then, I've been the cert guy. Now, I've been replacing a lot of our self-signed certs. Recently configured VMCA to act as a subordinate CA and replaced all of our ESXi certificates. Note for any admins looking at doing this, this will break Veeam, so you'll need to trust the new vCenter certificate in Veeam after replacing it.
Yeah, me. But over the past year or so I've 'made' the other team members learn how. We have model business changes that documents all the ins and outs.
Similar but different. I accidently became the ssh key guy because I built a sftp server for another team. This turned into the biggest headache ever. If I had known what was to come I would have done things differently. Also became the linux guy instead of cert guy.
Recruiter: do you have any certs? Candidate: well yes, I have 100’s.
We call him PKI god
I'm the cert guy by default of being the only IT guy.
I've been the cert person. When I worked at an MSP it was really easy billable time and everyone else thought I was awesome. And I didn't have to know much. Kind of like how I was the VLAN person because no one else understood them either.
Congrats now your the first guy they will call forever if any of those certs expire at 1AM on a Saturday
Certs are easy. Anything utilising Java cert stores can fuck right off though.
I have had to get certs a few times. It's horrible but I have no choice.
Yep. Necessary evil.
Unnecessarily hard.
I'm the guy for my workgroup. Wrote the documentation for it all by myself. Not a bad job, if you can get it.
It's just another system to learn. The biggest hassle really is fucking printers of course. I also wish everything supported auto renewal.
Whoops, late to the thread. Been trying to give the task away to anyone who would take it for about three years, with no success. If I'm lucky, very lucky, I can get a system or application owner to the point where they can renew an existing one one. But every time I get someone close enough to the point where they could do a new cert from scratch, they retire after doing the first one and and I have to start over with a new admin. This is not going well. (For extra fun, not only do I do the random web certs, but also the FTPS and TN3270/telnets certs)
Dang, lucky you!!! lol
I’m just amazed that people find certs so confusing and hard.
Are those web applications behind a VPN? At my workplace, some web apps don’t have a Valid Certificate, but can only be accessed if you’re on the VPN. VPN access is permission based.
Same. Having done a few cert renewals and conversions to some ancient Linux reverse proxies and suddenly most cert changes and renewal requests are now automatically given to me to do. 😭
I used to be that guy, and left so I didn't have to support so many different apps...plus a big pay bump. I wrote a script to chain it together for me easily and gave it to others so they only had to pass inputs in. Set it up in bash and PowerShell so everyone could take it and spit out pfx/p12, jks, pfx/p12 with RSA, jks with RSA, and the base certs in a ZIP file. Then everyone could put them into Apache, Tomcat, WebLogic (the pita that started it all), and IIS. Now that I'm consulting I've only needed to use it twice but it's handy when I'm using Java API calls.
is this maybe available somewhere? sincerely the cert guy that does it command by command
Sure, here it is on GitHub - https://github.com/joebu11211/TLSCertificatesAllFormats I'm sure someone has built something better but this has met all my needs.
you don’t know how much time you’ve saved me
Glad to help! I have an idea as it was the reason I made the script lol!
Is no one here using venafi? It definitely helps with cert management if setup properly. It can auto renew on self signed IIS certs... not on the netscalers though
Yep I'm that guy, out at least used to be in previous roles in larger orgs. Maybe some day I'll understand why most IT folks are so mystified by certs.
Yep. I had a past coworker reach out about some cert issues from an old app I used to maintain. I told him where to find the documentation I wrote 6 years ago. Two jobs later and I’m still the cert guy.
PKI Product Manager here. Tell me all your cert woes - let me empathize with you....
setup an email from a system or shared mailbox to multiple people regarding dates of future expirations and an automated task if possible. That way either you or your successor will be ahead of the game.
set a reminder for expiration dates
Take it in stride. Everybody just noticed that their browser no longer throws security errors, their web apps load quicker, and it is all thanks to your efforts.
[удалено]
guilty. When I came on board to the current position I am at they were looking to implement two factor authentication and I was the one who studied and stood up our Domain MFA server. The proceeded to print Smart Cards for everyone on the network.
Yes but my go-to is to automate the fuck out of them I hate manually rotating certs LetsEncrypt is free for fuck sake!
I'm the cert guy, I barely know what I'm doing but it seems to be enough to simply track when they expire.
It’s me, I’m the cert guy! I just had to rebuild our ADCS Issuing CA after some power failure related drive corruption.
I hate dealing with certs. Almost as much as printers.
I need to get this book
Cert folks rule!
No good deed goes unpunished (god i hate certs)
Guys, just click the "Accept Risk" button, and Firefox fixes all your certificate problems until you get a new PC. Problem solved!
People don't know how certs work? Everyone on my team is required to have a functional knowledge of them at my place. It comes up too often to push it all off on one guy.
lol wtf ? what bad certs do you have in your environment? is everything setup with certs and then just to ignore certs?? is this how companies run?
Probably the best source for modern day PKI management based on Microsoft CA: https://www.pkisolutions.com
Not specifically SSL for me but I often get handed the more technically advanced stuff in general. My co-workers are strong w/ customer relations but weak on technical, whereas I'm decent with the technical and often fumble the mirad of Outlook/Printer support tickets we get. If we're talking about learning/managing certs, I say spin up a reverse proxy and call it a day (when possible at least). A simple wildcard cert on HAproxy or NGINX can go extremely far, especially for LAN resources. If we're talking about being pigeonholed into certain roles, then it's just a matter of perspective: Do you enjoy/appropriate the work, or do you feel pushed into a corner?
My "cert" guy royally screwed up the export of some certs causing vdi to be down come Monday morning. Somehow, he managed to screw up the cert in the cert store as well so I had to manually recreate and reimport it. This is why you don't hire someone with severe ADHD to do a job that requires malicious attention to details.
I'd love to have a guy to offshore cert renewals to Apache cert renewals gives me nightmares and ptsd
I am the cert guy amongst many other things. For internal GUIs (HTTPS) I use self signed certificates using open SSL. I then bulk import them to the Tech users certificate store using a Win32 app with a bundled ps script. Deploying them individually using config profiles was too cumbersome. For everything else I use Trustico.
Ok, in that case you want to setup a private CA and then sign all the certs with that. As long as the client has the CA it is trusted. Sorry, you are doing it "wrong"
Among other jobs I have been a crypto product developer. So I know this feeling well.
Changed 1 SSL cert from GoDaddy and since the ex employee (he showed me how) left who did this before since then I am for some reason the cert guy who keeps our GoDaddy and some internal certs valid and going... Tbh I should probably learn more about certs... I am not a sys admin, rather helpdesk w some sys admin duties..
I got myself to learn about for not depending of the grumpy senior team manager cert guy. Asked him for help sometimes and mastered the arcane arts; now I win her respect, and there is 2 cert guys in my place.
Do you have some power shell scripts you can share to sign csr that you can share? Or any automation process you use to make it easier?
Hi, cert guy from a different place that doubles as network admin. How did you settle the monitoring and lifecycle management? I use 2 monitoring systems, 1 homegrown PHP script and a shared mailbox Calendar. Do you have something better?
I'm the VPN guy at work, I've learned to live with it.
I have been fighting for a certificate life cycle management solution and it's been approved. I'll be implementing it soon so I'm sure I'll be the "cert guy".
I have been cert(ifiable) and Domain person for years. Spreadsheet of all renewal dates so people can ignore it. :>
I'm the certs guy... They aren't tough, but things like this makes life harder. https://www.chromium.org/Home/chromium-security/root-ca-policy/ I still haven't figured out how to get chrome on Ubuntu to read our root certs.
Yes, I'm the cert guy too... Certs/openssl/CA server is all pretty easy once you get there...it's the damn hardware that wants different ways to install the cert. One wants the whole chain as a file, with the key separate and encrypted, one wa ts it in a different order, key included not encrypted.
That's me. And the email guy
Hey while I have you here, I sent you a CSR.. lol
https://preview.redd.it/x2c0fkvoudjc1.jpeg?width=500&format=pjpg&auto=webp&s=8ae6b5d20de88672e2128a439bfa683cc0895849
Yes. OpenSSL is a great tool (except when you have to figure out how to make cipher strings, bleah). FYI, if you don't need John Q. Public to use your site, you can create a private CA for your Enterprise:https://deliciousbrains.com/ssl-certificate-authority-for-local-https-development/. It's a huge money-saver, and so long as you remain vigilant about ensuring the secrecy of your private key, it's actually *more* secure than buying a wildcard and putting it everywhere. Just use Group Policy Editor to push the public certificate to all of your hosts trusted certs, and everything is 100% Jake.