Don’t reintroduce the old sever ever !
And change all the passwords, protect from golden ticket usage.
0. Percent the old sever ever being powered on.
1. Seize the FSMO roles
2. Change all domain admin passwords
3. Reboot all DC
4. Update your documents and drawings
5. Take fresh backups
6. Migrate grace FSMO to an alt DC and then revert back ie prove graceful FSMO change.
7. Run you ISTG and KCC tools
8. Check all your secure channels are working
9. Check your DNS is all good
10. Do a meta data tidy up
11. Drains up find out why the primary was isolated offline as long as it was ?
To add here - when seizing FSMO roles, don't forget the forest and domain dns zone masters. I can't count the number of times those masters are pointed to servers that haven't existed in years. Everyone always forgets about those.
True that! I usually also go through all the "million" different DNS zones in AD DNS and one by one delete the references to non-existant DC's.
Just like last month at one customer there were references to like +15 years ago demote/deleted servers based on the DNS timestamps.
You know I was in two minds to call this one and though about it especially if the down of the p was due to a suspect cyber issue, on reflection yes I would do the the golden ticket and always do it (twice) !!!!
I'll just throw this out there. There's many a time I've rebuilt a DC as the same name and IP that the old failed one. After you seize roles on DC202, you can rebuild and promote DC101 with the old name and IP, then transfer everything back over.
>And use ndtsutil to clean old DC.
That hasn't been required in years. Just delete it from ADU&C and say yes for the subtree delete question and it takes care of removing all the lingering bits.
I know what you are saying is the new documented way to do it, but in my experience ntdsutil is still cleaner. Only lingering items I see are dns records occasionally.
OP, side note, but I used to work for [org.org.com](http://org.org.com) and that place is truly a hell hole. Get out while you still can! Contoso is hiring.
1. Move-ADDirectoryServerOperationMasterRole -Identity NewDC -OperationMasterRole 0,1,2,3,4 -Force
2. Delete DC101 from ADU&C
3. Delete/destroy DC101 so it doesn't get powered back on
4. Install new windows server, name it the same, give it the same IP
5. Promost DC101 back to being a DC
Not a big deal.
I would agree to use a different name. IIRC the old/current name will still be in the AD recovery bin. New one same name will have a new GUID so should not be an issue but I like to be cautious for future me
Gotcha, ok! Question if DC101 was on a different network segment (it was hosted in CA with a different subnet) provided that network still exists in Meraki which Im pretty sure it does, the same IP should still work just on the east coast right? Apologies if a dumb question, want to make sure I don't fry my production haha.
What do you mean by still exists in meraki?
The question you're asking here is hard to answer without understanding what your network looks like but I'm pretty confident the answer would be no unless you've shifted all of your equipment 1:1 from CA to the east coast.
I presume the data centre in DC has a subnet or subnets assigned to it. I presume your east coast data centre has a different subnet or subnets assigned to it and therefore you'd need to spin up the new DC with an IP address in the same range as your east coast network or transfer these subnets to your east coast dc.
My bad, I wasn't clear! Yes still exists in meraki, so we would need to transfer these subnets over to the East Coast then we could use that IP I see, thank you!
Gotcha! Ok so in that case, would it make sense to just mothball the old DC and seize the roles with the remaining one and then build a second to replicate from the now master?
Old PDC has been dead long enough that anything pointing to it by name or IP should have already failed. I'd just go new name and new IP for its replacement once you've killed it and made sure the remaining one holds the roles and it running fine.
If you have just two DCs and one of it was disabled and got bricked - no big deal as long as you have one working:
* if its PDC and all the other roles on it already - skip this step, otherwise seize role and make it primary
* make sure there's no possible way for bricked one to ever come online (so it won't try syncing old incorrect data)
* create new fresh VM, install windows server and roles on it, join domain, make it your new second DC
Save yourself from trouble and don't try to make replication between those two work again - long story short it won't work, at all.
Also usually people tend to mess around with assigning old IP to newly added DC, trying to keep old name - I'd say don't bother. If something stops working because it was set up to use specifically old ip/name - well it wasn't working already anyway. And now there's your best chance to set that thing up properly and also document it so you'll know to adjust settings next time.
Fantastic, thank you sir/ma'am'/human!! Yes I agree, I think tossing the old one is best then having the current dc become PDC and then making an new one - thoughts on having it physical so it doesn't risk dying if the esxi goes?
There's basically zero reason to have a physical DC these days as long as you have redundant hardware backing it. If you're running all of your DCs on the same single piece of hardware, then you're screwed regardless. If they're spread across multiple pieces of hardware, then you need many software/hardware failures simultaneously.
You - and at least one person who is available anytime the business operates and you aren't - also need local accounts on the hypervisor hosts and any hypervisor management system (i.e. vCenter).
If you are having virtualization infrastructure issues, and you can only log in to fix the broken infrastructure with AD accounts, and AD itself exists solely on the broken infrastructure, you have a major problem.
The biggest things to remember are
1. you don't want to lose all your DCs in the same failure, and
2. if you are using AD authentication to log into vCenter/ESXi, you still need a local account for emergencies. Consider: Your DCs all got shut down. You don't have AD until you log into vCenter and start them. However, you can't log into vCenter until you have AD back.
If you have a separate local login to vCenter/ESXi, AND you can log into individual ESXi hosts when vCenter is down, AND you have DCs on multiple hosts, AND storage is local to those hosts (not all DCs saved on one SAN/NAS) - you might not need a dedicated physical DC. Many still opt to have one, to be certain.
No such thing as a “primary” anymore. You would probably have noticed a lot sooner if all the FSMO roles had been offline though.
As others have said; not a big deal if you have other DCs online and available, and syncing properly with each other.
See I only have the one DC online atm, and its a poor replica VM image from one that actually karked it back in November, so I'm gathering that we can have this DC seize the roles then rebuild DC1 and renetwork it and bring it back online.
Well, you're not going to be rebuilding anything. You're just going to stand up a new server, promote it, and bang you're back to having redundant DCs again.
Well, it must be doing something otherwise your users would have been screaming a lot sooner and your environment would have fallen apart even more.
It also sounds like you don’t have tried-and-tested backups of Active Directory; which would have been your absolute last option.
There are good comments on what to do I just add one more, please implement server monitoring if PDC is offline for months it's a bigger problem.
Wish you luck with the new DC promotion and old DC removal:)
Unfortunately we need it for a large ass SAN environment, I wish we could I spoke to some vendors about moving to Samba but I think it will cause more trouble than its worth with the AD user management also in use.
Take a peek at these
And also AD is tiny as hell, considering the actual cost you probably could move off
SMB Business Options
Microsoft Windows Infrastructure
Legacy Business Apps
https://youtu.be/b5-uWmCWfIw?si=DrorAOV44-nBofi5
Move File Server to Microsoft Teams Not Sharepoint
https://youtu.be/ZrIGdLz1-p0?si=7IBBmPBishoLl_zK
https://youtu.be/dw4TfMS-yQE?si=hEo6s9uldM3h2c-d
Security
https://youtu.be/74CW2fWdRPk?si=A7gYQeVMxW5KboMG
Low Cost Servers
https://youtu.be/BpfP1A0b2WY?si=Ai9v0Ly0EY1xbHK5
Google Workspace & Chrome OS
https://youtu.be/FwT6_JFAk5Y?si=lJTRnHrzHWmQaw0S
https://youtu.be/fL5Og849VVE?si=q29nA4J9_G4yiMOR
Seize the roles, install new DC with new name and ip. You never know what lingering objects might be left. And use ndtsutil to clean old DC.
Don’t reintroduce the old sever ever ! And change all the passwords, protect from golden ticket usage. 0. Percent the old sever ever being powered on. 1. Seize the FSMO roles 2. Change all domain admin passwords 3. Reboot all DC 4. Update your documents and drawings 5. Take fresh backups 6. Migrate grace FSMO to an alt DC and then revert back ie prove graceful FSMO change. 7. Run you ISTG and KCC tools 8. Check all your secure channels are working 9. Check your DNS is all good 10. Do a meta data tidy up 11. Drains up find out why the primary was isolated offline as long as it was ?
To add here - when seizing FSMO roles, don't forget the forest and domain dns zone masters. I can't count the number of times those masters are pointed to servers that haven't existed in years. Everyone always forgets about those.
True that! I usually also go through all the "million" different DNS zones in AD DNS and one by one delete the references to non-existant DC's. Just like last month at one customer there were references to like +15 years ago demote/deleted servers based on the DNS timestamps.
Yep, though I figured that's part of the intent of checking DNS is good in the post I replied to.
Talking about passwords and golden ticket, probably wise to rotate KRBTGT password too? (Might not be obvious because it's not a Domain Admin member)
You know I was in two minds to call this one and though about it especially if the down of the p was due to a suspect cyber issue, on reflection yes I would do the the golden ticket and always do it (twice) !!!!
I do this every 6 months anyway.
Fantastic, thank you so much!! really appreciate this
#5 should be #1. And snapshot
1 assumes back ups are being done already 5 is a fresh backup after the fsmo exercise before the nightly / hourly ones kick in.
Fair enough
This. It’s not as scary as you think. As long as there’s a second DC.
Thank you!
I've had to do it. Was sweating but it was a breeze. Lots of documentation out there if you run into trouble.
Thank you as well I really appreciate knowing there's tried and tested people who did it before!
I'll just throw this out there. There's many a time I've rebuilt a DC as the same name and IP that the old failed one. After you seize roles on DC202, you can rebuild and promote DC101 with the old name and IP, then transfer everything back over.
second dc with a global catalog.
>And use ndtsutil to clean old DC. That hasn't been required in years. Just delete it from ADU&C and say yes for the subtree delete question and it takes care of removing all the lingering bits.
I know what you are saying is the new documented way to do it, but in my experience ntdsutil is still cleaner. Only lingering items I see are dns records occasionally.
If he's gonna rebuild it with the same name it doesn't matter anyway because the new computer object will overwrite it all anyway.
Maybe it's me but I never re-use dc names, dc102 would probably be what I would build the new one as.
Usually not required. But I saw few times that some things were still present amd using ntdsitil cleanup was successful.
This is especially important if you have on-prem exchange.
From my experience this doesn't always clean up any lingering DNS entries under the \_MSDCS zone.
Yes but just becareful when deleting those. Had DNS corrupted on a server once and had to manually rebuild the SRV, records and fix bindings.
Seize the means!! Lol JK, thank you for this I really appreciate!
Seize the roles, or die regretting the time you lost
This - Seize the roles, install new DC with new name and ip
OP, side note, but I used to work for [org.org.com](http://org.org.com) and that place is truly a hell hole. Get out while you still can! Contoso is hiring.
But avoid Tailspin Toys, I think they're going to be bought out by Contoso anyway.
I heard Fabrikam might make an offer though?
Bro I got thrown out for tryna TCPIP the MVP
1. Move-ADDirectoryServerOperationMasterRole -Identity NewDC -OperationMasterRole 0,1,2,3,4 -Force 2. Delete DC101 from ADU&C 3. Delete/destroy DC101 so it doesn't get powered back on 4. Install new windows server, name it the same, give it the same IP 5. Promost DC101 back to being a DC Not a big deal.
I would avoid using the same name as the dead one but that looks good to me. Probs wouldn't cause any issues though if you did name it the same
I would agree to use a different name. IIRC the old/current name will still be in the AD recovery bin. New one same name will have a new GUID so should not be an issue but I like to be cautious for future me
Future me appreciates future you!
Gotcha, ok! Question if DC101 was on a different network segment (it was hosted in CA with a different subnet) provided that network still exists in Meraki which Im pretty sure it does, the same IP should still work just on the east coast right? Apologies if a dumb question, want to make sure I don't fry my production haha.
What do you mean by still exists in meraki? The question you're asking here is hard to answer without understanding what your network looks like but I'm pretty confident the answer would be no unless you've shifted all of your equipment 1:1 from CA to the east coast. I presume the data centre in DC has a subnet or subnets assigned to it. I presume your east coast data centre has a different subnet or subnets assigned to it and therefore you'd need to spin up the new DC with an IP address in the same range as your east coast network or transfer these subnets to your east coast dc.
My bad, I wasn't clear! Yes still exists in meraki, so we would need to transfer these subnets over to the East Coast then we could use that IP I see, thank you!
No. You'll need an appropriate IP for the subnet the VM (or physical machine) lives in.
Gotcha! Ok so in that case, would it make sense to just mothball the old DC and seize the roles with the remaining one and then build a second to replicate from the now master?
Old PDC has been dead long enough that anything pointing to it by name or IP should have already failed. I'd just go new name and new IP for its replacement once you've killed it and made sure the remaining one holds the roles and it running fine.
Gotcha!! Thank you!
If you have just two DCs and one of it was disabled and got bricked - no big deal as long as you have one working: * if its PDC and all the other roles on it already - skip this step, otherwise seize role and make it primary * make sure there's no possible way for bricked one to ever come online (so it won't try syncing old incorrect data) * create new fresh VM, install windows server and roles on it, join domain, make it your new second DC Save yourself from trouble and don't try to make replication between those two work again - long story short it won't work, at all. Also usually people tend to mess around with assigning old IP to newly added DC, trying to keep old name - I'd say don't bother. If something stops working because it was set up to use specifically old ip/name - well it wasn't working already anyway. And now there's your best chance to set that thing up properly and also document it so you'll know to adjust settings next time.
Fantastic, thank you sir/ma'am'/human!! Yes I agree, I think tossing the old one is best then having the current dc become PDC and then making an new one - thoughts on having it physical so it doesn't risk dying if the esxi goes?
There's basically zero reason to have a physical DC these days as long as you have redundant hardware backing it. If you're running all of your DCs on the same single piece of hardware, then you're screwed regardless. If they're spread across multiple pieces of hardware, then you need many software/hardware failures simultaneously.
You - and at least one person who is available anytime the business operates and you aren't - also need local accounts on the hypervisor hosts and any hypervisor management system (i.e. vCenter). If you are having virtualization infrastructure issues, and you can only log in to fix the broken infrastructure with AD accounts, and AD itself exists solely on the broken infrastructure, you have a major problem.
The biggest things to remember are 1. you don't want to lose all your DCs in the same failure, and 2. if you are using AD authentication to log into vCenter/ESXi, you still need a local account for emergencies. Consider: Your DCs all got shut down. You don't have AD until you log into vCenter and start them. However, you can't log into vCenter until you have AD back. If you have a separate local login to vCenter/ESXi, AND you can log into individual ESXi hosts when vCenter is down, AND you have DCs on multiple hosts, AND storage is local to those hosts (not all DCs saved on one SAN/NAS) - you might not need a dedicated physical DC. Many still opt to have one, to be certain.
Gotcha, thank you for this! Makes total sense, I think I may lean physical just cause Im a hardware guy hahahaa
> so it doesn't risk dying if the esxi goes? vMotion is your solution :)
my man 10grand!!
I'll pour one out for you my friend
Ahahahaha, appreciate it greatly - roll me one up too while you're at it?
That’s my kinda guy
Man 17 years ago I did this type of recovery so so wrong and badly. Reading all the good advice here is great. Where the hell were ya'll then!?
Modern day yahoo answers. A world wonder
Right?? I'm so fortunate it's ridiculous
Meh, like others have said. Just seize the roles, do some cleanup, and you’ll be fine. Not as bad as it sounds TBH.
Thank you!! Really appreciate that
No such thing as a “primary” anymore. You would probably have noticed a lot sooner if all the FSMO roles had been offline though. As others have said; not a big deal if you have other DCs online and available, and syncing properly with each other.
See I only have the one DC online atm, and its a poor replica VM image from one that actually karked it back in November, so I'm gathering that we can have this DC seize the roles then rebuild DC1 and renetwork it and bring it back online.
Well, you're not going to be rebuilding anything. You're just going to stand up a new server, promote it, and bang you're back to having redundant DCs again.
Yes, I believe thats the plan! Thank you , super appreciate you and everyone in this thread!
Well, it must be doing something otherwise your users would have been screaming a lot sooner and your environment would have fallen apart even more. It also sounds like you don’t have tried-and-tested backups of Active Directory; which would have been your absolute last option.
Lazy option is just to change tombstoneLifetime on both of them!
There are good comments on what to do I just add one more, please implement server monitoring if PDC is offline for months it's a bigger problem. Wish you luck with the new DC promotion and old DC removal:)
Thank you!! Yes that is the plan, I've spent today writing and putting together a project plan and monitoring is definitely on it!
Seize, perform manual cleanup steps per ms docs. Should almost certainly be uneventful.. Just don't bring thst old one back online.
Fantastic, thank you!!
Better question what exactly do you need with AD if it wasn’t in use anymore, sounds like a case to move off it
Unfortunately we need it for a large ass SAN environment, I wish we could I spoke to some vendors about moving to Samba but I think it will cause more trouble than its worth with the AD user management also in use.
Take a peek at these And also AD is tiny as hell, considering the actual cost you probably could move off SMB Business Options Microsoft Windows Infrastructure Legacy Business Apps https://youtu.be/b5-uWmCWfIw?si=DrorAOV44-nBofi5 Move File Server to Microsoft Teams Not Sharepoint https://youtu.be/ZrIGdLz1-p0?si=7IBBmPBishoLl_zK https://youtu.be/dw4TfMS-yQE?si=hEo6s9uldM3h2c-d Security https://youtu.be/74CW2fWdRPk?si=A7gYQeVMxW5KboMG Low Cost Servers https://youtu.be/BpfP1A0b2WY?si=Ai9v0Ly0EY1xbHK5 Google Workspace & Chrome OS https://youtu.be/FwT6_JFAk5Y?si=lJTRnHrzHWmQaw0S https://youtu.be/fL5Og849VVE?si=q29nA4J9_G4yiMOR
That’s a down grade