Hi everyone,
I recently got into trouble with this useful feature and figured that maybe if I share my experience, I can save someone the headache I went through.
AD Connect Staging Mode was introduced to provide an active-passive HA design to AD Connect – critical for anyone using Microsoft cloud services with hybrid Active Directory identities. If you aren’t familiar with AD Connect, then you can read about it here.
To utilize the Staging Mode feature, you configure a primary AD Connect instance and an identically configured Staging instance. The Staging instance runs scheduled synchronizations (just as the primary does) with the difference being the Staging instance doesn’t actually export anything to Azure AD unless Staging Mode is turned off (which should only be done if the primary instance is decommissioned or otherwise unavailable).
So far so good, right?
As an added bonus of Staging Mode, if you are introducing AD Connect into an existing hybrid AD environment and you want to confirm what the consequences will be once you unleash it, there is a reporting mechanism providing you a preview of what would happen if you were to disable Staging Mode. Some refer to it as a “what-if” mode.
This is where our story gets interesting…
I’ve used Staging Mode many times in the past to confirm what I was about to do wouldn’t cause any harm in the way of adds/deletes of users/groups/contacts etc. and it always gives me a nice warm and comfy feeling to see that a catastrophe is not about to occur – I’m a big fan of warm and comfy!
Recently, I was involved with a project and had to re-sync an on-prem AD environment with Azure AD which was disconnected several years prior after completion of a hybrid Exchange migration. The client was finally ready to stop managing two sets of identities and wanted to implement an identity management and SSO solution, so these estranged directories needed to be reunited like a happy family once again!
We had a lot of identities to marry up and needed to determine which attributes should survive as well as which groups, contacts, and users should be synced, and which items were not going to be synced (such as distribution groups). Lots of fun PowerShell scripts to write here, folks! Maybe that can be another blog sometime.
In the final implementation stage, while running a Staging Mode report, I came across some legacy Mail Enabled Security groups in AD (which existed prior to the hybrid migration) which were going to update the Office 365 versions of these groups which were actively being used for email distribution lists throughout this global organisation.
The group memberships were managed in Office 365 and the last thing we wanted to do was overwrite the memberships of these groups from the AD version of them as it would be waaaaaaaaay out of date. So, we deleted those groups from AD – which in my mind meant the Office 365 version of them would stay as cloud and un-synced. Sounds good, right?
Well, not quite…
AD Connect maintains something called a Metaverse which is a dynamic database of both AD and Azure AD directories and what the merged directory consists of (down to the attribute level on an ongoing basis, even in Staging Mode). So, since I deleted these groups in AD, AD Connect naturally decided I wanted them to be deleted in the cloud as well and it updated the Metaverse. Once I removed Staging Mode my assumption was that AD Connect would reconcile what is current in AD and Azure AD and do a fresh synchronization. What actually happened was that AD Connect already knew what its next order of business was after removing Staging Mode… pushing out whatever changes were queued up in the metaverse and DELETING THESE GROUPS in Office 365!!!
So much for Staging Mode being a harmless what-if tool! It was ready to replicate any changes processed once Staging Mode was removed, which in hindsight makes perfect sense. Sometimes what I think I know about how something works, distorts my ability to confirm how it actually works. Again, another topic for another blog!
Let’s get this cleaned up…
Since security groups are not a restorable object in Azure AD, I was forced to recreate those groups using data I had backed up at the beginning of the project. In the course of my career, I have learned to always plan for the worst while hoping for the best, so I indeed had a backup of all Azure AD group memberships – whew!
To make matters worse, I had paused the AD Connect sync schedule while doing this work and when I finally re-recreated those groups, I re-enabled Staging Mode and ran a sync and wouldn’t you know it, AD Connect once again was eager to delete those groups!
I just uninstalled and reinstalled AD Connect which created a new Metaverse and all was good from then on but once again Staging Mode to the rescue! I believe there is some irony here in using Staging Mode to help protect me from, well, Staging Mode itself!
Lessons Learned
After this ordeal, in chatting with an AD Connect SME at Microsoft, I was advised that instead of uninstalling and reinstalling I could have saved a few minutes and just cleared the connector spaces for both AD and Azure AD and essentially accomplished the same thing. But I haven’t tried this yet… please let us know if you have.
To do this, you would clear the connector space for both connectors (AD and AAD) by opening the Synchronization Service Manager, opening the Connexions tab, right-clicking on each connector and selecting “Delete…” followed by “Delete connector space only”.
So, go forth and stage away but be careful and ALWAYS plan for the worst!
