I learned a lot about Windows Time Service and related items last night.
For all the domain controllers and computers to talk nicely to each other on a Windows network (and other networks relying on Kerberos/most modern authentication mechanisms), the system times need to be relatively close to each other. Not to mention if your mobile devices (say Blackberries) are more than two minutes or so off of your workstations you will have all sorts of users who are irritated because they looked at their workstation and thought they were late, only to get to the meeting and see on their mobile device they weren’t late at all.
The general idea of time synchronization on a Windows network (assuming 2003 Server) is that there is one server for the domain (the one that holds the PDC Emulator role) that is authoritative for time synchronization. That system’s clock is to be set with a hardware device, the system BIOS clock or by using NTP to talk to a reliable time source over IP. All of the other servers get their time using "domhier" which is the domain hierarchy Microsoft has created that hopefully efficiently synchronizes time (talk to the closest server, make sure there is a small variance in what you expected, etc.). Then client workstations synchronize their time with available domain controllers. This is over-simplified, of course (there are all kinds of neat algorithms and such Windows uses to try to make this process efficient).
In the issue I was working on last night, the PDC Emulator role had been moved from one server to another to another. Microsoft has knowledge base articles (http://technet2.microsoft.com/windowsserver/en/library/4a63190b-c594-4d43-9195-e54e4cb89d251033.mspx?mfr=true, Link ID 91969 and others) that explain what to do on the old PDC Emulator so that it does not think it is the authoritative time server for the domain anymore. I completed these steps (along with the steps required on the new PDC Emulator) and confirmed every domain controller in the domain except the PDC emulator was set to point to domhier. I configured the PDC emulator (domain controller on a virtual machine) to point to a reliable NTP server online. Happy I had built a Microsoft-approved Windows time configuration, I waited for the time synchronization to happen, then I forced it with a command line (w32tm /resync /rediscover). The PDC emulator’s time would synchronize with the NTP server and then the time would change back within a minute.
I re-verified everything three times. Everything is right, but the whole domain is getting the wrong time (relative to atomic time) because the PDC won’t keep the correct time with the NTP server.
I was pulling my hair out.
I found out how to turn on debugging in the Windows Time Service (http://support.microsoft.com/kb/816043).
After turning it on, there was not much help in the logs. They would just show the time changing backward about 4 minutes after every time synchronization. It just didn’t make sense.
Until I had a revelation…
Here’s the reason for this article…because after hours of work and searching online, no one seems to have documented this…
Microsoft Virtual Server 2005 was changing the time on the guest operating system (Windows Server 2003) running the PDC Emulator WITHOUT LOGGING ANYTHING ON THE GUEST.
There you have it. I haven’t dug enough to see if it logged anything in Virtual Server or not, but it would have been really really nice if it had let me know somehow on the guest OS that it was changing the time.
I setup the Virtual server to sync it’s time with the NTP server and all is well.