K
KielRandor
Guest
I recently enabled SNTP on my DHCP server to broadcast my time server's address (DHCP option 042) and TZ offset(DHCP option 2) to non-Windows/Non-AD devices (Switches, Routers, VOIP Devices, Etc) that don't know to look for their time server from my AD servers. I miscalculated the Offset because it rejected the Offset on the switches that got the right UTC time, but I didn't really care if the switches reported UTC time so I left it alone.
The next morning, a handful of my users reported that their Windows 10 Desktops were showing their time in UTC. I quickly jumped on my DHCP server and removed the DHCP Options from the server to prevent it from pushing it out to additional clients. However about a dozen of my 200+ desktops appear to have got this change pushed out by DHCP.
I looked in Date & Time Settings and saw that the time was set to UTC. I tried to manually reset the Time zone to our correct time zone "(UTC-04:00) Atlantic Time (Canada)" from the GUI for the effected users but a few hours later the time zone reset to UTC.
Rebooting does nothing to cure this, in fact doing so seems to reset the Time Zone back to UTC.
I did some research and I found where DHCP stores its Options in the Registry:
\\HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\Interfaces\<HASH>\DHCPInterfaceOptions\
However the value is stored in Binary and is unreadable and apparently undocumented. However research indicated that this gets repopulated when you get a DHCP lease. So thinking that this was where the DHCP options were hiding out I deleted the key, cleared my DHCP lease and requested a new one. the values in this key looked different. And my Time zone seemed to be holding at Atlantic time however the next day, my same pool of users all reported their time had reverted back to UTC.
I tried manually setting the Time Zone in the registry here:
\\HKLM\CurrentControlSet\Control\TimeZoneInformation\
But again by the next day, all my effected users reported the issue again.
I was pretty certain that my Windows time server was working properly because none of my other users/desktops were reporting this issue. It was contained to just this handful of users. However I started to collect some additional diagnostic information from both desktops that were experiencing the issue and those that were behaving normal.
First I queried DHCP using a tool I found online called DHCPTest on both good and bad clients. This tool outputs every option that your DHCP server is presenting to DHCP clients. This was to confirm that I had successfully removed the DHCP Options from my server and that they were not still being pushed out somehow. DHCPTest also confirms that DHCP is working properly as part of its testing. Both sets of clients confirmed that DHCP was not pushing the Options out to anyone and that DHCP seems to be working fine.
Next I used TZUTIL to confirm my Time Zone was correct. I also used it to set the Time Zone on my effected clients. But the time quickly reset to UTC by next day.
Finally I confirmed my Windows Time Server and client machine Time Services were working properly using W32TM. This confirmed that my time Server was configured correctly, that my clients were receiving their time from the Time server and that they were in fact properly in sync.
I did alot of research found several articles that seemed to point to Windows 10's Automatic Time Zone behavior as a potential cause. All of these seemed to point to a recent Windows Update as a cause(we didn't receive any updates on these effected clients that were not also pushed out to all my clients that are working properly.) or that you have the option "Set Time Zone Automatically" turned on. (We have it off on our clients)
I learned that the Automatic time zone feature may default to UTC if it can not accurately resolve the Time Zone using GPS, Cell Towers, Region or IP Addresses, but since it's off and manually set, this shouldn't be our issue. For giggles I did cycle it on on a couple of our effected clients to see if they might autodetect our proper time zone, but this doesn't seem to have worked so I turned it back off.
So by this point I felt like I'd verified that it was not a setting in our Registry or a setting from our DHCP or Windows Time Server causing the continual change so I started looking closer at my users.
We use VMWare Horizon managed virtual desktops and VMWare Horizon UEM to manage user profile settings. I confirmed that my Golden Images were all fine and set to the correct Time Zone and Time.and built several new desktops.
In UEM, I purged all of the effect user's UEM captured user settings. Then I gave my effected users completely new virtual machines and had them log back in. This is effectively like sitting a brand new user with no profile down at a brand new Windows 10 machine. It felt like overkill but I was really at the end of things to try. I configured the users on their new machines, confirmed their Time Zone was correct and then touched the nearest piece of wood for luck.
However the next day the issue came back.
None of my servers(on the same AD network) have been effected and no new users have been effected. It is contained fully to this handful of users.
I'm at a complete loss. I've done everything I can think of. My colleagues are out of ideas as well. I know this could easily be any number of things. It might be a VMWare thing, an AD thing, a DHCP thing or a Windows 10 thing.
Any suggestions would be appreciated.
Crossposted to respective Reddit and VMWare forums as well.
More...
The next morning, a handful of my users reported that their Windows 10 Desktops were showing their time in UTC. I quickly jumped on my DHCP server and removed the DHCP Options from the server to prevent it from pushing it out to additional clients. However about a dozen of my 200+ desktops appear to have got this change pushed out by DHCP.
I looked in Date & Time Settings and saw that the time was set to UTC. I tried to manually reset the Time zone to our correct time zone "(UTC-04:00) Atlantic Time (Canada)" from the GUI for the effected users but a few hours later the time zone reset to UTC.
Rebooting does nothing to cure this, in fact doing so seems to reset the Time Zone back to UTC.
I did some research and I found where DHCP stores its Options in the Registry:
\\HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\Interfaces\<HASH>\DHCPInterfaceOptions\
However the value is stored in Binary and is unreadable and apparently undocumented. However research indicated that this gets repopulated when you get a DHCP lease. So thinking that this was where the DHCP options were hiding out I deleted the key, cleared my DHCP lease and requested a new one. the values in this key looked different. And my Time zone seemed to be holding at Atlantic time however the next day, my same pool of users all reported their time had reverted back to UTC.
I tried manually setting the Time Zone in the registry here:
\\HKLM\CurrentControlSet\Control\TimeZoneInformation\
But again by the next day, all my effected users reported the issue again.
I was pretty certain that my Windows time server was working properly because none of my other users/desktops were reporting this issue. It was contained to just this handful of users. However I started to collect some additional diagnostic information from both desktops that were experiencing the issue and those that were behaving normal.
First I queried DHCP using a tool I found online called DHCPTest on both good and bad clients. This tool outputs every option that your DHCP server is presenting to DHCP clients. This was to confirm that I had successfully removed the DHCP Options from my server and that they were not still being pushed out somehow. DHCPTest also confirms that DHCP is working properly as part of its testing. Both sets of clients confirmed that DHCP was not pushing the Options out to anyone and that DHCP seems to be working fine.
Next I used TZUTIL to confirm my Time Zone was correct. I also used it to set the Time Zone on my effected clients. But the time quickly reset to UTC by next day.
Finally I confirmed my Windows Time Server and client machine Time Services were working properly using W32TM. This confirmed that my time Server was configured correctly, that my clients were receiving their time from the Time server and that they were in fact properly in sync.
I did alot of research found several articles that seemed to point to Windows 10's Automatic Time Zone behavior as a potential cause. All of these seemed to point to a recent Windows Update as a cause(we didn't receive any updates on these effected clients that were not also pushed out to all my clients that are working properly.) or that you have the option "Set Time Zone Automatically" turned on. (We have it off on our clients)
I learned that the Automatic time zone feature may default to UTC if it can not accurately resolve the Time Zone using GPS, Cell Towers, Region or IP Addresses, but since it's off and manually set, this shouldn't be our issue. For giggles I did cycle it on on a couple of our effected clients to see if they might autodetect our proper time zone, but this doesn't seem to have worked so I turned it back off.
So by this point I felt like I'd verified that it was not a setting in our Registry or a setting from our DHCP or Windows Time Server causing the continual change so I started looking closer at my users.
We use VMWare Horizon managed virtual desktops and VMWare Horizon UEM to manage user profile settings. I confirmed that my Golden Images were all fine and set to the correct Time Zone and Time.and built several new desktops.
In UEM, I purged all of the effect user's UEM captured user settings. Then I gave my effected users completely new virtual machines and had them log back in. This is effectively like sitting a brand new user with no profile down at a brand new Windows 10 machine. It felt like overkill but I was really at the end of things to try. I configured the users on their new machines, confirmed their Time Zone was correct and then touched the nearest piece of wood for luck.
However the next day the issue came back.
None of my servers(on the same AD network) have been effected and no new users have been effected. It is contained fully to this handful of users.
I'm at a complete loss. I've done everything I can think of. My colleagues are out of ideas as well. I know this could easily be any number of things. It might be a VMWare thing, an AD thing, a DHCP thing or a Windows 10 thing.
Any suggestions would be appreciated.
Crossposted to respective Reddit and VMWare forums as well.
More...
