Here is a recap of what we covered in part 2. We followed a Citrix article relating to the issue of Citrix Director not reporting logon time for new 1909 Citrix environment. We looked for potential corrupt WMI providers. This testing showed the Citrix UPM WMI plugin was functioning correctly, albeit reporting zero times for all metrics of a user login. Next, we covered another Citrix article. We performed tests using Microsoft scripts to report the logon times from the operating system. This test reported the results correctly again albeit showing zeros for all metrics.
After following the recommended troubleshooting steps from Citrix on the subject, as well as numerous other posts which support Citrix recommendations as to the common causes of this issue. No steps so far have resolved the problem, we have though identified that the issue is likely caused by something within the operating system rather than a Citrix component fault.
Change in focus
Focusing now on Windows, I first reviewed the state of all of the services for the system, running, disabled, and the services which were set to automatic but were not running. Two items had been applied to the operating system up to this point. Both items make significant changes to the behavior of Windows, though usually for the better. They are Citrix optimisers, and the Citrix image sealing script, BIS-F had been applied to the image. There are a number of changes which these both make to the standard Windows server running state. The majority of the changes they make are registry settings and disabling unused services. The status of all services did, however, check out, and the services expected to be running, and disabled were correct.
Moving the troubleshooting on to event viewer, Application and System event logs were producing no error or warning event logs relating to the logon of a user. Digging deeper into the application-specific event logs, the Citrix and FSLogix logs also did not show any faults during the login process. Moving onto the extended Microsoft event logs, I reviewed the Group Policy/Operational log first. I found that the event log was not functioning and displayed the following error.
All other event logs up to this point were functioning, so I did not believe the event log service was stopped as per the description in the error. Next, I went to the properties of the event log and found the path for the event log was not set correctly. I changed the path for the event log to the correct path, refreshing the group policy / Operational event log, and it was now functioning correctly.
I retested a logon and went through the same tests as I previously had, running the following command from PowerShell.
get-wmiobject -namespace rootcitrixprofilesmetrics -class logontimings
This time it reported a slightly different result
With fixing the fault with the Group Policy/Operational event log, the user’s login session was now able to record the group policy processing times as shown above.
Looking into what other event logs may be involved in the process of logons, and potentially recording login times as well as to what other event logs were not working. Of the 300+ event logs, running on a Windows server, I found that approximately 50% of these were not functioning and producing the same error.
The cause of the event logs failing was the path of the events logs which were not functioning were incorrect. The event logs which were working had a valid path. This pointed pretty clearly to what the cause of this issue was. The path of all event logs had been redirected as part of the BIS-F script. The first time the image was sealed, and the script was run the event logs were redirected to (D:). This seems to have worked correctly and is the path shown in all of the event logs that were not functioning. At a later point, the persistent disk was changed to an (E:), when the script was next run, the event logs were redirected to the new path, well, so it seemed. When rerunning the BIS-F script after becoming aware of the issue, the script reported all logs were successfully redirected, but not all event logs were redirecting. Rechecking a few of the logs I knew to be producing the error, they were still not working, so the BIS-F script had changed some but not all event logs.
The issue impacted one hundred fifty event logs. Before attempting to fix all of them, I took the opportunity to try and determine precisely which event logs were required to allow the logon times to be recorded within the WMI object, which in turn would allow Director to record the results also.
After a LOT of trial and error, I finally found the culprit. The event logs required to successfully record logon time for Windows, are
- Microsoft-Windows-Group Policy-Operational
- Microsoft-Windows-TerminalServices-LocalSessionManager/Operational
Now with both event logs functioning, I again re-ran the earlier test I had run, executing the following command from PowerShell.
get-wmiobject -namespace rootcitrixprofilesmetrics -class logontimings
The result returned this time was.
Success!!, As you can see all five metrics required to record a users logon time successfully, GroupPolicyStart, GroupPolicyComplete, ProfileLoadStart, ProfileLoaded, DesktopReady, are finally populated. I moved back to Citrix Director next to check on the status and, success; it was also now showing the logon time correctly.
Moving back to the broader issue, I found with the event logs. I was recommended a script by a colleague to achieve a global reset for all event logs. You can find this script at Change the path of the Event Log file. Executing the script allowed all event logs to be set to the default path %SystemRoot%System32WinevtLogs. I was then able to rerun the sealing script BIS-F, which successfully redirected all of the event logs to the persistent drive for the system as was originally intended.
What I learned
Learnings I have found from this issue have been.
Citrix
The mechanism that Director uses to be able to record the user login times is built upon many layers and systems, involving.
- Citrix Director
- Citrix Broker
- Citrix VDA
- Citrix UPM WMI plugin
- Windows WMI counters
Windows Operating System
I found all Citrix components leverage WMI metrics provided by the operating system. No metric used to measure a users logon time is generated or unique to Citrix components. I also found that all WMI metrics involved are based on events that are recorded in the event logs of the operating system
Event Logs
And finally, the larger unknown fact was in the case of the logon metrics WMI is built to pull the details from event logs, specifically the following event logs
- Microsoft-Windows-Group Policy-Operational
- Microsoft-Windows-TerminalServices-LocalSessionManager-Operational
However, if the logs are not functioning correctly, as was my case, then no metrics will flow up to the higher components. This results in no logon times being recorded at any level.
