Citrix XenApp 6.0 and the Cries of "I Can't Log In!"

Note: The post below describes an issue with Citrix XenApp 6.0 on Windows Server 2008 R2.  The environment has a variety of client devices, including Wyse Winterm R90LW and Winterm 9150SE thin clients, as well as a number of PC's with Citrix Online Plugin 12.1 or Citrix Receiver Enterprise 3.3.0 (also known as "Citrix Receiver (Legacy PNA)").

Recently my fellow IT people and I started receiving some support calls from our users that they weren't able to log into Citrix.  This struck us all as odd, because we were able to log in without issues, and not everyone at a given location was affected--often only one at each office and at several offices no complaints at all.  After a few tries, everyone was supposedly able to finally log in.  We took a few minutes to scratch our heads and compare notes on the few calls we'd received.  There were no immediately obvious common threads based on the information we had.  We wrote it off as a fluke, as it had happened the morning after we deployed patches to the XenApp farm.  We've seen Microsoft patches that have caused some strangeness with XenApp until a second post-install reboot happens.

After writing this off, we went about our business for a few days.  Again, some of our users were having trouble logging in.  This time we were able to track it down to logon failures.  The users in question were entering the wrong username or password.  Once that happened, the screen went black and no further feedback was presented to the user.  Given this, we tried to find a resolution.  We found several vague references to XenApp hotfixes and updated client applications.  Eventaully we determined that the referenced hotfixes were included in the Hotfix Rollup package we had already installed several months before, and the clients, the recently-released Citrix Receiver 3.3.0, supposedly contained the client-side fixes necessary.

At this time, no further progress was made on a solution.  A coworker did determine, however, that the issue did not appear until we had rolled out the newest image to our thin clients.  Every person who was affected was logging in from a Wyse Winterm R90LW terminal with the newest image and all relevant security patches as of October 15, 2012.  This image included the Citrix Receiver Enterprise 3.3.0 client software.  The prior image in play used the Citrix Online Plugin 11.1, the last client software version that included pn.exe.  The importance of the upgrade from Online Plugin to Receiver in relation to this problem was not evident yet.

A day or two later, the issue returned.  This time, we tracked it down to the problem being users whose passwords had expired.  Otherwise, all symptoms were the same.  This prompted more investigation.

As a result of this investigation, both the previously-mentioned coworker and I determined that we could sometimes reproduce the issue, but not always, and never with our own user accounts.  The same user experiencing the problem could, on occasion, get the desired behavior (green screen indicating that the password had expired with an OK button to allow the user to proceed to the password change "screen") through no special action--merely persistence.  The only thing we could determine with absolute certainty was that the Wyse Winterm 9150SE devices we had weren't affected--it was completely impossible to reproduce the problem there.  We weren't sure why.

After some more research, I determined that I needed another Citrix Hotfix Rollup package that had just been released three weeks before.  Not only were there security related fixes in the rollup package, there were also some logon status fixes that seemed at least somewhat relevant to our environment.  I scheduled the installation for January 7.  Due to unforseen circumstances, this got pushed back to January 8.  Initial results were promising, so I wrote it off as fixed.

Finally, on January 14, I received another phone call with the same symptoms, clearly indicating that the hotfix rollup (rollup 2, for the curious).  I helped the user work around her problem and then I set to consulting the Google machine again.  This time, I stumbled across this post, and the comment there struck a chord.  I remembered that the Citrix Receiver used a Citrix-provided status box instead of the Windows-provided status reporting you see when logging in with Remote Desktop.  That prompted me to make the registry change described there, and one more for safety.  The changes, in .reg format, are:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Citrix\Logon]
"DisableStatus"=dword:00000001

[HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\Logon]
"DisableStatus"=dword:00000001

I applied that change to a single VM in our XenApp farm, rebooted it, and tested the login.  The logon process now showed the Windows-provided logon status messages instead of Citrix's.  Given that success, I applied the registry change to the remaining 3 VM's in our XenApp farm and let the daily automatic reboot take care of rebooting them for me.  I've been in touch with several users who entered their password incorrectly and several whose passwords had expired since I made the registry changes, and I've confirmed that this appears to have completely fixed the problem.

Now to the more interesting question--why did this not appear until we rolled out Citrix Receiver Enterprise 3.3.0 to all the users?  Well, it appears that this is due to the previous versions of the Citrix clients:

• The Winterm 9150 uses a version 8.x Program Neighborhood Agent client.  This version doesn't support the "advanced" status messages XenApp 6.0 is capable of sending to the client.
• The Winterm R90LW clients' previous image used the Citrix Online Plugin 11.1 client.  This client also doesn't support the XenApp 6.0 status messages.
• The Online Plugin 12.1 that we had on all the PC's does support those status messages, but no one ever ran into the problem there because they log onto the PC first, where they would be prompted to change their paswords before a Citrix login ever appeared.

A few closing notes:

• I wish the Citrix management console had a checkbox somewhere in the published application configuration (for the Desktop "application") to turn on or off the "advanced" status messages instead of having to make registry changes.
• I also wish the generic Citrix Receiver supported serial port redirection, but instead I have to deploy Citrix Receiver Enterprise, which has other features we don't want or need, to my thin clients because I have serial port printers and serial port signature capture pads that need to work with ICA sessions.
• And let's not forget that I'm still complaining about Citrix taking away pn.exe, which made configuring ICA connections for the thin clients stupidly simple; instead I now have to use an annoying tool called Citrix Quick Launch and save a .ica file to deploy to the clients.  Don't get me wrong--Citrix Quick Launch is a useful tool for troubleshooting purposes, but it's annoying if you're trying to actually produce a configuration that's usable long-term.

At any rate, I guess I should quit rambling now...