More IE 10 and Group Policy Preferences

As the title of this post suggests, I'm going to take another stroll down the IE 10 and Group Policy lane today.  After my last post, a friend and former co-worker of mine e-mailed me to thank me for the post, as it saved him some time researching to solve the same problem.  He also posed a question about how to handle managing settings since the Internet Explorer Maintenance item gets yanked out of the Group Policy Management Editor once IE 10 is installed, and any settings defined there don't apply to IE 10.  I had run into this, but hadn't needed to worry about it yet so wasn't sure of the answer.  I set to work on it and found a rather annoying answer to the problem.

In order to use Group Policy to control settings like the Home page (non-forcibly), Proxy settings, and the SSL/TLS options on the Advanced tab of the Internet Options control panel applet, you use our new friend, Group Policy Preferences (GPP).  However, Microsoft has decided to give us all a giant middle finger by making it impossible to edit GPP for IE 10 anywhere except Windows Server 2012 and Windows 8.  I don't understand how Microsoft can, in good conscience, do this to their customers--I know for sure that if I harmed my employer's customers intentionally like Microsoft is doing here, I would no longer be employed (and for good reason).

The specific problem for him is that he doesn't have Windows Server 2012 or Windows 8 in his environment.  In other words, it is impossible for him to create a GPO with GPP settings to manage IE 10 in his production environment.  Or is it?  Sometimes, you can give Microsoft the middle finger and work around their unwillingness to facilitate their customers' needs.

Note that Microsoft does not support any of what I'm about to tell you here.  Test it thoroughly before deploying it in a production environment.  Because this is an unsupported configuration, if/when it breaks, you get to keep all the pieces.  This is a hack, and it is theoretically possible that Microsoft could break it at any time with a future IE 10 patch, or even a future group policy client patch.

Fire up your Group Policy Management Console and create a new GPO in your domain.  Edit the GPO, then go to User Configuration > Preferences > Control Panel Settings > Internet Settings.  In the right pane, right-click in the white area, point to New, and select "Internet Explorer 8."  Configure as desired.  Note that even on a computer with IE 10 installed, you'll see the Internet Settings window as it appeared in IE 8, so if a setting from IE 8 no longer exists in IE 10, it will not apply to IE 10, even after finishing the hack.  After all, you can't change a setting that doesn't exist.  Note that the F5 through F8 keys will control the green and red underlines on the dialog.  Red means that setting won't apply, so make sure to edit the option and hit F6 afterward to enable it.

Now that you have your settings as you want them, close the Group Policy Management Editor.  Now go back to the Group Policy Management Console, find and select your GPO, and go to the Details tab.  Look for the "Unique ID" and copy it.  In my case, the GPO's Unique ID is {2EEFF6D1-9F81-4624-B227-103465CB4D41}.  Now that you have this copied, open Notepad as an administrator (right-click it in the Start menu, click "Run as Administrator").  Now go to File > Open, and go to \\domain\sysvol\domain.fqdn\Policies\Unique ID\User\Preferences\InternetSettings (domain is either the fully-qualified domain name of your domain or your domain's NetBIOS name, domain.fqdn is your domain's fully-qualified domain name, and Unique ID is the ID you copied from above--in my test case, this path ended up being \\testdomain\SYSVOL\testdomain.internal.rekkanoryo.net\Policies\{2EEFF6D1-9F81-4624-B227-103465CB4D41}\User\Preferences\InternetSettings).  There you'll find an XML file InternetSettings.xml.  Select and open it.  Scroll to the right until you see min="8.0.0.0" max="9.0.0.0".  Change the 9.0.0.0 to 11.0.0.0, then save the file.  (Post-publishing note: If you're planning to use the Windows 8.1 Preview at some point, you can change this value to 12.0.0.0 and it will work, however, at that point it's easier to just use the RSAT on Windows 8.1 to edit the GPO the supported way.)

Now go link this GPO somewhere and test it before you roll it out to production, and keep in mind that this is NOT supported!  The only Microsoft-sanctioned way to make these settings is to create/edit a GPO using a Windows Server 2012 server with the Group Policy Management tools installed or a Windows 8 Pro workstation with the Remote Server Administration Tools for Windows 8 installed.  That gives you a nice interface that looks nearly identical to the actual IE 10 settings interface and doesn't need any XML file hacking to force the application.

Now some of you may be inclined to point out the settings that are available in User Configuration > Policies > Administrative Templates > Windows Components > Internet Explorer, and you'd be correct that some management can be done from there.  Unfortunately these settings, while plentiful, are woefully inadequate to replace all of the old Internet Explorer Maintenance functionality.  And yes there's also the IE Administration Kit, but it doesn't have the periodic reapplication flexibility that Group Policy has, nor does it fit in with the ideal of doing as much as possible through Group Policy.  Although Group Policy is a behemoth, it is definitely an elegant solution if handled properly.

Post-publishing note: I forgot to mention at the original publication of this article that there is a homepage setting in the Administrative Templates for Internet Explorer and that setting applies to all IE versions from 5 to 11.  That setting, however, is forcible.  When using this policy setting, the user does not have the ability to change the homepage.  Granted, this is almost certainly what most administrators want, but I'm sure there are a few out there who would appreciate the equivalent of the Internet Explorer Maintenance behavior, which allowed the homepage to be set at policy application but gave the user the flexibility to change it.

Create Favorites for Internet Explorer 10 Using Group Policy

I know Windows 8 and Internet Explorer 10 have both been out for a while, but I also know a lot of administrators out there are trying to hold off as long as possible on deploying them into their environments.  I know I have been, because IE 10 breaks a number of older line-of-business applications that work fine in IE 6, 7, 8, and 9.  Of course, if these applications could be upgraded to a more recent version that supports IE 10, that would be great, but that's not always possible.  However, that doesn't mean you aren't testing Windows 8 and IE 10 in your test labs (even if that is just some virtual machines and old clunky boxes you rescued from a junk pile).

One of the first, and most frustrating, things I noticed about IE 10 is that installing it takes away the "Internet Explorer Maintenance" section from the Group Policy Management Console (GPMC or gpmc.msc to those of us who've been around for a while).  This was particularly frustrating to me because I use this area to define Favorites for my users.

It took a little bit of time with Google and putting the pieces together, but I found how to work around this annoyance.  The answer is Group Policy Preferences (GPP).  The beauty of this solution is that it should work with any IE version, as long as the OS supports Group Policy Preferences.  That means you need XP SP2 or 2003 SP1 or newer and The KB943729 GPP Client Side Extensions (CSE) Update if applicable.  Windows 7, 8, 2008, 2008 R2, and 2012 do not require this update.  If you still have Windows 2000 workstations, this won't work there, as the GPP CSEs are not available there.  Now that you have your clients ready for this, let's get to the actual policy creation bit.

For this example, I'm going to create a new Group Policy Object (GPO) for testing, called "IE 10 Favorites," and link it to the root of my testing domain.  An illustrative image:

Now I'm going to right-click the GPO and select "Edit" so I can actually put some settings in there.  Expand the "User Configuration" node in the tree at the left, then expand "Preferences" and "Windows Settings" to see the areas to use.

If you want to create folders in the Favorites menu/pane, you'll need to use the "Folders" option that you see in the left pane of the Group Policy Editor.  Click it and you'll see the right pane of the window change to reflect that you're in the GPP Folders section.  Right-click in the white area there, point to "New," then select "Folder."  Since you want to create the folder within Favorites, enter the path as %FavoritesDir%\FolderName in the "Path" field.  In the example below, I used "%FavoritesDir%\Test Favorites" for the name.  I chose the "Create" actoion here, as it made more sense than "Update" would have, although both would have worked.  The promised example:

Click OK to save the setting.

Now to actually create the Favorites themselves.  This one gets a bit tedious and repetitive.  In the left pane of the Group Policy Editor, select "Shortcuts."  Right-click on the white area in the right pane, point to "New," and select "Shortcut."  I created two to illustrate how the folders work.  Set the Action to "Update" to fix the Favorite automatically in the event a user edits it.  Don't set the Name quite yet.  Set the Target Type to "URL" and the Location to "Explorer Favorites" first, otherwise the changes will wipe out whatever you've typed for the Name.  Now you can enter the Name.  For my first shortcut, I used "Google" for the name, then set the Target URL to "http://www.google.com/".  I didn't change any other options.  Click OK to save the shortcut configuration.

To create the favorite inside a folder, I created another shortcut here in the Group Policy Editor.  This time, though, I used "Test Favorites\Gmail" for the Name and "https://www.gmail.com/" for the Target URL.  Another illustration:

Once the GPO is created, wait for it to replicate to your other domain controllers (you do have more than one, don't you?  And yes, I'll forgive only one DC in a test lab!).  Replication should be complete within the Active Directory site within 15 minutes, or 5 minutes if you have only two DC's.  Now log onto a workstation as a user that the policy will apply to.  Verify that the favorites were created.  If they weren't, run "gpupdate /force" to force the workstation to reapply its Group Policy settings.

If you see something similar to this, you did it right.

Enjoy the torture of migrating your old settings to this new method, because it will definitely take you a while!

Citrix XenApp 6.0 and the Cries of "I Can't Log In!"

Note: The post below describes an issue with Citrix XenApp 6.0 on Windows Server 2008 R2.  The environment has a variety of client devices, including Wyse Winterm R90LW and Winterm 9150SE thin clients, as well as a number of PC's with Citrix Online Plugin 12.1 or Citrix Receiver Enterprise 3.3.0 (also known as "Citrix Receiver (Legacy PNA)").

Recently my fellow IT people and I started receiving some support calls from our users that they weren't able to log into Citrix.  This struck us all as odd, because we were able to log in without issues, and not everyone at a given location was affected--often only one at each office and at several offices no complaints at all.  After a few tries, everyone was supposedly able to finally log in.  We took a few minutes to scratch our heads and compare notes on the few calls we'd received.  There were no immediately obvious common threads based on the information we had.  We wrote it off as a fluke, as it had happened the morning after we deployed patches to the XenApp farm.  We've seen Microsoft patches that have caused some strangeness with XenApp until a second post-install reboot happens.

After writing this off, we went about our business for a few days.  Again, some of our users were having trouble logging in.  This time we were able to track it down to logon failures.  The users in question were entering the wrong username or password.  Once that happened, the screen went black and no further feedback was presented to the user.  Given this, we tried to find a resolution.  We found several vague references to XenApp hotfixes and updated client applications.  Eventaully we determined that the referenced hotfixes were included in the Hotfix Rollup package we had already installed several months before, and the clients, the recently-released Citrix Receiver 3.3.0, supposedly contained the client-side fixes necessary.

At this time, no further progress was made on a solution.  A coworker did determine, however, that the issue did not appear until we had rolled out the newest image to our thin clients.  Every person who was affected was logging in from a Wyse Winterm R90LW terminal with the newest image and all relevant security patches as of October 15, 2012.  This image included the Citrix Receiver Enterprise 3.3.0 client software.  The prior image in play used the Citrix Online Plugin 11.1, the last client software version that included pn.exe.  The importance of the upgrade from Online Plugin to Receiver in relation to this problem was not evident yet.

A day or two later, the issue returned.  This time, we tracked it down to the problem being users whose passwords had expired.  Otherwise, all symptoms were the same.  This prompted more investigation.

As a result of this investigation, both the previously-mentioned coworker and I determined that we could sometimes reproduce the issue, but not always, and never with our own user accounts.  The same user experiencing the problem could, on occasion, get the desired behavior (green screen indicating that the password had expired with an OK button to allow the user to proceed to the password change "screen") through no special action--merely persistence.  The only thing we could determine with absolute certainty was that the Wyse Winterm 9150SE devices we had weren't affected--it was completely impossible to reproduce the problem there.  We weren't sure why.

After some more research, I determined that I needed another Citrix Hotfix Rollup package that had just been released three weeks before.  Not only were there security related fixes in the rollup package, there were also some logon status fixes that seemed at least somewhat relevant to our environment.  I scheduled the installation for January 7.  Due to unforseen circumstances, this got pushed back to January 8.  Initial results were promising, so I wrote it off as fixed.

Finally, on January 14, I received another phone call with the same symptoms, clearly indicating that the hotfix rollup (rollup 2, for the curious).  I helped the user work around her problem and then I set to consulting the Google machine again.  This time, I stumbled across this post, and the comment there struck a chord.  I remembered that the Citrix Receiver used a Citrix-provided status box instead of the Windows-provided status reporting you see when logging in with Remote Desktop.  That prompted me to make the registry change described there, and one more for safety.  The changes, in .reg format, are:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Citrix\Logon]
"DisableStatus"=dword:00000001

[HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\Logon]
"DisableStatus"=dword:00000001

I applied that change to a single VM in our XenApp farm, rebooted it, and tested the login.  The logon process now showed the Windows-provided logon status messages instead of Citrix's.  Given that success, I applied the registry change to the remaining 3 VM's in our XenApp farm and let the daily automatic reboot take care of rebooting them for me.  I've been in touch with several users who entered their password incorrectly and several whose passwords had expired since I made the registry changes, and I've confirmed that this appears to have completely fixed the problem.

Now to the more interesting question--why did this not appear until we rolled out Citrix Receiver Enterprise 3.3.0 to all the users?  Well, it appears that this is due to the previous versions of the Citrix clients:

• The Winterm 9150 uses a version 8.x Program Neighborhood Agent client.  This version doesn't support the "advanced" status messages XenApp 6.0 is capable of sending to the client.
• The Winterm R90LW clients' previous image used the Citrix Online Plugin 11.1 client.  This client also doesn't support the XenApp 6.0 status messages.
• The Online Plugin 12.1 that we had on all the PC's does support those status messages, but no one ever ran into the problem there because they log onto the PC first, where they would be prompted to change their paswords before a Citrix login ever appeared.

A few closing notes:

• I wish the Citrix management console had a checkbox somewhere in the published application configuration (for the Desktop "application") to turn on or off the "advanced" status messages instead of having to make registry changes.
• I also wish the generic Citrix Receiver supported serial port redirection, but instead I have to deploy Citrix Receiver Enterprise, which has other features we don't want or need, to my thin clients because I have serial port printers and serial port signature capture pads that need to work with ICA sessions.
• And let's not forget that I'm still complaining about Citrix taking away pn.exe, which made configuring ICA connections for the thin clients stupidly simple; instead I now have to use an annoying tool called Citrix Quick Launch and save a .ica file to deploy to the clients.  Don't get me wrong--Citrix Quick Launch is a useful tool for troubleshooting purposes, but it's annoying if you're trying to actually produce a configuration that's usable long-term.

At any rate, I guess I should quit rambling now...

Windows Server 2008 R2: Server Service Fails to Start (Error 67)

Note that the screenshots below are taken from a VM I built in a test environment specifically for the purposes of this post.

This bit a coworker and I just a couple days ago.  My coworker was attempting to deploy patches to a Windows Server 2008 R2 virtual server using VMware vCenter Protect.  Repeatedly this failed.  We went through the troubleshooting processes we were already familiar with and have seen success with in the past:

• See if traffic will pass both from the vCenter Protect console to the troublesome VM and from the troublesome VM back to the vCenter Protect console (using ping; traffic did pass correctly).
• Reboot the troublesome VM.  The problem remained.
• Reboot the VM running the vCenter Protect console.  He did this twice due to an unrelated issue; the problem remained.

After a short while, my coworker checked the Services console (services.msc) and realized that the Server service was not running.  Every attempt to start the service failed and presented this dialog:

Error 67: The network name cannot be found.

From there we proceeded to several other troubleshooting measures:

• Verify that the VM's IP address and DNS record matched, and that there were no additional IP addresses registered to the name in DNS.  DNS records were what they should have been.
• Verify that the VM's DNS server IP addresses were correct.  There was a mistake here; I corrected this but it didn't solve the problem.
• Try re-enabling NetBIOS over TCP/IP on a temporary basis.  This didn't solve the problem; of course, we didn't expect it to!
• Change the IP address and ensure the new address was properly reflected in DNS.  Again, no dice, although I again didn't expect this to have an effect.  I set the IP address back to its original value.
• Google!

Of course, when everything you can reasonably think of fails, Google is the next resort.  My coworker searched for a while and found a number of other things that all didn't help.  At this point I was letting him do everything himself and only tossing some ideas his way here and there, as I was working on another project that I'll probably post about later.  One of the more notable things he tried was to compare the Server service's registry settings (located at HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\LanmanServer) on the troublesome VM with an identically-configured VM that was working properly and adjust accordingly.  It seemed promising, as there were inconsistencies, but resolving these inconsistencies didn't help.

Eventually his Google well ran dry.  He finally tried to replace svchost.exe and srvsvc.dll on the troublesome VM with the ones on the identical, working VM.  Again, no dice.  At this point, I decided it was time for me to take a look.  I re-did a few of the troubleshooting steps above just to confirm neither of us were completely nuts yet, then I tried to start the service to generate a log message so I could consult the Event Viewer (eventvwr.msc).  I found this message:

Event ID 7023 from Service Control Manager

I deciced it couldn't hurt to double-check the Google results, so I copied the exact text from that log message and ran it through Google.  Interestingly enough, this Microsoft Knowledge Base article was the first result.  (My coworker hadn't copied the entire message when he searched, so this article ended up not appearing in the first two or three pages of results.)  It seemed odd, but I fired up cmd.exe and ran the path command.  Sure enough, at the tail end of the path was an entry that was a UNC path.  I proceeded to check the system-wide environment variables through Control Panel and found that the system PATH variable did indeed include the UNC path, similarly to what I've shown here:

UNC path in System PATH variable

I removed the UNC path and another invalid path that referenced a network drive letter, then rebooted the server.  Magically, everything worked again.

Now that everything worked again, it was time to figure out why it happened and how to prevent it again.  This server is for a testing instance of an application and is thus very lightly used.  According to our event log archive, the coworker I've mentioned here is the only person besides myself who had logged onto this server either interactively or via Remote Desktop in the entire calendar year and his only log on events were from the day this issue occurred.  Therefore, it had to have been something I did.  This had been working just three weeks ago when he did the initial patch scan with vCenter Protect, so it had to be fairly recent activity.

I consulted my to-do list and found a potential culprit--I had installed a service pack for the application on the server just a week and a half ago.  I then checked our notes related to the application and found that the two entries I removed from PATH were added as part of migrating the app from another server to this one.  It appears that the person who did this migration (over a year ago!) forgot to remove these entries after the migration was complete. I still can't figure out why it took over a year for this to become a problem, but I'm confident this was the root cause.

In closing, I'd like to point out that Microsoft's article linked above states the following: "A system path that contains a UNC path may cause severe system problems and severe software problems. Therefore, a system path that contains a UNC path is unsupported."  Given that this is unsupported and known to cause problems, shouldn't the control panel applet at least give a warning when a UNC path is entered into the system PATH variable?

Introduction, More or Less...

I'm a System Administrator by "trade."  I'm a generalist--I administer primarily Windows machines, but I also have responsibility over a couple AIX boxes, a slowly growing contingent of Linux boxes, some HP PoE switches, a bunch of Cisco gear, a Microsoft Exchange 2010 environment utilizing a Database Availability Group (DAG) and fail-over clustering, and a VMware environment consisting of 10 ESX servers, one ESXi server, and roughly 50 virtual machines.  I'm also responsible for the backup operations, using a few different products that don't always play nice with each other.  In many ways, my peers and co-workers consider me to be a resident expert, which is a role I'm not always comfortable with.

I learned Linux by using it at home starting in 2001, back in the days when 128 MB RAM was sufficient for a workstation, X11 was provided by XFree86 3.3.6, and GNOME 1.2.x was "current."  I started with what was then called LinuxMandrake 7.2.  I eventually upgraded to Mandrake Linux 8.1, tried RedHat 8, resided on Fedora Core for a while, experimented with Ubuntu long enough to discover I hated it with a passion (around 2006), then finally settled on Debian.

I've used what I learned on Linux to spread out into other UNIX flavors.  I've briefly messed with IRIX and HP-UX, and I worked on an NCR MP-RAS system for several years.  At one point, per my boss' request, I wrote a custom backup solution for the MP-RAS system in Bourne Shell that used cpio, OpenSSL, and OpenSSH to get the same backup written to an encrypted tape and securely copied to a remote MP-RAS system as a disaster recovery option.  I even included multiple verification of each backup with md5sum, sha1sum, and sha256sum (long story), followed by a test extract into a temporary destination on the remote system.

I learned Active Directory design and administration at a local community college, on Windows 2000--I just finished the Windows 2000 courses a month or two after Windows Server 2003 was released.  I never pursued certification, because I didn't expect that I would need it and I was too cheap to pay for all the exams.  I've kept current via experimentation with evaluation editions and sometimes at-work test environments.  I'm incredibly happy that although Microsoft has added a ton of new features and changed a bunch of UI elements they have chosen to keep Active Directory in a state where all of the basic concepts I learned 10 years ago still apply.

I am a developer on the open-source IM client Pidgin.  I don't do a whole lot over there anymore, but I've worked on a few areas I thought were worthwhile.  I once rewrote huge chunks of the man page (you Linux and UNIX guys out there are probably nodding in approval now), and I also at one point reworked the preferences window to fit in the ridiculously tiny screens on netbooks, amazingly without slashing any options in the process.

As part of my administration job, I occasionally write scripts to do work for me.  I'll write in whatever language I feel I can get the right results in the most quickly.  Sometimes that's a plain old .bat file (ugh), VBScript (ugh, again!), occasionally good old Bourne shell, and more frequently now Python.

I started this blog primarily because I wanted to be able to post administration-related ramblings without having to remember to properly apply labels to my other blog to avoid being picked up in Pidgin's news feed.  I'm hoping someone finds this stuff useful.