Here’s a little bit of fun I had while trying to find out who would be playing my beloved Detroit Tigers in the World Series. I needed to update our Lync servers, so installation of the October 2012 Cumulative Update was in order.
As soon as the Cumulative Update was done installing on one of our front end servers, SCOM started throwing alerts every 30 seconds. Essentially, they looked like this:
Oct 20 13:13:56 cltlb01 l4d: VS 22.214.171.124:443(TMG Lync External HTTPS meetContosoCom) Taken out of service due to failed Real Servers
Looking at the servers, the first front end server wasn’t reporting any issues. But the second was flooded with these in the application log:
Every time the HLB tried to do the health check, another ASP.NET exception would show up in the event log. We checked the HLB config, and it was properly set to connect to https://*/meet/blank.html for its deep health check. This is important because if you have it set to just check /meet or https://*meet/, I’ve seen this identical problem pop up. I verified that the file was at the correct location (C:\Program Files\Microsoft Lync Server 2010\Web Components\Join Launcher\Int), and that I could open it in IE. The file didn’t have a recent modified date. So, we changed the health check just to do a TCP bind to the server, and the problem went away. HLB was happy, and ASP.NET alerts stopped in the event log. Synthetic tests came back fine. Back to the baseball game, I figured.
Fast forward to Monday morning, when two users simultaneously contacted me to say that they were attempting to join a scheduled Lync meeting and getting the below error. Not the same meeting – two separate meetings.
They sent me the links, and I was able to successfully join both meetings, but I’m homed on a different server than those two users. The plot thickens.
I looked at the front end servers again, and sure enough, whenever someone would try to join either of those meetings, that trouble server would throw an ASP.NET error again. It wasn’t as fixed as I hoped.
I drained the server using Stop-CsWindowsService -Graceful, then stopped IIS using net stop w3svc. I navigated to C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727 and ran aspnet_regiis.exe -i. I restarted the server, and suddenly the problem was gone. Everyone was able to join their meetings, and the server no longer was throwing ASP.NET errors. We changed the health check back to looking at /meet/blank.html, and all was good.