SSL Connection to AutoDiscover slow or times-out on domain joined computer with invalid cached credentials

Once again, I found myself coding. A common occurrence since I am always assisting customers and partners whom hit a technological roadblock that can ultimately be solved with IT.  A similar roadblock was reached when I uncovered a frustrating issue with AutoDiscover while working on some updates to our own tools in order to support mapiHTTP connections.

In making updates to Priasoft Migrator so it would   support mapiHTTP it became necessary to retrieve data about mailboxes and public folders from AutoDiscover.  Since the most common connection to this service is conducted over a secure route (SSL/TLS), we wrote our code to do the same, essentially connecting to https://server/autodiscover/autodiscover.xml.

However, the computer where I do a bulk of the development tasks is not a domain-joined PC.  I do this by choice as it assures that if I hit any issues, they are worked through and make it into the product.

Once the new code was written and our team was initiating testing and production readiness, I began to test these updates on a domain-joined host.  Initially I found that the connections to the AutoDiscover service began timing out…from a domain joined machine!  One would not normally expect two machines that sit on the same LAN to timeout accessing a website or webservice.

Fiddler and Wireshark were brought in to scope to troubleshoot, and to our surprise we discovered that the SSL/TLS handshake between the client code and the IIS instance on the Exchange server was taking way too long.  As the code was written, the conversation with the webserver was handled synchronously such that further processing could not continue until the secure handshake complete.  By the time the handshake completed and the client code resumed, the internal timeout had expired and the whole thing unwound.

Searching the web proved to be a waste of time.  There was far too many variables that could give similar results, and all of the common topics that bubbled up didn’t match the experience I had.

To be clear, this did NOT occur from a non-domain-joined host.  At first glance I thought it was some difference between the OSes (dev is Win8, test host is Server2008) or some difference between .NET or a difference between updates or something…none of which presented an issue.

I also considered that maybe the issue was a factor of the authentication routine, since   AutoDiscover requires authentication before it can return information.  However, after much trial and error, we found that the authentication was not a variable either since the code was never reaching the authentication point; it was timing out before that could occur.

Raising the timeouts on the client code DID work.  However, there was such a stark contrast in the responsiveness between my dev host and the test host that increasing the timeouts just didn’t seem like the right answer.  So, as engineering led techies further investigation ensued.

Once it was determined that the issue lied solely in the SSL/TLS handshake, the focus move to that.  ILSpy was used to investigate some of the Microsoft code to see what it does and to see if there were flags or options to control how it worked.  We then encountered an anomaly from the Wireshark traces that became a focal point.  In those traces in became clear that before any TLS handshaking would occur, initial  attempts at a Kerberos/RPC connection to IIS would occur, , then finally a TLS handshake would take place.

In contrast, a non-domain-joined host never even attempted the Kerberos/RPC connections.  It would go immediately for the TLS handshake.  This seemed to be the key difference.  The question was how and why was my client code doing that?  We certainly didn’t write any code to try to make an RPC connection to the webserver!

So, as it is typical with such troubleshooting, many hours were consumed trying to control the connection to prevent the RPC connection attempt.  Basic authentication was looked at, and as mentioned before, had no bearing on the issue since that action comes AFTER the secure handshake.  But, intuitively it seems that such would possibly side-step this other connection action.

Wireshark would clearly show every time that “windows authentication” was being attempted first before the TLS handshaking.  One of the recurring trace elements from Wireshark was also several “SMB” logon attempts, but those were initially discarded as related because there were other communications outside of this client application that could connect with the same server.  Eventually the patterned showed that these SMB logons only occurred when the client code was running.

Upon looking at the details of these events, it was seen that the windows host where this client code was running was attempting to validate and authenticate to the Exchange server using “Windows Networking”, meaning a secure “integrated” connection that can occur between windows hosts in the same domain.  This ultimately is a part of LSA and SSPI and THIS IS ALWAYS DONE!  One cannot bypass this because the lower level code that handles the secure connection from one host to another detects that the computer is domain-joined.

So, the question then became, why is this connection failing?  Why is it taking so long to figure out that the “domain authentication” won’t work?  In the end it was because of a bad credential.  The critical variable that was never considered was a cached credential in the windows “Credential Manager”.  The long delay, due to many retries, before the TLS handshake was due to windows having to retry for each installed authentication provider (Kerberos, NTLM, Digest, etc.).  Windows will exhaust all of those before dropping to the TLS handshake.

In the windows credential manager, there as a “domain” credential for the target Exchange server, but with a bad password.  To be clear, this was not stored for use by web browsers and generally such credentials are stored i the “Web Credentials” or “Generic Credentials” areas, so this credential didn’t immediately come to mind.  However, since Windows was trying to establish a computer-to-computer connection between domain-joined hosts, that action occurs (or is attempted) before any other.  It is this same feature that allows for single-sign-on to an internal webservice/website, provided that the webserver can handle “Windows Integrated Authentication”.  But, don’t get the wrong idea here.  Even if you turn off Windows Integrated Authentication on the target webserver, this lower level handshaking still occurs.  The option in IIS for Windows Integrated Authentication comes into action AFTER the computer-to-computer connection is made.

In the end, removing the bad credential changed experience immediately.  Once the two computers could validate each other at a domain level, the TLS connection could then be started.