Once again, I found myself coding. A common occurrence since I am always assisting customers and partners whom hit a technological roadblock that can ultimately be solved with IT. A similar roadblock was reached when I uncovered a frustrating issue with AutoDiscover while working on some updates to our own tools in order to support mapiHTTP connections.
In making updates to Priasoft Migrator so it would support mapiHTTP it became necessary to retrieve data about mailboxes and public folders from AutoDiscover. Since the most common connection to this service is conducted over a secure route (SSL/TLS), we wrote our code to do the same, essentially connecting to https://server/autodiscover/autodiscover.xml.
However, the computer where I do a bulk of the development tasks is not a domain-joined PC. I do this by choice as it assures that if I hit any issues, they are worked through and make it into the product.
Once the new code was written and our team was initiating testing and production readiness, I began to test these updates on a domain-joined host. Initially I found that the connections to the AutoDiscover service began timing out…from a domain joined machine! One would not normally expect two machines that sit on the same LAN to timeout accessing a website or webservice.
Fiddler and Wireshark were brought in to scope to troubleshoot, and to our surprise we discovered that the SSL/TLS handshake between the client code and the IIS instance on the Exchange server was taking way too long. As the code was written, the conversation with the webserver was handled synchronously such that further processing could not continue until the secure handshake complete. By the time the handshake completed and the client code resumed, the internal timeout had expired and the whole thing unwound.
Searching the web proved to be a waste of time. There was far too many variables that could give similar results, and all of the common topics that bubbled up didn’t match the experience I had.
To be clear, this did NOT occur from a non-domain-joined host. At first glance I thought it was some difference between the OSes (dev is Win8, test host is Server2008) or some difference between .NET or a difference between updates or something…none of which presented an issue.
I also considered that maybe the issue was a factor of the authentication routine, since AutoDiscover requires authentication before it can return information. However, after much trial and error, we found that the authentication was not a variable either since the code was never reaching the authentication point; it was timing out before that could occur.
Raising the timeouts on the client code DID work. However, there was such a stark contrast in the responsiveness between my dev host and the test host that increasing the timeouts just didn’t seem like the right answer. So, as engineering led techies further investigation ensued.
Once it was determined that the issue lied solely in the SSL/TLS handshake, the focus move to that. ILSpy was used to investigate some of the Microsoft code to see what it does and to see if there were flags or options to control how it worked. We then encountered an anomaly from the Wireshark traces that became a focal point. In those traces in became clear that before any TLS handshaking would occur, initial attempts at a Kerberos/RPC connection to IIS would occur, , then finally a TLS handshake would take place.
In contrast, a non-domain-joined host never even attempted the Kerberos/RPC connections. It would go immediately for the TLS handshake. This seemed to be the key difference. The question was how and why was my client code doing that? We certainly didn’t write any code to try to make an RPC connection to the webserver!
So, as it is typical with such troubleshooting, many hours were consumed trying to control the connection to prevent the RPC connection attempt. Basic authentication was looked at, and as mentioned before, had no bearing on the issue since that action comes AFTER the secure handshake. But, intuitively it seems that such would possibly side-step this other connection action.
Wireshark would clearly show every time that “windows authentication” was being attempted first before the TLS handshaking. One of the recurring trace elements from Wireshark was also several “SMB” logon attempts, but those were initially discarded as related because there were other communications outside of this client application that could connect with the same server. Eventually the patterned showed that these SMB logons only occurred when the client code was running.
Upon looking at the details of these events, it was seen that the windows host where this client code was running was attempting to validate and authenticate to the Exchange server using “Windows Networking”, meaning a secure “integrated” connection that can occur between windows hosts in the same domain. This ultimately is a part of LSA and SSPI and THIS IS ALWAYS DONE! One cannot bypass this because the lower level code that handles the secure connection from one host to another detects that the computer is domain-joined.
So, the question then became, why is this connection failing? Why is it taking so long to figure out that the “domain authentication” won’t work? In the end it was because of a bad credential. The critical variable that was never considered was a cached credential in the windows “Credential Manager”. The long delay, due to many retries, before the TLS handshake was due to windows having to retry for each installed authentication provider (Kerberos, NTLM, Digest, etc.). Windows will exhaust all of those before dropping to the TLS handshake.
In the windows credential manager, there as a “domain” credential for the target Exchange server, but with a bad password. To be clear, this was not stored for use by web browsers and generally such credentials are stored i the “Web Credentials” or “Generic Credentials” areas, so this credential didn’t immediately come to mind. However, since Windows was trying to establish a computer-to-computer connection between domain-joined hosts, that action occurs (or is attempted) before any other. It is this same feature that allows for single-sign-on to an internal webservice/website, provided that the webserver can handle “Windows Integrated Authentication”. But, don’t get the wrong idea here. Even if you turn off Windows Integrated Authentication on the target webserver, this lower level handshaking still occurs. The option in IIS for Windows Integrated Authentication comes into action AFTER the computer-to-computer connection is made.
In the end, removing the bad credential changed experience immediately. Once the two computers could validate each other at a domain level, the TLS connection could then be started.
It’s hard to believe that it has been almost 2 years since we first architected the Public Folder sync as an ancillary offering to our Priasoft Migration Suite. Back then, it was an urgent issue that a customer encountered, which we quickly committed to resolve. As an ever-present, 20 year veteran in the world of how complex migrating Microsoft Exchange can be – we never imagined that our new solution would redefine a facet the migration world. More importantly, this feature would take the laborious process of migrating public folders to migration simplicity. Since then, we’ve continued to refine the solution based on customer experience and feedback and by taking a forward-looking approach to how the challenges can be solved. This evolutionary process was recently experienced when we decided to support one of the more complex preparatory aspects of Public Folder synchronization during the migration.
When Microsoft first changed the storage architecture of Public Folders, it seemed to be a well-timed and necessary change to a system that had been virtually the same for almost 15 years. They first introduced an immediate change in architecture which was the use of mailbox objects for the storage of Public Folder data instead of a single database, which allowed IT administrators to plan for resiliency and recovery in the same manner they would plan for normal mailboxes. And while this change may seem inconsequential to the IT novice, seemingly overnight, this shift enabled backups of Public Folder data to be efficiently segmented, making the days of the “2 terabyte data file” a thing of the past. For earlier on-premises deployments of Microsoft Exchange this change in architecture had an immediately positive impact, but for those customers migrating to later versions of Microsoft Exchange or Office 365 this presented a whole new set of migration considerations and factors that needed addressing before their expedient shift to the Microsoft Cloud. Now, let’s review why.
The new architecture found in Microsoft Exchange 2013 or later, including Office 365, requires the creation of one or more special mailbox objects – Public Folder Mailboxes – to contain the Public Folder information. In an on-premises deployment, this is a fairly easy task since an administrator or Microsoft Exchange architect could choose to set very high size limits on these special mailboxes, or none at all. If you had a Public Folder Database in your source environment that was 400 gigabytes in size, you could have a single Public Folder Mailbox of that size, or two 200 gigabyte mailboxes, or four 100 gigabyte mailboxes and so on. Again, a fairly easy issue to solve and as the administrator you would just do what felt natural or necessary.
However, with Office 365 (Cloud computing anyone?) and the many limits and restrictions it imposes, a scenario like the previous on-premises setup becomes much more arduous. In its early roll-out, Microsoft Office 365 was based on Exchange 2010 which used the Public Folder database architecture. Then, when the service transitioned to Exchange 2013 and 2016 as a base platform, different limits were required, some of which were simply the inherited structural limits of a mailbox.
One of the biggest headaches with the use of Public Folder Mailboxes in relation to a migration, was how to plan and distribute a large monolithic database of Public Folders data across several smaller mailbox objects. We would often be the first to introduce this headache to our customers and would provide the recipe as to how a customer *should* tackle the issue, but it was not until very recently that we actually addressed the issue with technology. Resolving the dilemma seemed simple and we defined three steps for customers to consider at kick start:
1. Analyze the source folders and gather statistics
2. Find the larger folders and assign them to a specific mailbox
3. Add more mailboxes as necessary
These recommendations were well received, as the feedback was always positive and grateful for exposing something that, on their own, the customer would not have known. Yet, we knew this was going be just part of a recurring issue that would continue to come up with customers choosing to move to the Microsoft Cloud. The First Sync In time and after many customer support calls that were not product specific in nature, but were set-up and preparation related, we came to the realization that not only had our support burden skyrocketed, but most of the calls were education of deep technical topics. Most customers didn’t even know of the challenge in front of them and the complexity behind it. As such our support conversations became more about the broader task of migrating to Office 365 and educating customers about issues that they would likely run into and how they, the customer, could tackle those challenges.
Realizing that this could be a substantial burden on customers, what would any good Expert in Microsoft Migrations do…? I know, I know, there’s a ton of “so called” experts, but after 20 years in the game, we tackle such issues by writing code and creating tools and solutions. However, before you can throw code at it, you must take yourself down the rabbit hole, flush out the anomalies and define a recipe for success that will work for all, if not most.
As the primary architect of this new build, I was immediately faced with taking myself through the process that our many partners and customers were facing. Specifically, how to choose which folders go to which mailbox, and what percentage of a mailbox should be filled up before moving on to another one. What seemed like a simple discussion point, at least conceptually, turned out to be very complicated when it came time to actually figure it out.
Initially, the idea was to treat this like a card dealer, where an even amount of data would be distributed across one or more Public Folder mailboxes. The distribution of the folders was actually easy, but was horribly inefficient. When assigning a folders to a mailbox, you don’t have to assign every folder individually, and really shouldn’t. You only need to assign the parent folder to the mailbox. Any action to create child folders under that parent would automatically assign those children to the same mailbox as the parent folder. While the card analogy works, it doesn’t quite represent the issue completely. A better analogy to the issue is with farming and watering of crops.
Here in the southwest, much of the crop irrigation is done by flooding, not sprinklers. Farmers dig small canals and channels to direct water to plots of land that have been planted. In this method, one does not have to water every plant individually – as the card analogy would be – but one only needs to define where the water should begin its flow. Once the canals are dug, the water will then flood and fill up each field.
This is where the first level of complication came in…how to collapse as many child assignments to as few “primer” folder assignments as possible. It was figured out, but took a few tries to get right. The next set of complication came from doing several “limit checks” to make sure that no rules were broken with regards to max size, or max item/folder count, etc. and to also not exceed any support limits imposed by Microsoft.
Needless to say and an update later, we figured it out and can rest assured that we can continue to follow our Priasoft moniker with the Experts in Microsoft Migrations more than metaphorically. Today, not only do we provide the simplest way to migrate and synchronize Public Folders, but we can stake the claim that we are the only ones doing it properly and most importantly, intuitively to Microsoft Office 365 or Microsoft Exchange.
