Technical

Migrate Exchange 2010 To 2016 With Priasoft’s Migration Suite For Exchange

The driving event that prompts the need for migration may vary, but the demand to execute does not. Whether the event is a merger, acquisition, divestiture, or upgrade, the success of such a project depends upon many factors – of which some of the most influential are perception elements like the end-user experience afterwards.

Priasoft has been deeply involved in this category for nearly two decades now (starting in 1999 with a large enterprise migration of a petroleum company). Through those number of years, our understanding of what makes a migration successful, versus not, has grown and we have been able to provide a “run book” of sorts that helps enterprises, both large and small, orchestrate their migrations with minimal disruption.

Migrating email between environments, or even as an upgrade, is not without challenges. The elements that increase or decrease these challenges vary between businesses, but there are several elements that are the same for nearly all migrations. The typically common elements are usually technical in nature, exposed as features of Microsoft Exchange, SMTP, Active Directory, and so on. Elements that are more business, policy, or market specific are the ones that make a migration unique.

Priasoft’s nearly twenty year timespan in this space means that we’ve seen most of the unique and mundane challenges that are part of migrations, and while there may not be a technological or automated solution to all the elements, our experience is there to fill the gaps with either guidance, expectation setting, scripting, or education.

What Determines Success?

The success of a migration is generally more of a “perception” of success than any measurable technical outcome. If end users are complaining after a migration, regardless of the validity of the complaint, the perception is soured and might not be considered very successful, especially if there is a productivity or support cost related to the complaints. Even a flood of simple questions to the help desk after a migration, like “why does it seem to take longer for Joe to get my emails now?”, can be costly as it can create a spike in support which may get in the way of more critical support issues.

One of the first steps then for migration should be a document of “success criteria”. This may be as simple as “all mailboxes migrated, and no more than a 10% increase in post migration support”. However, the more criteria that can be added to the document, the better. Adding dependent elements like compliance, eDiscovery, and or sales integration may be part of a given success criteria. Do what is appropriate for the business needs.

Is Exchange 2010 Unique?

The previous paragraphs likely seem very generic, and in truth they are and would apply equally to any migration. Migrating from Exchange 2010 to 2016 is a fairly unique path, not because it is not performed very often (quite the opposite!), but because there are many architectural differences between the versions. This document will draw out those differences, and will provide some fundamental guidance in this regard.

Some primary differences between these versions that should be noted are:

• Differences in how item, folder, and mailboxes sizes are calculated by Exchange.
• Differences in how the server roles operate (CAS, HUB, MBX, etc.)
• Differences in how Public Folders are stored, configured, and managed.
• Differences in how Outlook (on Windows) can connect to mailboxes
• Differences in RAM and CPU utilization
• Differences in management interfaces

There are many other more subtle differences, but none that are so critical that can affect a migration or post-migration experience. Many of the other differences are purely technical or architectural differences, while others are more of a style difference.

We are experts in 2010 to 2016 Migration, to learn more about our solutions follow the link below:

Vocabulary And Conventions

Before diving in to the details of how to migrate and the order of operations, it is probably best for some vocabulary to be introduced so that there is less possibility of confusion since some of the terms may have different meanings for different readers.

Is There A Recommended Order To This?

Of course! While migrating in a different order than we propose is possible, and can even be as successful, we have found through the years that migration order that differs from our recommendation tends to have more issues, support, and “perception souring”.

In the basest and simplest of terms we recommend migrating as follows:

1. Exchange Address Book
2. Public Folders
3. Mailboxes
4. Outlook/mail applications

However, this is obviously greatly oversimplified, and nonetheless still very accurate and true. Let’s add some detail and additional steps to ensure success and to prevent “perception souring”.

Here’s a revised order:

1. Determine Migration Type

Same-forest: This is an upgrade of Exchange 2010 to 2016 where the new Exchange 2016 servers are simply going to be added to the existing Active Directory forest/domain.
Cross-forest upgrade: This is a migration of Exchange 2010 to 2016 where Exchange 2016 will exist in a new or possibly pre-existing Active Directory forest/domain, but is, for the most part, an empty target environment.
Cross-forest merge: This is a migration of Exchange 2010 to 2016 where Exchange 2016 exists in an already established environment and the event is driven by a merger or acquisition type of scenario.

The migration type changes the complexity and challenges that one will face and none of the types above are inherently more or less complex or risky. The choice of tools used, understanding of the capabilities and limitations of Exchange, and business tolerances have more influence on complexity than anything else.

2. Discovery And Categorization

This topic covers the discovery and identification of “what should migrate” as well as “what is the use and purpose of the objects to be migrated”. We find very often, in the early development of a migration strategy, only “user mailboxes” are initially considered. However, such is not the entirety of Microsoft Exchange. Review the vocabulary list for more detail.

Our suggestion on this topic is to identify ALL MAIL-ENABLED OBJECTS that are or could be, or could have been used by Exchange. It is important to understand that when an email is sent or received thru Microsoft Exchange, the recipients and sender data embedded in the message are not always SMTP (joe@coolstuff.com) addresses. When messages are sent between objects in Exchange, the embedded address is actually an x500 address (/o=OrgName/ou=AdminGroupName/cn=recipients/cn=objectID). Every mail-enabled object in Exchange has an X500 address. It is rarely seen, and Microsoft does not show it in any of the normal user interface dialogs of either its management tools nor in Outlook or mail clients. The value is stored in Active Directory in the attribute named ‘legacyExchangeDN’. It’s also important to note that ALL objects that are part of Exchange have this value: servers, policies, users, contacts, groups, etc. This is the unique address for all objects in Exchange. Failing to include all the objects in a migration can create NDRs (non-delivery reports) to end users when the reply or reply-all to migrated messages.

The identification of all mail objects is important because they may exists in unexpected or forgotten locations in Active Directory. We have seen many times where some non-user mailboxes or special mail groups exist in Active Directory containers outside of the normal business structure.

The categorization part of this topic involves the separation of objects by type, by delimiters that might influence the success of the migration. For example, there may be 1000 people with mailboxes in an organization, but perhaps 50 of them – due to their job function – use encrypted email or have some mail-enabled application that creates a need for special handling of those users’ data. We have found it best to start with what is generally known at an organization with regards to uniqueness as just a set of categories. As purely an example and suggestion a table similar to the following could be used, and checkboxes or ‘x’ chars used to mark all the categories for which a user or object belongs.

Producing a table similar to the above can help identify how many users and how many different groups of users you have that may need special treatment thru the migration. Users that have desktops that are not domain-joined, for example, might not be able to be automated for updates to Outlook profiles after a migration. Such may require the user to do their own action, or may require help from IT support services.

3. Address Book Sync

This migration element is the act of capturing all the mail-enabled objects from the source environment and pre-staging them in the target in a working fashion. The Priasoft Collaboration Suite is a set of tools that facilitate this effort. This task ensures that all the related “mail-able” objects are represented in the target environment. Note that in a same-forest migration this is ignored as there is only the one Active Directory and so no need to copy or sync anything.

For cross-forest migrations however, this is a critical element. Consider that mail-enabled groups, contacts, mail-users, mailboxes, and even mail-enabled public folders exist in Exchange and ALL of these object types have their respective Exchange attributes stored in Active Directory.

If the mailboxes were simply migrated without this consideration, users’ replies to old messages will likely have non-delivery responses for those other object types.

This is listed here in this position in the order of things due to its importance, and possibility of challenges. There is a common case where user accounts were migrated or setup in the target Active Directory prior to any consideration for the Exchange migration that was due to come later. One challenge created by this is matching. If the user accounts created have no matching details of source objects, such forces some tedious “human actions” to match objects between the source and target environments. This issue is exacerbated by merger/acquisition scenarios since the target will already have existing users with and without mailboxes – conflicts on name, use, and purpose tend to always be present and have to be discovered, analyzed, and passed up to object owners for decisions on what actions to take.

Another possible challenge are with Contacts in the target Active Directory. It is common also for administrators to “get a head start” on things by creating Contact objects in the target Active Directory for all the mailboxes in the source. Unfortunately, it is not realized until later the problem that is created: a contact cannot be converted to a mailbox because a contact cannot receive a password. Only user accounts can have mailboxes. This creates a pre-migration “cleanup” whereby all the contacts have to be removed so that mailboxes can be created – Exchange will not allow two objects to have the same email address, mail alias, and a few other attributes. Furthermore, deleting these contacts should not be treated as a trivial inconvenience. Each contact has a unique x500 value (legacyExchangeDN) value which could be embedded in the mailboxes that exist in the target environment. Deleting contacts, without consideration for this value can cause non-delivery messages for existing users, before a migration has even occurred! Further to that, the contacts may be members of mail groups and may have additional email addresses or other display attributes for which the actual source mailbox does not have. Blindly deleting these contacts often ends up with many “odd” support issues that appear difficult to explain.

Priasoft tools and support staff help customer tackle this first major migration task with relative ease.

We are experts in 2010 to 2016 Migration, to learn more about our solutions follow the link below:

4. Testing Preparations

A good plan is only as good as it has been tested before unleashed on production objects. Testing is extremely valuable to success, however as the value of the testing is dependent upon the quality and types of testing. Testing Preparations is a step where the testing framework is setup and configured based on technical needs, business goals, and desired answers to be validated.

We do NOT recommend doing first tests, or even later tests with real users! From 20 years of experience, we have seen nearly every case of “real user tests” or “pilot groups” create problems instead of solving them. One of the primary reasons a pilot group creates issues is because those users will be in a new environment that is distinctly not the same as the one they left. There are certain features of Microsoft Exchange – mostly collaboration features – that were never designed to work between different environments and sometimes not even between different versions of Exchange.

We recommend creating as many different types of “test objects” as necessary to satisfy all the different concerns and categories (consider step 2 above). In reality, the architects and executors of the migration will know more about what to test and how to test items than a user would. Additionally, most pilot group users – regardless of them agreeing to participate and ‘there may be some issues’ – don’t expect there to be any issues, and when there are, can report such soured perceptions back to their peers, creating a potential “progress blocker”.

Test preparations then should include not only the creations of mailboxes, group, etc. in Exchange, but also the end-user situations as much as possible. Creating typical user “desktops” and situations are key to good testing. If it has been identified that a group of users exist for which group policies cannot be applied to make changes, the testing of such a desktop scenario should be tested.

This type of setup will also for a full “dress rehearsal” of the migration, but without the potential of affecting any real users.

5. Mail Client Updates – Automation/Execution

The next migration element to handle is how updates to mail applications will occur. Outlook (on Windows) can be updated automatically in many ways and Priasoft has a unique and very important utility application that can be used to update Outlook profiles to reconfigure them to point to the new migrated mailboxes. The choice of automation of this update though is left to the decision of the environment owners and administrators. Group Policy logon scripts are the most common way to handle this, but any process that can cause an application to run “as the user” will suffice. Desktop automation software like Microsoft SCCM can also be used.

Referring once again to the importance of categorization, identifying users and situations where automation cannot be applied is also important. For those cases, some alternative method must be introduced, which can mean providing instructions to users to “self-service” the updates, or thru interactive help-desk sessions.

Mobile clients are an especially taxing complexity of migrations. Unless the organization being migrated already has an automation platform for mobile (known as MDM or Mobile Device Management solutions), such devices may have to be updated manually by “somebody”, which could be the user, the help-desk staff, the migration team, or consultants. A special note on mobile devices: the importance of the mobile experience as it relates to mail can be a determining factor as to “how fast can we migrate”. It has been seen many times that while mailboxes and updates to Outlook clients are proven to occur for 1000s of users in a 24 hour period, if the importance of mobile is very high, such can cause a migration to change from a single-event migration to a multiple-event migration possibly spanning many weeks. The longer a migration is split between environments, the great the chance the perception of success will sour. It is highly recommended to balance between these concepts.

Outlook Profile updates are very important to migration success. Outlook tends to be the primary user interface for Microsoft Exchange and as such is usually the most visible element of Exchange. Failures with Outlook after a migration can obviously be very disruptive. Additionally, it should be know that Outlook – being simply a software program for email – has no built-in understanding of migrations. In a same-forest migration (upgrade Exchange 2010 to 2016), Outlook can query Active Directory to determine new mailbox database details and can usually update itself properly. However, this success is dependent upon the user account remaining as the same Active Directory account thru the migration. In a cross-forest migration, Outlook has no understanding of this.

AutoDiscover is often thought to be a bit of “magic” for handling Outlook profiles. Unfortunately, this is not true. An Outlook profile will store many values that related to the source environment and which are never seen in any configuration dialogs. As such, if Outlook gets only new servers names for which to connect (which is all AutoDiscover really provides), it will simply try to connect to those endpoints, but without clearing out any of the previously cached information. In this ‘half-baked’ state, Outlook will attempt to make connections to both environments and if the source environment resources for which Outlook is still trying to use become unavailable, Outlook will lock up. Additionally, there are other Outlook features that can be affected by this ‘half-baked’ state such as folder delegation, calendars, free-busy lookups, offline-address-books, etc.

Priasoft profile update utility is a valuable and necessary component of a successful migration.

6. Public Folders

If the organization that is to be migrated has Public Folders, the migration of them can become a very complicated topic. We have also heard many organization state that they will simply “not keep them” and simply leave them behind, or will attempt to shift the public folders to shared mailboxes. While both ideas have their merits, attempting to do this work in conjunction with the overarching email migration often ends up being a bad idea. Public Folders by their nature can be used by users in ways and on frequencies that are extremely difficult or impossible to retrieve. Leaving pubic folders behind not only can be disruptive to users, but can also be a failure in a compliance audit or subpoena for information.

We suggest saving these ideas – culling or converting to shared mailboxes – for a post-migration project or projects. Doing so will separate any issues that arise from the experience as it relates to the migration, and can remove unnecessary pressure on timelines and milestone achievements.

Priasoft has public folder synchronization tools which, as the term implies, can manage updates to Public Folders on one or both sides of a migration (source and target environments). In the specific case of Exchange 2010, the migration/sync of public folders become important and complex due to Microsoft change in architecture from “Public Folder Databases” to “Public Folder Mailboxes”. The Microsoft tools do not provide an elegant solution to this and create a situation where there is forced “downtime” for Public Folders during its “cutover” event, which takes ALL PUBLIC FOLDERS offline for the duration of that cutover event. The Priasoft solution does not do this and allows users on both sides, should the migration be a multi-event migration pattern, to work with public folders.

In addition, please review the later topic on Exchange 2010 key differences later in this article as there are some very important sizing and planning for Pubic Folders that must occur.

7. Dry-Run Migrations

Dry-run migrations are specifically different than the testing which was referred in step 3 above. Dry-run migrations are a feature of the Priasoft Migration Suite for Exchange that allows for the migration of mailboxes into dry-run accounts for a “sandbox” migration experience. This ability provide several very important outputs that become critical in the development of the migration plan and schedule.

1. Functional Tests: This use of the dry-run provides “sanity checks” to show that migrations still work. Consider that the business is still “live” and it could be that environmental changes have occurred between the last use of the migration tools and now. A quick functional test will prove whether the environmental changes affect the ability to migrate, or not.

2. Performance Tuning: This use of the dry-run provides a way to determine the maximum performance of the migration by forcing all of the environmental variables to participate. The concept is to gradually increase the “run rate” of the migration to a point where the environment(s) show that it cannot do more.

3. Fidelity Checks: This use of the dry-run provides a way to determine which mailboxes will succeed versus fail. It will also expose any environmental issues. Essentially, any issues that occur in a dry-run are issue that would also occur in a production run. However, there is now the ability to discover those issues and either make changes to correct them, or work around them.

4. Duration and Timing: Since a dry-run migration actually copies data – but without interference or changes to end user experiences – the duration of mailboxes and of an entire batch is accurate down to the minute. A dry-run batch of 1000 users showing a completion time of 24 hours can be accurately used for scheduling purposes a week or two later, plus or minus only a few minutes for possible new data since the last dry-run.

5. Target environment load testing: The target databases used for the dry-run will be filled with real data from source mailboxes, possibly exposing any storage configuration or hardware issues for which a traditional ‘load gen’ scenario could never do. We have seen many times where the results of a dry-run leads to the discovery of issues with the target storage system, and where other tests did not.

8. Sample And End-To-End (E2E) Tests

Although dry-runs tests provide much value, by design, it does not make any end-user changes and so cannot expose any issues with the production migration that are related to those elements. End-to-End tests with test mailboxes and desktops becomes important for such elements. These tests can be done concurrently with dry-runs if desired and if sufficient staff and bandwidth are available to do such. It is often better, however, to do E2E testing after dry-runs since the dry-run exercises may lead to discoveries of issues that can be resolved before E2E tests.

9. Migration Scheduling And Orchestration

Dry-run migrations, by design, will provide timing data so that scheduling of actions and resources – human and technology – are managed efficiently. This also helps set expectations better for end-users, department leaders, application owners, etc.

Additionally, once timing information is available, batching and orchestration work can occur which is simply a process of determining which users migrate first, second, or third and when each group should start. Consider that without detailed timing information provided by the dry-runs, this type of work cannot really occur, at least not with any accuracy.

So, now we have a 9-step process for migrations that hopefully provide a better overall context on how to be successful.

Exchange Key Differences And Points Of Concern

As an article focused on Exchange 2010 to 2016 migrations, it is important that some key topic be shown that can influence the migration in various ways.

Size Information

Exchange 2010 and earlier, used a different algorithm for reporting item, folder, and mailboxes sizes than does later versions of Exchange. Exchange 2010’s size calculations were from an idea of “how many bytes to send this somewhere”. Exchange 2013 and later changed to “how many bytes consumed on disk”. Note that the size of the item stored in the database on the disk is not different, only the reporting of its size. This difference in reporting means that the exact same item will be reported larger by 30-40% in Exchange 2013 and later.

Consider this change in a numeric way:

Now consider the many features of Exchange that uses size information. For example, if the source environment has a mailbox database policy that limits mailboxes to 2GB, a mailbox of 1.65GB is obviously safely below that limit. However, after a migration and if the same policy values were set on target databases, the mailbox will likely exceed the limit and users will be impacted afterwards.

The same is true for message size limits, attachment limits, etc. it is highly recommended to review all the elements of the target environment that are setup to react to size information to ensure that it is accommodating this difference in calculation.

Public Folders

Public folders are also affected by this. If the source public folder “database” is 200GB, as measured from the actual database file in the file system, then it should not be expected to change. But if the size information about public folders has been gathered by summing the folder sizes using PowerShell, those numbers need to be inflated before used for settings in the target environment.

Public Folders also have new limits since the data is not contained in one or more public folder “mailbox” objects. Since the data is now in mailboxes, it inherits all the limits of mailboxes as well, some of which a public folder database never had. It is highly recommended to review the public folder limits of Exchange 2016 before starting: https://docs.microsoft.com/en-us/exchange/collaboration/public-folders/limits

If your source public folder data is more than 50 GB, you will likely need to have multiple public folder mailboxes which adds new complexity. Unfortunately, if a public folder mailbox fills up, data does not “spill over” into a new mailbox – data stops being added and users receive cryptic messages of a simple “failed to save item” pattern. Priasoft’s tools have unique and automated handling of this challenge by calculating and mapping source folders to specific target mailboxes.

There are additional topics to discuss about public folders. Contact our team if you’d like to know more.

Memory/RAM

Exchange 2010 works with a concept of “shared memory”. This was not necessarily because of any explicit design choice by Microsoft, but is due to the fact that all mailboxes databases were loaded and managed by a single background process (store.exe). In windows, multiple threads of an application can access the same memory region, if desired. As such, it was common to see store.exe reserve 80% of available host RAM for caching of information and quick responses. This reserved ram could be adjusted by Exchange as needed to service different database and the demand for data being requested from them.

However, this also meant that it was a single-point-of-failure or attack. If an issue were to occur with one database, all databases would be affected. Additionally, for some other system operations and if a configuration change were necessary, it could require that store.exe be restarted, which would drop access to all databases until it was restarted.

Exchange 2013 and later use a separate “worker” process for each mailbox database that is mounted. While this added some valuable resiliency and database-level configuration management, is also means that each database has a fixed amount of reserved RAM. Running 20 database on Exchange 2010 can be done efficiently with a set amount of RAM. However, the same amount of RAM on Exchange 2016 might not be as efficient for the same number of databases.

If Exchange has insufficient RAM, then it will start to use the windows “paging file” to provide more “virtual RAM”. This means that databases, some, most, or possibly all can be sluggish for reads and writes. Furthermore, if you consider an exchange backup situation where the backup calls cause the fetching of data from the paging file, backups can become very slow. If the setup is a simple environment where the database files for Exchange use the same disk as the operating system, then the performance of the host can suffer even more. If you layer the effects of a migration on top of that as well, we hope it is seen how sufficient RAM-per-database is important.

Failing to properly consider this ratio can lead to very intermittent issues. On one day, the demand on the databases is low, so the effect of the paging file is not noticeable. However, on a later date, when heavy demand arrives, things seem sluggish. Maybe a reboot of the server is made and things seem ok again for a while, but the repeat again after some time. These are common symptoms of this topic.

CPU/Core Count

Exchange 2010, when compared to Exchange 2016, had a much simpler “footprint” on a server with regards to processing and services. A default install of Exchange 2010, with all 3 roles (CAS, HUB, MBX) has about 24 services that are “Exchange” (this is not even including the IIS related services). In contrast Exchange 2016 has over 30 core services, and that’s is without the inclusion of any database “worker” processes. Exchange 2016 has added many more health services to provide much better resiliency and issue forecasting that any prior version, but the cost is CPU utilization. Also, Exchange 2016’s full text indexing seems to be more aggressive and adds to the CPU load when a flood of new data arrives (think migration). Lastly, the IIS (web) components used by Exchange have also expanded.

The point we are trying to make is to not simply adopt the same paradigm for capacity planning and host resource allocations as was done with prior versions of Exchange. We find in even a simply lab setting, that Exchange 2016 only runs smoothly with an 8-core host, or more, and sufficient RAM as described above.

It is especially important consider this for virtualized hosts since the memory allocations can be dynamic (not recommended either). We suggest monitory the paging file related performance counters on exchange servers to see how often and if there are any trends with regards to Exchange having to use the paging file. This is true both for migrations and for normal daily use.

Connection Protocols

Exchange 2010 supported two different connection types for Outlook/MAPI: RPC-over-TCP, and RPC-over-HTTP. The older legacy RPC-over-TCP is the original MAPI communication protocol that uses Windows RPC conversations. The protocol is quite secure, but not very flexible especially for the more disconnected and loosely coupled environments we now see. The latter connection type, RPC-over-HTTP is also known as Outlook Anywhere. This was first introduced in a service pack for Exchange 2003. This protocol simply wrapped the RPC calls in HTTP conversations. This allows for Outlook to access mailboxes securely from outside the organization using SSL/TLS security mechanisms to encrypt the wire traffic.

Exchange 2013 and later no longer support RPC-over-TCP connections from client applications (it is still used on the server itself, but only on the local host). Exchange 2016 provides a newer protocol called mapi-HTTP. This newer protocol is more resilient and provides faster connections and authentications and essentially remove the “RPC” conversation and replaced it with newer, purposely designed conversations directly in the HTTP conversations. If nothing else, it removed a layer of translation. Exchange 2016 still provides OutlookAnwhere connections, but Microsoft will slowing fade that protocol away as well over time. It is expected that the next version of Exchange will likely have no support and no option for OutlookAnywhere.

The importance of this consideration is that if Outlook clients before migration are using RPC-over-TCP, it becomes even more important to have a process to update those profiles properly. It also requires a compatible version of Outlook. Outlook 2010 was updated by a hotfix in 2015 to support mapiHTTP. Outlook 2013 received the support in Service Pack 1. Outlook 2016 and later have it available from the first release. If end users do not have these updates, they will fail to connect to the target mailbox after migration, even though the profile is properly updated by Priasoft.

An additional consideration is for applications that use MAPI. Such applications likely do not rely upon AutoDiscover to detect changes and so may not know that a new protocol need to be used with Exchange 2016.

Conclusions

Test! Test! Test!

The old saying, those who fail to plan, plan to fail, could be rewritten to: those who fail to test, will fail the test.

Priasoft provides the most robust testing capability yet seen, and with 20 years of direct experience in migrations, also provide guidance for testing in ways that many other cannot due to that same lack of tenure in these events.

We are experts in 2010 to 2016 Migration, to learn more about our solutions follow the link below:

We’d be happy to discuss how this pattern can work for your next or current migration project. We can also show this process first hand in a lab setting if desired.