Exchange Migration Knowledge BaseCategory: Mailbox Migration QuestionsWe’re seeing item failures in our migration logs that cause mailboxes to fail. What does this mean and what is the best way to handle this?
Anonymous asked 11 years ago

We are seeing logs that have entries like the following.

2015-08-26T10:09:30

Error

The mailbox has exceeded the threshold for messages that have failed to copy due to an exception. The threshold is 15 and has been exceeded. To change the Skip Message threshold please modify or add the SkipMessagethreshold setting in the registry in the following location HKEY_CURRENT_USER\Software\Priasoft\Mailbox Migration Manager 6.0\Control
Is this due to corruption, or something else?

1 Answers
Eriq VanBibber Staff answered 8 years ago

Item failures tend to happen for only a few reasons:

  • Corrupted Items
  • An “on-access” Exchange virus scanner
  • Item manipulation during migration
  • A transient issue

Let’s detail each of the above…
Corrupted Items
This is rarely seen by our solutions and tools.  It does happen, but is just not common.  The only real way to determine corruption is by looking at a pattern.  If each restart of the mailbox results in the same failure, at the same folder, and after the same amount of time, then it is likely corruption.
The issue with corruption though is that if an item is truly corrupted, there’s no way to handle it other than to skip it.  A truly corrupted item will cause an issue in Outlook just the same so attempting to remove the items by deleting them within outlook rarely works.
There is no marking or return code that identifies a corrupted item.  MAPI has simplistic return codes for operations like “Failed”, “Timeout”, “Too Big”, etc.  There is no code for “corrupted”.  So there is no ability mark or preemptively ignore corrupted items – it is only done thru discovery.
This lends to the importance of Priasoft’s Dry-Run feature.  Since during a dry-run, data is actually touched and migrated, corruption can be found before it affects an end user during a production event.  Notice can then be sent to users, if deemed necessary, about the issue and helps control the perception during the production migration efforts.
On-Access Virus Scanners
This is the most common reason for what appears to be item corruption.
Exchange 2007 and earlier supported a feature called the MS Virus Scanning API.  It allowed virus scanning software to be notified any time an item was being requested.  Exchange would hand the request to the virus scanner for inspection and would wait for the virus scanner to return.
Most of the virus scanners worked with the use of a tracking database to understand if an item had been recently scanned, and with which revision of virus signatures.  As such, an item that was scanned an hour ago would quickly be handed back to Exchange since the virus scanner would know that it was scanned.  However, for older items the issue is that there maybe days, weeks, or years worth of virus signatures with which to scan.  The older the item, the longer it takes to scan the item.  If the item has complex attachments, it can take even longer.
MAPI and exchange have some hard-coded, unchangeable timeout values for things, one of them being how long to wait for an item request to return.  When you consider that a migration will access all the items in the mailbox, including the oldest items, it should be evident then that a virus scanner can make it appear like there is item corruption.  When a request is may by MAPI for an item in a folder, if a response from the Exchange server is not received in a timely fashion, MAPI will time out and return a simple “Failed” error code in this case.  One might expect a “timeout” error code to be returned, but that error is reserved for other network related issues, not cases of waiting for exchange to respond.
The pattern that can be seen in the migration logs when a virus scanner is an issue is one where each restart of the mailbox progresses the a bit further thru the mailbox.  In addition, this will also cause the virus scanner to update its tracking info such that the next time the items are requested, they are seen as “recently scanned”.  However, this approach skews the timing and value of dry-runs when trying to determine how long a migration will take.
Furthermore, even if all items are “recently scanned”, there is still a performance hit to the migration since each request of an item is still handed over to the virus scanner for inspection.  We have see as much as a 30% difference in speed between no virus scanner active and on that is enabled.
It is our recommendation then, at least for the duration where mailboxes are actively being migrated, to disable any on-access virus scanner software.  However, this may prove to be non-trivial.  Considering how this works where a DLL file is loaded with the MS Exchange Store process, and that DLLs are not simply “unloaded”, it is necessary to properly disable a virus scanner.
The first way is by the use of the vendor’s suggested operation, usually by clicking a button in their control panel to disable the scanner.  However, this does not cause the DLL to be unloaded.  As such, exchange may still hand messages to the virus scanner, but being disabled it hand the item immediately back to exchange.
If you simply “stop” the virus scanner “service” in windows, this will not (in most cases) cause the virus scanner to stop scanning.  This usually only prevents the reporting and tracking of the state of an item.
Item Manipulation During Migration
This is very rare, but is a potential case.
When a mailbox is being migrated, the process moves folder-by-folder and item-by-item in those folders.  In order to process items in the folder, the tools request a listing of all the items for a given folder and works off of this in-memory list.  However, if at exactly the moment after we get the list of items, a user begins to delete, move, or modify those items, there’s a chance of a conflict of access to the item such that it could cause an item failure.
It is our recommendation that some strategy then be applied with regards to when a migration occurs and how to communicated to end users about it.
The first part of the strategy is to do migrations during periods of low utilization by the users that are to be migrated.  For many organizations, nights and weekends are opportune times for such activity.  This nearly eliminates the chance of a user making changes right at the same time as we are working a folder.
The second part of the strategy is to communicate with those users that might likely need to be in their mailbox at the same time as the migration and inform them that it is ok to send and receive mail and read mail, but activities that make broad changes to the mailbox should be avoided until the mailbox completes the migration.  Importing or exporting data with PST files, moving folders or otherwise restructuring the mailbox, or making mass changes to items should be avoided during the short timeframe where we are actively migrating data for the mailbox.
Transient Issues
Because the source and target environments are live systems, there can be small issues that happen in the environment and for which MAPI and Exchange do not respond well.  Network interruptions, VPN key changes, exchange database failovers, and a myriad of other potential environment issues can occur.  Such things are often very short-lived and are generally not felt or discovered by end-users or even administrators.  The reason is due to timing and processing speed.  Most user activity runs as “human speed”.  However, migrations run a “computer speed”.  As such, a network interruption that lasts 1 or 2 seconds could affect a dozen item copy actions for a migration.
The Priasoft migrator automatically retries mailbox failures 3 times before giving up and presenting the mailbox as “Failed”.  When this happens, the item is sent to the bottom of the migration queue and will bubble up to be migrated again as part of the normal process.  In most batches enough time passes just waiting for the re-queued item to bubble up such that the transient issue passes and the mailbox succeeds.  However, if the batch is very small – perhaps only a dozen or less mailboxes – there may not be a sufficient amount of other work before the re-queued item is restarted.  In such a case, and when the transient issue is still present, the mailbox may be retried the 3 times and ultimately fail.  However a manual restart of the failed item then succeeds because by the time an admin discovers the failed items and makes the motions to restart, sufficient time has passed.
It is then the recommendation, especially for small batches, to let a few minutes elapse after a mailbox shows as failed and then to restart it.  Chances are high that it will simply succeed on the subsequent pass.
Resolution Options
Give the above likely reasons for item issues, the first resolution is to restart the failed mailbox.
If corruption is determined, or if the environment is unstable and transient issues are more likely, increasing the “skip count” can help.  However, please understand that while this can more readily allow the mailbox to complete, there is also a greater chance that data is being left behind that could have been copied.
If a virus scanner is at issue, and management cannot be convinced to disable it, it may be worthwhile to set the skip count very high and then run a dry-run across all mailboxes first just to “seed” the tracking database in the virus scanner such that all items appear as “recently scanned”.  Then, more traditional use of dry-runs (for performance tuning, fidelity, and duration) can be executed.
One important point to note with the previous comment is that if time elapses between the dry-run “seeding” pass and other activities, updates to the virus scanner’s signature database could be made and somewhat invalidates the value of the “seeding”.
Lastly, the detail in some of the older log files reference a location in the registry that is incorrect and should be the following location:
HKEY_CURRENT_USER\Software\Priasoft\Mailbox Migration Manager 6.5\Control
Notice the version number difference in the above path.
The registry value of SkipMessageThreshold may not exist, and if not should be added as a new DWORD value set to the maximum number of items failures before failing the mailbox.  The default value, when missing in the registry, is 15 which means that on the 16th item failure, the mailbox is failed.
Regards,
Eriq