Playing with RSG | Database Restore

This came across few days back when one of our production Exchange 2003 backend server went down due to hardware failure and was totaly dead.The Backup team was shortly in action to carry out the restore. The restore was done on the different server in a Recovery Storage Group (RSG).

Now the question is how to merge these mailboxes in RSG with the production ones... certainly direct merge was not possible as we didnt opt for Dialtone method as the production server was totaly dead and we didnt set up a new server to replace that one.

The one option was to to extract the mailboxes from RSG using Exmerge, delete the existing mailboxes of users and create new ones on another server, and then import that extracted mailboxes. But this is very time consuming.
However here's an another option to tweak the RSG to make it behave as the production store and move that mailboxes over.

The Recovery Storage Group is identified by the msExchRestore Attribute. The msExchRestore = TRUE property tells us if a database is a recovery database.
As such we cannot directly connect the mailbox from the RSG to user direcly. We have to modify this msExchRestore Attribute and make its value to Not Set so that it cannot be identified as RSG and we can connect the mailbox to the user.

The detailed steps are as follows:

1. Reset the MsExchRestore attribute on both the Recovery Storage Group, and the recovered mailbox store using Adsiedit:

Expand the Configuration Container node, and then browse the hierarchy to:
DC=Domain
DC=com
CN=Configuration
CN=Services
CN=Microsoft Exchange
CN=Administrative Groups
CN=Administrative Group Name
CN=Servers
CN=ExchangeServerName
CN=InformationStore
CN=Recovery Storage Group

Right click for Properties on the Recovery Storage Group, ensure that the checkbox for "Show Optional Attributes" is checked. Scroll down to the "MsExchRestore" attribute. Double-click the MsExchRestore attribute and check the Not Set parameter. Click OK, Apply, and OK to exit the properties pane.
Right click the "CN=MailBox Store(ServerName)" under the Recovery Storage Group, and select Properties. Ensure that the checkbox for "Show Optional Attributes" is checked. Scroll down to the "MsExchRestore" attribute. Double-click the MsExchRestore attribute and check the parameter. Click OK, Apply, and OK to exit the properties pane.
Close the AdsiEdit mmc

2. Now from the ADUC>Exchange Tasks, delete the mailboxes of the users that are in question.

3. In Exchange System Manager select the Recovery Storage Group, right click and select Refresh. Expand the Recovery Storage Group, then the mailbox store.

4. Select "Mailboxes" under the Mailbox Store, then right click the mailbox to be recovered, and select "Reconnect". The applet "Select a New User for this Mailbox" applet will appear. Enter the alias of the user you wish to associate with the recovered mailbox into the "Enter the Object Name to select" data entry field, the click "Check Name". The alias of the user should be resolved to the full display name. Click OK. You will see a pop-up stating "The operation has completed successfully".

5. Once its done for all the users, you can successfully move the mailboxes over to any of the servers.

Storage Group not responding | Version Store Issues

This happend few days earlier on our exchange server... all users in a particular Storage Group were not able to connect to Outlook/OWA. The mails queued up for them on the server...
On looking at the server logs... there were lot of logon error messages... users were not able to logon to their mailbox... and at the beginning when it all started, there was one error message with Event Id 623. It says:

Information Store (2984) Storage Group 1: The version store for this instance (3) has reached its maximum size of 155MB. It is likely that a long running transaction is preventing cleanup of the version store and causing it to build up in size. Updates will be rejected until long-running transaction has been completely committed or rolled back.

Analysis

The Version Store keeps an in-memory list of modifications made to the database. It gives ESE the ability to track and manage the current transactions. Thus the Version Store is where transactions are held in memory until they can be written to disk.

Event 623 is the result, typically of a long running transaction. The result of this long running transaction is to exhaust resources in the version store. As a result, the Version Store no longer reaps deleted records causing unneeded data, which is marked as deleted, to accumulate in the database. The accumulation of unneeded data can exacerbate performance problems which can lead to event id 623. No more transaction can continue until this is clear.

Thus we will see 623 event indicating that the maximum Version Store size has been reached . All the Write operations to the database will fail because there's no more version store space to record the operation.

Why this happens

This can happen for one of the two reasons:

1) In order to properly reconcile write-conflicts and properly support repeatable reads, a given entry in the version store cannot be cleaned up until it is older that the oldest active transaction.
2) Version Store cleanup simply cant keep up with the load on the machine.

Possible Causes

- Online Maintenance Tasks running at peak times.
- Backups running at peak times.
- Disk I/O performance.
- Large Mails

Any of these can add up simultaneously and add to the performance degrade of the server.

More details and troubleshooting on this.

Find Users not using Default Exchange Storage Limit set via Mailbox Policy

Here's an LDAP query that I use frequently to search and display all the users in the domain that do not have the default mailbox store policy set.

(&((mailNickname=*)(mDBUseDefaults=FALSE)))

Public Folders not able to receive External Emails

When we set the Mail Enabled Public Folders, it receive all the internal emails fine but is unable to receive any mail from outside and throws the NDR to the sender:

This is an automatically generated Delivery Status Notification.

Delivery to the following recipients failed.
abc@xyz.com

Final-Recipient: rfc822;abc@xyz.com
Action: failedStatus: 5.2.1
X-Display-Name: abc

This happens when Permission to Anonymous is set to None. Make sure Anonymous has atleast Contributor rights.

Mail Delivery Slow: Messages Waiting to be Routed Queue Filled up

This issue stuck us 2 weeks back and we went all over by this. Suddenly we found out that the emails sent are being received after a lot of delay. Looking at the Exchange Server, I saw thousands of messages in Messages Waiting to be Routed Queue for Default SMTP Virtual Server and the number kept on increasing and the messages were going out very slow.

Slow mail delivery and mass queuing of mail in the Messages Waiting to be Routed queue in Exchange is typically caused by either Anti-Virus software on the Exchange server, by Distribution List expansion problems, or by connectivity problems between Exchange and Active Directory.

Antivirus was already ruled out as we tried disabling it from the registry as well but with no affect.

I opened up a case with MS PSS... turned up the diagnostics logging for MS Transport and MS DSAccess but nothing conclusive from the logs.

WE monitored the LDAP read and search times, SMTP categorizer queue length as it seemed to be the performace issue. Here is excellent MS guide for Troubleshoting Exchange Server Performance Issues.

Then we collected the Hang Dumps for Store.exe and Inetinfo.exe.

In the Store dump we see that we are waiting on WLAP calls to the GC’s.
From the Inetinfo dumps, we are waiting for the HrCheckRestrictions which means that the mail was probably to a DL that had restrictions placed on it.

So we figured out that Delviery Restrictions might be the cause of mail delivery being slow as we have applied the delivery restrictions on some Distribution Lists quite recently.

This problem occurs when lots of Lightweight Directory Access Protocol (LDAP) searches are initiated. Lots of LDAP searches are initiated when you send mail to distribution groups that include lots of users who have delivery restrictions configured on their mailboxes.

If you send a message to a group that contains many recipients, and if each of those recipients is also configured with a delivery restriction to reject messages from the members of a distribution group that contains many members, Exchange 2000/2003 Server must expand the restricted distribution group one time for each member of the group to which you sent the message. Also, if a failure that can be retried occurs during this process, Exchange Server stops the group expansion process, and then retries the connection an hour later. This causes the messages to be held in the categorizer queues, delays message processing, puts load on transports' Advanced Queuing and SMTP components, and eventually causes system queues to start backing up.

Here is the Excellent MS Exchange Team Blog for Performance issues due to connector restrictions.

Well, the solution to this above was the hotfix (already included in Exchange SP2) and change in one registry entry that defines the new Expansion Logic for restricted Distribution Group.

Here are the details:

1. Locate and then click the following registry subkey:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MSExchangeTransport
2. On the Edit menu, point to New, and then click Key
3. Type Parameters, and then press ENTER to name the new registry subkey.
4. Right-click Parameters, point to New, and then click DWORD Value.
5. Type RestrictionMethod, and then press ENTER to name the new registry entry.
6. Right-click RestrictionMethod, and then click Modify.
7. Type 2, and then press ENTER .

Reference: http://support.microsoft.com/default.aspx?scid=kb;EN-US;895407

Troubleshooting Fast Growing Transaction Logs

Here is a pretty good technet that details various causes:

http://technet.microsoft.com/en-us/library/aa996191.aspx

Also when we mount/dismount the stores manually, the transaction logs files are replayed on to the database thus they get committed and we are able to mount stores.

We can check this status by using eseutil: eseutil /mk "path to checkpoint file"

It will give you the transaction log file uptill which all the logs have been committed. This is helpful in the case we are running out of disk space and when the full backup is also a long way to finish… so that we can move out the committed logs and keep them on a different location. In case of recovery scenario, we need these log files to make database up-to-date.

Disaster Recovery (Soft Recovery) - Exchange 2003

After an unexpected shutdown of the Exchange Server, the Information Stores in One Storage Group goes dismounted. Gave an error message while mounting.
Checked the state of the databases by eseutil /mh and there it shows dirty shutdown.

Here it can be that one of the database is not up-to-date and the server got rebooted when it was processing one of the transactions and got corrupted.

Our next step in this case is to identify the store that has been corrupted.

We will try to replay the logfiles on the each of the stores individually:

• Renamed priv2.edb and priv2.stb.
• Replayed the logfiles: eseutil /r /i E00
• Error Encountered. So log files cannot be replayed for Information Store 1 (priv1.edb, priv1.stm)
• Now Renamed priv1.edb and priv1.stm.
• Replayed the logfiles: eseutil/ r /i E00
• Completed Successfully. So the Information Store 2 is up to date now and we are able to mount it.

Now as the Information Store 1 is corrupted, the best practice is to restore from the last backup and then make the database up to date by recovering from the present log files.Eseutil /p should be the last resort as it performs the hard recovery and punches off the corrupted tables or entries from the database. So there is always a good chance of loss of some data.

As we have renamed priv1.edb and priv1.stm, the database restore from the backup will create new set of these files. We should not overwrite these files from the restore.

The Restore of backup (Veritas in this Case) should be set with the following options:

• Do not replay logs automatically
• Do not mount databases
• No Loss restore (don't overwrite or delete anything)
  

After the restore, we have the new set of edb and stm (a day old) and the current log files that will make them up to date.
Now we will use "ESEUTIL /CC" command to replay to restore.env file and continue on to production log files. There are temporary log files and restore.env file created when we run the restore. In case of Veritas restore, they are created in C:\temp\

Restore.env file is just a temporary environmental file holding path information about the data. Its purpose is to help the Exchange 2003 restore process find its files and match them with the corresponding email stores.

Now run the command eseutil /cc "path of restore.env"

Mount the store back after it completes successfully.

Calculate amount of disk space to be reclaimed by Offline Defrag...

By using the command eseutil /ms, we can calculate the exact amount of disk space that can reclaimed by doing the offline defrag.

Here are the steps:

1) Run eseutil /ms command from the bin directory on the dismounted database. It will always be run on the edb file.

2) The first section of the space report is the SLV SPACE DUMP. This reports on free space in the streaming database file (.stm).
TOTALS:
Free: 1528814
Reserved: 2238
Deleted: 228
Committed: 11888
Unknown: 0

There are 1528814 free pages in the .stm file, and each page is 4096 bytes in length. Therefore, there are 6,262,022,144 bytes of empty space in the file (1528814 x 4096).

3) As for the EDB file dump you'll see the following information in the last line in the column 'Available'
-------------------------------------------------------------------------------
3244272

The number at the lower right of the output (3244272) is the total of all free pages in the database. If you multiply this number by 4096, you will see that defragmenting this database will recover 12.3 gb of space.

4) Hence, based on the above example you will recover 12.3 GB + 6GB after you do the offline defrag.(this is including the STM also)

System Attendant Mailbox Missing: Users getting NDRs, Cant Move Mailboxes...

In this scenario, the users are getting NDRs that System Attendant cannot be reached... and when we try to move the mailbox to and from the server, the error says:
"The attempt to log on to the Microsoft Exchange Server computer has failed. The MAPI provider failed. Microsoft Exchange Server Information Store ID no: 8004011d-0512-00000000"

The application log has the following error messages:

Event ID : 9175
Raw Event ID : 9175
Record Nr. : 1167709
Category : MAPI Session
Source : MSExchangeAdmin
Type : Error
Message : The MAPI call 'OpenMsgStore' failed with the following error:
The attempt to log on to the Microsoft Exchange Server computer has failed.
The MAPI provider failed.
Microsoft Exchange Server Information Store
ID no: 8004011d-0512-00000000

These symptoms indicate that the ‘mailbox store’ that holds the System Attendant mailbox is down or that the SA mailbox is disconnected, but actually all the mailbox stores are up and running on the server.
There is only one mailbox store in an Exchange server that holds SA mailbox... but you cannot find this mailbox on any of the stores.
And when you verify the attributes of System Attendant using ADSIEdit under Microsoft System Attendant, it is found that homeMDB was ‘Not Set’

Recommended Plan:
=================

• Using ADSIEdit, browse to the location – Configuration – Microsoft Exchange - - Administrative Groups - - Servers – – Information Store -, on the right side you’ll see a list of mailbox stores, click on any one of them and copy the value of ‘Distinguished Name’ attribute
• Using ADSIEdit, browse to the location – Configuration – Microsoft Exchange - - Administrative Groups - - Servers – – Microsoft System Attendant, right click here and find the attribute ‘homeMDB’, paste the value that you copied earlier (From ‘Distinsguished Name’) here
• Restart ‘Microsoft Exchange System Attendant’ service
• After the exchange services come up, browse to the store whose DN value you copied earlier and find the System Attendant mailbox under ‘Mailboxes’ list, if you don’t find it run a ‘Cleanup Agent’

If you don’t see the SA mailbox in the ‘Mailboxes’ list on the store even after performing ‘Run Cleanup Agent’. Using ADSIEdit, once you get to the properties of the ‘Microsoft System Attendant’ you can note the value of the the attribute ‘mail’. This is essentially the primary SMTP address of the SA mailbox. You can send an email to this address, that should initialize the mailbox and you should see it under ‘Mailboxes’ tab in ESM.