Troubleshooting Back Pressure Issues | Exchange Server 2007 Hub/Edge Transport Servers

Back Pressure is an important resource monitoring feature of Microsoft Exchange Transport Service that exists on Hub/Edge Transport servers. The following system resources are monitored by this:

• Free space on the hard disk drive that stores the message queue database.
• Free space on the hard disk drive that stores the message queue database transaction logs.
• Number of uncommitted message queue database transactions that exist in memory.
• Memory that is used by the EdgeTransport.exe process.
• Memory that is used by all processes.

All configuration options and threshold values for back pressure are available in the EdgeTransport.exe.config application configuration file that is located in the C:\Program Files\Microsoft\Exchange Server\Bin directory. You can even enable or disable this resource monitoring by changing the EnableResourceMonitoring parameter in EdgeTransport.exe.config file.

There might be situations and i also ran into one when you notice that the mail has stopped flowing from the Hub server and the queues are disabled. In the event logs, there is a warning message from MSExchangeTransport with event id 15004 which states:

Resource pressure increased from Normal to Medium.

Resource utilization of the following resources exceed the normal level:

Version buckets = XXX [Medium] [Normal=80 Medium=120 High=200]

Back pressure caused the following components to be disabled:

Inbound mail submission from the Internet

Mail submission from the Pickup directory

Mail submission from the Replay directory

Mail delivery to remote domains

Please note that all these threshold values for normal, medim and high are defined in the EdgeTransport.exe.config file. There can be number of reasons that can cause resource pressure to jump high and disable the queues... here are most expected reasons for this:

1. Running out of disk space or disk performance issues.
2. Running out of memory
3. Antivirus software running on Exchange Server with incorrect configuration
4. Virus issues.
5. A very large message being processed by Exchange Server.

The first 4 are are very visible and you can easily verify if one or the other is running out or spyking anything. In most of the cases, its the fifth one that is likely to hit. Once that large message gets in the submission queue, it is being processed by exchange server and this causes the problem.

There are various things that can prevent this type of scenario... for e.g. Outlook 2010 wouldn't even let you attach the attachment that is larger than the limits set on an exchange server, however if you are running earlier version of Outlook and Exchange Server is also not at current update level, this can happen.

Here are the few steps that you need to do for the quick resolution for this:

1. Find out the mailbox that has sent the large email.
2. Disable that mailbox.
3. Restart the Information Store service on the mailbox server.
4. Change the location of Queue Database path and log file path. It can be done from EdgeTransport.exe.config file by changing the value for "QueueDatabasePath" and "QueueDatabaseLoggingPath".
5. Restart the MSExchange Transport Service on the Hub Transport Server.
6. You might need to perform the recovery on the queue database to get the emails that are in uncommitted state on the previous queue database from the logs.

If you want to learn more on backup pressure, here's the technet for you.

Understanding Back Pressure

Exchange 2010! Released to Manufacturing

Great news for everyone... described as the most advanced and the best messaging application by many - Exchange 2010 has been Released to Manufacturing... announced on the MS Exchange Team blog.

Update Rollup 9 for Microsoft Exchange Server SP1 Released

Microsoft has released Update Rollup 9 for Microsoft Exchange Server 2007 Service Pack 1 (SP1).

This update rollup contains some imortant bug fixes.

KB 970162 details about the issues fixed in this release of update rollup.

Read more on MSExchange Team Blog.

Standby Continuous Replication - On Standalone Mailbox Servers - Part 3

In Part 2, we discussed about the Preperation of the SCR Target server, in this Part 3, we will talk about SCR Activation for the Database Portability scenarios and in the end, we will discuss about SCR Activation using the Recover Server switch.

Database Portability is a new feature of Exchange 2007 and it allows us to mount the exchange database on any of the exchange servers in same organization. So in case of a database corruption say in SG1, we can manually perform the activation of the SCR target database.

Here are detailed steps for SCR Activation for the database portabilty scenarios:

1. Dismount the SCR Source database.

2. Now we need to disable the SCR and need to make the target database available for mounting.

This is done by the following command:

Restore-StorageGroupCopy SourceServerName\SG1 -StandbyMachine TargetServerName

Note: If the SCR source is not available, the Force parameter should be used with the Restore-StorageGroupCopy command.

3. After this has completed, we need to verify whether the database is in a Clean Shutdown state. If the database is in a Dirty Shutdown state, we can bring the database to a Clean Shutdown state by running Eseutil recovery mode (Eseutil /r) against the database.

4. Now after the database is in the Clean Shutdown state, we need to update the Active Directory with the new locations of the Storage Group and database files.

Here are the commands to change the paths for SG1Target and DBTarget from the temp paths to the paths for the SCR target's storage group and database files:

Move-StorageGroupPath TargetServerName\SG1Target -SystemFolderPath Path -LogFolderPath Path -ConfigurationOnly

Move-DatabasePath TargetServerName\SG1Target\DBTarget -EdbFilePath Path -ConfigurationOnly

5. Now we need to make the database to allow itslef to be overwritten during a restore operation. We can do this from its properties from the EMC.

6. Next, we can mount this database.

7. Ok now, the database is mounted, but we need to rehome the mailboxes to point to SCR Target Server. Please note that Microsoft Exchange System Attendant and System mailboxes should not be included in this.

Here is the command to do this:

Get-Mailbox -Database SourceServerName\SG1\DB I where {$_.ObjectClass -NotMatch '(SystemAttendantMailboxExOleDbSystemMailbox)'} I Move-Mailbox -ConfigurationOnly -TargetDatabase TargetServerName\SG1Target\DBTarget

So now this information has been replicated across the Active Directory, users can access the mailboxes now.

Please note if the Source server is not accessible, for clients running Outlook 2003 and earlier versions, we need to change their outlook profile to point to new target server.

Now, there might be scenarios where the whole source server is completely down and is not accessible pertaining to any hardware failure or something else, in these cases we can use Recover Server to activate the SCR target.

Here are the detailed steps for SCR Acttivation using Recover Server:

1. Disable the SCR and make the target databases available for mounting.
This is done by the following command:

Restore-StorageGroupCopy SourceServerName\SG1 -StandbyMachine TargetServerName -Force

2. Uninstall exchange server from the target server.

3. Rename the target server as the name of the source server if you have Outlook 2003 clients and want them to rehome automatically.

4. Run Setup /recoverserver. This should be run from the location where exchange binaries are located.

5. If the databases are in Dirty Shutdown state, use Eseutil to recover the databases.

Standby Continuous Replication - On Standalone Mailbox Servers - Part 2

In the Part 1 of this article, we discussed about SCR and how to enable it. In this Part 2, we will focus on the second step i.e. Prepare the SCR Target Server.

We can have the target server prepared initially so that in case of a failure, the SCR activation can be done pretty quickly that will save us precious time.

For all the Storage Groups (from the source server) whose copy has been setup on the SCR target server, we will create the Storage Groups on the Target server and create the mailbox database on it. We can keep the path of the log files and the database on any temp location as we will mount this store, dismount it and remove that temp directory.

Steps and Commands:

As in the part 1 of this article, we have enabled the SCR for SG1 on the Source Server. Here we will create the Storage Group SG1Target and mailbox database DBTarget on the Target Server.

1. New-StorageGroup -Server TargetServerName -name SG1Target -LogFolderPath C:\SG1Target -SystemFolderPath C:\SG1Target
2. New-MailboxDatabase -StorageGroup TargetServerName\SG1Target -Name DBTarget -EdbFilePath C:\SG1Target\dbtarget.edb
3. Mount-Database DBTarget
4. Dismount-Database DBTarget
5. Delete everything from C:\SG1Target

Please note that these storage groups that we have set up on the target server are only temporary objects that will be used to carry out the database portability recovery during the SCR Activation Process. We can create them later on also but creating them now will save us the precious time while performing the recovery.

Standby Continuous Replication - On Standalone Mailbox Servers

SCR - Standby Continuous Replication, the wonderful new feature introduced in Exchange Server 2007 SP1, introduces a new dimension in High Availabilty for Exchange Server 2007. SCR is designed for scenarios that use standby recovery servers and allows replicating a copy of Storage Group to a number of remote servers.

SCR uses the same log shipping and replay technology as local continuous replication (LCR) and cluster continuous replication (CCR).

It is very useful in the DR planning especially if we don’t want to go for full fledged clustering with automatic failover technologies like CCR because of various reasons like cost constraints of servers, OSs that are needed to have these types of setups.

SCR introduce redundancy in two scenarios for the Mailbox Servers:

• One Storage Group or Multiple Storage Groups go corrupt - Use Database Portability to activate SCR target databases.
• Full Server failure - Use Server Recovery to activate SCR target databases.

The starting point for SCR is called the source, which is any storage group, except a recovery storage group and the endpoint for SCR is called the target. SCR can support multiple targets per storage group. Although SCR can be combined with CCR and LCR, in this article we will focus and discuss its configuration on the standalone Mailbox Servers.

The whole process revolves around three activities:

1. Enable SCR
2. Prepare the SCR Target Server
3. SCR Activation

Enable SCR

Requirements:

1. The paths must be the same for both soource and the target like if source server is C:\Exchange\Data and C:\Exchange\Logs then these paths must be available on the target server.
2. There is a hard coded 50 log lag between the Source and Target, by default there is a 24 hour replay time which is configurable.
3. There can be only 1 database per storage group.

Command:

Enable-StorageGroupCopy SourceServerName\SG1 -StandbyMachine TargetServerName -ReplayLagTime 0.0:0:0

Parameters:

ReplayLagTime specifies how long the Replication Service waits before replaying replicated transaction logs to the replica Database (EDB) on the target. Default - 1 day

TruncationLagTime sets a lag time for truncating log files on that replica. Default - 0.

We can validate and check the status of the replication by running:

Get-storagegroupcopystatus -standbymachine TargetServerName

We will discuss the next two steps in the next parts of this article.

451 4.4.0 DNS query failed

I was working on this annoying issue few days back, the mails sent to few domains are getting stuck in the queues on the Hub Transport server. The error message on the queue states 451 4.4.0 DNS query failed.

Well, it looked liked from here that there might be some issue with the DNS, ran few nslookup tests from the HUB servers itself, and surprisingly the domain and its MX records were resolving fine and i was able to connect to them absolutely fine.
More interestingly, if you hardcode the entry of the MX Record for that domain in the hosts file, the mails went through straight away.

I researched around few forum posts stated below, it seem like that this is a Bug with Exchange Server 2007 on Windows Server 2008, some issue related to IPv6.

Reference Posts:

http://social.technet.microsoft.com/Forums/en-US/exchangesvrtransport/thread/2e9a6263-7863-4d13-9480-7611db222e9e
http://social.technet.microsoft.com/Forums/en-US/exchangesvrtransport/thread/5ac614ee-e4fb-41d9-88f0-363f912634b1

Update Rollup 8 for Microsoft Exchange Server SP1 Released

Microsoft has released Update Rollup 8 for Microsoft Exchange Server 2007 Service Pack 1 (SP1).

It contains the important bug fix which happened after Update Rollup 7 which caused the sender of the Delivery Status Notification to be unresolved.

KB 968012 details about the issues fixed in this release of update rollup.

Read more on MSExchange Team Blog.

Exchange Server 2007 SP2... Arriving Soon!

Exchange Server 2007 SP2 is set to arrive in Q3 this year... it contains many new features but some great news on the backup front. It is providing the new backup VSS-based plugin that will enable users to take Exchange aware VSS backups from Windows Server 2008. So we will be able to backup and restore Exchange 2007 SP2 databases from Windows Server 2008 backup tool.

For more details and other features, please read this MSExchange Team blog.

Outlook Anywhere or RPC over HTTPS configuration from Script

I was reading this interesting post from Oz Casey Dedeal about RPC over HTTPS Script. This prompted to me to write this post as i was working on this some time back and had another way of scripting RPC over HTTPS and Outlook Anywhere configuration.

This VB script will do the configuration for both Outlook 2003 and Outlook 2007. This script basically aims at changing the registry entries for your Outlook Profile that do all the outlook over https or outlook anywhere configuration.

Here are the Registry Keys and values that we are targetting at for doing this configuration in an Outlook Profile:

HKEY_CURRENT_USER\Software\Microsoft\Windows NT\CurrentVersion\Windows Messaging Subsystem\Profiles\\13dbb0c8aa05101a9bb000aa002fc45a

"00036601" - for Cached Mode
"00036623" - Enable Exchange Proxy settings and control various check boxes like "on fast networks…" etc.
"00036627" - For Authentication - Basic or NTLM
"001f6622" - for specifying Exchange proxy server name
"001f6625" - for specifying Exchange proxy server name according to principal cert name.

So the script will go like:

Const HKEY_CURRENT_USER = &H80000001
strComputer = "."
Set oReg=GetObject("winmgmts:{impersonationLevel=impersonate}!\\" & _
strComputer & "\root\default:StdRegProv")
strValue1Name = "00036601"
arrValue1 = Array(&H00,&H00,&H00,&H00)
strValue2Name = "00036623"
arrValue2 = Array(&H00,&H00,&H00,&H00)
strValue3Name = "00036627"
arrValue3 = Array(&H00,&H00,&H00,&H00)
strValue4Name = "001f6622"
arrValue4 = Array(&H00,&H00,&H00,&H00)
strValue5Name = "001f6625"
arrValue5 = Array(&H00,&H00,&H00,&H00)
strKeyPath = "Software\Microsoft\Windows NT\CurrentVersion\Windows Messaging Subsystem\Profiles"
oReg.EnumKey HKEY_CURRENT_USER, strKeyPath, arrSubKeys

For Each subkey In arrSubKeys
strFullPath = strKeyPath & "\" & subkey & "\13dbb0c8aa05101a9bb000aa002fc45a"
oReg.SetBinaryValue HKEY_CURRENT_USER,strFullPath,strValue1Name,arrValue1
oReg.SetBinaryValue HKEY_CURRENT_USER,strFullPath,strValue2Name,arrValue2
oReg.SetBinaryValue HKEY_CURRENT_USER,strFullPath,strValue3Name,arrValue3
oReg.SetBinaryValue HKEY_CURRENT_USER,strFullPath,strValue4Name,arrValue4
oReg.SetBinaryValue HKEY_CURRENT_USER,strFullPath,strValue5Name,arrValue5
Next

Replace the value of 00 after &H in arrValue1, arrValue2, arrValue3, arrValue4 and arrValue5 with your values by looking into the registry under mentioned reg keys.

Update Rollup 7 for Microsoft Exchange Server SP1 Released

Microsoft has released Update Rollup 7 for Microsoft Exchange Server 2007 Service Pack 1 (SP1).

This update has important fixes for SCR and IMAP4 issues.

KB 960384 details about the issues fixed in this release of update rollup.

Read more on MSExchange Team Blog.

Integration Of Exchange 2007 and OCS 2007

In the last post, I discussed about the Microsoft's core solutions on Unified Communications - Exchange Server 2007 UM and OCS 2007. I had a chance to setup a test environment where they both operate together. So here we will talk about their intergration and take a look how it looks like.

To have both of them work together, you need to have Exchange Server 2007 SP1. Here are the key steps that you need to perform to configure them:

1. Configure the new dial plan. To do this, run this command from the powershell:

   New-umdialplan -name OCSPlan -Uritype "Sipname" -Voipsecurity "SipSecured" -Numberofdigitsinextension 4



2. Go to properties of this dial plan and set the subscriber access number. Subscriber access is used by users to access their individual mailboxes to retrieve e-mail, voice messages, contacts, and calendaring information. Outlook Voice Access is the new Exchange 2007 Unified Messaging feature that lets subscribers access their Exchange 2007 mailbox.


3. Apply this dial plan to the UM server:
   set-umserver -id UM_Server_Name -dialplans OCSPlan


4. Run Powershell script "EXCHUCUTIL.PS1" from Exchange Management Shell. This is installed when you apply SP1 for Exchange 2007. To run this successfully you might need to modify the properties of the Start Menu shortcut for the Exchange Management Shell so that it launches directly from the Scripts folder "Program files\Microsoft\Exchange Server\Scripts".


5. On the OCS server, run OCSumutil with the following command line: ocsumutil /domain:yourdomain.com
   

Now you can configure your users for Unified Messaging. You can do this from Exchange Management Console and right click on the user to Enable Unified Messaging and select the dial plan.

So now when you log on with this user to Communicator 2007, you will see the additional call menu from where you can forward your calls or redirect incoming calls to voicemail.



And now you get a Missed Conversation notification when someone IMs you and you are offline or away:

Also, You get the missed call notifications or voice mails when someone calls you via OCS client:

The world of Unified Communications

Over the recent times, this has been one of the hottest topic of discussion in various organizations and among various professionals.

Microsoft and third-party vendors such as Cisco Systems, Avaya, Nortel, and Polycom offer products that support Unified Communications (UC), and have their own definitions for UC.

If we talk about Microsoft here, Microsoft offers two integrated solutions: Microsoft Exchange Server 2007 and Office Communications Server 2007 that are the key players in Microsoft's UC strategy. Together they bridge the gap between computers and telephones. They integrate with your company's exisiting phone system and deliver complete communication services using your existing data network.

So in the world of Unified Communications, Microsoft offers Exchange as the Unified Messaging component, handling voicemail, fax, and telephone access to messaging. OCS is the component that offers IM, conferencing, presence, and voice services.

Microsoft Exchange Server 2007 Unfied Messaging

Here are the features that Unified Messaging Role provides in Exchange Server 2007:

Outlook Voice Access (OVA): Outlook Voice Access allows a phone to be a Unified Messaging client. A text-to-speech engine is built into the UM server to allow email message to be read from a users mailbox. A UM-enabled user can call into their company's UM server and can access Voicemail, Email, Calendar, Contacts, OOO messages and security and personalization settings.

Call Answering: Say if a user receives any external call, this call will be routed via IP/PBX or a VoIP gateway, if the destination user is not available, this call will be routed to Unified Messaging Server. The external user will be given an option to leave a voice mail and it will be delivered to the mailbox server, else the user will be notified of the missed call.

Auto Attendant: The Auto Attendant in the Exchange Unified Messaging Server allows an organization to create customized menus, custom greetings, searching directory etc.

Fax Receiving: The UM server provides the ability to receive faxes in .tiff format via IP/PBX or VoIP gateway.

Microsoft Office Communications Server 2007

Office Communications Server 2007 manages all real-time (synchronous) communications including: instant messaging, VoIP, audio and video conferencing. It works with existing tele-communications systems, so business can deploy advanced VoIP and conferencing without tearing out their legacy phone networks. Microsoft Office Communications Server 2007 also powers Presence. With presence, you can see at-a-glance if someone is available and contact them with a click using instant messaging, a phone call or a video.

Microsoft unified communications technologies delivers voice and video conferencing through Microsoft Office Communications Server 2007 via IP. This means that conference call audio and video streams are captured and logged just like an instant message or an email.

The complete conferencing solution requires:

• Office Communications Server 2007
• Microsoft Office Communicator 2007
• Microsoft Office Live Meeting service (2007)
• Video-equipped computers and/or VoIP phones

Integration of Exchange 2007 and OCS 2007

Microsoft Exchange Server 2007 Unified Messaging and Microsoft Office Communications Server 2007 can be deployed together to provide voice messaging, Instant Messaging (IM), enhanced user presence, audio-video conferencing, and an integrated e-mail and messaging experience for users in your organization.

Here are the key features and benefits that they will provide together:

• You get the missed call notifications or voice mails when someone calls you via OCS client.
• You get a notification when someone IMs you and you are offline or away.
• OCS automatically uses the out-of-office (OOF) message text you set in Exchange.
• Presence information displayed in Microsoft Outlook 2003/2007 and in Windows Sharepoints Services team sites and document libraries.

So in a nutshell, you are Connected... Always!

Update Rollup 6 for Microsoft Exchange Server SP1 Released

Microsoft has released Update Rollup 6 for Microsoft Exchange Server 2007 Service Pack 1 (SP1).

This update has a fix for the critical security issue and a fix to allow IE 8 to be used for OWA 2007. This does not include OWA 2007 S/MIME control.

KB 959241 details about the issues fixed in this release of update rollup.

Read more on MSExchange Team Blog.

Some mailboxes show up as Shared or Linked... after Migrating to Exchange Server 2007

Here's a scenario where after migrating users from Exchange Server 2003 to Exchange Server 2007, some of the user mailboxes show as "Shared" Mailbox or "Linked" Mailbox within the Exchange Management Console... instead of showing them as "User" Mailbox.

A Linked mailbox is a mailbox that is associated with an external account. Thus Linked mailboxes are normal mailboxes that can hold an External account in a separate resource forest.

Few of these mailboxes here were shown as Linked mailboxes after migration because an orphan Active Directory Account has the "Associated External Account" right on the mailbox in question.

To fix this and have them show up as normal "User" mailboxes remove that orphan Active Directory account that has the "Associated External Account" right on the mailbox in question and run the following command from the Exchange Management Shell:

set-mailbox -id MAILBOX_NAME -ApplyMandatoryProperties

Now we come to why few mailboxes are shown up here as Shared mailboxes. If you have Self permission with associated external account checked, you get the shared mailbox on migration. That is the reason these mailboxes are showed up as shared.

To fix this, remove the "Associated External Account" permission from the Self and again run that command:

set-mailbox -id MAILBOX_NAME -ApplyMandatoryProperties

To read and understand more about the ApplyMandatoryProperties Parameter, read this excellent blog entry from Evan Dodds.

Exchange Server Remote Connectivity Analyzer

This looks pretty exciting... in beta though but worth keeping an eye on as it can be very useful for exchange admins... especially the amount of troubleshooting we need to do... :-)

https://www.testexchangeconnectivity.com/

Entourage for EWS

Entourage for Exchange Web Services is targeted at organizations using Exchange Server 2007. This means that we can finaly say goodbye to WebDav and hopefully to all the issues that we face as EWS is a stronger and more robust protocol than WebDAV, that enables client applications to communicate with the server running Microsoft Exchange Server 2007 SP1 or later – for better compatibility, performance and reliability.

Entourage EWS will launch a public beta this month. The final release will be available later this year as a free update to Office 2008 users.

More Details: http://www.microsoft.com/mac/itpros/entourage-ews.mspx