IMAP not working for users | Mailbox server in different AD site than CAS

Here we have a cross site scenario where users from one site are trying to connect to the CAS that is on the different site.

By default, POP3 and IMAP4 connectivity between a Client Access server in one Active Directory site and a Mailbox server in another Active Directory site is not enabled. The Client Access server prevents users from logging on to their mailboxes by using POP3 or IMAP4 if their mailboxes are located on a Mailbox server in a different Active Directory site.

And we get the following warning in the application log of the Client Access Server:

User "User-Name" was prevented connecting to his/her mailbox because the Mailbox Server resides in a different ActiveDirectory site (CN=Site A,CN=Sites,CN=Configuration,DC=domain,DC=com) than the Client Access Server (CN=Site B,CN=Sites,CN=Configuration,DC=domain,DC=com) to which he/she connected. Either arrange for the Client Access Server and the Mailbox Server to reside in the same site, or change the configuration setting of AllowCrossSiteSessions to true.

To enable cross-site connectivity for POP3 and IMAP4 clients:

1. On the Client Access server, locate the POP3 and IMAP4 configuration files. By default, the location is C:\Program Files\Microsoft\Exchange Server\ClientAccess\PopImap.
2. Use the text editor to open the configuration files. The two configuration files are as follows:
   Microsoft.Exchange.Pop3.exe.Config
   Microsoft.Exchange.Imap4.exe.Config
3. Search for the key AllowCrossSiteSessions. add key="AllowCrossSiteSessions" value="false" /
4. Edit the line of the configuration file as follows. add key="AllowCrossSiteSessions" value="true" /
5. Repeat the steps in this procedure for each protocol for which you want to enable cross-site connectivity.
6. Restart the services for each configuration file that you have updated to enable cross-site connectivity.

More Details at Microsoft Technet.