Our monitoring over the weekend detected a backlog of spooled emails. In order to clear the backlog, the Support Team have scaled down importing (Mailbox Reader and Import Services). Additional Processing Units (PU's) have been scaled up to help clear the backlog and the team are currently monitoring the progress. We anticipate that the backlog will take around a day or so to clear.
We understand the root cause of the problem; a misconfiguration of an LDAP setting caused PUs to timeout resulting in an initial backlog, we also identified mailboxes setup to Poll, which is not best practice for customers with a large number of mailboxes. We have shared the details with Titan Support.
Please note we have not scaled down the journaling of live emails.
I am pleased to report that the spool has cleared and all services are running with no issues.
Support teams will continue to provide heightened monitoring of the system for the next 24 hours.
The backlog is processing as expected, we anticipate that it will clear down within the next 24 hours, we will enable the default mailbox readers so that newly created tenants can download data. However the larger bespoke mailbox readers eg Harbour, Momentum Solar, thejoint will be disabled until the back log clears
The Support Team will continue to closely monitor the environment.
Processing of backlog emails is still in progress and the queue is reducing. Please note that mailbox readers may be in Waiting state as we have scaled down the services to allow the processes to clear the lag.
The teams are engaged monitoring the queue and will scale up the services as soon as the lag clears.
