Journaling of email is now stable, we are continuing to monitor the system
The traffic levels have remained at normal historical levels and there has not been a reoccurrence of the increased number of connection requests within the last 24 hours.
Our teams have enabled additional monitoring and will continue to identify the root cause using the information gathered.
The traffic level continue to remain within previous level.
We have not seen a reoccurrence of the increased number of connection requests. The teams will continue to monitor the environment.
The traffic levels have remained at the previous normal historical levels.
We have not seen a reoccurrence of the increased number of connection requests. The teams will continue to monitor the environment.
We have been extensively monitoring the system and have found the following details.
At around 3:00 PM UTC on Mondy 23rd Arctitan started receiving massively elevated connections via SMTP. Likewise, the flow of data into the system via SMTP has increased, this situation has been ongoing until around 3:30 AM UTC Thursday 26th.
The traffic levels have for the moment returned to normal historical values.The increased number of connection requests has the impact of consuming more memory on the SMTP servers and also generating a high queue for the requests to be serviced.
During this event, we have scaled up the number of SMTP servers which has mitigated the problem somewhat but not completely. The very high request rate results in some senders having their connections dropped when trying to deliver emails to the archive.
Now that connection rates have returned to normal, the James servers are at full performance and have consumed all emails waiting at the senders' systems. The SMTP servers processing emails very quickly once connections could get through.
At 22:00 UTC Wednesday 25rd we enabled detailed logging of inbound connection requests to the Amazon load balancer.
We have been able to analyse the IP addresses during the logged time and identified SIUMED to be the top IP address. We are leaving the Amazon Request logging system in place. If the elevated traffic were to resume, we will be able to identify the senders' IP addresses.
We will continue to monitor the system over the coming days, however, for the moment the incident appears to be over.
Some customers have reported issues with journaled email failing delivery. The Support Teams have increased the resources to process the increase of journaled email.
The back log is clearing and the systems are under heightened monitoring
