Advice: Networking performance issues: 2016-05-31 to 2016-06-03

Follow

The QRIScloud networks in Polaris are currently experiencing periods of high load.  Users may be seeing drop-outs in network connections to their virtual machines or when accessing QRIScloud RDS collections.

We are investigating potential solutions to the cause of the increased network traffic. We apologize for any inconvenience.

UPDATE - 2016-05-31 16:45 - we have implemented a change to rate-limit the source of the network traffic that was causing the problem.

UPDATE - 2016-06-01 11:00 - unfortunately, the rate-limiting that we implemented was in effective, and the network performance issues are still occurring.

UPDATE - 2016-06-01 16:30 - we have turned off off-site replication for QRIScloud collections, and this has reduced the network load significantly. However, we are still seeing short periods (up to a minute) of packet drops roughly once an hour.

UPDATE - 2016-06-02 16:10 - an hour ago (at 15:10) we made a configuration change on a critical data path within the Polaris data centre to try to address the problem.  So far, it seems that the change is working.

UPDATE - 2016-06-03 09:00 - unfortunately, we are still seeing problems.

UPDATE - 2016-06-03 17:45 - we made more configuration changes at around 11:30 today, and so far we have not seen any further large-scale dropping of packets.  Please let us know if you are still seeing problems.

UPDATE - 2016-06-06 10:15 - we did not seen any recurrences of the networking problems over the weekend. Please let us know if you see any more problems.

Have more questions? Submit a request

Comments

Powered by Zendesk