We are experiencing recurrences of the Tier2 NFS server problems from last week.
Operations staff are working with the vendor to diagnose and resolve the problem. (The changes made last week were apparently ineffective.)
We apologize for the inconvenience.
UPDATE - 2017-01-09 - 13:20 - We don't have a resolution yet. The vendor engineer is still working on the problem. We have also noticed that the Tier2 outage is impacting on Standard Access Tier3 collection when they are accessed via the "data", "ssh1" and "ssh2" servers.
UPDATE - 2017-01-09 - 15:10 - The Tier2 server appears to be working normally again.
UPDATE - 2017-01-09 - 15:30 - Then it broke again. The vendor engineer is still working on it.
UPDATE - 2017-01-09 - 16:10 - The vendor engineer reports that a fix for the problem is now in place, and the server does appear to be working correctly once more. However, we need to wait a day or so to be sure. Please let us know if you see NFS server issues.
UPDATE - 2017-01-11 - 12:00 - The vendor changes were sufficient to allow the NFS server to cope with the load spikes that we were experiencing. Also, yesterday we finally identified and addressed the source of the load spikes. We are now prepared to call the problem "resolved".