10/23/17

Tuesday

~9 minutes, ~7:24am.

Lulu reports power outage: ~ 7:24-7:33am

Q: A double power-down within a minute, around 7:32 and 7:33?

Campus-wide.

Math's server also down, so as of 9:20am, IT reporting (Chemistry and Physics) and Pumpkin Cam were down.

CornellALERT: Power Outage - Ithaca Campus
Oct 24, 2017 7:29 AM CORNELLALERT
Cornell Police has received reports of a widespread power outage impacting the Ithaca campus. Service personnel are aware of the outage and responding. Additional reports of the outage are not needed. Life threatening emergencies, including persons stuck in elevators, should still be reported to Cornell Police at 607-255-1111 or 911. For additional information and updates go to http://cuinfo.cornell.edu/alerts

Email's time-stamp was GOOD: Tue 10/24/2017 7:31 AM

4/23/17

Sunday

5-10 minutes., ~9:25am.

Some data points:

=============

Sunday, April 23, 2017, 9:28 AM, Micheal wrote: I’m betting we just had a power outage or blip on campus.

http://www.cornell.edu/cuinfo/specialconditions#2718

Sun, 04/23/2017 - 10:58: At 09:50 a.m. on April 23, 2017 power outage is impacting the Ithaca campus. Service personnel are aware of the outage and are responding.

Sun, 04/23/2017 - 11:57: Cornell Utilities reports power to all of the Ithaca campus has been fully restored as of 10:45 AM.

Almost 3 hours, 5:17-8:05pm.

We did get power temporarily on for 20 minutes near the end of this interval, 7:15-7:35pm.

Some data points:

=============

Sat 2/4/2017 7:44 PM, Micheal wrote: (Power went out at) 7:15pm to the (ed. room, per Synology records). I got in and was starting to look at what was up and what was down, but then I lost access to everything. I think the room lost power again at 7:35pm.

Sat 2/4/2017 9:12 PM, Michael wrote: Power appears to have come back at 8:05pm (ed. per Synology records).

TBD.

Note, for future reference: Early on, "NYSEG has estimated restoration around 8:15-8:30pm"

https://itservicealerts.hosting.cornell.edu/view/4655

CUAlert nofications:

(1) First CUAlert came about 30 minutes after actual outage started:

Sent: Saturday, February 4, 2017 5:46 PM
CornellALERT: ITHACA CAMPUS POWER OUTAGE
A widespread power outage is  currently impacting the entire Ithaca campus and parts of the Ithaca area.

(2) Our systems only got power 20 minutes after CUAlert stated power was back:

Sent: Saturday, February 4, 2017 7:52 PM
CornellALERT: UPDATE 7:45 p.m. ITHACA CAMPUS POWER OUTAGE
Power to Cornell’s Ithaca campus has been restored.

Michael had logged in, noted system had shut down. And was himself kicked out, indicating a second power outage during this time.

NetAdmin-L alerted potential ~10 minutes before Alert:

From: On Behalf Of Jamie Rosner Duong
Sent: Saturday, February 4, 2017 5:36 PM
To: NET-ADMIN-L
Subject: Power outage?

Are parts of Cornell experiencing a power outage?

4/11/16 Monday

About 40 minutes, starting shortly after noon. Official emails:

Mon, Apr 11, 2016 at 12:25 PM: CornellALERT: ITHACA CAMPUS POWER OUTAGE
----------------------------------------------------------------------
There widespread power outage is impacting the entire Ithaca campus and parts of Ithaca. Service personnel are aware of the outage and responding. Additional reports of the outage are not needed.  NYSEG and Cornell Utilities personnel are working to restore power as quickly as possible

Mon, Apr 11, 2016 at 12:59 PM: CornellALERT: ITHACA CAMPUS POWER OUTAGE - UPDATE 12:48
----------------------------------------------------------------------
As of 12:48 p.m on 04/11/16 Cornell Utilities reports that the power has beeAs of 12:48 p.m on 04/11/16 Cornell Utilities reports that the power has been fully restored to the Ithaca campus.

Sun article, quoting Melissa Hines:

http://cornellsun.com/2016/04/11/damaged-transmission-line-responsible-for-cornell-power-outage/

2/27/2014
Thursday

2 minutes
CU's record: Per 3/3/14 email: On Thursday 2/27/14 at 9:50am the campus experienced a disruption of electricity.  The whole campus experienced an approximately 30 second outage. Many buildings on the central, west and north campus areas remained without power for 30 minutes to an hour.
Oliver's record: Power outage at 9:53a. Restored at 9:55a (more than one minute).

Procedural error?
Per 3/3/14 email: The outage was caused as a result of routine maintenance activities which were being conducted at the Campus’ main substation which takes power from the NYSEG transmission system and provides it to campus.  This work has been conducted many times before without incident but in this case caused a major disruption of electricity supply.  Staff from Utilities and external technical resources are investigating the root cause of this unexpected event.

No link to info in 3/3/14 email?
Some delayed time-stamps, from the IT folks:
http://www.it.cornell.edu/services/alert.cfm?id=3072
Initial timing info, from the power folks:
http://www.cornell.edu/cuinfo/specialconditions/#2050

Michael led our effort to initially evaluate and restore systems, with Oliver adding to documentation and to-do's. Lulu completed the cluster restoration efforts.
Lost 1-2 hours each for Michael, Lulu, and Oliver.
Cornell called it a "power blip". In Oliver's books, any outage longer than seconds is not a "blip".
Q: Broke one GPU workstation?

1/27/2014
Monday

17-19 minutes
CU's record: Power outage at 2:22p. Restored around 2:41p.
Oliver's record: Power outage at 2:22p. Restored at 2:39p.

?

http://www.it.cornell.edu/services/alert.cfm?id=3040

Lulu, Michael, and Oliver shut down head nodes and other systems which were on UPS. (Those systems non UPS shut down hard, per usual.)
Lost 3 hours, for Lulu, Michael, and Oliver.
Roger away on vacation (out of the U.S.)

12/23/2013
Monday

2 minutes
CU's report: 08:36 AM - 8:38 AM
(ChemIT staff not in yet.)

Procedural error?

http://www.it.cornell.edu/services/alert.cfm?id=2982

Terrible timing, right before the longest staff holiday of the year.
ChemIT staff not present during failure.
Lost most of the day, for Roger and Oliver.
Michael Hint and Lulu way on vacation (out of the U.S.)

7/17/13

Half a morning (~2 hours)
CU's report: 8:45 AM - 10:45 AM

http://www.it.cornell.edu/services/alert.cfm?id=2711