Excerpt |
---|
Too many power outages in Baker Lab, and other Chem buildings! |
ChemIT's record of recent power outages
Date | Outage duration | Cause | Official link | ChemIT notes |
---|---|---|---|---|
2/27/2014 | 2 minutes | Procedural error? | No link to info in 3/3/14 email? | Michael led our effort to initially evaluate and restore systems, with Oliver adding to documentation and to-do's. Lulu completed the cluster restoration efforts. |
1/27/2014 | 17-19 minutes | ? | Lulu, Michael, and Oliver shut down head nodes and other systems which were on UPS. (Those systems non UPS shut down hard, per usual.) | |
12/23/2013 | 2 minutes | Procedural error? | Terrible timing, right before the longest staff holiday of the year. | |
7/17/13 | Half a morning (~2 hours) |
|
|
...
- Because we don't know the answer to this question following any specific power outage, we are reluctant to turn back on servers right away. Instead, we like to wait ~10-25 minutes.
What would it cost to UPS our research systems?
Assuming protection for 1-3 minutes MAXIMUM:
...
CCB head nodes' UPS status:
Remaining UPS's to invest in
Most have been done Spring 14, after the spate of power failures. For details on clusters and non-clusters respectively, see:
Wiki Markup \[CCB HPCs
../../../../../../../../../../display/chemit/CCB+Clusters+Information]
- CCB non-HPCs
Clusters
Cluster | Done | Not done | Notes |
---|---|---|---|
Loring |
| X | Unique: Need to do ASAP |
Abruna |
| X | Unique: Need to do ASAP |
Non-clusters
Stand-alone computers' UPS status:
...
Cluster | Compute node count | Power strip equivalents | Cost estimate, | Notes |
---|---|---|---|---|
Collum | 8 | 1 | $900 | |
Lancaster, with Crane (new) | 10 | 2 | $1.8K | |
Hoffmann | 19 | 2 | $1.8K |
|
Scheraga | 91 | 13 | $11.7K | |
Loring | 4 | 1 | $900 | |
Abruna | 9 | 1 | $900 | |
C4 head node: pilot | N/A | N/A | N/A | This CCB Community head node pilot has no compute nodes of its own. |
Widom | 2 | 1 | ? | Compute nodes are hanging off of "C4" head node, above. |
TOTALS | ~140? | 21 | $18K + Widom |
|
Procedures and reminders
Reboot switches?
- 8 switches support the clusters (out of 12 in the room).
Start head nodes, if not on already
- Only a few are on UPS. Those can obviously be left on.
- None should be set to auto-start on power-off.
Confirm head nodes accessible via SSH
PuTTY on Windows
Use FMPro to get connection info?! (not the right info there, though...)
Menu => List Machine / User Selections = > List Group or selected criteria
- Machine function => Server HN
Start compute nodes
If nodes done show up, consider:
...