Page History

To check sol head node hard drives status, go:

https://192.168.255.100:888

with Administrator as log in name, password is root password

Maintenance records:

3/8/2016: Lulu: No errors detected on hard drive checks. No security updates available via YUM. FSCK run and no errors.

3/8/2016: updated BIOS on rh027 to v2.0 from v1.0a, ran into serious problems, did not do others in same series but did do the IPMI firmware updates on them, updated rh012-rh019 from v3.0 to v3.2, ran out of time to do the BIOS updates on rh004-rh010 - meh26

7/26/2016: Lulu: No firmware update from Michael. There are no security updates available via YUM. Checked all hard drives are fine. Forced fsck, no errors found. User's home data occupied 92% data partition. Hoffmann group users need clean their home directories. Here are four users who had used most of the space:

705386768KB       lgr48

361360488KB       yt443

355886296KB       px32

354069408KB       tz265

10/18/2016: Lulu: No firmware update from Michael. There are no security updates available via YUM. Checked all hard drives are fine. Tried ddimage the root partition to an image file at external hard drive. Use sgdisk to backup disk partition table.

1/17/2017: Lulu: No firmware update from Michael. There are no security updates available via YUM. Checked all hard drives are fine. Forced fsck, no errors found. User's home data occupied 87% data partition. Checked root /usr & /var space, looks OK now.

4/18/2017: Lulu: No firmware update from Michael. There are no security updates available via YUM. Checked all hard drives are fine. User's home data occupied 93% data partition.

7/18/2017: Michael: no need to worry about any firmware on motherboards or RAID controller.

7/18/2017: Lulu: There are about 6 security updates via yum on head node. There are about four security updates on capsule but failed and aborted on updating. We got the version before yum update from Ezbackup and stored in /data/chroot/centos6-6-restore. We will see if we need revert it back to the version before update. Checked all hard drives are fine. Forced fsck,

1/16/2018: Lulu: No security updates via yum on head node. ("No packages needed for security;" - meltdown won't got patched by this) Checked all hard drives are fine. Forced fsck,

4/17/2018: Lulu: No security updates via yum on head node. Checked all hard drives are fine. Forced fsck, root partition was 90% full. Then I cleaned up many server and job logs. Root partition is 69% full.  Cannot access 3ware web interface with error message "The server at 192.168.255.100 is taking too long to respond. 3ware commands working. Need more investigation. 

10/16/2018: Michael did some firmware update on compute nodes. Michael replaced the router with EdgeRouter X. Lulu: No security updates via yum on head node. Checked all hard drives are fine. Forced fsck,  Root partition is 77% full.  Access 3ware web interface without problem.

1/17/2019: Lulu: No security updates via yum on head node. Checked all hard drives are fine. root partition was quite full. I cleaned up many server and job logs. Now root partition is 70% full. 

4/16/2019: Michael:

1. Update EdgeRouter X from 1.10.7 to 2.0.1
2. Update headnode BIOS from 3.2 to 3.3

      Lulu: Forced fsck. data is 93% full. rh001 cannot boot.

7/23/2019: Lulu: backintime /dev/sdb drive has I/O errors. I disabled mounting and disabled time machine for now. It has EZbackup now. data is 97% full. rh001 cannot boot.

10/15/2019: Lulu: forced fsck. backintime /dev/sdb drive has I/O errors.rh001 does not boot.

01/22/2020: Lulu: backintime /dev/sdb drive has I/O errors.rh001 does not boot