Unix Wizard: A Week in Whiteboard Sketches
Over the past few days, systems administrator Aphyr has been capturing the week's IT incidents as quick whiteboard doodles, prompted by a suggestion from his boss, Laird. The sketches chronicle a series of real-world troubleshooting sagas, from routine maintenance to a potentially catastrophic hardware scare.
Tuesday's work involved repairing a file transfer box after an apt upgrade broke SFTP by updating dependent libraries. The fix required reconstructing the chroot environment with the aid of ldd.
Wednesday was spent expanding the monitoring setup, adding fresh check scripts and data sources to Cacti to graph mail relay queue lengths and system load averages among other metrics.
By Thursday, the focus had shifted to cleanup duty, purging the heavily saturated file and download servers.
Friday presented the most dramatic event: one ESX server reported irreparable damage to four of its six disks, prompting an automatic shutdown of the entire array to guard against data loss.
In a fortunate twist, the RAID controller's grim diagnosis proved false; restoring the array configuration from disk brought everything back online without issue.
