Mail Relay Meltdown and Recovery
A recent incident at the office started with a routine ClamAV update going sideways. Freshclam skipped eight of its nine known mirrors, leaving a single unavailable mirror to retry for an hour. During that stall, clamd attempted a database refresh, failed to acquire the lock held by freshclam, and shut down. This cascade broke the two clamdscan processes used by amavisd-new, and the Postfix incoming queue swelled with roughly 6,000 undelivered messages.
Resolution came via debian-volatile, which maintains rolling stable packages for ClamAV and other fast-moving projects. The fix also prompted a broader revision of amavisd-new configuration, including restoring communication with SpamAssassin. The net effect is that all inbound mail now passes through spam and virus filtering before reaching the Exchange servers, reducing load and inbox clutter.
Beyond the immediate fix, I expanded monitoring for both Cacti and Nagios so that queue growth and service failures are caught earlier in the future.
Plugin Misadventures
The morning after the mail relay work brought its own trouble. After accidentally spilling a Coke into a colleague's laptop keyboard, I spent an hour writing and debugging a memory check plugin for Nagios. The effort was wasted when I forgot which machine I was on and overwrote the new plugin with the broken one I had been testing from NagiosExchange. The plugin had to be rewritten from memory.
Work calmed down after that, leaving the rest of the day uneventful. Outside the office, plans are in motion to catch a Blue October concert at the Crystal Ballroom tomorrow evening.
