Operation Heat Death

BLACKOUT · LINUX · difficulty 5 · ~35 min · 22 steps

Hall 3 is gaining a degree a minute. You are not going to save the uptime. Decide what you ARE going to save.

Briefing

01:14. Building management paged: hall 3's cooling is not keeping up. Inlet temperatures are climbing about 1.2 degrees a minute and have been for eleven minutes. Facilities have someone driving in; that is forty minutes away. At 40 degrees inlet the GPUs throttle and the training runs stop being useful. At 45 the drives start reporting errors. Somewhere past that the chassis protect themselves and drop power without asking you, mid-write, all at once — which is the outcome that costs an array rebuild rather than an afternoon. You have a management jump box with BMC reachability, cluster access, and read access to the building's plant network. You do not have a way to make the room colder. Work out what that means before you type anything.

Objectives

• Establish whether the temperature is real or a sensor telling stories
• Cut the heat you are generating, in the order that removes most first
• Move the work off the hall before the hardware decides for you
• Shut down in a sequence that protects the data, not the uptime
• Find out why the plant changed its mind at one in the morning

Return to the range lobby