Operation Dark Fiber

BLACKOUT · LINUX · difficulty 4 · ~30 min · 25 steps

The data centre has no uplink. Everything still works — on cached answers, for about forty minutes.

Briefing

04:52. The border router for hall 1 lost its carrier uplink during a maintenance window and did not come back. Nothing has failed yet: DNS is serving from cache, the directory has a replica on this side, and the licence servers hold a grace period. All three of those expire. When the resolver cache empties, name resolution stops; when the licence grace runs out, the hypervisors stop accepting new workloads; when the directory's cached credentials age out, nobody can log in to fix any of it. That is the order it falls over in, and the first one is roughly forty minutes away. You have console access to the border router and nothing else. Get the link back. Resist the urge to start reseating optics.

Objectives

• Find out which layer is actually broken before touching anything
• Explain why the interface the config names does not exist any more
• Get the port authenticated to the carrier's network access control
• Work out who changed the trust anchor, and when
• Restore service and record what happened while it is still fresh

Return to the range lobby