Software for an experiment you cannot rerun

A lead–lead collision in the LHC is over almost before it begins, and it happens exactly once. ALICE was upgraded to record them continuously at interaction rates of up to 50 kHz.¹ If something upstream was miscalibrated while that happened, the data is not wrong in an obvious way. It is quietly, plausibly wrong, and you cannot go back for another copy.

From July 2025 to September 2026 I wrote production C++ for ALICE's online quality-control system, the software that watches data as it is being taken and tells people on shift whether to trust it. My corner was FIT, the Fast Interaction Trigger: the detector that tells the rest of the experiment that a collision happened, when, and roughly how violent it was.¹

The river problem

"You cannot step twice into the same river."Heraclitus, as reported by Plato in the Cratylus²

Detectors age. Photodetectors lose gain as they accumulate charge, so the same particle produces a smaller signal in month nine than in month one. Nothing breaks. The numbers just drift, slowly enough that every individual run looks fine.

The fix is to give the detector a ruler it cannot fake. A minimum ionising particle, a MIP, deposits a well-known amount of energy, so its peak in the amplitude spectrum is a natural unit. FT0, one of FIT's sub-detectors, sees roughly 14 ADC counts per MIP.¹ Track where that peak sits over time and you are tracking the instrument itself. That is what the aging monitor and the ADC-to-MIP calibration workflows I worked on do: they turn "is the detector still the detector we calibrated?" from a question someone remembers to ask into a plot someone sees every shift.

Rebuilding the ship at sea

"We are like sailors who must rebuild their ship on the open sea, never able to dismantle it in dry-dock and reconstruct it from the best materials."Otto Neurath, "Protocol Sentences," 1932³

Neurath meant knowledge. It describes running experiments just as well. You do not stop ALICE to refactor its monitoring. Changes go into a system that is taking data, used by people who were not in the room when you made them. Three habits followed from that.

• Monitor the instrument, not only the data. A clean distribution from a drifting detector is the most dangerous plot in the building.
• Treat calibration as data. It is versioned, it has provenance, and it is wrong in ways you will want to reconstruct later.
• Write for the person on shift at 3 a.m. If a check needs an expert to interpret it, it is a research question, not a check.

That last point is why we also proposed an AI assistant for FIT operators, accepted at MIDI 2025.⁴ Not to replace the expert, but to make the expert's knowledge available at the hour when they are asleep.

Most software gets to assume the world holds still. Physics software doesn't, and I suspect it is right more often for exactly that reason.