Docs / Operating

When something is wrong

For whoever is holding the pager — on a small team that is the person who built the thing, on a larger one it is not. This page is about reading the state of an environment and clearing it; being told the moment something breaks is its own page, Reacting to failures.

An environment carries one vital sign, and every entity in it carries its own: a dot on its row, rolled up the tree so a group shows the worst of its members, and summed into the status bar as the one colour for the whole environment. No dot means nothing has been recorded since somebody last looked — not that last night went well because a screen says so this morning.

Two colours

The colour tells you how much is wrong, and the evidence tells you who raised it and what to read next.

• Red means something is not working. Usually a run failed: it threw, it went over its budget, or somebody stopped it, and the evidence is the run itself. It can also be the platform’s own observation about a system you call — a connection whose far end stopped answering, where there is no run to open because the fact is about the connection.
• Amber means something is worth knowing, and nothing is broken. Usually your own script said so: logError or logWarning reports the half of integration trouble that never throws — the far end answered politely, and the answer is wrong. The run finished; the script still flagged it, and the evidence is the line it wrote. It can also be the platform noticing something about a system you call while every call still works — a login server that granted your connection different permissions from the ones it asked for.

Either way the evidence says which: it names the run to open, or it says what the platform itself saw.

Only unattended work colours anything. A Test in the editor reports to the person pressing the button, and to nobody else — the answer is already in front of you.

The state stays until somebody clears it

A surface that shows the last run’s outcome is green by morning: the night’s one failure is followed by a hundred quiet successes, and by the time anybody looks, nothing looks wrong. So the mark does not work that way. Once an entity goes red or amber it stays red or amber — through every success that follows — until a person acknowledges it. The mark is the platform saying something happened here that nobody has seen yet.

From the colour to the line

The status bar’s vital sign is a door: it takes you to the health section, which lists each marked entity with the evidence itself — since when, and the failure’s own words, not just a colour. From there, open the run behind it.

A run’s log is where the story bottoms out. Under your own lines the platform writes one line for every call the run made — what was called, what it answered, how long it took — so the log filters are how you narrow a bad night: set the level to error and above, add a search word, and open the run behind the line that remains. The runs list is also where a script that wrote no log lines at all still shows what it received and what it returned.

Clearing it

Acknowledge is not “mark as read”. It deletes the episode: since when, what state, how many times it tripped — gone, and the entity is unmarked until something new happens. What remains afterwards is the audit trail’s record that you acknowledged it, and the runs themselves for as long as they are kept. If you want the details later, read them before you clear it, or find the runs again by their time.

That is the whole loop: the mark tells you something happened, the evidence tells you what, and acknowledging it is you signing that somebody has seen it.

Who clears what

Anyone who may change the environment clears one entity’s mark. The person who owns the script, read the failure and fixed it is usually the person who should say it has been seen, so clearing a single mark asks for the same access as changing the thing that failed. Someone with read-only access sees every mark and clears none — where you cannot clear one, the row says so rather than simply leaving the button out.

Clearing a whole environment’s marks at once is a tenant administrator’s. That is the act for the night a broken connection or a bad release marked half the environment, and it is a different decision from signing off one failure: it names how many marks it is about to delete, and then they are all gone. It clears what you were looking at — anything that fails while it runs is still a failure nobody has seen, and it stays.

An automated browser test performs this page’s loop against a running platform, every time the product changes — a real delivery fails a real script, the mark appears in the tree and survives two successes, the evidence is read, and the acknowledgement clears it. Two accounts differing only in access walk the paragraphs above: one clears the mark on their own work, the other is shown the sentence saying why they cannot, and a tenant administrator clears the environment whole with the count in front of them. A second walk narrows three runs to one log line and opens the run behind it. The page cannot outlive the behaviour it describes.