A thought experiment about software quality
If you couldn't read the source code,
how would you know it was good?
Treat the software as an opaque artifact. You can still observe whether a test notices a changed behavior, whether a contract holds, and what happens to real users. Each signal answers a different question. The Observatory maps what those signals can tell us — and what they leave unknown.
Without the source
What would change your mind?
Someone says the system works. You cannot inspect its implementation. What could you observe that would make you more confident?
No single observation settles the question. Each one rules out a particular way the software could be wrong.
A changed condition makes a test fail.
The suite notices that behavior changed. It cannot tell you whether the intended behavior was right in the first place.
Examine mutation testing → 02 / BOUNDARY AGREEMENTA consumer's contract still holds.
One system still meets another's expectations. The rest of the workflow may still be wrong.
Examine contract tests → 03 / LIVE BEHAVIORA canary behaves normally under real traffic.
The release has not shifted the signals you watch. That cannot establish that users get the outcome they need.
Examine canary analysis →A worked example
What does a successful payment response tell us?
Suppose the payment endpoint reports success, but the order never enters the paid state. The response tells us the endpoint answered. It does not establish that the business promise was kept.
Read the full example →The payment endpoint responded successfully.
Whether the order entered the paid state.
Check that every payment event has an order in state paid within an hour.
Examine business invariants →The confidence stack
You do not get one verdict.
You build a case.
Evidence arrives at different times and from different vantage points. Each observation narrows a different doubt. A green result from one says nothing about the claims measured by the others.
-
01 / BEFORE IT RUNS
Is the artifact internally coherent?
A type checker catches contradictions in the program. It cannot know whether the behavior is useful or intended.
-
02 / BEFORE IT SHIPS
Does the behavior meet a stated expectation?
Contract tests check a boundary another system depends on. They say nothing about expectations nobody stated.
-
03 / WHILE IT RUNS
What happened outside the test environment?
Events and canaries show live behavior. They only see what you chose to record and compare.
-
04 / AFTER USERS MEET IT
Did the system keep its promise?
Business invariants test outcomes the domain requires. A pass covers those promises, in that window, and no others.
Continue the inquiry
What would convince you about your system?
The catalog has 59 instruments. Each entry says what it can observe, what it cannot detect, and what evidence it produces. Start with a claim you want to test, then look for observations that do not share the same blind spot.