Three ways a calculator is wrong that its own tests can never catch

The distinction that does the work

Verification asks whether a system does what it was specified to do. Validation asks whether the specification was right. Test suites are overwhelmingly instruments of the first kind, and a model with a green suite has demonstrated exactly one thing: it is self-consistent.

That is worth having. It is what stops an edit to one bracket silently changing an unrelated output, and it is what our own build enforces on every commit, with two independent engines computing the same vectors and a one-cent mismatch failing the build. But it is a claim about the model's relationship to itself, and a tax calculator's value lies entirely in its relationship to something outside itself.

One: a whole rule replaced by a plausible approximation

Mexico's income tax was implemented as (gross − 8,000) × 15%. It is a defensible-looking expression. It produces sensible-looking numbers across the whole ordinary salary range. It is monotonic, continuous, and passes every structural property you might think to assert.

It is also not the tax. The real Article 96 tariff is eleven brackets of a fixed quota plus a marginal rate, and at a 15,000 MXN salary the approximation was out by a third.

No test derived from the model could find this, because the tests encode the approximation. What found it was a line in the model's own note admitting the real table "es más detallada". A note that concedes a simplification is a lead, not an excuse — it marks the exact place where the author already knew the model was incomplete.

Two: a missing rule rather than a wrong number

This was the most common non-drift defect, and it is the hardest to see, because everything present is correct. The error is an absence.

Saudi Arabia's end-of-service award computed the correct entitlement and ignored the Article 84 resignation reduction, which scales that entitlement to nothing, a third, two thirds, or the full amount depending on service length. The UAE model omitted the statutory cap of two years' wages. South Korea's vehicle tax never applied its 5%-a-year age discount, so an older car was billed at double.

Note what these have in common: in each case the page described the rule in prose. The author had read the statute, understood the provision, written it down for the reader, and not implemented it. Diffing the prose against the derivations finds this in minutes, and nothing else we tried found it at all.

Three: a brand-new line item

Japan's employee social-insurance rate moved from 14.66% to 14.69%. A 0.03-point change looks exactly like rounding noise, and would be easy to dismiss as such.

It was not a rate change. It was a new levy — the 子ども・子育て支援金 child-and-childcare support contribution of 0.23%, split evenly between employer and employee, which came into force in April 2026. A model refreshed by adjusting existing rates would never have found it, because the thing that changed was the list of things, not the value of any of them.

The general lesson: a refresh has to diff the derivations and the schema, not only the constants. Denmark and Slovakia both needed new fields in 2026 rather than new numbers.

What actually catches these

• Read the whole statute section, not just the rate you came for. Every missing-rule defect we found was adjacent to a figure that was already correct.
• Diff the explanatory prose against the implementation. When they disagree, the prose is usually right and the code is usually the one missing something.
• Treat any self-admitted simplification as an open defect with a location attached.
• Validate a conversion against a published worked example. Where an official source provided one, reproducing it to the cent is what separated a trustworthy rewrite from another guess.
• Refresh a rate set as a whole. A 2026 ceiling applied to a 2025 tariff is a model of no year that ever existed, and it will pass every test you have.

The honest framing of a verification badge

This is why our own calculators state their claim in the narrowest terms we can write, both on the page and inside the machine-readable model: the engine reproduces the model's stated formula through N published test cases, which asserts internal consistency and not that the formula matches current law. The cited sources, each carrying the date it was checked, are the evidence for the second claim, and they are a different kind of evidence.

A badge that blurred the two would be worth less than no badge, because it would be spending credibility it has not earned on a claim it cannot support.