Home / Articles / Calculations You Can Defend HES-01
Why Do Tests Approve Wrong Values?
An audit of this site's calculation engines found five calculation errors. Every one of them had a test, and every one passed. This article explains how a test can "approve" an error — and how to prevent it.
1. A confession
archi-civil.com's motto is "calculations you can defend". In October 2026 the site's calculation engines were audited systematically and five calculation errors were found. This article tells their story rather than hiding it, because each carries a general lesson about how correctness is not achieved in engineering software. All have been fixed, and all are now guarded by tests that catch them.
2. Five errors, five tests
| Error | Symptom | Why the tests did not catch it | Fixed |
|---|---|---|---|
| Sun azimuth | 180° at noon instead of 0° | The test expected 180° at noon | v565 |
| Kinetic facade | South facade "sun behind" at summer noon | The test represented the south facade with 180° | v566 |
| Truss member forces | Every force divided by member length | Tests checked only the reactions | v569 |
| Cross method | 15 kNm instead of 45 for two equal spans | The only correct case was the symmetric one that needs no distribution | v570 |
| Cross worked example | "M_B = −qL²/8 = −18" (qL²/8 = 54; correct is 36) | The validation table compared the correct result with a wrong expected value | v570 |
The striking point: none of the five came from untested code. Every one had a test, and every test passed.
3. The oracle problem
A test has two parts: the result the code produces, and knowledge of what the correct result is. In software engineering the second is called the "oracle". A test is only as good as its oracle. If the oracle is wrong, the test approves the wrong result — confidently, every time. Each of the five errors was a case where the oracle had been broken in some way.
4. Pattern 1: the expected value was copied from the code
In the sun azimuth test, the expected value at noon was written as 180°. Yet the page's own label said "south = 0°"; at noon the sun is due south and the correct value is 0°. Most likely the code was run when the test was written and its output was pasted in as the expected value. Such a test checks that the code has not changed, not that it is correct. The kinetic facade test followed the same pattern, representing the south facade with 180° to match the code's convention.
5. Pattern 2: the wrong thing was checked
The truss engine's tests checked the support reactions carefully, even their signs. But they never looked at the member forces — what the user actually sees. That is exactly where the error was: the stiffness matrix and the force recovery were written with different assumptions, so every member force came out divided by the member length. The reactions were correct because they were internally consistent.
6. Pattern 3: the easy case was checked
The Cross method engine gave the correct result for a symmetric beam fixed at both ends. In that case the middle joint is already balanced at the start, so the engine performs no distribution or carry-over at all. The error was exactly in that step: carry-over went to the neighbouring spans instead of the far end of the same member. Because only the easy case was checked, the error stayed invisible.
The example page was broken differently: its "analytical" solution was wrong both arithmetically (qL²/8 = 54, not 18) and mechanically (by symmetry the middle support does not rotate; the correct value is qL²/12 = 36). Its validation table compared the correct result with this wrong value and showed a 200–300% deviation.
7. What worked
- Independent reference solution: for the Cross method, a separate slope-deflection solver was placed inside the test; 50 random beams are compared by both methods. For the truss, a geometry-independent equilibrium check was added: ΣF = 0 at every joint.
- Mutation testing: after each fix the old error was deliberately reintroduced to confirm the new tests catch it. A test that does not catch the error is not a test.
- Physical facts: facts independent of the code that everyone knows — "on a summer noon the south facade gets sun and the north facade does not" — are the most robust oracles.
The next article covers the second lesson of the same audit: the difference between reading a calculation and verifying it.
References
- archi-civil.com release notes (README), v565–v572: symptom, cause, fix and added tests for each error.
- Barr, E. T., Harman, M., McMinn, P., Shahbaz, M. and Yoo, S. (2015). The Oracle Problem in Software Testing: A Survey. IEEE Transactions on Software Engineering, 41(5), 507–525.
- Jia, Y. and Harman, M. (2011). An Analysis and Survey of the Development of Mutation Testing. IEEE Transactions on Software Engineering, 37(5), 649–678.
The numbers in this article were produced by the site's engines; code values are given with their clause numbers.