cint

cint university / 201 / ced, the cint editor

201-05: tests and goldens

This is lesson 201-05 of CU-201.2, checked against cint 0.2.0.

You will write tests beside the code they check and save a golden of a run you have checked. Then you will see which mistakes each one catches.

the program

This is the bill split from 201-04, with its two calculations as functions and three tests at the end of the file:

// Split a bill in cents among people, with an 18 percent tip.
// The cents that do not divide evenly go to the first people, one each.

I64 tip(I64 bill) {
    return muldiv(I64, bill, 18, 100, half_even);
}

// What person p (counting from 1) pays when total is split among people.
I64 share(I64 total, I64 people, I64 p) {
    I64 pays = total / people;
    if (p <= total % people) {
        pays += 1;
    }
    return pays;
}

void main() {
    I64 bill = 10_000;
    I64 total = bill + tip(bill);
    I64 people = 3;
    "bill {bill:/100}, tip {tip(bill):/100}, total {total:/100}\n";
    for p in 1..=people {
        "person {p} pays {share(total, people, p):/100}\n";
    }
}

test "the shares add up to the total" {
    for people in 1..=12 {
        I64 paid = 0;
        for p in 1..=people {
            paid += share(11_800, people, p);
        }
        assert(paid == 11_800);
    }
}

test "a tip of half a cent rounds to even" {
    assert(tip(25) == 4);       // 4.5 cents rounds to 4
    assert(tip(75) == 14);      // 13.5 cents rounds to 14
}

test "nobody to split among is a fault" expect_fault E_DIV_ZERO {
    _ = share(100, 0, 1);
}

Run this program in ced

run it, then test it

$ cint run split.ci
bill 100.00, tip 18.00, total 118.00
person 1 pays 39.34
person 2 pays 39.33
person 3 pays 39.33
$ cint test split.ci
test split.ci:27 "the shares add up to the total" ... ok
test split.ci:37 "a tip of half a cent rounds to even" ... ok
test split.ci:42 "nobody to split among is a fault" ... ok
3 of 3 tests passed

cint run runs main and ignores the tests. cint test runs each test and ignores main.

tests

A test block is a function with a name in quotes and no parameters, written in the same file as the code it checks. Each test starts from the program's initial state, so one test cannot change what another sees.

  • assert(condition) stops the test with a fault when the condition is false.
  • expect_fault CODE after the name turns the test around: it passes only when its body faults with that code. The third test passes because share(100, 0, 1) divides by zero.

Change tip to the plain division return bill * 18 / 100; and test again:

$ cint test split.ci
test split.ci:27 "the shares add up to the total" ... ok
test split.ci:37 "a tip of half a cent rounds to even" ... FAILED
fault[E_ASSERT]: assertion failed
  --> split.ci:39:12
   |
39 |     assert(tip(75) == 14);      // 13.5 cents rounds to 14
   |            ^ assert.checked.bool
   |
   = operation: assert.checked.bool
   = operand-count: 2
   = operand: I64 13
   = operand: I64 14
   ...
test split.ci:42 "nobody to split among is a fault" ... ok
2 of 3 tests passed

A failed assertion is a fault, with a fault report like any other. For a comparison, its two operands are the values of the two sides: tip(75) returned 13 where the test wanted 14. The first assertion in the same test still passed, because 4.5 rounds down to 4 under both rules. cint test exits with status 3 when a test fails, so a script or a build can stop on it.

goldens

A test checks a case you chose. A golden checks a whole run: it is the saved output of a run that you checked by hand, and every run after it must match it byte for byte. Three rules govern a golden:

  1. Check the output by hand before you save it.
  2. Never edit a golden by hand.
  3. When a change is meant to alter the output, check the new output by hand. Save it as the new golden in a separate step, so the change and the new golden stay apart.

cint supports two kinds of golden.

the output of a run

The output of cint run, saved to a file, is the simplest golden:

$ cint run split.ci > split.golden

After a change, run again into another file and compare the two. A difference shows exactly which lines changed; 401-01 works through an example.

In Windows PowerShell, > saves text in a different encoding from the one cint writes. Both files are saved the same way, so VS Code's Select for Compare still shows the lines that differ, and fc.exe /b split.golden after.txt still says whether the files match. Type fc.exe in full, since fc alone is the PowerShell alias for Format-Custom.

the reference's record

The reference, the Python program that defines what every cint program means, writes a record of a run. It is in the ref folder of the cint source. Point Python at that folder once per terminal, replacing SOURCE with the folder you unpacked cint into:

On macOS and Linux, type this:

$ export PYTHONPATH="SOURCE/ref"

In Windows PowerShell, type this:

PS> $env:PYTHONPATH = "SOURCE\ref"

Then save the record:

$ python3 -m cint_ref run split.ci > split.expect
$ cat split.expect
case split
source reference
outcome value
stdout-bytes 97
stdout-sha256 266f03fdb752d2d04bc11f2ee54c21d60e281857c78c96ccaa0fadd181a1b439
fuel-consumed 9

In Windows PowerShell, write the first command of that block as cmd /c "python -m cint_ref run split.ci > split.expect", so the record is saved byte for byte.

The record holds how the run ended (outcome value: it finished) and the size and SHA-256 digest of everything it printed. It also holds fuel-consumed, a count charged at each call and at each pass through a loop, which comes out the same on the reference and on the compilers. For a run that faults, the record also holds every field of the fault report. check runs the program again and compares:

$ python3 -m cint_ref check split.ci split.expect
$ echo $?
0

Silence and status 0 mean the run matches the record. In PowerShell, $LASTEXITCODE shows the status.

a change only the record notices

Now compute the tip once and keep it in a variable. This change keeps the output the same but removes a call.

    I64 t = tip(bill);
    I64 total = bill + t;
    I64 people = 3;
    "bill {bill:/100}, tip {t:/100}, total {total:/100}\n";

Run this program in ced

Run it and compare with split.golden: the files match. The record disagrees:

$ python3 -m cint_ref check split.ci split.expect
split.expect: line 6 differs
  expected: fuel-consumed 9
  actual:   fuel-consumed 8

The output is the same, byte for byte, but the fuel is 1 lower, because tip is called once instead of twice. A change in fuel means the program took a different path. Here a new path was the point of the change, so the record is out of date: check the output, then save a new record in a step of its own. When you did not mean to change the path, the same line warns that the program now takes a different path.

which to use

Catches Shows
A test a wrong answer for a case you chose: an edge, a tie, a fault which assertion failed, with both sides
The output of a run any change in what the program prints which lines changed
The reference's record any change in the output, in how the run ended, in a field of a fault or in the path it took the first line of the record that differs

A program you intend to keep has tests for its hard cases and a golden of its main run, and both live beside it in version control.

elsewhere

Saved-output tests exist in most languages, as golden files, snapshot tests or approval tests. They are harder to keep where output is not the same from run to run. On 2026-10-10, running python3 -c "print({'apple', 'banana', 'cherry'})" five times with Python 3.13 on Ubuntu 24.04 printed the set in four different orders, because Python changes how it hashes strings each time it starts. A cint program has no such source of variation. One program, built by one release, prints the same bytes on every run and with each of the four C compilers used to check these lessons.

walk-through

for people in 1..=12 inside a test checks twelve cases in one test. A test is ordinary code, so loops and helper functions work in it.

paid, not sum. sum is a built-in function, and a local variable cannot take its name; the compiler says error[C3001]: duplicate name. count, min and max are taken the same way.

_ = share(100, 0, 1); calls the function and discards its result. The test exists only for the fault.

try this

  1. Add a test that splitting 1 cent among 3 people gives 1, 0 and 0.
  2. Change main to split among 4 people. Which goldens fail, and what does each one show?
  3. Save a record of a run that faults: set bill to I64.max in main and run the reference. Which fields does the record add, and which line of the program does it name?

reference

tests, assert, expect_fault, fault record, golden outputs, golden

Next: 201-06, the workbench.