setUp

What’s a Unit Test?

A unit test calls a function and checks the result.

PASS ☑ FAIL ☒

2017 tests passed in 225ms ☑

1/2017 tests failed in 228ms ☒

Why Most Unit Testing is Waste

James O Coplien (2014?)

Reliability is not the only measure

Questions

whoami

t = time.then()
assert self.job(t) == "@SNSystems #BRI"

Test Pyramid

whoami

t = time.now()
assert self.job(t) == "@ClinithinkWales #BGN"

more unit tests =
more code =
more bugs!

Unit Tests slow you down

git commit -m "Implement feature"

git commit -m "Fix the unit tests"

git commit -m "Fix the unit tests (again!)"

Why write tests for code you know will change?

Test Ice Cream

3rd Party Test Policies

SQLite

Boost

Python

Assertions

Assert False

• unit tests cannot touch the system (DB, files, sockets...)
• unit test all possible inputs
• unit test all code paths
• more tests = better
• more tests = more code = more bugs
• delete any test which has passed for a year

Assert False

• you can’t test legacy code
• you can’t test code which uses singletons
• you can’t test private methods
• the type system tests your code
• don’t write unit tests when prototyping
• don’t write unit tests for code which will change

Assert False

• you need the latest greatest unit test library
• you need to write your own unit test library
• you need IDE support
• your harness should tolerate flaky tests

Assert True

• unit tests run quickly
• unit tests run automatically
• unit test code is simple and flat
• but not repetitive
• more test less setup (please)

Assert Maybe

• unit tests suit low level code
• writing unit tests is boring
• testing is creative
• get another engineer to write the tests
• system test failure ⇒ unit test missing
• your harness should tolerate flaky tests

Unit tests which are brittle, large, slow, perpetually broken (and subsequently ignored), or flaky set bad examples which can get replicated through an entire test suite like a virus. Poorly-written tests can actually be worse than no tests at all, leaving the impression that testing is a waste of time.

Mike Bland, Goto Fail, Heartbleed, and Unit Testing Culture

Can you test quality into a product?

Unit tests are about design as well as correctness

What’s wrong with this test?

import random

def test_sort():
    xs = list(range(10))
    random.shuffle(xs)
    xs.sort()
    assert all(a < b
               for a, b in zip(xs, xs[1:]))

What’s wrong with this test?

def test_sort():
    ins_outs = (
        ([],              []),
        ([1],             [1]),
        ([1, 2, 3, 4, 5], [1, 2, 3, 4, 5]),
        ([5, 4, 3, 2, 1], [1, 2, 3, 4, 5]),
        ([4, 2, 1, 5, 3], [1, 2, 3, 4, 5]),
        ([6, 6, 6],       [6, 6, 6]))
    for i, o in ins_outs:
        i.sort()
        assert i == o

And the rest!

• test stability
• test with custom comparators
• ... with comparators which raise exceptions
• ... with comparators which mutate the input!

tearDown

Conclusions

Professor Sir Tony Hoare is right ✔

James O Coplien isn’t ✘

Donald Knuth is being serious!

Thank you

@thomasguest • wordaligned.org • @clinithinkwales