Unit Tests Questioned

@thomasguestwordaligned.org@clinithinkwales

Agenda

setUp

questions

assertions

tearDown

What’s a Unit Test?

A unit test calls a function and checks the result.

PASS ☑ FAIL ☒

2017 tests passed in 225ms ☑

1/2017 tests failed in 228ms ☒

Why Most Unit Testing is Waste

Coplien

James O Coplien (2014?)

Reliability is not the only goal

whoami

assert self.job(time.then()) \
    == "@SNSystems #BRI"

Test Pyramid

Test Pyramid

whoami

assert self.job(time.now()) \
    == "@ClinithinkWales #BGN"

more unit tests =
more code =
more bugs!

Unit Tests slow you down

git commit -m "Implement feature"
git commit -m "Fix the unit tests"
git commit -m "Fix the unit tests (again!)"

Why write tests for code you know will change?

Test Ice Cream

Test Ice Cream

3rd Party Test Policies

SQLite

Boost

Python

Assert False

  • unit tests cannot touch the system (DB, files, sockets...)
  • unit test all possible inputs
  • unit test all code paths
  • more tests = better
  • more tests = more code = more bugs
  • delete any test which has passed for a year

Assert False

  • you can’t test legacy code
  • you can’t test code which uses singletons
  • you can’t test private methods
  • the type system tests your code
  • don’t write unit tests when prototyping
  • don’t write unit tests for code which will change

Assert False

  • you need the latest greatest unit test library
  • you need to write your own unit test library
  • you need IDE support
  • your harness should tolerate flaky tests

Assert True

  • unit tests run quickly
  • unit tests run automatically
  • unit test code is simple and flat
  • but not repetitive
  • more test less setup (please)

Assert Maybe

  • unit tests suit low level code
  • writing unit tests is boring
  • testing is creative
  • get another engineer to write the tests
  • system test failure ⇒ unit test missing
Unit tests which are brittle, large, slow, perpetually broken (and subsequently ignored), or flaky set bad examples which can get replicated through an entire test suite like a virus. Poorly-written tests can actually be worse than no tests at all, leaving the impression that testing is a waste of time.

Mike Bland, Goto Fail, Heartbleed, and Unit Testing Culture

Can you test quality into a product?

Unit tests are about design as well as correctness

What’s wrong with this test?

import random

def test_sort():
    xs = list(range(10))
    random.shuffle(xs)
    xs.sort()
    assert all(a < b
               for a, b in zip(xs, xs[1:]))

What’s wrong with this test?

def test_sort():
    ins_outs = (
        ([],              []),
        ([1],             [1]),
        ([1, 2, 3, 4, 5], [1, 2, 3, 4, 5]),
        ([5, 4, 3, 2, 1], [1, 2, 3, 4, 5]),
        ([4, 2, 1, 5, 3], [1, 2, 3, 4, 5]),
        ([6, 6, 6],       [6, 6, 6]))
    for i, o in ins_outs:
        i.sort()
        assert i == o

And the rest!

  • test stability
  • test with custom comparators
  • ... with comparators which raise exceptions
  • ... with comparators which mutate the input!

Conclusions

Professor Sir Tony Hoare is right ✔

James O Coplien isn’t ✘

Donald Knuth is being serious!

Thank you

@thomasguestwordaligned.org@clinithinkwales