Writing Programs That Write Tests: Better Testing With Scala

Adam Rosien @arosien
Inner Product LLC inner-product.com

17 Oct 2019

About you

  1. Introduction
  2. Types and Tests
  3. Properties, Part 1
  4. Properties, Part 2
  5. Generators
  6. Case Study: ToDo List
  7. Testing Stateful Systems
  8. Summary


Follow along

Install sbt: https://www.scala-sbt.org/release/docs/Setup.html

Clone repo: github.com/inner-product/essential-testing-code

$ git clone git@github.com:inner-product/essential-testing-code.git
$ cd essential-testing-code
$ sbt
sbt:essential-testing-code> testOnly AbsSpec
[info] ! Abs.non-negative: Falsified after 3 passed tests.
[info] > ARG_0: -2147483648
[info] Failed: Total 1, Failed 1, Errors 0, Passed 0

Types and Tests

Types and Tests

what do they do?

why are we here?


Bug-elimination Procedure

write code that doesn’t compile

change the code

when the code compiles, done!


Bug-elimination Procedure

make a failing test

change the code

when the test passes, done!

Program testing can be used to show the presence of bugs, but never to show their absence!

Edsger Dijkstra


Pros: correct for all inputs; compiler-checked

Cons: doesn’t handle all properties; the compiler can be annoying


Pros: can prove complex properties we cannot express in types

Cons: more effort, more boilerplate; test inputs are cherry-picked; can only prove the presence of bugs

A Problem with Example-Based Tests

extract the test inputs into a fixture

assert over all fixture inputs

what problems still exist with this style?

Property-Based Testing

example-generation: test fixture → generators

assertion specification: assertions → properties

Properties, Part 1

Integer Addition

x + y

what properties can you think of?

  • closure: for any two integers x and y, z = x + y is also an integer
  • identity: x + 0 = x = 0 + x
  • associativity: (x + y) + z = x + (y + z)
  • commutativity: x + y = y + x
  • invertibility: for every x there is an integer -x such that x + -x = 0

Int Addition

same properties?

  • closure: 👍
  • identity: 👍
  • associativity: 👍
  • commutativity: 👍
  • invertibility: ?

Assertions are boolean formulas over a concrete type:

Properties are universally quantified over a concrete type:

Laws are universally quantified over an abstract type:

Preconditions: additional restriction on the input

when the type doesn’t express this restriction itself

Properties: Exercises

  • List length (length and size in the standard library)
  • List append (++ in the standard library)
  • Sorting a list (sorted in the standard library)

patterns: closure, identity, associativity, commutativity, invertibility, …

Properties, Part 2

Properties: Asserting Invariants


Generators: Producing Values to Test

Generator Primitives

and more at https://www.scalacheck.org/files/scalacheck_2.11-1.14.0-api/index.html#org.scalacheck.Gen$

Generator combinators

(or pass generators as the values)

Expressing Preconditions

Generator Exercises

  • Generate a String of lowercase alphabetic characters
  • Generate a String of numeric characters
  • Generate integers between 100 and 200, inclusive
  • Generate strings that start with one of “a”, “b”, or “c”
  • Generate odd integers
  • Generate a list of between 3 and 5 positive integers

Building blocks available at https://www.scalacheck.org/files/scalacheck_2.11-1.14.0-api/index.html#org.scalacheck.Gen$ (yes, terrible URL)

More Complicated Situations

Create a Gen[User], where User is

Make justifiable decisions for the choice of generators for the fields.

Generator Challenges

Generators are a source of feedback!

too “wide”

  • these inputs can never happen
  • narrow the type via refinement: List => NonEmptyList, etc.

too “narrow”

  • doesn’t represent the variation in the actual domain
  • framework can’t generate enough samples to satisfy a precondition

Case Study: ToDo List

>>> POST /todos?value=get+milk&due=2018-05-13 HTTP/1.1

<<< HTTP/1.1 201 Created
<<< Location: /todos/68
>>> GET /todos/68

<<< HTTP/1.1 200 OK
<<< {
<<<   "value": "get milk"
<<<   "due": "2018-05-13"
<<< }

>>> DELETE /todos/68

<<< HTTP/1.1 204 No Content

POST   /todos?value=<value>&due=<due>
GET    /todos/<id>
DELETE /todos/<id>

What properties should hold?

Testing Stateful Systems

How do we know the counter is incremented correctly, no matter what commands are given to it?

Exercise: Think about what properties we want to prove, and what state we need to keep track of in the test in order to to make these assertions.

Writing a Stateful Test

For our counter, we need to:

  • Track the expected value of the counter in the test itself.
  • Generate, using Gen, commands that represent calls to the inc, dec, get, and reset methods.
  • For each command, we need to call the actual method on the counter, and update our state according to what command was run. For the command that invokes get, we need to assert if our computed value of the counter matches what the counter outputs.

See stateful/CounterProperties.scala

Writing a Stateful Test

  • Define the type of state we will track, no matter how the system is tested.
  • Generate commands for each operation of the system we want to test.
  • For each command, link it to the actual methods of the system, properly update our locally-tracked state, and make assertions using the computed and actual state.

Exercise: Testing an ATM

(See stateful/ATM.scala)

What state is hidden, that our test should track?

What pre- or post-conditions should we assert after running any of these commands?

Work with the stateful test template stateful/ATMProperties.scala, get it to compile by replacing all ??? implementations with real implementations.

Extra exercise: Add a deposit method and update the tests


Tests: examples checked via assertions

Tests prove the presence of bugs.

Properties can’t be proven, but they can be falsified.

Don’t write examples, write example-generating programs!

Thank you!

Adam Rosien @arosien

Inner Product LLC inner-product.com

Hire us to teach your team! ☝︎

References and Links: