Follow along

Install sbt: https://www.scala-sbt.org/release/docs/Setup.html

Clone repo: github.com/inner-product/essential-testing-code

$ git clone git@github.com:inner-product/essential-testing-code.git
$ cd essential-testing-code
$ sbt
sbt:essential-testing-code> testOnly AbsSpec
[info] ! Abs.non-negative: Falsified after 3 passed tests.
[info] > ARG_0: -2147483648
[info] Failed: Total 1, Failed 1, Errors 0, Passed 0
...

Types and Tests

what do they do?

Bug-elimination Procedure

write code that doesn’t compile

Bug-elimination Procedure

make a failing test

change the code

when the test passes, done!

Types

Pros: correct for all inputs; compiler-checked

Cons: doesn’t handle all properties; the compiler can be annoying

A Problem with Example-Based Tests

class MyTest extends SomeTestThing {
  val systemUnderTest = ???

  test("my code") {
    assert(
      systemUnderTest.doSomething(
        12, "yes", true, true, false) == null)
  }
}

trait TestFixture {
  val inputs = List(
    DoSomethingInput(12, "yes", true, true, false),
    ...
  )

  case class DoSomethingInput(i: Int, s: String, b1: Boolean, b2: Boolean, b3: Boolean)
}

extract the test inputs into a fixture

class MyTest extends SomeTestThing with TestFixture {
  val systemUnderTest = ???

  test("my code") {
    for {
      input <- inputs
      (i, s, b1, b2, b3) = input
    } assert(systemUnderTest.doSomething(
        i, s, b1, b2, b3) == null)
  }
}

assert over all fixture inputs

Property-Based Testing

example-generation: test fixture → generators

Integer Addition

x + y

what properties can you think of?

Int Addition

same properties?

Assertions are boolean formulas over a concrete type:

val x = 12

x == 12

val alwaysTwelve =
  forAll { (x: Int) => x == 12 }

Laws are universally quantified over an abstract type:

def commutativity[A: Arbitrary](f: (A, A) => A) =
  forAll { (x: A, y: A) =>
    f(x, y) == f(y, x)
  }

Properties: Exercises

• List length (length and size in the standard library)
• List append (++ in the standard library)
• Sorting a list (sorted in the standard library)

patterns: closure, identity, associativity, commutativity, invertibility, …

Properties: Asserting Invariants

import org.scalacheck.Prop.forAll

val invertibility =
  forAll { (x: Int) =>
    x + -x == 0
  }

Generators: Producing Values to Test

sealed abstract class Gen[+T] {
  def sample: Option[T]
}

Generator Primitives

import org.scalacheck.Gen

Gen.alphaStr
Gen.numStr
Gen.posNum[Int]
Gen.const(1)
Gen.choose(0, 9)
Gen.oneOf('a', 'b', 'c')

and more at https://www.scalacheck.org/files/scalacheck_2.11-1.14.0-api/index.html#org.scalacheck.Gen$

Generator combinators

Gen.posNum[Int].map(x => x * 2)

Gen.choose(0,2).flatMap { 
  case 0 => Gen.alphaStr
  case 1 => Gen.numStr
  case _ => Gen.const("Hi!")
}

Expressing Preconditions

Gen.posNum[Int].map(x => x * 2)

Gen.posNum[Int].suchThat(_ % 2 == 0)

Gen.posNum[Int].filter(_ % 2 == 0)

Generator Exercises

• Generate a String of lowercase alphabetic characters
• Generate a String of numeric characters
• Generate integers between 100 and 200, inclusive
• Generate strings that start with one of “a”, “b”, or “c”
• Generate odd integers
• Generate a list of between 3 and 5 positive integers

Building blocks available at https://www.scalacheck.org/files/scalacheck_2.11-1.14.0-api/index.html#org.scalacheck.Gen$ (yes, terrible URL)

More Complicated Situations

Create a Gen[User], where User is

case class User(name: String, age: Int, email: String)

Make justifiable decisions for the choice of generators for the fields.

Generator Challenges

Generators are a source of feedback!

>>> POST /todos?value=get+milk&due=2018-05-13 HTTP/1.1

<<< HTTP/1.1 201 Created
<<< Location: /todos/68

>>> GET /todos/68

<<< HTTP/1.1 200 OK
<<<
<<< {
<<<   "value": "get milk"
<<<   "due": "2018-05-13"
<<< }

>>> DELETE /todos/68

<<< HTTP/1.1 204 No Content

POST   /todos?value=<value>&due=<due>
GET    /todos/<id>
DELETE /todos/<id>

What properties should hold?

val c = new Counter
// c: Counter = repl.Session$App3$Counter@2b42ed66
c.inc
c.inc
c.get
// res23: Int = 2

How do we know the counter is incremented correctly, no matter what commands are given to it?

Exercise: Think about what properties we want to prove, and what state we need to keep track of in the test in order to to make these assertions.

Writing a Stateful Test

For our counter, we need to:

• Track the expected value of the counter in the test itself.
• Generate, using Gen, commands that represent calls to the inc, dec, get, and reset methods.
• For each command, we need to call the actual method on the counter, and update our state according to what command was run. For the command that invokes get, we need to assert if our computed value of the counter matches what the counter outputs.

See stateful/CounterProperties.scala

Writing a Stateful Test

• Define the type of state we will track, no matter how the system is tested.
• Generate commands for each operation of the system we want to test.
• For each command, link it to the actual methods of the system, properly update our locally-tracked state, and make assertions using the computed and actual state.

Exercise: Testing an ATM

trait ATM {
  def authorize(user: User, pass: Password): Boolean

  def balance(user: User): Either[ATM.Error, Amount]

  def withdraw(user: User, amount: Double): Either[ATM.Error, Amount]
}

(See stateful/ATM.scala)

What state is hidden, that our test should track?

What pre- or post-conditions should we assert after running any of these commands?

Work with the stateful test template stateful/ATMProperties.scala, get it to compile by replacing all ??? implementations with real implementations.

Extra exercise: Add a deposit method and update the tests

Tests: examples checked via assertions

Thank you!

Adam Rosien @arosien

Inner Product LLC inner-product.com

Hire us to teach your team! ☝︎

References and Links: