Taking your unit tests to the next level.

I have been using ScalaCheck testing library for at least 2 years now. It allows you to take your unit tests to the next level.


  • You can do Property-based testing by generating a lot of tests with random data and asserting properties on your functions. A simple code example is described below.
  • You can do Law testing that is even more powerful and allows you to check mathematical properties on your types.

Property-based testing

Here is our beloved User data type:

case class User(name: String, age: Int)

And a random User generator:

import org.scalacheck.{ Gen, Arbitrary }
import Arbitrary.arbitrary
implicit val randomUser: Arbitrary[User] = Arbitrary(for {
  randomName <- Gen.alphaStr
  randomAge  <- Gen.choose(0,80)
  } yield User(randomName, randomAge))

We can now generate a User like this:

scala> randomUser.arbitrary.sample
res0: Option[User] = Some(User(OtwlaaxGbmdhuorlmgvXitbmGfbgetm,22))

Let’s define some functions on the User:

def isAdult: User => Boolean = _.age >= 18
def isAllowedToDrink : User => Boolean = _.age >= 21

Let’s claim that:

All adults are allowed to drink.

Can we somehow prove this? Is this correct for all users?

This is where property testing comes to the rescue. It allows us not to write specific unit-tests. Here they would be:

  • 18-year-olds are not allowed to drink
  • 19-year-olds are not allowed to drink
  • 20-year-olds are not allowed to drink

All of these statements can be replaced by a single property check:

import org.scalacheck.Prop.forAll
val allAdultsCanDrink = forAll { u: User =>
    if(isAdult(u)) isAllowedToDrink(u) else true }

Let’s run it:

scala> allAdultsCanDrink.check()
! Falsified after 0 passed tests.
> ARG_0: User(,19)

It fails as expected for a 19-year-old.

Property testing is awesome for a few reasons:

  • Saves time by writing less specific tests
  • Finds new use cases generated by Scala check that you forgot to handle
  • Forces you think in a more general way
  • Gives you more confidence for refactoring than conventional unit tests

Law testing

It gets better: let’s take it to the next level and define an Ordering between Users:

import scala.math.Orderingimplicit val userOrdering: Ordering[User] = Ordering.by(_.age)

We want to make sure that we didn't forget any edge cases and that we defined our order properly. This property has a name, and it’s called a total order. It needs to holds for the following properties:

  • Totality
  • Antisymmetry
  • Transitivity

Can we somehow prove this? Is this correct for all users?

This is possible without writing a single test!

We use cats-laws library to define the laws we want to test on the ordering we defined:

import cats.kernel.laws.discipline.OrderTests
import cats._
import org.scalatest.FunSuite
import org.typelevel.discipline.scalatest.Discipline
import org.scalacheck.ScalacheckShapeless._
class UserOrderSpec extends FunSuite with Discipline  {
//needed boilerplate to satisfy the dependencies of the framework
  implicit def eqUser[A: Eq]: Eq[Option[User]] = Eq.fromUniversalEquals
//convert our standard ordering to a `cats` order 
  implicit val catsUserOrder: Order[User] = Order.fromOrdering(userOrdering)
  
//check all mathematical properties on our ordering
  checkAll("User", OrderTests[User].order)
}

Let’s run it:

scala> new UserOrderSpec().execute()
UserOrderSpec:
- User.order.antisymmetry *** FAILED ***
  GeneratorDrivenPropertyCheckFailedException was thrown during property evaluation.
   (Discipline.scala:14)
    Falsified after 1 successful property evaluations.
    Location: (Discipline.scala:14)
    Occurred when passed generated values (
      arg0 = User(h,17),
      arg1 = User(edsb,17),
      arg2 = org.scalacheck.GenArities$$Lambda$2739/1277317528@41d7b4cf
    )
    Label of failing property:
      Expected: true
  Received: false
- User.order.compare
- User.order.gt
- User.order.gteqv
- User.order.lt
- User.order.max
- User.order.min
- User.order.partialCompare
- User.order.pmax
- User.order.pmin
- User.order.reflexitivity
- User.order.reflexitivity gt
- User.order.reflexitivity lt
- User.order.symmetry
- User.order.totality
- User.order.transitivity

Sure enough, it fails on the antisymmetry law! Same age and different names are not supposed to be equals. We forgot to use the name in our original Ordering, so let's fix it and rerun the laws:

implicit val userOrdering: Ordering[User] = Ordering.by( u => (u.age, u.name))
scala> new UserOrderSpec().execute()
UserOrderSpec:
- User.order.antisymmetry
- User.order.compare
- User.order.gt
- User.order.gteqv
- User.order.lt
- User.order.max
- User.order.min
- User.order.partialCompare
- User.order.pmax
- User.order.pmin
- User.order.reflexitivity
- User.order.reflexitivity gt
- User.order.reflexitivity lt
- User.order.symmetry
- User.order.totality
- User.order.transitivity

And now it passes :)

If you are wondering what can you test besides Orders, go check out the docs here: https://typelevel.org/cats/typeclasses/lawtesting.html

Summary

  • Property tests are more powerful than unit tests. They allow us to define properties on functions and generate a large number of tests using random data generators.
  • Law testing takes it to the next level and uses the mathematical properties of structures like Order to generate the properties and the tests.
  • Next time you define an ordering and wonder if it’s well-defined, go ahead and run the laws on it!