Abdu's Garden

Learning Rust - Ch. 11 - Automated Tests

Properties1
tagsrust/tests

13 min read

Writing Tests

They’re functions that verify that other non-test code functions exactly as expected.
Most tests follow a similar routine:

  • Set up.
  • Run code to test, usually specific functions.
  • Assert results are as expected.

Test Functions

Tests are functions that are annotated with the test attribute.

Attributes

These are metadata annotations (labels) that the code.

We annotate by adding #[test] before the function header (one line before) and then when everything’s set we use cargo test to run all our tests.
Cargo creates a test module with the general template whenever we make a new library project, found at src/lib.rs file.

For example:

$ cargo new adder --lib
     Created library `adder` project
$ cd adder

Then navigating to src/lib.rs:

pub fn add(left: u64, right: u64) -> u64 {
    left + right
}
 
#[cfg(test)]
mod tests {
    use super::*;
 
    #[test]
    fn it_works() {
        let result = add(2, 2);
        assert_eq!(result, 4);
    }
}

The assert_eq! macros asserts that the value received by the test is equal to the expected value and its result determines if the test passes or fails. It does this by evaluating an expression that results in a Boolean.
If that bool is False, then the assert_eq! macro calls the panic! macro and the test fails, otherwise it passes just fine.

Tests & Threads

Each test runs on a separate thread, and when one test fails, it panics on that thread using the panic! macro.
The main thread listens and when it finds a test thread’s died, it marks that test as failed.

In the test results, there are two sections that inform us of failing tests and some useful information about them

  • A section displays the failed test and details on why it failed, such as fail location and message.
  • Another section that lists the names of the failing tests, useful when testing extensively.

For example:

$ cargo test
   Compiling adder v0.1.0 (file:///projects/adder)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.72s
     Running unittests src/lib.rs (target/debug/deps/adder-92948b65e88960b4)
 
running 2 tests
test tests::another ... FAILED
test tests::exploration ... ok
 
failures:
 
---- tests::another stdout ----
 
thread 'tests::another' panicked at src/lib.rs:17:9:
Make this test fail
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
 
 
failures:
    tests::another
 
test result: FAILED. 1 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
 
error: test failed, to rerun pass `--lib`

Module Visibility in Tests

The default test module is an inner-module and follows the same visibility rules other modules do.
So when testing, we need to use the members of the other modules we intent to test.

To save the hassle, we can use use super::*; to import everything from the outer scope into the test module’s scope.
For example, a test that checks if one rectangle can fit another:

#[cfg(test)]
mod tests {
    use super::*;
 
    #[test]
    fn larger_can_hold_smaller() {
        let larger = Rectangle {
            width: 8,
            height: 7,
        };
        let smaller = Rectangle {
            width: 5,
            height: 1,
        };
        
        // Rectangle::can_hold checks if the dimensions of self are larger than the other 
        assert!(larger.can_hold(&smaller));
    }
    
    #[test]
    fn smaller_cant_hold_larger(){
	    let larger = Rectangle {
            width: 8,
            height: 7,
        };
        let smaller = Rectangle {
            width: 5,
            height: 1,
        };
        assert!(!smaller.can_hold(larger)); // negated because we want to assert we get False
    }
}

This example uses the assert! macro instead because it already expects boolean values.

Testing Equality, or Not

There are two macros in question, one you’ve seen already.

  • assert_eq!: asserts two values are equal.
  • assert_ne!: asserts two values are unequal.
    Both macros print the results if they fail.

The failure message printed will also convey the values received by these macros, labelled left and right for your convenience. Unlike other languages and frameworks, there’s not actual and expected, instead it’s positional to how you passed the arguments to the macro, and therefore the order doesn’t matter…

The assert_ne! is rather useful when you don’t know what the resulting value from a piece of code will be but you certainly know what it shouldn’t be.

Underhood Implementation

Both of these macros employ the == and != operators underneath.
The printed results use debug formatting so must implement PartialEq and Debug traits, so they can be checked and printed.

Custom Failure Messages

We can add custom messages to all 3 of the assert macros, by simply passing optional arguments after the required ones.
Any arguments specified after the required arguments are passed along to the format! macro, you can pass a format string that contains {} placeholders and values to go in those placeholders.

For example:

#[test]
fn greeting_contains_name() {
	let result = greeting("Carol");
	assert!(
		result.contains("Carol"),
		"Greeting did not contain name, value was `{result}`"
	);
}

This failure message is more useful than simply stating where the that the test failed and what line the assertion is on.

And this is how it’ll display when it does fail:

$ cargo test
   Compiling greeter v0.1.0 (file:///projects/greeter)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.93s
     Running unittests src/lib.rs (target/debug/deps/greeter-170b942eb5bf5e3a)
 
running 1 test
test tests::greeting_contains_name ... FAILED
 
failures:
 
---- tests::greeting_contains_name stdout ----
 
thread 'tests::greeting_contains_name' panicked at src/lib.rs:12:9:
Greeting did not contain name, value was `Hello!`
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
 
 
failures:
    tests::greeting_contains_name
 
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
 
error: test failed, to rerun pass `--lib`
  • The name of the failing test
  • The line where the assertion fails
  • Custom fail message

Checking If Tests Panic

Sometimes we want our code to panic, especially when things go wrong and are unrecoverable. See Learning Rust - Ch. 9 - Error Handling & Panicking.

When this happens or is expected to happen, we need to be able to test this, that’s where the should_panic attribute comes in.
This attribute annotates the test function and allows the test to pass if the code inside panics.

Example:

 
#[cfg(test)]
mod tests {
    use super::*;
 
    #[test]
    #[should_panic]
    fn greater_than_100() {
        Guess::new(200);
    }
}

The #[should_panic] attribute is placed immediately after the #[test] attribute, on its own line.
Now when the test greater_than_100 runs and panics, the test actually passes.

Making Precise Panic Tests

Tests using the #[should_panic] attribute are imprecise because sometimes code panics for different reasons.
In order to make these tests more precise, we need to introduce an optional expected parameter.
By adding this parameter, the test harness will make sure that the failure message contains the provided text.

 
impl Guess {
    pub fn new(value: i32) -> Guess {
        if value < 1 {
            panic!(
                "Guess value must be greater than or equal to 1, got {value}."
            );
        } else if value > 100 {
            panic!(
                "Guess value must be less than or equal to 100, got {value}."
            );
        }
 
        Guess { value }
    }
}
 
#[cfg(test)]
mod tests {
    use super::*;
 
    #[test]
    #[should_panic(expected = "less than or equal to 100")]
    fn greater_than_100() {
        Guess::new(200);
    }
}

Using Result Type In Tests

Instead of just panicking, we can actually use Result<T, E> in our tests.

#[cfg(test)]
mod tests{
	use super::*;
	
	#[test]
	fn it_works() -> Result<(), String> {
		let result = add(2,3);
		if result == 4{
			Ok();
		} else{
			Err(String::from("Two plus two is four minus one dats three quick maffs!"));
		}
	}
}

By returning a Result type we don’t need to assert to determine if the test passes or not!

Take Note!

  • You can’t use the #[should_panic] annotation on tests that use Result<T, E>.
  • To assert that an operation returns an Err variant, don’t use the question mark operator on the Result<T, E> value. Instead, use assert!(value.is_err()).

Controlling How Tests Run with Cargo

Similar to how how cargo run will compile and run our project, using cargo test will compile the code in test mode and run the tests in parallel (multi-threaded) and captures the output of the tests.

However, we can pass arguments to the command that control how the test are ran and even control how many lines of output we want viewed.
Arguments are listed using cargo test followed by the separator -- and then the ones that go to the test binary. Running cargo test --help displays the options.

Parallel or Consecutive

Default is for tests to run in parallel so that they don’t block each other, and feedback’s returned sooner.
This mode’s suitable if your tests aren’t dependent on one another or have shared state or environment.

To specify the number of threads used, you can send the --test-threads=n argument to cargo test specifying that the tests run on.

$ cargo test -- --test-threads=3

if set to 1 then you’re basically telling it to not use any parallelism, runs on a single thread consecutively, but the tests won’t interfere with each other if they share state.

Showing Outputs

Rust’s test library captures all output from tests, that’s why even a println! in a test’s body won’t print anything.
If a test fails, then we’ll see the test’s output and the failure cause like we’ve been seeing.

To see printed values in tests that pass, we use the --show-output argument.

Subsets By Name

If you have extensive test suites, it may become tedious to run all of them when you’re only concerned with a subset. To run only the subset, we can call them by name.

Take the following example:

pub fn add_two(a: u64) -> u64 {
    a + 2
}
 
#[cfg(test)]
mod tests {
    use super::*;
 
    #[test]
    fn add_two_and_two() {
        let result = add_two(2);
        assert_eq!(result, 4);
    }
 
    #[test]
    fn add_three_and_two() {
        let result = add_two(3);
        assert_eq!(result, 5);
    }
 
    #[test]
    fn one_hundred() {
        let result = add_two(100);
        assert_eq!(result, 102);
    }
}

Running a single test

$ cargo test one_hundred
   Compiling adder v0.1.0 (file:///projects/adder)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.69s
     Running unittests src/lib.rs (target/debug/deps/adder-92948b65e88960b4)
 
running 1 test
test tests::one_hundred ... ok
 
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 2 filtered out; finished in 0.00s

Filtering To Run Multiple

This is done by specifying parts of a test name of the tests to rub.

$ cargo test add
   Compiling adder v0.1.0 (file:///projects/adder)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.61s
     Running unittests src/lib.rs (target/debug/deps/adder-92948b65e88960b4)
 
running 2 tests
test tests::add_three_and_two ... ok
test tests::add_two_and_two ... ok
 
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 1 filtered out; finished in

Ignoring Tests

To ignore tests unless specified we add the ignore attribute to them.
Ex:

    #[test]
    #[ignore]
    fn expensive_test() {
        // code that takes an hour to run
    }

When running cargo test the test will be ignored unless its name is specified like so:
cargo test -- --ignored

To run ALL tests regardless of ignored or not

Run cargo test -- --ignored:

Organising Tests

Rust is concerned with two types of tests, unit and integration.

  • Unit Tests: small and focused on testing one module in isolation, and can test private interfaces.
  • Integration Tests: external to your library and use your code similar to how external code would, and only uses public interfaces and can exercise multiple modules per test.

Unit Tests

These tests live in the src directory, in the same file of the code they’re testing. // just like in prev examples
By convention, we create a module named tests in each file to contain the test functions and to annotate the module with cfg(test).

Annotation

The cfg stands for configureation. This tells Rust to compile and run the code only when cargo test is run, and not when cargo build to save time and space since they won’t be included in the compiled artifact.

Since Integration tests go in a different directory, they don’t need the #[cfg(test)] annotation.

Private Function Tests

Unlike some languages, Rust permits testing private functions.
Consider this example:

pub fn add_two(a: u64) -> u64 {
    internal_adder(a, 2)
}
 
fn internal_adder(left: u64, right: u64) -> u64 {
    left + right
}
 
#[cfg(test)]
mod tests {
    use super::*;
 
    #[test]
    fn internal() {
        let result = internal_adder(2, 2);
        assert_eq!(result, 4);
    }
}

Despite internal_adder being private, the test module imports it from its parent regardless.
Because tests are just Rust code in regular modules, they’re capable of bringing the members of their parent module into scope.

Integration Tests

These tests are external to your library, they use it like others’ code would.
All they can do is call functions from your library’s public interface (API).

“Units of code that work correctly on their own could have problems when integrated, so test coverage of the integrated code is important as well.”

Start by creating a tests directory.

Tests Directory

This should be a top-level directory in your project, right next to src.
Inside we make as many files as needed, Cargo is aware of its existence and will compile all of them as individual crates.

Here’s an example structure:

adder
├── Cargo.lock
├── Cargo.toml
├── src
│   └── lib.rs
└── tests
    └── integration_test.rs

Notice lib.rs…

Then enter Filename: tests/integration_test.rs

use adder::add_two;
 
#[test]
fn it_adds_two() {
    let result = add_two(2);
    assert_eq!(result, 4);
}

Since each file’s compiled into a separate crate, the library must be brought into scope for each one.
Also, no need to annotate the tests with cfg(test).

Running cargo test will show the output for all three test types, unit, integration, and doc.
However, note that

“…if any test in a section fails, the following sections will not be run. For example, if a unit test fails, there won’t be any output for integration and doc tests, because those tests will only be run if all unit tests are passing.”

To run all the tests in a particular integration test file, pass --test argument followed by the name of the file to:

$ cargo test --test integration_test

Submodules

There are cases where you would likely need to make some common functionality for all your integration tests to use. Gathering all this logic into a single file /tests/common.rs will end up in that module being treated like a test crate and then gets 0 passed. Not ideal…

Instead, create tests/common/mod.rs

├── Cargo.lock
├── Cargo.toml
├── src
│   └── lib.rs
└── tests
    ├── common
    │   └── mod.rs
    └── integration_test.rs

You likely recall this naming convention from Learning Rust - Ch.7 - Packages, Crates, and Modules, it’s the older naming style. This style tells Rust not to create a common integration test crate.
Actually, all files in subdirectories of the tests directory don’t get compiled as separate crates or have sections in the test output.

To use the common logic simply bring it into scope use common;.

Int tests and Binary crates

If your project’s a binary crate with only scr/main.rs and no src/lib.rs then we cannot create integreations test for it.
Because only library crates expose functions that other crates can use; binary crates are meant to be run on their own.

This is why it’s common for binary crates with a main.rs to simply call functions living in lib.rs, that way they can be tested with use.

Summary

  • Rust test assert outputs, check for panics, or check Result type.
  • Can run tests in parallel or consecutively.
  • Outputs can be modified.
  • Tests can be selectively tested.
  • Rust has unit tests and integration tests.

Graph View

Backlinks