Fuzz / Invariant Tests | The New Bare Minimum For Smart Contract Security

What is fuzz testing? What are invariant tests? We introduce how to use these tools in Web3 & Solidity and explain why they are essential, especially for security. Every project should have stateful fuzz tests moving forward, and auditors can use understanding invariants to find critical bugs before code is deployed.

--

Shout-out to Trail of Bits and Horsefacts for all the fuzzing content.

Introduction

Most of the time, hacks come from scenarios you didn’t think about and write a test for.

What if I told you that you could write one test that would check for almost every possible scenario?

Let’s get froggy.

As always, you can watch my video on this subject and view a full sample repo here.

Basic of Fuzzing

What is a Fuzz Test?

Fuzz Testing or Fuzzing is when you supply random data to your system in an attempt to break it.

For example, if this balloon is our system/code, it would involve doing random stuff to the balloon to break it.

Now, why would we want to do all that?

Let’s look at an example.

`// SPDX-License-Identifier: MITpragma solidity ^0.8.0;contract MyContract {    uint256 public shouldAlwaysBeZero = 0;    uint256 private hiddenValue = 0;    function doStuff(uint256 data) public {        if (data == 2) {            shouldAlwaysBeZero = 1;        }        if (hiddenValue == 7) {            shouldAlwaysBeZero = 1;        }        hiddenValue = data;    }}`

Let’s say we have this function named `doStuff`, which takes an integer as input. We additionally have a variable named `shouldAlwaysBeZero` that we want always to be zero.

The fact that this variable should always be zero is known as our invariant, or “property of the system that should always hold.”

Invariant: The property of the system that should always hold.

Our invariant (also known as property) in this contract is that:

`Invariant: `shouldAlwaysBeZero` MUST always be 0`

In our balloon example, if we market our balloon as “indestructible,” our invariant might be that “our balloon should never be able to be popped.”

`Invariant: `balloon` should never be popped`

In DeFi, a good invariant might be:

• A protocol must always be overcollateralized
• A user should never be able to withdraw more money than they deposited
• There can only be 1 winner of the fair lottery

Example in Foundry

Let’s look at a normal unit test in Foundry.

`// SPDX-License-Identifier: MITpragma solidity ^0.8.0;import {MyContract} from "../src/MyContract.sol";import {Test} from "forge-std/Test.sol";contract MyContractTest is Test {    MyContract exampleContract;    function setUp() public {        exampleContract = new MyContract();    }    function testIsAlwaysZeroUnit() public {        uint256 data = 0;        exampleContract.doStuff(data);        assert(exampleContract.shouldAlwaysBeZero() == 0);    }}`

With this single-unit test `testIsAlwaysZeroUnit`, we might think our code has enough coverage, but if we look at the `doStuff`function again, we can see that if our input is 2, our variable will not be zero.

`function doStuff(uint256 data) public {        // WHAT IS THIS IF STATEMENT???        // 👇👇👇👇👇👇        if (data == 2) {            shouldAlwaysBeZero = 1;        }        // 👆👆👆👆👆👆        // Ignore this one for now        if (hiddenValue == 7) {            shouldAlwaysBeZero = 1;        }        hiddenValue = data;    }`

This seems obvious with our example function, but more often than not, you’ll have a function or system that looks like this:

`function hellFunc(uint128 numberr) public view onlyOwner returns (uint256) {        uint256 numberrr = uint256(numberr);        Int number = Int.wrap(numberrr);        if (Int.unwrap(number) == 1) {            if (numbr < 3) {                return Int.unwrap((Int.wrap(2) - number) * Int.wrap(100) / (number + Int.wrap(2)));            }            if (Int.unwrap(number) < 3) {                return Int.unwrap((Int.wrap(numbr) - number) * Int.wrap(92) / (number + Int.wrap(3)));            }            if (Int.unwrap(Int.wrap(Int.unwrap(Int.wrap(Int.unwrap(Int.wrap(1)) / Int.unwrap(Int.wrap(Int.unwrap(Int.wrap(Int.unwrap(Int.wrap(numbr)))))))))) == 9) {                return 1654;            }            return 5 - Int.unwrap(number);        }        if (Int.unwrap(number) > 100) {            _numbaar(Int.unwrap(number));            uint256 dog = _numbaar(Int.unwrap(number) + 50);            return (dog + numbr - (numbr / numbir) * numbor) - numbir;        }        if (Int.unwrap(number) > 1) {            if (Int.unwrap(number) < 3) {                return Int.unwrap((Int.wrap(2) - number) * Int.wrap(100) / (number + Int.wrap(2)));            }            if (numbr < 3) {                return (2 / Int.unwrap(number)) + 100 - (Int.unwrap(number) * 2);            }            if (Int.unwrap(number) < 12) {                if (Int.unwrap(number) > 6) {                    return Int.unwrap((Int.wrap(2) - number) * Int.wrap(100) / (number + Int.wrap(2)));                }            }            if (Int.unwrap(number) < 154) {                if (Int.unwrap(number) > 100) {                    if (Int.unwrap(number) < 120) {                        return (76 / Int.unwrap(number)) + 100 - Int.unwrap(Int.wrap(uint256(uint256(uint256(uint256(uint256(uint256(uint256(uint256(uint256(uint256(uint256(uint256(numbr))))))))))))) + Int.wrap(uint256(2)));                    }                }                if (Int.unwrap(number) > 95) {                    return Int.unwrap(Int.wrap((Int.unwrap(number) % 99)) / Int.wrap(1));                }                if (Int.unwrap(number) > 88) {                    return Int.unwrap((Int.wrap((Int.unwrap(number) % 99) + 3)) / Int.wrap(1));                }                if (Int.unwrap(number) > 80) {                    return (Int.unwrap(number) + 19) - (numbr * 10);                }                return Int.unwrap(number) + numbr - Int.unwrap(Int.wrap(nunber) / Int.wrap(1));            }            if (Int.unwrap(number) < 7654) {                if (Int.unwrap(number) > 100000) {                    if (Int.unwrap(number) < 1200000) {                        return (2 / Int.unwrap(number)) + 100 - (Int.unwrap(number) * 2);                    }                }                if (Int.unwrap(number) > 200) {                    if (Int.unwrap(number) < 300) {                        return (2 / Int.unwrap(number)) + Int.unwrap(Int.wrap(100) / (number + Int.wrap(2)));                    }                }            }        }        if (Int.unwrap(number) == 0) {            if (Int.unwrap(number) < 3) {                return Int.unwrap((Int.wrap(2) - (number * Int.wrap(2))) * Int.wrap(100) / (Int.wrap(Int.unwrap(number)) + Int.wrap(2)));            }            if (numbr < 3) {                return (Int.unwrap(Int.wrap(2) - (number * Int.wrap(3)))) + 100 - (Int.unwrap(number) * 2);            }            if (numbr == 10) {                return Int.unwrap(Int.wrap(10));            }            return (236 * 24) / Int.unwrap(Int.wrap(Int.unwrap(Int.wrap(Int.unwrap(Int.wrap(Int.unwrap(number)))))));        }        return numbr + nunber - mumber - mumber;    }`

This was one of the Cyfrin Security Challenges.

Here, it’s not quite so obvious if there even is an input that will cause a revert. It would be insane to write a test case for every single possible integer or scenario, so we need a programmatic way to find any outlier.

There are two popular methodologies to find these edge cases programmatically:

1. Fuzz / Invariant Tests
2. Formal Verification / Symbolic Execution

We will save “Formal Verification” for another video.

In Foundry, you’d write a solidity fuzz test like so:

`    function testIsAlwaysZeroFuzz(uint256 randomData) public {        // uint256 data = 0; // commented out line        exampleContract.doStuff(randomData);        assert(exampleContract.shouldAlwaysBeZero() == 0);    }`

Foundry will automatically input semi-random values to `randomData` and over x number of runs, input them to the`doStuff` and check that the assertion holds.

This would be equivalent to writing many tests where `randomData` had different values, all in one test!

Now I say “semi-random” because the way your fuzzer (in our case, foundry) picks the random data isn’t truly random, and should be somewhat intelligent with the random numbers it picks. Foundry is smart enough to see the `if data == 2` conditional, and pick `2` as an input.

At the moment, I think the trail of bits hybrid echidna is the best fuzzer out there due to its intelligent random number selection, but Foundry’s fuzzer (in my opinion) is easier to write code for at the moment. The echidna logo is also the best logo I’ve ever seen. Even better than the ripped Jesus logo.

Anyways, so if we run our fuzz test, it tells us exactly what input fails our test:

`\$ forge test -m testIsAlwaysZeroFuzzFailing tests:Encountered 1 failing test in test/MyContractTest.t.sol:MyContractTest[FAIL. Reason: Assertion violated Counterexample: calldata=0x47fb53d00000000000000000000000000000000000000000000000000000000000000002, args=[2]] testIsAlwaysZeroFuzz(uint256) (runs: 6, μ: 27070, ~: 30387)`

We can see that it found out if it passed `args=[2]` to the test, it was able to break our `assert(exampleContract.shouldAlwaysBeZero() == 0)`. So now, we can go back into our code, and realize we need to fix the edge case where `data == 2`, and now we are safe from the exploit of input data being 2!

Basics of Fuzzing Summary

In summary, to write a fuzz test, we did the following

1. We understood our invariant or “property of our system that must always hold”

2. We wrote a test that would input random values into our function to try to break our invariant

Stateful vs Stateless Fuzzing

Stateless Fuzzing

Now you may notice that there is another scenario where our code could have an issue, and that’s when `hiddenValue == 7`. In order for this revert to happen, you have to first set `hiddenValue` to `7`, by calling `doStuff` with the value `7` which sets `hiddenValue` to `7` and then call this function again.

`uint256 hiddenValue = 0;function doStuff(uint256 data) public {        // Fixed this part by removing it        // if (data == 2) {        //     shouldAlwaysBeZero = 1;        // }        // Wait what's this??        // 👇👇👇👇👇👇👇        if (hiddenValue == 7) {            shouldAlwaysBeZero = 1;        }        // 👆👆👆👆👆👆👆        hiddenValue = data;    }`

It takes 2 calls for our invariant to be broken.

1. Call `doStuff` with `7`
2. Call `doStuff` with any other number

Our fuzz test written above will never be able to find this example because as it’s currently written, our test is what’s known as a “stateless fuzz test.” Which is where the state of a previous run is discarded for the next run.

Stateless Fuzzing: Fuzzing/Fuzz Testing where the state of a previous run is discarded for the next run.

If we go back to the balloon example, stateless fuzzing would be similar to doing something to balloon A for one random attempt to break it, then blowing up a new balloon B and attempting to break it differently.

In the balloon example, you’d never try to break a balloon you already tried to break in the past. This seems a little silly as if our balloon invariant is that “the balloon can’t be popped” we’d want to make multiple attempts on the same balloon.

Stateful Fuzzing

So, in software engineering, we could do “stateful fuzzing” instead. Stateful fuzzing is where the state of our previous run is the starting state of our next run.

Stateful Fuzzing: The state of our previous fuzz run is the starting state of our next fuzz run.

A single stateful fuzz run would be similar to writing a test with all your assets in the same test.

`function testIsAlwaysZeroUnitManyCalls() public {        uint256 data = 7;        exampleContract.doStuff(data);        assert(exampleContract.shouldAlwaysBeZero() == 0);        data = 0;        exampleContract.doStuff(data);        assert(exampleContract.shouldAlwaysBeZero() == 0); // this would fail    }`

To write a stateful fuzz test in Foundry, you’d use the `invariant` keyword, and it requires a little more setup.

`// SPDX-License-Identifier: MITpragma solidity ^0.8.0;import {MyContract} from "../src/MyContract.sol";import {Test} from "forge-std/Test.sol";import {StdInvariant} from "forge-std/StdInvariant.sol";contract MyContractTest is StdInvariant, Test {    MyContract exampleContract;    function setUp() public {        exampleContract = new MyContract();        targetContract(address(exampleContract));    }    function invariant_testAlwaysReturnsZero() public {        assert(exampleContract.shouldAlwaysBeZero() == 0);    }}`

Instead of just passing random data to function calls, a stateful fuzz test (invariant) test will automatically call random functions with random data.

We use the `targetContract` function to tell Foundry that it can use any of the functions in `exampleContract`. There is just one function for this example, so it will just call `doStuff` with different values.

If we run this test, we can see the output as such, and we can see it finds out that if you call `doStuff` twice (once with the value 7), it will throw an error!

`\$ forge test -m invariant_testAlwaysReturnsZeroFailing tests:Encountered 1 failing test in test/MyContractTest.t.sol:MyContractTest[FAIL. Reason: Assertion violated]        [Sequence]                sender=0x000000000000000000000000000000000000018f addr=[src/MyContract.sol:MyContract]0x5615deb798bb3e4dfa0139dfa1b3d433cc23b72f calldata=doStuff(uint256), args=[7]                sender=0x0000000008ba49893f3f5ba10c99ef3a4209b646 addr=[src/MyContract.sol:MyContract]0x5615deb798bb3e4dfa0139dfa1b3d433cc23b72f calldata=doStuff(uint256), args=[2390]`

Wait, what’s an invariant again?

Now, important aside on how Foundry uses the term invariant. As we’ve described, an invariant is a property of the system that must always hold, but Foundry uses the term to mean “stateful fuzzing.” Just keep this in mind.

• Foundry Invariant Tests == Stateful Fuzzing
• Foundry Fuzz Tests == Stateless Fuzzing
• Invariants == Property of the system that must always hold

So in an actual smart contract, your invariant won’t be that a balloon shouldn’t pop or some function should always be zero; it’ll be something like:

• New tokens minted < inflation rate
• There should only be 1 winner of a random lottery
• Someone shouldn’t be able to take more money out of the protocol than they put in

And let me tell you what, at this point, you now know all the basics of Fuzzing! Congratulations! Maybe now you take a break and try writing some tests yourself, and then come back to the video.

This is the new bare minimum

This is the new floor for security in web3. It’s systematic to do, anyone can learn how to do it, and it can save a LOT of headaches.

2. Write stateful fuzz tests for them
3. Don’t go to an audit before.

Now you understand the basics of fuzzing & invariant tests, you can go use the tools you like! To learn more about advanced stateful fuzz testing, be sure to stay tuned, we have an advanced video coming up soon! Additionally, read the Foundry documentation on the Handler method, as that’s the recommended way to build the most sophisticated stateful fuzz tests.

Have Fun!