Fuzz / Invariant Tests | The New Bare Minimum For Smart Contract Security
What is fuzz testing? What are invariant tests? We introduce how to use these tools in Web3 & Solidity and explain why they are essential, especially for security. Every project should have stateful fuzz tests moving forward, and auditors can use understanding invariants to find critical bugs before code is deployed.
Shout-out to Trail of Bits and Horsefacts for all the fuzzing content.
Introduction
Most of the time, hacks come from scenarios you didn’t think about and write a test for.
What if I told you that you could write one test that would check for almost every possible scenario?
Let’s get froggy.
As always, you can watch my video on this subject and view a full sample repo here.
Basic of Fuzzing
What is a Fuzz Test?
Fuzz Testing or Fuzzing is when you supply random data to your system in an attempt to break it.
For example, if this balloon is our system/code, it would involve doing random stuff to the balloon to break it.
Now, why would we want to do all that?
Let’s look at an example.
// SPDX-License-Identifier: MIT
pragma solidity ^0.8.0;
contract MyContract {
uint256 public shouldAlwaysBeZero = 0;
uint256 private hiddenValue = 0;
function doStuff(uint256 data) public {
if (data == 2) {
shouldAlwaysBeZero = 1;
}
if (hiddenValue == 7) {
shouldAlwaysBeZero = 1;
}
hiddenValue = data;
}
}Let’s say we have this function named doStuff, which takes an integer as input. We additionally have a variable named shouldAlwaysBeZero that we want always to be zero.
The fact that this variable should always be zero is known as our invariant, or “property of the system that should always hold.”
Invariant: The property of the system that should always hold.
Our invariant (also known as property) in this contract is that:
Invariant: `shouldAlwaysBeZero` MUST always be 0In our balloon example, if we market our balloon as “indestructible,” our invariant might be that “our balloon should never be able to be popped.”
Invariant: `balloon` should never be poppedIn DeFi, a good invariant might be:
- A protocol must always be overcollateralized
- A user should never be able to withdraw more money than they deposited
- There can only be 1 winner of the fair lottery
Example in Foundry
Let’s look at a normal unit test in Foundry.
// SPDX-License-Identifier: MIT
pragma solidity ^0.8.0;
import {MyContract} from "../src/MyContract.sol";
import {Test} from "forge-std/Test.sol";
contract MyContractTest is Test {
MyContract exampleContract;
function setUp() public {
exampleContract = new MyContract();
}
function testIsAlwaysZeroUnit() public {
uint256 data = 0;
exampleContract.doStuff(data);
assert(exampleContract.shouldAlwaysBeZero() == 0);
}
}With this single-unit test testIsAlwaysZeroUnit, we might think our code has enough coverage, but if we look at the doStufffunction again, we can see that if our input is 2, our variable will not be zero.
function doStuff(uint256 data) public {
// WHAT IS THIS IF STATEMENT???
// 👇👇👇👇👇👇
if (data == 2) {
shouldAlwaysBeZero = 1;
}
// 👆👆👆👆👆👆
// Ignore this one for now
if (hiddenValue == 7) {
shouldAlwaysBeZero = 1;
}
hiddenValue = data;
}This seems obvious with our example function, but more often than not, you’ll have a function or system that looks like this:
function hellFunc(uint128 numberr) public view onlyOwner returns (uint256) {
uint256 numberrr = uint256(numberr);
Int number = Int.wrap(numberrr);
if (Int.unwrap(number) == 1) {
if (numbr < 3) {
return Int.unwrap((Int.wrap(2) - number) * Int.wrap(100) / (number + Int.wrap(2)));
}
if (Int.unwrap(number) < 3) {
return Int.unwrap((Int.wrap(numbr) - number) * Int.wrap(92) / (number + Int.wrap(3)));
}
if (Int.unwrap(Int.wrap(Int.unwrap(Int.wrap(Int.unwrap(Int.wrap(1)) / Int.unwrap(Int.wrap(Int.unwrap(Int.wrap(Int.unwrap(Int.wrap(numbr)))))))))) == 9) {
return 1654;
}
return 5 - Int.unwrap(number);
}
if (Int.unwrap(number) > 100) {
_numbaar(Int.unwrap(number));
uint256 dog = _numbaar(Int.unwrap(number) + 50);
return (dog + numbr - (numbr / numbir) * numbor) - numbir;
}
if (Int.unwrap(number) > 1) {
if (Int.unwrap(number) < 3) {
return Int.unwrap((Int.wrap(2) - number) * Int.wrap(100) / (number + Int.wrap(2)));
}
if (numbr < 3) {
return (2 / Int.unwrap(number)) + 100 - (Int.unwrap(number) * 2);
}
if (Int.unwrap(number) < 12) {
if (Int.unwrap(number) > 6) {
return Int.unwrap((Int.wrap(2) - number) * Int.wrap(100) / (number + Int.wrap(2)));
}
}
if (Int.unwrap(number) < 154) {
if (Int.unwrap(number) > 100) {
if (Int.unwrap(number) < 120) {
return (76 / Int.unwrap(number)) + 100 - Int.unwrap(Int.wrap(uint256(uint256(uint256(uint256(uint256(uint256(uint256(uint256(uint256(uint256(uint256(uint256(numbr))))))))))))) + Int.wrap(uint256(2)));
}
}
if (Int.unwrap(number) > 95) {
return Int.unwrap(Int.wrap((Int.unwrap(number) % 99)) / Int.wrap(1));
}
if (Int.unwrap(number) > 88) {
return Int.unwrap((Int.wrap((Int.unwrap(number) % 99) + 3)) / Int.wrap(1));
}
if (Int.unwrap(number) > 80) {
return (Int.unwrap(number) + 19) - (numbr * 10);
}
return Int.unwrap(number) + numbr - Int.unwrap(Int.wrap(nunber) / Int.wrap(1));
}
if (Int.unwrap(number) < 7654) {
if (Int.unwrap(number) > 100000) {
if (Int.unwrap(number) < 1200000) {
return (2 / Int.unwrap(number)) + 100 - (Int.unwrap(number) * 2);
}
}
if (Int.unwrap(number) > 200) {
if (Int.unwrap(number) < 300) {
return (2 / Int.unwrap(number)) + Int.unwrap(Int.wrap(100) / (number + Int.wrap(2)));
}
}
}
}
if (Int.unwrap(number) == 0) {
if (Int.unwrap(number) < 3) {
return Int.unwrap((Int.wrap(2) - (number * Int.wrap(2))) * Int.wrap(100) / (Int.wrap(Int.unwrap(number)) + Int.wrap(2)));
}
if (numbr < 3) {
return (Int.unwrap(Int.wrap(2) - (number * Int.wrap(3)))) + 100 - (Int.unwrap(number) * 2);
}
if (numbr == 10) {
return Int.unwrap(Int.wrap(10));
}
return (236 * 24) / Int.unwrap(Int.wrap(Int.unwrap(Int.wrap(Int.unwrap(Int.wrap(Int.unwrap(number)))))));
}
return numbr + nunber - mumber - mumber;
}This was one of the Cyfrin Security Challenges.
Here, it’s not quite so obvious if there even is an input that will cause a revert. It would be insane to write a test case for every single possible integer or scenario, so we need a programmatic way to find any outlier.
There are two popular methodologies to find these edge cases programmatically:
- Fuzz / Invariant Tests
- Formal Verification / Symbolic Execution
We will save “Formal Verification” for another video.
In Foundry, you’d write a solidity fuzz test like so:
function testIsAlwaysZeroFuzz(uint256 randomData) public {
// uint256 data = 0; // commented out line
exampleContract.doStuff(randomData);
assert(exampleContract.shouldAlwaysBeZero() == 0);
}Foundry will automatically input semi-random values to randomData and over x number of runs, input them to thedoStuff and check that the assertion holds.
This would be equivalent to writing many tests where randomData had different values, all in one test!
Now I say “semi-random” because the way your fuzzer (in our case, foundry) picks the random data isn’t truly random, and should be somewhat intelligent with the random numbers it picks. Foundry is smart enough to see the if data == 2 conditional, and pick 2 as an input.
At the moment, I think the trail of bits hybrid echidna is the best fuzzer out there due to its intelligent random number selection, but Foundry’s fuzzer (in my opinion) is easier to write code for at the moment. The echidna logo is also the best logo I’ve ever seen. Even better than the ripped Jesus logo.
Anyways, so if we run our fuzz test, it tells us exactly what input fails our test:
$ forge test -m testIsAlwaysZeroFuzz
Failing tests:
Encountered 1 failing test in test/MyContractTest.t.sol:MyContractTest
[FAIL. Reason: Assertion violated Counterexample: calldata=0x47fb53d00000000000000000000000000000000000000000000000000000000000000002, args=[2]] testIsAlwaysZeroFuzz(uint256) (runs: 6, μ: 27070, ~: 30387)We can see that it found out if it passed args=[2] to the test, it was able to break our assert(exampleContract.shouldAlwaysBeZero() == 0). So now, we can go back into our code, and realize we need to fix the edge case where data == 2, and now we are safe from the exploit of input data being 2!
Basics of Fuzzing Summary
In summary, to write a fuzz test, we did the following
- We understood our invariant or “property of our system that must always hold”
2. We wrote a test that would input random values into our function to try to break our invariant
Stateful vs Stateless Fuzzing
Stateless Fuzzing
Now you may notice that there is another scenario where our code could have an issue, and that’s when hiddenValue == 7. In order for this revert to happen, you have to first set hiddenValue to 7, by calling doStuff with the value 7 which sets hiddenValue to 7 and then call this function again.
uint256 hiddenValue = 0;
function doStuff(uint256 data) public {
// Fixed this part by removing it
// if (data == 2) {
// shouldAlwaysBeZero = 1;
// }
// Wait what's this??
// 👇👇👇👇👇👇👇
if (hiddenValue == 7) {
shouldAlwaysBeZero = 1;
}
// 👆👆👆👆👆👆👆
hiddenValue = data;
}It takes 2 calls for our invariant to be broken.
- Call
doStuffwith7 - Call
doStuffwith any other number
Our fuzz test written above will never be able to find this example because as it’s currently written, our test is what’s known as a “stateless fuzz test.” Which is where the state of a previous run is discarded for the next run.
Stateless Fuzzing: Fuzzing/Fuzz Testing where the state of a previous run is discarded for the next run.
If we go back to the balloon example, stateless fuzzing would be similar to doing something to balloon A for one random attempt to break it, then blowing up a new balloon B and attempting to break it differently.
In the balloon example, you’d never try to break a balloon you already tried to break in the past. This seems a little silly as if our balloon invariant is that “the balloon can’t be popped” we’d want to make multiple attempts on the same balloon.
Stateful Fuzzing
So, in software engineering, we could do “stateful fuzzing” instead. Stateful fuzzing is where the state of our previous run is the starting state of our next run.
Stateful Fuzzing: The state of our previous fuzz run is the starting state of our next fuzz run.
A single stateful fuzz run would be similar to writing a test with all your assets in the same test.
function testIsAlwaysZeroUnitManyCalls() public {
uint256 data = 7;
exampleContract.doStuff(data);
assert(exampleContract.shouldAlwaysBeZero() == 0);
data = 0;
exampleContract.doStuff(data);
assert(exampleContract.shouldAlwaysBeZero() == 0); // this would fail
}This is bad practice btw, don’t write your asserts in the same test, please.
To write a stateful fuzz test in Foundry, you’d use the invariant keyword, and it requires a little more setup.
// SPDX-License-Identifier: MIT
pragma solidity ^0.8.0;
import {MyContract} from "../src/MyContract.sol";
import {Test} from "forge-std/Test.sol";
import {StdInvariant} from "forge-std/StdInvariant.sol";
contract MyContractTest is StdInvariant, Test {
MyContract exampleContract;
function setUp() public {
exampleContract = new MyContract();
targetContract(address(exampleContract));
}
function invariant_testAlwaysReturnsZero() public {
assert(exampleContract.shouldAlwaysBeZero() == 0);
}
}Instead of just passing random data to function calls, a stateful fuzz test (invariant) test will automatically call random functions with random data.
We use the targetContract function to tell Foundry that it can use any of the functions in exampleContract. There is just one function for this example, so it will just call doStuff with different values.
If we run this test, we can see the output as such, and we can see it finds out that if you call doStuff twice (once with the value 7), it will throw an error!
$ forge test -m invariant_testAlwaysReturnsZero
Failing tests:
Encountered 1 failing test in test/MyContractTest.t.sol:MyContractTest
[FAIL. Reason: Assertion violated]
[Sequence]
sender=0x000000000000000000000000000000000000018f addr=[src/MyContract.sol:MyContract]0x5615deb798bb3e4dfa0139dfa1b3d433cc23b72f calldata=doStuff(uint256), args=[7]
sender=0x0000000008ba49893f3f5ba10c99ef3a4209b646 addr=[src/MyContract.sol:MyContract]0x5615deb798bb3e4dfa0139dfa1b3d433cc23b72f calldata=doStuff(uint256), args=[2390]Wait, what’s an invariant again?
Now, important aside on how Foundry uses the term invariant. As we’ve described, an invariant is a property of the system that must always hold, but Foundry uses the term to mean “stateful fuzzing.” Just keep this in mind.
- Foundry Invariant Tests == Stateful Fuzzing
- Foundry Fuzz Tests == Stateless Fuzzing
- Invariants == Property of the system that must always hold
So in an actual smart contract, your invariant won’t be that a balloon shouldn’t pop or some function should always be zero; it’ll be something like:
- New tokens minted < inflation rate
- There should only be 1 winner of a random lottery
- Someone shouldn’t be able to take more money out of the protocol than they put in
And let me tell you what, at this point, you now know all the basics of Fuzzing! Congratulations! Maybe now you take a break and try writing some tests yourself, and then come back to the video.
This is the new bare minimum
This is the new floor for security in web3. It’s systematic to do, anyone can learn how to do it, and it can save a LOT of headaches.
- Understand your invariants
- Write stateful fuzz tests for them
- Don’t go to an audit before.
- If you do, make sure your auditors help you understand your invariants!
Now you understand the basics of fuzzing & invariant tests, you can go use the tools you like! To learn more about advanced stateful fuzz testing, be sure to stay tuned, we have an advanced video coming up soon! Additionally, read the Foundry documentation on the Handler method, as that’s the recommended way to build the most sophisticated stateful fuzz tests.
This article from Horsefacts also does an amazing walkthrough.
Have Fun!
😸😸Follow Patrick!😸😸
Book a smart contract audit: Cyfrin
