SHA-256 (vs. Jolt)

As of May 2024

This note gives a benchmark comparison of proving execution of SHA-256 on a 32-byte input, on Valida and on Jolt. In this benchmark, single-core Valida proving was estimated to be about 6.4 times more efficient than multi-core Jolt proving, in terms of the time and energy expended on computing the proof. Multi-core Valida proving was estimated to be about 5.4 times faster than multi-core Jolt proving on this example. These are not the best results that could be obtained on this problem with Valida, since much work remains to be done on optimizing the prover and the code generated by the compiler.

---

Results

The raw data is available. The following table records the means, medians, and standard deviations of the various sample groups. All measurements are denoted in seconds.

Prover	Measure	Mean	Median	Standard deviation
Valida serial	User t.	1.1237	1.1235	0.01787
Valida parallel	User t.	2.7835	2.7865	0.06926
Jolt	User t.	7.1879	7.1945	0.2194
Valida serial	Wall clock t.	1.151	1.1505	0.0219
Valida parallel	Wall clock t.	0.249	0.2485	0.0051
Jolt	Wall clock t.	1.3565	1.357	0.0139

The following table displays the ratios between the mean measurements for Valida and Jolt of the user time and the wall clock time. A number greater than 1 indicates that Valida is faster; a number less than 1 indicates that Jolt is faster.

Measure	Valida condition	Valida advantage
User time	Serial	6.397
User time	Parallel	2.582
Wall clock time	Serial	1.179
Wall clock time	Parallel	5.443

---

Methodology

The following zk-VM versions were used:

• Valida: git commit https://github.com/valida-xyz/valida/commit/54f3f5a0c8a499569d0f8c22ec37b8d73bb0631d

• Jolt: git commit https://github.com/a16z/jolt/commit/09d11a44ebb88666c42e2b1bca0000862771a906

The following version of the Valida LLVM compiler was used: 45bce621680189d5d006f88cbadbe9cbef403b89

The C implementation of SHA-256 which was compiled to run on Valida is a modified version of the reference implementation of SHA-256 by Brad Conte. That modified version is available here: https://github.com/lita-xyz/valida-c-examples/blob/main/sha256_32byte_in.c

The Rust implementation of SHA-256 which was compiled to run on Jolt is available in this repository.

The inputs for both Valida and Jolt are the same and are a 32-byte array of 5s.

The following commands were run to execute the benchmarks:

For Valida:

# excluded in the benchmarking time
./llvm-valida/build/bin/clang -c -target delendum ../../valida-c-examples/sha256_32byte_in.c -o ./buildValidaTests/sha256_32byte_in.o
./llvm-valida/build/bin/ld.lld --script=./llvm-valida/valida.ld -o ./buildValidaTests/sha256_32byte_in.out ./buildValidaTests/sha256_32byte_in.o

# included in the benchmarking time, run as a bash script and timed
RAYON_NUM_THREADS=32 ./valida/target/release/valida run ./buildValidaTests/sha256_32byte_in.out ./buildValidaTests/sha256_32byte_in.log
RAYON_NUM_THREADS=32 ./valida/target/release/valida prove ./buildValidaTests/sha256_32byte_in.out ./buildValidaTests/sha256_32byte_in.proof
RAYON_NUM_THREADS=32 ./valida/target/release/valida verify ./buildValidaTests/sha256_32byte_in.out ./buildValidaTests/sha256_32byte_in.proof

The above command shows multi-threaded execution; for single-threaded execution, 32 is replaced with 1.

Note that Valida currently fails to verify this program, but the output is checked to be correct by examining the log time. We are working on fixing this problem but the verification time should not make a meaningful impact on the results.

For Jolt, the following command was run from within the source repo, in the root directory:

git clone [e-mail]:lita-xyz/benchmarks.git
cd sha256/jolt
time cargo run --release

In order to measure the run time of a program, this study used GNU time. The test system has a AMD Ryzen 9 7950X 16-Core Processor, with 32 threads, and 124.9 GB DDR5 RAM. During the tests there was no other programs running on the system other than the tests themselves. Tests were performed sequentially, one after another.

For some unknown reason, running cargo run --release on the test system caused the host program to be recompiled every time, which is not supposed to happen. This made it a little harder to measure the execution time of the Jolt prover. The build time listed in the output is therefore subtracted from the recorded time such that only the execution, proving, and verifying times are counted.