Upstream EVM benchmarking

[chfast]

The current scripts for running benchmarks on EVM implementations are in https://github.com/cdetrio/benchmarking-wasm-ewasm-evm (private repo).

• The scripts could be moved to this repo.

They are based on out-of-tree changes that should be upstreamed to:

• geth: Evm code bencher ethereum/go-ethereum#20151 and cmd/evm: Add --bench flag for benchmarking ethereum/go-ethereum#20330
• parity (https://github.com/cdetrio/parity/tree/evm-code-bencher)
• cita-vm (https://github.com/cdetrio/cita-vm/tree/evm-bencher)

Also consider:

• Building individual docker images for each EVM instead of having single one.

Recent results: https://docs.google.com/presentation/d/1MgXd159bmx37Q70Ng_ggHcClcYLafYqrxNEBFKAMqWI/edit#slide=id.g64bd7b8151_0_1

• evmone results from docker build and from native build are different - the difference is big enough that only usage of different compiler version is not enough to explain it.

The EVM should accept for benchmarking

• EVM bytecode
• EVM input

From each EVM we need following data:

• Total time of execution
• EVM output (to check correctness)
• gas used
• EVM exception/success status (not strictly needed)

[axic]

Things have improved since October, now all the sources are in this repository and in https://github.com/ewasm-benchmarking. Yet it would be nice to upstream some of the benchmarking code.

Out of all the forks we have, I think upstreaming to openethereum is one case which could make sense: https://github.com/ewasm-benchmarking/openethereum/tree/evm-code-bencher

@chfast @gumb0 are you interested?

[axic]

@chfast And can this evmone fork https://github.com/ewasm-benchmarking/evmone/commits/bench-evm-codes replaced by upstream?

[gumb0]

I would be interested in messing with openethereum/evm-code-bencher.

Should we make a separate tool out of it or rather new mode in evmbin tool? (currently it looks to replace most of evmbin features)

[axic]

@gumb0 I were you I'd ask openethereum first whether they are interested in such features, either in evmbin or in a separate tool. If they are, then perhaps it is worth the effort, if we can timebox into some low number of hours.

[axic]

Updates:

1. I don't think it make sense trying to upstream to cita-vm.

2. evmone was upstreamed and the repo uses the 0.4.1 release

3. parity/openethereum would be nice to upstream, the latest code we have is: https://github.com/ewasm-benchmarking/openethereum/releases/tag/v2.5.1-benchmarking