OpenAI and Paradigm Introduce EVMbench for AI-Powered Smart Contract Security Evaluation

A new benchmark for evaluating AI agents' ability to detect and exploit smart contract vulnerabilities has been introduced by OpenAI and Paradigm. The EVMbench benchmark measures performance in economically meaningful scenarios using reproducible, sandboxed deployments and programmatic grading.

What Happened

The EVMbench benchmark was developed as a response to the growing use of AI agents in smart contract security. Smart contracts on public blockchains manage large amounts of value, and vulnerabilities in these systems can lead to substantial losses. As AI agents become more capable at reading, writing, and running code, it is natural to ask how well they can navigate this landscape.

The EVMbench benchmark draws on 120 curated vulnerabilities from 40 audits, with many sourced from open audit competitions. It also includes additional scenarios from the security auditing process for the Tempo blockchain (a payments-oriented L1), to reflect where agentic stablecoin payments and on-chain financial activity may grow.

Background and Context

The use of AI agents in smart contract security is a rapidly growing field. These agents can be used to detect vulnerabilities, patch them, or even exploit them for malicious purposes. However, the effectiveness of these agents depends on their ability to accurately identify and respond to vulnerabilities.

OpenAI's Codex model was evaluated using EVMbench in three capability modes: Detect, Patch, and Exploit. In the Detect mode, the agent audited a smart contract repository and was scored on recall of ground-truth vulnerabilities and associated audit rewards. The results showed that Codex achieved a score of 72.2% in the Exploit mode, compared to GPT-5 at 31.9%.

Why it Matters

The EVMbench benchmark is significant for several reasons. Firstly, it provides a standardized evaluation framework for AI agents' performance in smart contract security. This will enable developers and researchers to compare the effectiveness of different models and identify areas for improvement.

Secondly, EVMbench addresses the need for economically meaningful security evaluation. The benchmark measures performance in scenarios that reflect real-world economic consequences, making it a more accurate representation of AI agents' capabilities.

What Comes Next

The release of EVMbench marks an important step forward in the development of AI-powered smart contract security tools. As the field continues to evolve, we can expect to see further innovations and improvements in AI agent performance.

OpenAI has made the EVMbench code available on GitHub, allowing developers to contribute to its development and use it for their own projects. The benchmark's open-source nature will facilitate collaboration and knowledge-sharing among researchers and practitioners in the field.

Key Facts

• EVMbench is a new benchmark developed by OpenAI and Paradigm to evaluate AI agents' ability to detect and exploit smart contract vulnerabilities.
• The benchmark draws on 120 curated vulnerabilities from 40 audits, with many sourced from open audit competitions.
• EVMbench measures performance in economically meaningful scenarios using reproducible, sandboxed deployments and programmatic grading.
• OpenAI's Codex model achieved a score of 72.2% in the Exploit mode, compared to GPT-5 at 31.9%.
• The EVMbench code is available on GitHub, allowing developers to contribute to its development and use it for their own projects.