Can ChatGPT really replace crypto audits? Not yet, say researchers
Everyone is experimenting with ChatGPT, even crypto exchanges.
Coin base recently turned to artificial intelligence to experiment with how accurately ChatGPT could perform a token security review – a requirement for all tokens listed on the exchange.
After reviewing 20 different smart contracts, the mega-popular AI tool produced the same results as the manual review 12 times.
However, five of the eight errors were cases where ChatGPT incorrectly labeled a high-risk asset as low-risk, which is the worst error.
The experiment also revealed that the AI sometimes produced inconsistent results, with the same request generating different outcomes, especially when moving from one iteration of ChatGPT to the next.
Still, the Coinbase team is optimistic that – with further rapid development – they can increase the accuracy of ChatGPT to the point where it can be used as a secondary quality assurance check.
“We are not surprised because such smart contracts can also be automatically audited by [other traditional programming] tool,” said a spokesperson from BlockSec, a blockchain security infrastructure firm Decrypt. “However, it may not work for complex business logic, which are the main attack surfaces and key loopholes that smart contract audits should focus on.
However, Coinbase’s optimism about using the tool for additional assurances was echoed by other security experts in the crypto security space.
“At this stage it is [AI] cannot replace a person, but it is an indispensable aid, including for tired or inattentive auditors,” said independent security researcher Officer’s Notes. Decrypt via Twitter. “I think that Q/A [quality assurance] and fuzzing definitely won’t do without AI tools in the future.”
ChatGPT replaces engineers
Although preliminary, the blockchain security sector appears to be accepting the possible implementation of AI tools.
But can AI replace manual security auditors in the future?
“Maybe one day we’ll get to that point, but we’re still a long way off. What’s more likely is a complementary approach. There are some things humans do better than machines and vice versa,” Certik’s head of solution architecture Connie Lam told Decrypt via email, “Tools help us build new things, but they don’t replace us. The invention of the calculator didn’t make accountants obsolete, it made them better at their jobs.”
For now, though, non-AI security tools are still far more useful than anything that just hits the market to pinpoint vulnerabilities.
“Current security audit tools are still superior to OpenAI,” said an OpenZeppelin spokesperson Decrypt. “The [Coinbase] tests it to list ERC-20 tokens, which is a well-known pattern. That makes it more suitable for automation.”
However, that may change.
The rapid progress seen even between ChatGPT 3.5 and ChatGPT 4 is palpable, suggesting that further upgrades will continue to impress.
And while they do, integration of these tools should even be “encouraged”.
“Usage of ChatGPT should be encouraged during the development phase. It is a powerful tool and refusing to work with it and learn what it can do would be a setback,” said Lam. “ChatGPT is also a very powerful tool for searching for information and building a knowledge center. It can help users quickly understand complicated topics and stay up-to-date on the latest security information.”
