Bug causes chaos on the Ethereum blockchain

The event raised systemic concerns that user’s funds were at risk as the bug forced several of the network’s largest applications to become inoperable.

Ethereum is now back to full functionality after the scare which lasted a few hours but frustration across the community remains. The bug caused the Ethereum chain to temporarily split and for several hours users were dealing with two versions of the Ethereum blockchain with different transaction histories.

Major exchanges including Binance suspended ETH withdrawals and the Ethereum community faced hours of panic before it was revealed that a bug in the Go Ethereum (GETH) client was the cause of the chaos.

It was revealed the chain split occurred because of a code change that had been inserted into the client during an upgrade in July. The change was incompatible with the previous versions of Geth and caused consensus to break for operators who were running earlier releases. Geth Node operators who had ignored the upgrade were affected by the bug and broke into a minority chain.

Most affected node operators said when they had viewed the upgrades released on July 20th 2020 Geth v1.9.17 update, they considered them to be minor and had not realized the upgrade included a major change to Ethereum’s consensus design. An initial post-mortem from the Geth team states that a “consensus issue was (deliberately) triggered on the Ethereum network.”

A team member from Optimism, builders of Layer 2 scaling solution Optimistic Roll-ups, posted that the project had knowingly decided to “test the bug and see what would happen” explaining “we didn’t realize the impact of the few nodes that were not upgraded.”

One of the node service providers affected was Infura, which provides infrastructure for a number of major Ethereum applications including Metamask, OpenZeppelin, Maker, Coinbase wallet, CryptoKitties, and the 0x protocol. Infura reports that it had a major outage on its Ethereum services which temporarily crippled a number of key DeFi and wallet applications.

Geth developer and Ethereum security expert Martin Swende said the incident is a “reminder to keep your node(s) up to date!” and later continued that the developers did not announce the big change to avoid drawing attention to the vulnerability.

Péter Szilágyi a team lead at Ethereum explains in the incident report that in Ethereum’s case “it takes a lot of time (weeks, months) to get node operators to update even to a scheduled hard fork.” This means the network is exposed to attacks from potentially malicious actors during the time lag before all operators can upgrade. He continues “Security via obscurity is definitely not something to aim for, but delaying a potential attack by enough to get most node operators immune may be worth the temporary "hit" to transparency.”

The consensus bug was apparently dormant in the code for over a year before it was discovered and fixed. In the view of the Geth team that chances someone would accidentally trigger it was “tiny”.

The Ethereum community has questioned why the bug could not have been disclosed discreetly and teams building and developing on Geth and get them to upgrade to newer, safer versions of the client. Matt Corallo, the founder of Thesis, expressed his view on Twitter saying. “According to this not only did ETH have one of its worst bugs ever but the devs botched the disclosure of it and didn’t even mention it to likely the largest operator of ETH full nodes.”