THORChain Network back online after 20 hours of outage due to a software bug

On October 27, several blockchain developers reported the unavailability of the THORChain network on Twitter owing to a software bug. 

In an update, the THORChain team reported that the problem was caused by sources of nondeterminism between nodes. The company added that the interruption has nothing to do with its financial competence and that they are trying to debug the network quickly.

The team explained the issue by stating, “Consensus halts in a distributed state machine are from sources of non-determinism between individual nodes and prevent the ledger from becoming corrupted.” 

THORChain outlines solution

THORChain network operators went on to list procedures that would need to be taken to fix the issue, including locating the non-determinism’s origin, launching an update, and restarting the state machine. 

They said that they were almost close to identifying the source followed by an update after 3 hours announcing that they had identified the problem as string manipulation.  

“Turns out to be string manipulation: code was pushing a cosmos. Uint [instead of a uint64] into a string, which causes the string to get the point of the big int instead of the actual value, causing the memo string to be different on different nodes.”

The developers further admitted: “Didn’t see this in stagenet because the bad memo never gets written to disk/block, because it gets swapped immediately. Where at mainnet has a queue and therefore the swap to synth doesn’t get executed the same block, and the bad memo is written to the block.”

After almost 20 hours, the THORChain team returned online by tweeting that it is “back online and producing blocks.”

THORChain is not the first of its kind to be affected since numerous other blockchains have frequently gone offline, with Team Finance, a crypto vesting platform, being the most recent in the list. It briefly halted all platform operations after facing a severe attack, in an effort to prevent any further potential hacks.