On 2020-02-25 an Omni Core integrator, Antoine Le Calvez from Coin Metrics made us aware of a potential issue with Omni Core through responsible disclosure. After further inspection, we believed Omni Core v0.6 and later versions were affected by a potential replay of transactions after a block reorganization. The developer was provided a bounty for reporting the issue promptly and responsibly to the Omni Layer team.
When a block reorganization occurs, the internal state of Omni Core is rolled back and the effects of the affected block or blocks are reverted. Omni Core then tries to catch up and scans for new transactions. When a new block is found or a longer chain is connected, the new transactions are processed.
However, we believe that due to an optimization of the locking code in Omni Core in an update from v0.5 to v0.6 multiple events may occur simultaneously, resulting in a faulty state:
- A block is orphaned
- Omni Layer transactions from the orphaned block are rolled back and an older state is restored
- A new block arrives and is registered
- Omni Core scans and executes new transactions up to the current chain tip, in case there is a gap between the old restored state and the current chain
- The transactions from the new block are processed
The correct order would be:
- Omni Layer transactions are reverted
- An older state is restored
- If the state is too old, Omni Core catches up and reparses the old transactions
- A new block arrives or a longer chain connects
- The new transactions are processed
In this edge case the new block is registered before Omni Core scans for potential transactions in the gap between the old state and the new chain tip. This leads to the transactions from the newly connected block getting executed, even though they will be executed again when the new block is processed in the regular way. The end result is that the effect of some transactions may be executed twice by the parsing engine of individual clients, depending on the order in which the blocks were received.
Note: No new money is created out of thin air and all rules for sending tokens are still in place. For example: when A had a balance of 10 tokens and sent 10 tokens to B, then the second execution of the transaction would simply be considered as invalid and A still ends up with 0 tokens and B with 10 tokens.
However, if the second execution of the transaction was valid, for example if A had started with a balance of 100 tokens, then it’s effect would be applied twice, potentially moving more tokens than the sender originally intended.
Conclusion: The reach of this issue is difficult to determine. The local state may vary from node to node as it depends on which block the local node saw first during a block reorganization. This edge case was patched and released in Omni Core v0.8 and all future releases. Users of Omni Core versions prior to v0.6 and users of the OmniExplorer API were not affected.
Additionally, specific instructions for integrators on how to validate their local nodes data were included in the release notes of the upgrade and a previously planned protocol level upgrade (Feature ID 15: Activate trading of any token on the distributed exchange) was used to ensure affected clients were not subjected to further possible errors.
The developers have not been made aware of any balance discrepancies resulting in loss of funds by users. The upgrade proceeded successfully and all running clients share the correct consensus state.
One thought on “Omni Core Incident Report March 2020”
[…] A more in-depth post is available here. […]
Comments are closed.