Faulty Oracle Transaction Incident Postmortem
Incident Summary:
Liqwid protocol experienced a faulty oracle update transaction at approximately 23:26 PM UTC on January 10.
The oracle price feeds for DJED and iUSD had large fluctuations stemming from our use of a CoinGecko price feed API which received incorrect data from a broken API managed by the WingRiders DEX team.
The incident was detected by a Liqwid developer. The core team took immediate action and within 15 minutes the first actions were taken to pause the batching system and shortly after the Liqwid Admin Multisig was utilized to pause the DJED and iUSD markets.
During this time we also contacted the CoinGecko DevOps team who manages their price feed API and requested they remove WingRiders from all Cardano native assets supported on Liqwid until they confirm they’ve fixed and tested their API.
The faulty oracle contract update transaction can be found here: https://cexplorer.io/tx/a8747c7356613d2f70f1a6f511362b813811e0426fa70d4ddef8cf07844fec9b
We communicated the faulty oracle update and our immediate next steps to secure the price feeds and bring the protocol back online on Liqwid’s Twitter and Discord Announcements channel:
https://x.com/liqwidfinance/status/1745253381352919173?s=20
https://discord.gg/PZ3GUWamY8
Incident Response:
After completing additional testing and updates on the Liqwid oracle smart contract we immediately implemented Charli3 oracle price feeds for ADA, DJED and iUSD. These have been live on mainnet since January 19th with no issues. Alongside SHEN, Liqwid protocol now uses Charli3 price feeds for 4 of 10 supported markets.
Long term we are implementing a price floor and cap model on the Liqwid oracle smart contract to filter price data received from Charli3 and Orcfax oracles based on price deviation percent from previous price. This will serve as an ultimate line of defense in the event of incorrect price data being published on-chain by either one of these oracle service providers. We are also implementing a trailing price model to protect the protocol from market manipulation in Cardano native token markets for the purpose of inflating collateral value.
Implementing Charli3 and Orcfax decentralized oracle price feeds along with the floor and cap model and trailing prices to the Liqwid oracle smart contract represent a significant improvement to the resiliency and security of price data used to determine Liqwid collateral and liquidation values.
Incident Corrective Actions
The total amount owed to liquidated users and liquidators impacted is estimated to be ~$7,000. Proposal 37 to fully compensate affected users with LQ from the staking rewards wallet (Safety Pool) has passed an on-chain governance vote: https://app.liqwid.finance/governance/proposal/37
All affected users have now been fully reimbursed and the protocol has no bad debt.