October 2022 lending pool exploit postmortem

Brought to you by Sovryn

October 24, 2022

In this security incident postmortem, we are sharing information regarding the exploit that happened on October 4th. This information includes the exact nature of the vulnerability that was exploited, the detection of and response to the exploit, and the steps taken to patch the vulnerability and strengthen systems and processes to reduce the likelihood of this kind of vulnerability from making it into production again in the future.

The vulnerability

There are two relevant smart contracts that together created the vulnerability. One is the lending pool aka the loan token contract, which has the public endpoints for borrowing, margin trading, minting (providing liquidity), and burning (removing liquidity). The other one is the lending, borrowing, and margin trading protocol, which provides the functionality to open, maintain, and close positions. For simplicity, we will call the latter contract “the protocol”.

The lending pool contract and protocol contract both interact with each other. Borrowing and margin trading create positions on the protocol. Positions are being closed on the protocol, which returns the principal and interest to the lending pool.

The iToken price of the lending pools is defined by the ratio of the underlying asset to the total iToken supply. At the beginning the two are equal, but as interest is generated the amount of underlying assets grows, changing the ratio and therefore increasing the iToken price. 

The underlying assets are held partially on the lending pool (as lending deposits and accrued interest), partially on the protocol (as open positions), and partially as accounting entries if the assets have been withdrawn by borrowers. The share of assets which are currently lent out is either held by the protocol or by borrowers, and the non-borrowed assets are held by the lending pool. When reading the total asset supply on the lending pool in order to calculate the price, it needs to read the protocol’s state as well. This is what made the exploit possible.

The exploit

The pools affected by the exploit are the iRBTC and iUSDT lending pools. In the following section, we first explain how the iRBTC pool was exploited, and will then explain the subtle difference in how the iUSDT pool was exploited.

Based on analysis of the exploit transactions, the exploit on the iRBTC pool was conducted in the following steps:

  1. The exploiter deploys an “exploit contract” and uses the exploit contract to borrow RBTC from the iRBTC lending pool with XUSD collateral.
  2. The exploiter closed the position by swapping part of its XUSD collateral for RBTC, but chose to receive back the remaining collateral (after repaying the loan) in XUSD. There is collateral remaining because positions on the protocol need to be overcollateralized.
  3. There is no way to buy a certain amount of RBTC on the Sovryn AMM. So, the protocol estimates the amount of XUSD it needs to sell to receive just enough RBTC to cover the loan, always with some excess RBTC received. For small loans the excess RBTC is basically a dust amount. But for larger loans, the excess RBTC can be a significant amount. If the amount of excess RBTC from the swap is under a certain threshold, it is sent to the lending pool as “profit” for iRBTC holders. If the amount of excess RBTC is over the threshold, the excess is returned to the borrower as a refund. This refund happens after the state is updated on the protocol contract, but before the principal is actually returned to the lending pool.
  4. The refund called a fallback function (the default function triggered when receiving RBTC) on the exploit contract used by the exploiter to carry out the attack. Once the fallback function is called, the exploit contract deposited RBTC into the lending pool and minted iRBTC. As mentioned above, the iToken price of the lending pools is defined by the ratio of the underlying assets to the total iToken supply. Because the protocol state was updated when the exploiter paid back their RBTC loan in step 2, but the RBTC did not yet arrive in the lending pool, the underlying RBTC balance of the lending pool calculated by the contract was lower than it should have been and the iRBTC price was therefore lower than its actual value. As a result, the exploiter was able to purchase iRBTC “at a discount”.
  5. After minting the iRBTC, the execution of the closing function resumed, transferring the RBTC principal to the lending pool. This increased the lending pool’s underlying asset balance and consequently the iRBTC price.
  6. Finally, the exploiter redeemed/burned their iRBTC and took the difference between the “discount” price at which they acquired the iRBTC and the new, higher iRBTC price as (wildly inflated) profit.

We decided to call this a cross-contract reentrancy attack. It is not a classical reentrancy in the sense that the attacker executes a function on the same contract. The protocol as well as the lending pools are guarded against reentrancy individually. This attack was only possible because the exploiter interacted with two different contracts with separate storage, where one contract depended on the other.

The iRBTC scenario worked because there was a native currency (RBTC) transfer to the attacker, which triggered the fallback function. Regular ERC20 transfers do not notify the receiver and are therefore not offering the same opportunity. This is the reason why the DOC, BPRO, and XUSD lending pools remained untouched by the exploiter.

The exploit on the iUSDT pool used the same underlying vulnerability but worked slightly differently. The underlying asset of iUSDT is rUSDT. rUSDT is an ERC777 token, which calls the receiver when tokens are transfered. This allowed the exploiter to perform the same exploit as the one used on the iRBTC pool but with some small differences.

To exploit the iUSDT pool the exploiter borrowed rUSDT with RBTC collateral and closed the loan in the same transaction with a deposit. So, in contrast to the scenario above, there was no swap involved, but the principal was returned by the borrower in the same currency (rUSDT). The transfer of rUSDT notified the sending contract (the exploit contract) before actually processing the transfer. This allowed the exploiter to mint iUSDT “at a discount” in the same way as described above for the iRBTC exploit.

The fix

Fix transactions

We decided to implement two independent protections against this kind of attack. The first is a global reentrancy protection, which is not limited to securing one contract, but a system of interconnected contracts. The second is the introduction of invariant checks to the closing functions. We have been discussing the benefits of invariant checks for our smart contract security internally for a while and see the current situation as an opportunity to start introducing them.

The global reentrancy protection

The heart of the global reentrancy protection is a simple mutex contract that holds a counter and exposes a function to increment the counter and return its value. A shared reentrancy guard contract defines a modifier that increments the mutex counter in the beginning and verifies at the end of the function execution if the value returned by the mutex remains unchanged. If another function, which is protected in the same way, is called from within a protected function, it increments the counter and therefore leads to a revert at the end of the first function execution. With this solution, it is possible to prevent nested calls of protected functions while still allowing their subsequent execution.

The invariant check

The invariant check provides an additional layer of protection. It was added to both closing functions of the protocol. At the beginning of the function execution it reads the total supply of the lending pool’s iTokens and at the end, it ensures that the total supply remains unchanged.

Funds recovery

While the exploiter was trying to exit the Rootstock blockchain with the funds they took from the lending pools, they made a critical error: they used the Sovryn bridge to try to send about 29 RBTC to BNB Smart Chain (BSC). What the exploiter did not realize ahead of time was that BTCs (the version of BTC received on BSC after sending RBTC across the bridge) only had one use: trading in the Sovryn Perpetuals protocol, which at the time of the exploit was actually paused pending a v2 protocol update. So really, BTCs was useless.

Realizing after crossing the bridge that BTCs was useless, the exploiter tried to send the BTCs back to Rootstock. Here, the exploiter failed at making another critical realization ahead of time: the Sovryn bridge had a transfer limit imposed on BTCs of 0.1 BTCs per transaction. The exploiter began sending hundreds of 0.1 BTCs transactions across the bridge. Around the time that about half of the BTCs had been sent back to Rootstock this way, Sovryn devs were alerted about unusually high volume on the Sovryn bridge and began to investigate. Due to the strange behavior of the address that was generating the unusually high bridge volume, the decision was made to contact the bridge federators and ask them to pause bridge node operations until the investigation could be completed. The federators agreed and turned off their federator bridge nodes.

Once it was determined conclusively that most of the BTCs stuck on the bridge came from an exploit of a vulnerability on the lending pool, the decision was made to recover those funds from the bridge federation multisig and return them to the lending pool. Additionally, the exploiter still had some BTCs in their BSC address. Because the Perpetuals protocol was paused, there were no other holders of BTCs other than Sovryn Exchequer. The decision was made to recover the RBTC backing those BTCs from the bridge federation multisig and return the RBTC originating from the exploit to the lending pool and transfer the RBTC backing the BTCs held by Exchequer to the Exchequer multisig on Rootstock. The BTCs token contract that was previously used for Perpetuals will now be deprecated and replaced with a new BTCs token contract, if needed.

In addition to attempting to use the Sovryn bridge to exit Rootstock with RBTC, the exploiter also attempted to convert RBTC to BTC using the Powpeg bridge. Coincidentally, at the time that the exploit occurred, a bug in Bitcoin Core was causing all Powpeg peg-out transactions to be rejected by bitcoin full nodes. As such, none of the peg-out transactions were being confirmed on bitcoin. 10 RBTC transferred by the exploiter via Powpeg remains in limbo while the Rootstock community waits for the bug to be fixed.

The last remaining funds that could be recovered came from BNB, ETH, and USDT transfers that the exploiter sent over the BSC and Ethereum bridges and were still pending at the time that the bridge federators turned off their federator bridge nodes to pause bridge operations.

Final accounting

Amount taken out of the lending pools via the exploit:

44.9368 RBTC 

282,351.9644 rUSDT

Amount recovered:

26.7610 ETH 

60.8476 BNB 

17.717 RBTC 

19,511.1505 USDT

Incident timeline

2022-08-18 06:50:03 UTC - The exploiter funds their Rootstock address via FastBTC.

2022-10-04 01:30 UTC - The exploiter deploys their first exploit helper contract and sends their first wave of exploit transactions.

2022-10-04 02:56 UTC - The exploiter deploys their second exploit helper contract and sends their second wave of exploit transactions.

2022-10-04 08:05 UTC - Sovryn community members notice and report unusually high trading volume on the Sovryn AMM. A Sovryn community manager reports the unusual activity to Sovryn developers. The initial conclusion is that the trading volume, while unusual, is probably from legitimate users.

2022-10-04 08:12 UTC - The exploiter converts some of the assets they received from the exploit into BNB, ETH, and USDC and sends the funds across the Sovryn bridge to Ethereum and BSC. Most of the funds ultimately end up in Aztec and Tornado Cash. Some of the ETH is converted into BTC using THORChain. A spreadsheet investigating where the funds went can be found here.

2022-10-04 12:50 UTC - While investigating the unusual trading activity, Sovryn developers notice unusual bridge activity and begin to suspect that a vulnerability has been exploited. The exact nature of the vulnerability is unknown at this time. Sovryn developers contact the Sovryn bridge federators and FastBTC federators and ask them to turn off their federator nodes to pause the processing of transactions, which they do.

2022-10-04 14:00 UTC - Sovryn developers conclusively determine that a vulnerability has been exploited on the Sovryn lending protocol. The exact nature of the vulnerability is still unknown at this time. The lending protocol is paused and the frontend is put into maintenance mode. The BabelFish developers are contacted and given a recommendation to pause the BabelFish aggregator in case it shares the same vulnerability as the lending pools, which they do. IOV Labs is also contacted and given a recommendation to pause the RSK Token Bridge both as a security precaution and to shut down another avenue of escape, which they do. Sovryn developers also ask the BabelFish and IOV Labs developers for help investigating the exploit to determine the root cause and develop a fix.

2022-10-04 16:00 UTC - The Sovryn community is notified that Sovryn developers are aware of a vulnerability exploit and that the system has been put into maintenance mode until it is considered safe to unpause the contracts and lift maintenance mode.

2022-10-05 16:00 UTC - Because the vulnerable contracts were originally forked from Ooki (fka bZx), a Sovryn core contributor reaches out to the Ooki developers to alert them about the exploit.

2022-10-06 12:00 UTC - Sovryn developers reverse-engineer the exploit and learn the exact nature of the vulnerability. They begin working on a fix for the vulnerability.

2022-10-05 20:00 UTC - After determining that the Sovryn AMM and FastBTC do not share the same vulnerability as the lending pools, maintenance mode is lifted on these parts of the system.

2022-10-06 23:00 UTC - After consultation with Sovryn developers, IOV Labs unpauses the RSK Token Bridge and the bridge resumes normal operation.

2022-10-07 20:00 UTC - The Sovryn bridge federators unpause the pending transactions that were not sent by the exploiter, but leave the rest of the bridge functionality paused. BabelFish devs unpause the BabelFish aggregator.

2022-10-10 18:00 UTC - The Ooki developers are privately informed about the details of the vulnerability and how the exploit worked.

2022-10-12 17:00 UTC - The Sovryn bridge federators fully unpause the bridge and it resumes normal operation.

2022-10-13 13:00 UTC - Sovryn developers publish a proposed fix for the vulnerability for public review. BabelFish and IOV Labs developers contribute to the review.

2022-10-24 14:14 UTC - Sovryn developers deploy a fix for the vulnerability and the Exchequer multisig triggers the contract logic update to implement the fix. All of the funds removed from the lending pool by the exploit transactions are returned. The returned funds come from the funds recovered, with the balance covered by the Sovryn treasury. The protocol is unpaused, and maintenance mode is lifted on the lending, borrowing, and margin trading frontend application, and the protocol resumes normal operation.

2022-10-24 20:00 UTC - This incident response report and postmortem is published.

You May Also Like

Leave A Reply