Coinbase Logo

Euler Compromise Investigation - Part 2 - The Redemption

Tl;dr: The Unit 0x team, a Coinbase team dedicated to investigating and preventing attacks in the broader crypto ecosystem, investigates the Euler Finance compromise which serves as a great case-study in both general risks to lending protocols as well as incident handling applicable to other DeFi projects. In this two part analysis we will first perform an in-depth analysis of the smart contract vulnerability caused by a new feature introduced after the audit. In part two of the series we will explore a whirlwind of on-chain activity and communication leading up to the return of stolen assets as well as critical lessons learned from the incident handling.

By Heidi Wilder, Peter Kacherginsky, Anto Joseph

Engineering

, April 25, 2023

, 15min read time

Screenshot 2023-04-19 at 3.15.38 PM

While this compromise does not directly target Coinbase Exchange custodial services or systems, we strongly believe that it is important to address any attacks or vulnerabilities within the crypto community as a whole and hope the information in the blog will help strengthen and inform the entire ecosystem.

On March 13, 2023, Euler Finance, a lending protocol on the Ethereum blockchain, was hacked, resulting in the theft of over $197 million by a single actor in just 15 minutes. The attackers skillfully exploited a vulnerability and unique borrowing and liquidation mechanisms to completely drain DAI, USDC, WETH, and other token pools. For in-depth discussion of the vulnerability and how it was exploited see Part 1 of the blog.

Continuing our discussion of the Euler compromise, let's follow along attackers’ activity before and after the exploit while retracing the events leading up to the return of all assets. The story takes a fascinating twist involving North Korean state actors, dissenting members, and concludes with a real celebration for the Euler team and the DeFi community.

How was the attack funded?

The Euler attacker address used to deploy the attacker contract received 1ETH from Tornado Cash at 8:43 UTC on February 13 (tx).

“MEVbot” attacker

Before the Euler attacker could execute the attack, an MEV bot front ran their attack by automatically detecting and redeploying a slightly modified exploit contract. However, because the reward address 0xb66cd9 was hard-coded in the contract, the bot ended up helping the attacker siphon off DAI for them.

Further delving into the contracts, we find that the MEV bot introduced minimal changes to the attacker’s bytecode by replacing all instances of the deployer address with their own. There were no structural changes to the bytecode indicating that the MEV bot did not have access to the source code and likely replayed the deployment transaction with the above changes clearly visible in decompiled code.

Screenshot 2023-04-25 at 9.36.07 AM

The MEV contract that helped siphon off funds to the attackers was deployed seconds later and was funded via Fixed Float and appears to have funded the FCDEP EPMAX flash loan attack on BSC back in February (tx). The funder address of the second contract deployed received funds from 0xbcaa6c. The same day of the Euler attack, this address sent a message to themselves reading the following (tx):

Screenshot 2023-04-25 at 9.36.44 AM

Where did the money go?

Unlike a usual operation conducted by North Korea, this attacker seems to have tested out various methods and strategies over the following days. This resulted in the attacker sending funds to an address associated with the Ronin Bridge Exploit, which is connected to North Korea, a response from them, and the eventual negotiation which resulted in the attacker sending the majority of proceeds back to the protocol.

March 13, 2023 - 100 ETH to Tornado

On March 13 following the exploit, funds were primarily sitting in the following addresses:

At 10:38 UTC, however, 100 ETH from the Exploiter address 1 was sent to Tornado via a holdover address, 0xc66dfa (Exploiter address 3) [tx]. However, the rest of the funds remained idle.

March 16, 2023 - 1,000 ETH deposited to Tornado and 100 ETH given to victim

Starting at 1:19 UTC on March 16, 2023, funds from Exploiter address 1 began to be moved to a new Exploiter address 3 in increments of 100 ETH each across 10 transactions.

This 1,000 ETH was sent to the 100 ETH Tornado contract between 01:31 UTC and 01:39 UTC from Exploiter address 3.

Back on March 15, an address, 0x2af24e5, begged for their 78 wstETH back to the exploiter via a message on chain [tx]:

Screenshot 2023-04-25 at 9.39.01 AM

The next day at 01:43 UTC, the attackers sent 100 ETH from Exploiter address 1 to the “victim’s” address [tx].  Later on March 16, 2023 at 21:33 the “victim” attempted to send 12 ETH to one of the Euler multisigs with the following message  [tx]. The transaction, however, reverted.

Screenshot 2023-04-25 at 9.39.43 AM

Next at 21:38 UTC the “victim” was able successfully send back 12ETH to the Euler deployer with the same message [tx].

March 17, 2023 - 100 ETH sent to well known Ronin Bridge Exploiter address

On March 17, 2023 the attackers attempted to confuse investigators by first sending 1,000k ETH from 0xb66cd (Exploiter address 1) to 0xc66df (Exploiter address 3) between 03:43 UTC and 3:46 UTC and finally sending 100 ETH to the Ronin Bridge Exploiter address at 3:48 UTC (tx).

March 18, 2023 - 3,000 ETH returned to Euler

On March 18, 2023 the Euler attackers returned 3,000 ETH from Exploiter address 1 to the Euler deployer address over the course of three separate transactions between 6:53 UTC and 6:56 UTC [tx, tx, tx]. We suspect this was done to open the conversation with Euler and show good faith. 

A fourth transaction was sent at 10:20 UTC for 0 ETH [tx]. It is unclear why this was sent but likely was to perhaps show a sign of life or confirmation that indeed the attacker was in control of this particular address.

March 20, 2023 - Euler attacker reaching out to Euler to negotiate

ON March 20, 2023 at 16:10,  the attackers sent a message reading “T_27” 0xb2698c to an EOA associated with the attackers: 0x6014a (Exploiter address 4) [tx]. 

This address is referenced in the attacker’s initial contract [contract]:

Screenshot 2023-04-25 at 9.40.47 AM

At 16:41 UTC, the attackers sent a memo to the Euler deployer asking to negotiate via Exploiter address 1 [tx].

Screenshot 2023-04-25 at 9.41.25 AM

At 19:14 UTC, the Euler Deployer responded asking to negotiate over Blockscan [tx].

Screenshot 2023-04-25 at 9.41.55 AM

March 21, 2023 - Tripartite negotiations On March 21, 2023, at 17:02 UTC, the Ronin Bridge attackers responded to the Euler attackers’ exploiter address 1 by sending 2ETH and a message implying that they were willing to negotiate [tx].

Screenshot 2023-04-25 at 9.42.38 AM

The Ronin attackers likely recovered the public key of the Euler attackers via txs the Euler attackers previously sent (how to recover).

We suspect that this may have been North Korean actors attempting to recruit the attacker at the very least and at worst may be an attempt to phish the attacker’s private key from them. This further implies that the Euler Finance attackers and Ronin Bridge attackers are not the same actors. This likely led to the Euler exploiters panicking. 

In response, at 18:18 UTC the Euler attackers reached out to the Euler team to confirm that he indeed is willing to still negotiate with them via Exploiter address 1 [tx].

Screenshot 2023-04-25 at 9.43.20 AM

March 22, 2023 - Attacker publicizes their email address  On March 22, 2012, at 14:24 UTC the Euler attackers published another message to the Euler deployer reading via Exploiter address 1 [tx]:

Screenshot 2023-04-25 at 9.44.15 AM

According to Protonmail’s PGP directory, the email appears that it was likely registered on the same day as the message:

Screenshot 2023-04-25 at 10.18.29 AM
Screenshot 2023-04-25 at 10.18.24 AM

It may also be the case that the user reset the PGP keys, but it’s unlikely.

The attackers likely published their email address in order to prove that the emails they sent to Euler are indeed legitimate.

March 25, 2023 - Partial return of funds and partitioning off

On March 25, 2023, the Euler exploiters and Euler team likely came to a failed agreement on chain.

At 15:08 UTC, only 51,000 ETH was returned to the Euler deployer from the Euler Exploiter address 1 [tx].

Later at 15:10 UTC the exploiters provided their contact information [tx]:

Screenshot 2023-04-25 at 9.46.16 AM

Between 15:25 UTC and 15:30 UTC, the exploiters began to partition out the 7,738.25 ETH and 10,787,465.25 DAI and sent them to fresh addresses:

ETH was first sent in the same increments to the above four exploiter addresses, followed by DAI. 

March 25, 2023 - Dissent in the ranks

Following the partitioning of stolen assets into four wallets each one of the wallets returned all of the assets and began communicating to the Euler team with increasingly desperate messages.

At 15:38 UTC exploiter address 7 0x46e0b returned 7738.25 ETH to Euler deployer [tx]. This was followed by the same address returning 1.23m DAI to the Euler deployer at 16:08 UTC [tx]

At 16:09 UTC the same address sent the following message to themselves, indicating that they were a lone wolf of the team and were willing to provide intel on the exploiters in exchange for a bounty [tx]. The likely intended this message to be sent to the Euler team.

Screenshot 2023-04-25 at 9.47.11 AM

At 16:10 UTC, exploiter address 7 sent the same message to Euler team [tx]. However, at 16:15 UTC exploiter address 7 sent another message to Euler team, this time indicating that they would accept a lower reward [tx].

Screenshot 2023-04-25 at 9.47.44 AM

It appears that the holder of exploiter address 7 may have been getting increasingly anxious in awaiting a response. They therefore at 17:06 UTC sent a message to themselves indicating that they no longer wanted a bounty but just wanted to be contacted at the below email [tx].

Screenshot 2023-04-25 at 9.48.25 AM

This email routes to tempumail[.]com. This is a temp email service that anyone can access. While the emails for this account have been deleted since, various Twitter users discovered messages sent to the exploiter by several emails. Note: the individual appears to be signing up for protonmail.

Screenshot 2023-04-25 at 9.49.23 AM
Screenshot 2023-04-25 at 9.49.48 AM

Note also, that this is the third address that the exploiters partitioned out funds to and does not reference the “Euler exploiter 3” address on Etherscan. 

At 17:07 UTC, exploiter address 7 sent the above message to the Euler Deployer [tx].

March 25, 2023 - The group reaches out to Euler

Later that day at 19:43 UTC a message was also sent from Exploiter address 1 to the Euler deployer reading [tx]:

Screenshot 2023-04-25 at 9.50.27 AM

This note indicates that the other attackers likely coordinated their response in response to the lone wolf returning a portion of the funds. 

This was followed by a message at 20:35 UTC from the same exploiter address reading [tx]:

Screenshot 2023-04-25 at 9.51.02 AM

March 27, 2023 - Return of partitioned funds

On March 27, 2023 at 18:03 UTC,  exploiter addresses 6, 8 and 9 that had received partitioned ETH and DAI returned them to the Euler Deployer. This indicates likely that the group was persuaded to in fact return the bounty that they had originally kept for themselves. 

March 28, 2023 - An apology

On March 28, 2023 at ​​01:56 UTC, Exploiter address 1 wrote the following apology [tx]:

Screenshot 2023-04-25 at 9.52.07 AM

At 01:57 UTC, the same exploiter address followed it with 23 ETH and the following message [tx]:

Screenshot 2023-04-25 at 9.52.35 AM

It appears that Jacob may be a part of the team, but that the name “Jacob” itself may be a pseudonym.

March 31, 2023 - Dust returned

On March 31, 2023 between 14:41 UTC and 14:43 UTC, the exploiters returned dust in three of the above partitioned out addresses to the Euler Multisig.

At 15:12 UTC, Exploiter address 1 also sent the following message to the address 0x74cde6 [tx].

Screenshot 2023-04-25 at 9.53.19 AM

This address appears to have been topped up using Huobi on both BSC and Ethereum. We suspect it likely belongs to the exploiters (exploiter address 8)

April 3, 2023 - Last of funds returned

On April 3, 2023 at 22:54 UTC, 8,080 ETH was returned from Exploiter address 2 to the Euler multisig [tx]. Another 2,500 ETH was returned from Exploiter address 2 to the Euler multisig five minutes later at 22:59 UTC [tx]. 

In total, the following was returned to Euler:

  • 95,532.2 ETH

  • 43,149,861 DAI

The exploiters currently holds the following dust balances on the following addresses:

  • Exploiter address 1 - 0xb66cd- 1.02 ETH

  • Exploiter address 2 - 0xc66dfa - 0.784 ETH

  • Exploiter address 3 - 0xb2698c - 0.99 ETH

  • Exploiter address 4 - 0x6014a - 0 ETH

  • Exploiter address 5 - 0xa1b44d - 0.144 ETH

  • Exploiter address 6 - 0x46e0be - 0.144 ETH

  • Exploiter address 7 - 0x8765a3 - 0.141 ETH

  • Exploiter address 8 - 0xc4e04a - 0.143 ETH

The exploiters additionally received 1.1k ETH, which we suspect is still sitting in Tornado Cash. For more uptodate data, please see here

Negotiation and the return

While we do not have access to the negotiation discussions between the Euler team and the exploiter, we do have some insights into the Euler attacker’s rationale based on their blockchain transactions alone. 

Recall, on March 17 the exploiters sent 100 ETH to the Ronin Bridge exploiter’s address. The exploiters likely did this to throw investigators off their tracks and to suggest that the exploit was related to North Korea and therefore that funds were not recoverable. This led to a flurry of media articles and increased government concern over the hack. We suspect that this sudden rise in media attention was what the attackers weren’t exactly aiming for and possibly led to increased  law enforcement interest in the exploit.

The attackers the next day sporadically sent 3,000 ETH to the Euler team, suggesting a good faith effort to open negotiations with the team. On the 20th, the attackers sent a message to the Euler team to negotiate and Euler responded that they’d be open to negotiating over Blockscan. It’s unclear what was discussed.

The following day on March 21 the Ronin Exploiter address reached out on chain with 2 ETH and a message indicating that the attacker should install an application from a specific Github repo to discuss further. This likely panicked the attackers, leading to them then reaching out again and showing a willingness to discuss. We suspect that by this time, the attackers knew their only option was to return some of the stolen funds. 

The next day the attackers anxiously sent the Euler team their protonmail email address on chain, likely to indicate which email sender they should look for in their inbox. Three days later on March 25, about 50% of the stolen funds were returned. After this, the rest of the ETH and DAI were split amongst four addresses, each likely meant for one attacker within the group. Taking a step back, the only reason the attackers would split the funds pre-laundering them if there had been internal disagreements between them.

As such it was not surprising, one of these four addresses promptly returned the funds it received to the Euler team and reached out to them to provide intel on the attackers. We suspect that the holder of this address did not agree with the exploiter groups’ negotiation strategy and decision to take such a large cut and was panicked. This led to the larger group later reaching out with another contact email address to negotiate.

Two days later, on March 27, the attackers sent over another ~30% of the stolen funds along with another email address and a message indicating that they would return all of the funds. We suspect that Euler may have had some additional information or leverage over the attackers, possibly provided by the holder of the dissenting address, that led to them returning this additional amount. The following day, the attacker wrote an apology message on-chain.

On April 3, ~20% of the rest of the stolen funds were returned - totalling to almost 100% of funds being returned - making the protocol mostly whole again. We suspect that this was done to clear the attackers’ names who at this point highly likely were identified by Euler and referred to law enforcement. 

Who is behind it?

Based on the on-chain communication and wallet partitioning, there are likely multiple actors (up to 4) involved in the exploit. For example, on March 20th, 2023 attackers sent messages referring to themselves as “We”:

Screenshot 2023-04-25 at 9.54.23 AM

As mentioned above, the group included a dissenting party which identified themselves as “Euler Attacker #3” on March 25th, 2023:

Screenshot 2023-04-25 at 9.54.49 AM

As well as another actor switching to individual requests for forgiveness:

Screenshot 2023-04-25 at 9.55.19 AM

The Euler exploiters clearly exhibit knowledge in on-chain opsec such as the use of Tornado Cash, transaction messages for negotiations, and not interacting with entities requiring KYC. Due to the highly technical nature of the exploit, the exploiters are highly proficient at writing liquidation bots, understand how lending/borrowing protocols work and how to profit from them.

Smart contracts deployed by attackers employed access controls and hard-coded functions unique from deployer addresses to protect against MEV bots and front-running transactions.

There were no apparent test transactions prior to the exploit indicating that exploiters likely tested their exploit off-chain using a tool like Foundry.

Assessing Euler liquidations over the past year, we find that in the past year, there were 271 liquidations. 44 of which involved smart contracts performing the liquidation. Of these 44 transactions, 11 unique contracts were used. 

While none of the liquidator contracts are identical to the Euler exploiters’ contract, there are some similarities between the Euler exploiters’ and other liquidators contracts. Likely indicating that the exploiters built their contract using liquidator contracts as inspiration.

You can find a complete list of indicators in the Appendix A below.

Rippling impact

The Euler compromise left users with collateral and debt positions without underlying tokens backing them. Without the ability to redeem their assets, users rushed to remove any available liquidity on the protocol creating an appearance of additional compromises with assets leaving the platform en masse. 

Another immediate effect of the compromise was the sudden drop in Euler’s governance token - EUL. At 10x normal volume level, the token lost more than half of its value.

Composability is one of the key features of DeFi protocols including Euler. Below are just a few of the secondary effects on the entire DeFi ecosystem following the hack:

  • Angle Lab’s stablecoin depegged and the project paused after it lost 17m USDC.

  • Balancer rushed to pause its Euler Boosted Pools but not before some of the liquidity providers dumped a single token pair increasing damage to the remaining investors.

  • Yield Protocol shut down its mainnet app since it relied completely on Euler.

  • Inverse Finance, SwissBorg, Yearn Finance, Harvest Finance, Idle, Overnight, USD+, Opyn, Mean, Sense, and many others had to shut down pools and in some cases to compensate users.

What can we learn from this?

Attackers are becoming increasingly clever and don't necessarily need to rely on compromised private keys to conduct attacks. Exciting and increasingly complex protocols with new features are being launched daily. New features like Euler’s mint() and donate() functions, while elegant, user and gas friendly on their own, created a gap for attackers to exploit. We implore protocols and auditors to consistently review new features added and how together these features can create risks for the entire ecosystem. 

The entire Euler attack lasted under 30 minutes. Interestingly PeckShield, a 3rd party blockchain security boutique firm, alerted the team on Twitter less than 9 minutes after the initial DAI theft. Unfortunately, the Euler team was not fast enough to respond to the incident which allowed subsequent theft of USDC, WBTC, and other tokens. It is clear that having a combination of a monitoring and alerting system as well as an incident response plan to take advantage of them is crucial for modern DeFi protocols to minimize losses from compromises caused by inevitable bugs.

Last but not least are lessons on post-compromise communication with attackers. A sufficiently strong bounty offer not to press charges combined with on-chain investigation efforts to uncover attackers’ identities often results in the recovery of a significant portion of stolen assets. The decision to make a bug bounty offer is similar to that of ransomware victims’ where the alternative may have significantly worse consequences further encouraging the pattern.

The constantly evolving landscape of decentralized finance presents both opportunities and challenges. While new protocols and features bring innovation and convenience, they also open doors for attackers to exploit vulnerabilities. It is crucial for protocols and auditors to remain vigilant in their review of new features and their potential risks to the ecosystem. As we continue to embrace the future of defi, let’s also make sure to prioritize the security and protection of protocols and user’s funds. Ultimately, prevention is always better than cure.

Congratulations again to the Euler team!

Appendix A: Indicators

Attackers:

Ethereum: Exploiter address 1 - 0xb66cd966670d962c227b3eaba30a872dbfb995db Ethereum: Exploiter address 2 - 0xc66dfa84bc1b93df194bd964a41282da65d73c9a Ethereum: Exploiter address 3 - 0xb2698c2d99ad2c302a95a8db26b08d17a77cedd4 Ethereum: Exploiter address 4 - 0x6014a3bb4a17adaeb95b10f731d6542a6a4df288 Ethereum: Exploiter address 5 - 0xa1b44d4b5b4c361f51e029b81bf2db9cf4d8e676 Ethereum: Exploiter address 6 - 0x46e0be2df97dac791fc8e30cf2b2e4f58c50cf55 Ethereum: Exploiter address 7 - 0x8765a35394c98e81b9d56d44248e1199d8e38a4c Ethereum: Exploiter address 8 - 0xc4e04ac48639ff077ebb36e7cfe0c4993b7b208e Ethereum: Exploit Contract - 0x036cec1a199234fc02f72d29e596a09440825f1c Email: INTACH3ZZ@PROTONMAIL[.]COM Email: sheeps4music@xyzmailhub[.]com Email: sheeps4music@mail2tor[.]com Email: xxxyyy990@umail.edu[.]pl

(User)name: Jacob

MEV Bot:

Ethereum: 0xebc29199c817dc47ba12e3f86102564d640cbf99 (Exploit Contract) Ethereum: 0x583c21631c48d442b5c0e605d624f54a0b366c72 (Violator Contract) Ethereum: 0xa0b3ee897f233f385e5d61086c32685257d4f12b (Liquidator Contract) Ethereum: 0x5f259d0b76665c337c6104145894f4d1d2758b8c (Contract Creator) Ethereum: 0xbcaa6ce6fdf7a393dc903049324e443168e17c5c (EOA Funder)

Coinbase logo