Being an engineer at Coinbase Custody means solving challenging problems at the intersection of security, reliability, and user experience. Coinbase’s security-first culture mandates that all trade-offs be made in favor of security. This approach is taken to the next level at Custody. After all, security is the main reason why large institutions and high net worth individuals trust Custody to hold on their behalf billions of dollars worth of crypto assets.
While security dominates all Custody technical decisions, customers experience our product day-to-day primarily through the lense of reliability and convenience. Our users frequently perform high-value, time-sensitive withdrawals and expect transactions to be confirmed on the blockchain without delays and have clean predictable amounts arriving at the destination (and with no fees deducted from the amount).
Fortunately, Bitcoin’s advanced technique called Child-Pays-For-Parent (CPFP) allows us to satisfy all of the above requirements: zero-fee transactions with reliable confirmation times, all while keeping the fundamental property of secure Cold Storage — using each private key strictly once.
Each transaction submitted to the Bitcoin network includes a fee paid to miners for processing this transaction and including it into a block. A block is produced every 10 minutes on average. When the size of all pending transactions exceeds the maximum block size limit of 1MB miners prioritize transactions with higher fees for inclusion while kicking low-fee transactions down the next block. Such a process can repeat multiple times, causing transaction to become stuck indefinitely until the network congestion clears up.
If you transacted Bitcoin in 2017–2018 you may remember multi-hour and sometimes multi-day delays during peaks of price volatility and other important market events. During those times it was not uncommon for transactions to pay up to 100x regular fees in order to be confirmed without delays. The chart below shows how volatile Bitcoin fees can be.
Historical transaction fee distribution (credit: https://jochen-hoenicke.de/queue/#0,all)
At Coinbase Custody we recognize this obstacle and continuously innovate to give our customers an edge when they need it the most. One obvious solution is to calculate transaction fees immediately before broadcasting it to the Bitcoin network — knowing the current network conditions enables us to predict an optimal fee. Unfortunately, Cold Storage security requirements make it difficult to estimate the fee right before the broadcast.
The Cold Storage security model requires that complete transaction payload (including amounts and destinations) is fully known beforehand, guarding against any changes to the payload after the key restore procedure is initiated. On the flip side, since the transaction fee is encoded in the payload, it means that the fee must be calculated long before the transaction is broadcast. If during that time gap the network suddenly becomes congested, the calculated fee may become inadequate for the transaction to be confirmed in a timely manner.
One interesting property of Bitcoin’s UTXO model is that transactions can be chained together creating a “bundle” that miners evaluate as a whole with respect to the fees. For example, if one (child) transaction spends an output of another (parent) transaction, the child transaction can pay fees for both — miners have no choice but to include the parent transaction if they want to collect the excess fees from the child transaction. This approach is commonly known as Child-Pays-For-Parent (CPFP).
The Coinbase Consumer platform has been successfully using CPFP to accelerate withdrawals for quite some time and you can learn more about it in the blog post. However, Custody’s unique challenges forced us to rethink the approach and come up with a different design.
In Custody we use CPFP to shift the fee estimation to immediately before the broadcast. In addition, we use a special Gas Station service to cover the transaction fees. Here’s how it works step-by-step:
When a withdrawal from a cold address is initiated we do an initial fee estimate and use the Gas Station service to send 10x estimated fees to that address.
When constructing the withdrawal (parent) transaction in addition to the destination and change outputs we add an output that moves the pre-gassed amount (10x the fees) to a special “fee” address (this address is generated for each withdrawal and is used only once). Let’s call this output a “CPFP link”.
Once the main transaction is signed at the end of the key restore ceremony and right before the broadcast, the fees get re-estimated again based on the current network conditions.
The fee address generates a child transaction spending the CPFP-link output, paying the fees for both transactions and sending the remainder of the pre-gassed amount back to the Gas Station.
Let’s work through an example to better understand the mechanics of the solution. Note that some minor details are omitted for simplicity. Let’s say we have a customer who has a balance of 15 BTC and they would like to withdraw 12 BTC from their Custody account into an external Bitcoin address. Here are the steps that will happen (note that ‘satoshi’ or ‘sat’ is the smallest unit in Bitcoin and equals to 0.00000001 BTC):
Custody estimates the fee to complete this transaction to be 10,000 sat.
Gas Station sends 10x that amount or 100,000 sat to the cold address (the address which holds the funds to be withdrawn).
Custody generates the withdrawal transaction with two main outputs: 12 BTC goes to the destination address, 3 BTC “change” is returned to Cold Storage.
We add a third “CPFP-link” output that moves the 100,000 sat to a newly generated fee address — this amount will later be used to pay the fee.
The “key restore” ceremony commences. The result of it is a signed transaction payload ready to be broadcasted to the network.
Right before the broadcast we estimate the fee again. As it turns out, the network suddenly became congested and now the fees are 2x what we initially estimated — 20,000 sat instead of the original 10,000 sat.
We create a child transaction that spends the “CPFP link” output, pays 20,000 sat fee and moves the remaining 80,000 sat back to the Gas Station.
We broadcast both transactions to the network.
Note that the destination address receives the clean 12 BTC amount. From the user perspective neither fees nor child transaction exist s— they are abstracted away and work behind the scenes to enable reliable zero-fee Bitcoin withdrawals.
This is just one example of interesting technical problems that Coinbase Custody engineers have an opportunity to work on. If you’re interested in joining the Coinbase Custody team, check out open roles here.
This website contains links to third-party websites or other content for information purposes only (“Third-Party Sites”). The Third-Party Sites are not under the control of Coinbase, Inc., and its affiliates (“Coinbase”), and Coinbase is not responsible for the content of any Third-Party Site, including without limitation any link contained in a Third-Party Site, or any changes or updates to a Third-Party Site. Coinbase is not responsible for webcasting or any other form of transmission received from any Third-Party Site. Coinbase is providing these links to you only as a convenience, and the inclusion of any link does not imply endorsement, approval or recommendation by Coinbase of the site or any association with its operators.
Unless otherwise noted, all images provided herein are by Coinbase.
Product,
Dec 4, 2024