Coinbase now storing 87% of customer funds offline

Offline storage provides an important security measure against theft or loss of bitcoins. Today I’d like to talk about how we implement offline storage at Coinbase and give you a behind the scenes look at some of our processes.

What is offline storage?

Offline storage (sometimes called “cold storage”) is the process of taking money that is connected to the internet and moving it offline for safe keeping.

Why do this? Well, any computer that is connected to the internet with money on it (or bitcoins) is a target for hackers. Keeping servers secure involves many variables and is it’s own area of study and research. Under normal conditions this level of security should be sufficient, but payment apps are held to a higher standard.

This is where it’s helpful to have multiple layers of security. And offline storage is just that — another layer of security we use so that if our first layer fails, we have still not lost all customer funds.

How does it work?

During a normal week, the largest transaction on Coinbase might only account for 5% of total customer deposits. So it’s not necessary to keep 100% of funds available at any given time.

Instead, we can safely move about 90% of those funds offline. We do this by taking the sensitive data that would normally reside on our servers (the “private keys” which represent the actual bitcoins) and moving it to USB sticks and paper backups. We then take these to a safe deposit box at an actual bank. In this case we use the bank more like a vault instead of for storing any traditional currency.

(Example paper copy shown for the photo — we don’t actually use any data on that page.)

What we actually store on the USB drives is a number of pre-generated bitcoin addresses along with their private keys. These keys are never generated on the live servers — they are generated offline so they never have a chance to get accidentally left on a server.

How often do you have to go to the safe deposit box?

Not very often. What’s nice is that each address has a public and private component. For us to move the money offline we actually can do it remotely without visiting the bank just by sending excess funds to the public bitcoin address. (This happens each night with any excess funds deposited that day.) In this sense we can “top up” our offline storage remotely without having to open our safe deposit box.

The only time we need to physically access it is to restore funds to the live servers if someone (or several people) want to withdraw more than 10% of all customer deposits in a day. Since customer deposits have been going up each week, this is actually a rare occurrence (and has only been necessary twice so far).

What if I want to withdraw all my money at once from Coinbase?

As long as you control less than 5–10% of all customer deposits on Coinbase, you shouldn’t see any delay and your withdrawal will happen instantly. If you happen to be one the largest users of Coinbase and want to send out all of your funds at once (or if many people decide to withdraw their bitcoins at the same time) it is certainly possible that we will exceed the amount of funds stored live on the server. In this case you may experience delays of up to 48 hours while we visit our safe deposit box and restore your funds live to the server for withdrawal.

This seems to be a reasonable tradeoff given the large security benefit, but of course we’ll continue to evaluate if it becomes a hassle for users.

One thing worth noting is that Coinbase does not loan out customer funds or use them in any way — we have 100% of customer funds in either online or offline storage at all times.

We have a dashboard internally showing various stats on customer deposits — including what percentage is stored online, offline, and in user wallets, along with a list of currently pending transactions due to insufficient funds live on the server. They automatically go out once funds are restored to the live servers.

Why three USB drives and paper as well?

There is a small chance a USB drive can fail, so we keep multiple copies. The paper is there in case who knows what happens — giant solar flares, magnets? We’re not sure, but it seems safer.

Don’t bank employees have access to this data?

No. We also encrypt the data on the USB drives and paper backups using AES-256 encryption. The USB drives contain text files (and the paper is simply print outs of the encrypted text) which look like very long passwords. Bank employees (or anyone else who gained access to the safe deposit box without decryption passwords) wouldn’t be able to use the data on them.

Could you store more than 90% of funds offline?

We’ll continue to experiment with this and find the right balance between convenience and security. As deposits continue to grow it may be possible to store 95% or even 99% of funds offline. Our goal is to be able to cover the amount of online funds in the event of loss, so we will use this as a guideline as we continue to improve our procedures.

But wait, this is not safe because…

You are probably correct. What we have today is certainly not perfect and we’re still refining the process. In particular we need to address:

• where to keep redundant copies so they aren’t all in one location

• employee access procedures

• getting audits done by security professionals

Offline storage is a great step in the right direction but hardly sufficient by itself. It is just one small part of our overall security plans.

Security is something that we will continue to improve over the coming years (and hopefully share this journey with you on this blog as we make progress). We’ll continue to post updates as we make corrections and get closer.

Hopefully this gives you a better sense of how we’re thinking about security today.

P.S. And if you haven’t already, you can try creating a free bitcoin wallet on Coinbase.