Security PSA: Securely designing and integrating with protocol services

Coinbase is the world’s most trusted crypto exchange due to its longstanding commitment to consumer protection, security, and global regulatory compliance. To that end we have the CryptoSec team, a dedicated team within Blockchain Security, whose main focus is to securely design and integrate cryptocurrency services across blockchain products at Coinbase. 

Cryptocurrency services are defined as any service within a stack that enables read/write to blockchains.  In our previous blog post we discussed how we classify and define different crypto services; in this post, we will dive more deeply into protocol services, services that act as a portal to the blockchain and the only set of services that implement network specific configurations. We will illustrate how certain protocol service designs can be more secure than others, explore security controls across varying levels and review some of the best practices that we at Coinbase implement to maintain our reputation as the world’s most trusted crypto exchange.

Understanding Node vs Indexer Services

Before we discuss best practices, it is essential to understand the components of protocol services and understand the difference between the two.

Protocol Services Security Best Practices

A common misunderstanding is that protocol services are significantly less security critical than other parts of the crypto stack, such as signers.  However, the reality is that an attacker that can exploit data integrity violations in protocol services in many cases would be able to steal a similar amount of funds as if the private keys themselves were compromised.

Below we have highlighted some of the best practices that we at Coinbase implements to secure interactions with blockchain protocol services. 

Node Integrity: Validating State 

In the context of blockchain, validating state refers to the process of checking and ensuring that the data on the blockchain is accurate, complete, and not tampered with. This process involves assessing and affirming each block and transaction's correctness in the chain. The node service checks and authenticates numerous parameters like block hashes, transaction details, nonce values, and more. The node’s state validation process is vital because it maintains the integrity and security of blockchain data, making sure the information used or relayed by blockchain services is reliable and secure.

Scenario: An exchange suffers downtime on its nodes. To catch back up to the most recent chain tip, engineers download last week’s blockchain data from a third party snapshot provider. 

Risk: An exchange assumes the state has been fully validated and credits users based on the blockchain data in the snapshot. It turns out that the third party suffered a breach and the snapshots have been altered to include spoofed transactions. 

What does Coinbase do?

Coinbase runs canonical node implementations that validate full chain state from genesis, wherever possible. In cases where this is not possible, Coinbase validates block data against other independent external sources.

Fully Codify Node Configurations

A compromised node can steal funds, therefore nodes should be treated as first class citizens within a production environment.  Standard production safeguards should apply, such as codification of build, deploy, and production access.  Nodes have advanced configuration settings that should be defined explicitly in configuration files rather than configured manually at runtime. These settings include the node's peering rules, validation settings, available RPC endpoints, consensus settings, and more. Ensuring all relevant settings are version controlled removes ambiguity, increased consistency, ensures auditability, and reduces the potential for errors or misconfigurations that could be exploited by malicious actors. 

Scenario:  A node has various default configurations that improves performance by disabling security features. When running this node, the engineer that deploys the node has to remember to re-apply a particular flag to enable a security feature every time a new instance is deployed.  

Risk: An engineer could forget to set the security flag or leave the company and it is likely that the configuration would revert to the default the next time the node instance needs to be redeployed, or an update needs to be applied.

What does Coinbase do?

Coinbase has security flags codified as part of our consensus-controlled deploy pipeline, so they get applied whenever nodes are deployed. 

Indexers: Operating Nodes at Scale

Operating blockchain services at scale can potentially overwhelm nodes with data requests, thereby necessitating the use of indexers, which are designed to manage high volumes of data. Additionally, nodes do not always provide user-friendly APIs. For instance, by default, Bitcoin nodes do not offer an API endpoint for "What is the balance of address XYZ?" A query like this, though, is often indispensable for service interfaces, prompting us to leverage indexer services for more sophisticated, efficient APIs.

Indexers don't validate the chain state.  Because of this, it’s important that they run "on top" of node services, ensuring the data they process has been validated at the node level, thereby maintaining blockchain network integrity. Indexers essentially serve as "translators" of blockchain data, faithfully conveying the blockchain state from nodes to downstream services, similar to a language translator. 

Scenario: Operations are solely reliant on nodes to query information from the blockchain

Risk: Heavy demand suddenly destabilizes previously under-utilized nodes, causing crashes and slow responses, damaging the ability to interact with the blockchain.

What does Coinbase do?

We tackle this issue by strategically deploying indexers. By doing so, we ensure that our APIs remain performant, and our blockchain operations can handle the scale that our platform demands. Furthermore, it allows us to offer more sophisticated services without compromising the security and stability of our nodes.

Node Security: Secure Software Lineage

Node services, while central to the operation and maintenance of a blockchain, are typically third-party code. As such, they can introduce multiple risks to the integrity and security of a blockchain system. 

Scenario:  A third party node repository gets compromised and a malicious release is merged.

Not-so-good: A node operator automatically downloads and runs the malicious release binaries published to node projects’ github page as soon as they are released.

What does Coinbase do?

At Coinbase, we employ a 'defense in depth' approach to mitigate these risks with several commonly referred guidelines:

• Build nodes from source code where possible.

• Pin node builds to a commit hash.

• Utilize the primary/incumbent node implementation.

• Monitor for node updates and run the most recent stable release.

• Isolate nodes from critical services such as key management systems.

Providing Best in Class Security

At Coinbase, we understand the inherent risks associated with handling blockchain data. That's why we design our protocol services with security as a top priority.  We believe that by sharing these practices, we can encourage better security standards throughout the crypto industry, ensuring a safer environment for everyone involved in blockchain technology. Stay tuned for the next post in our series, where we will continue to explore the landscape of blockchain security.