Operating staking nodes on Kubernetes

In 2022, Coinbase wrote a blog post which confirmed its belief that Kubernetes could offer a more flexible and scalable system than Coinbase's homegrown deployment solutions. Not too long before that post, Coinbase acquired Bison Trails, a staking node provider that had been using Kubernetes since 2019 to achieve decentralized and cloud-agnostic deployments across Google Cloud Platform (GCP) and Amazon Web Services (AWS). 

In this article, we share our experiences operating staking nodes on Kubernetes while achieving these goals: preventing slashing, ensuring security and compliance, maximizing uptime, and realizing enhanced decentralization.

Preventing slashing

Slashing is a feature of proof-of-stake blockchains which deters validator misconduct and promotes security, availability, and network involvement. Penalties of this nature arise from two main misbehaviors: downtime and double signing. Though specifics vary per blockchain implementation, slashing typically results in forfeiting a set percentage of a validator's tokens for abnormal behavior. Double signing penalties, which occur when a validator duplicates a block, are generally more severe than downtime penalties.

To manage our stateful blockchain staking nodes within Kubernetes, we utilized StatefulSets, allowing us to address unique requirements like preserving critical runtime data and ledger copies across blockchain client upgrades. StatefulSets, in conjunction with PersistentVolumes and PersistentVolumeClaims (PVCs), enable durable storage attachment and guarantee data safety and continuity during container restarts caused by updates to their containing pods.

We found that we could create a configuration using native Kubernetes primitives that reduced the likelihood of getting slashed. Rather than use the "volumeClaimTemplates" feature of StatefulSets, we opted to explicitly define and apply PVC manifests and reference them directly in the "volumes" field of the StatefulSet's template specification. We configured the PVC's access mode as ReadWriteOnce, which allows the volume to be mounted as read-write by only one Kubernetes node. At this point, we nearly eliminated the possibility of having the same pod run on two different nodes at any given time. However, because this access mode permits multiple pods to access the volume when running on the same node, we needed to reduce the likelihood that one pod would be spun up on any given node to avoid the risk of slashing. To achieve this, we carefully defined node taints and resource requests and limits, creating a configuration that makes it highly unlikely to spin up two identical pods on the same node.

To further secure our staking nodes, we introduced Gatekeeper, an open-source policy controller for Kubernetes. Gatekeeper enabled us to enforce policies that prevented the replica count of our StatefulSets from being edited, providing an additional layer of protection against double signing.

By establishing Gatekeeper in our Kubernetes clusters, we achieved:

• Policy enforcement: Gatekeeper allowed us to define and enforce policies that prevented undesirable changes to our Kubernetes resources, such as increasing the replica count of StatefulSets.

• Audit capabilities: Gatekeeper's audit functionality provided insights into any policy violations, enabling us to monitor and maintain the security of our staking nodes.

• Extensibility: Gatekeeper's extensible design allowed us to create custom policies tailored to our specific requirements, ensuring the highest level of protection for our staking nodes.

Security and Compliance

Security was of paramount importance in our staking node deployment. To ensure the highest level of security, we implemented several best practices, including:

• Role-Based Access Control (RBAC): We used RBAC to manage access to our Kubernetes resources, ensuring that only authorized users could perform actions on our staking nodes. By defining roles and binding them to users, we achieved fine-grained access control, improving the overall security of our infrastructure.

• Network Policies: We implemented network policies to control the flow of traffic between our staking nodes and other services within our Kubernetes clusters. This approach allowed us to define rules for ingress and egress traffic, ensuring that only authorized connections were permitted.

• Security Contexts: We employed security contexts to define security settings for our staking node pods, such as running containers with a non-root user and enforcing read-only file systems. These security contexts enhanced the security posture of our staking nodes and reduced the attack surface.

Backup and Disaster Recovery

To protect our customers' staking rewards and ensure the availability of our staking nodes, we implemented robust backup and disaster recovery strategies. Some key components of our strategy included:

• Regular backups: We performed regular backups of our staking node data using our Global Blockchain Sync product. By backing up our data to offsite storage, we ensured that we could quickly recover from any data loss or corruption event.

• Multi-region deployment: To enhance the availability of our staking nodes, we enabled deployment across multiple regions in both GCP and AWS. This approach reduced the impact of regional outages and improved the overall reliability of our infrastructure.

Secret Manager

In our efforts to manage sensitive data within our Kubernetes clusters, we decided to build a custom secrets management system instead of relying on Kubernetes' Secret object type. Our decision was primarily driven by security concerns, as Kubernetes' Secrets would have stored sensitive information in plaintext within the etcd data store back in 2019. Instead, our custom solution provided granular access control and improved security by employing encryption-at-rest and in-transit to protect private keys and other sensitive data. 

We also implemented role-based access management for fine-grained control over access. One of the key benefits of our custom solution was its ability to seamlessly integrate with any blockchain client we deployed into our clusters, regardless of the team or language it was written in. This flexibility was crucial given the diversity of design philosophies and languages within the crypto ecosystem.

Maximizing Uptime

Initially, our engineers were uncertain about employing single-replica StatefulSets to reduce the likelihood of slashing, as it seemed to contradict Kubernetes' conventional scaling benefits. However, we found that the benefits of single-replica StatefulSets were well-suited for our particular use case. Specifically, the stable network identity and seamless integration with PersistentVolumes provided inherent advantages that aligned with our needs.

Our upgrade process demonstrated that even with one replica, the StatefulSet maintained the desired availability level. The true breakthrough occurred when we needed to revert to a previous version. Should any issues arise with a new release, we can rapidly roll back by restoring the previous pod configuration which mounts the same preserved state as the failed upgrade. For context, syncing an Ethereum node from scratch can take several days without a purpose-built blockchain snapshot system (e.g., Global Blockchain Sync).

If we had built automations around Kubernetes to deploy a replacement StatefulSet, we could have attained nearly seamless upgrades, but the trade-off would have been significant:

• Increased complexity: To ensure no double signing occurred, a mechanism would need to be built to prevent the new node from proposing a duplicate block before taking over as the primary validator.

• Increased upgrade time: With the necessity to wait for the blockchain to sync from scratch during every upgrade and rollback, maintaining single-replica StatefulSets would have prolonged the duration of each upgrade or outage by minutes, hours, or even days, depending on the blockchain protocol.

Coinbase prioritizes double signing prevention over downtime prevention due to its higher associated penalties. The advantages of swift rollbacks during failed upgrades outweigh the potential drawbacks, such as brief interruptions in block production. By only keeping one node running per validator deployment, we protect our customers' staked assets while maintaining minimal disruption.

Cloud-agnostic, Multi-region Deployments

In our efforts to achieve cloud-agnostic, multi-region deployments, we found that utilizing Kubernetes primitives such as StatefulSets, PersistentVolumeClaims, and ConfigMaps was the way to go. One of the major benefits of this approach was that it streamlined our integration into GCP and AWS. By leveraging Kubernetes primitives wherever possible, we were able to avoid the need for custom operators and CRDs that might have introduced complexity and created gaps between the different cloud environments — which was more of a concern in 2019 and is less so today.

To manage our Kubernetes infrastructure, we adopted Infrastructure as Code (IaC) and GitOps methodologies, streamlining our deployment process. Leveraging tools like Terraform, Atlantis, and Helm, we defined our infrastructure and application deployments in code, allowing us to review, test, and iterate on our infrastructure more efficiently. By embracing GitOps, we benefited from a declarative approach to managing our Kubernetes resources, which simplified our deployment process and increased our staking nodes' overall reliability.

Conclusion

Leveraging our automations, we successfully executed approximately 1,000 staking node upgrades per week, streamlining the process and reducing the risk of human error. Our CI/CD pipelines automated the building, testing, and deployment of our staking node upgrades, ensuring each upgrade met our quality standards. With automated rollouts, monitoring, alerting, and testing, we managed to quickly identify and address potential issues during and after upgrades.

By utilizing configuration management tools like Helm, we automated the deployment of our staking node configurations, ensuring consistency across our staking nodes during upgrades. Our commitment to automation has allowed us to scale our operations and deliver a seamless staking experience for our customers, regardless of the cloud provider they choose.

Operating staking nodes on Kubernetes since 2019 has been a journey of continuous innovation, learning, and growth. By leveraging Kubernetes primitives and building custom solutions, we have achieved cloud-agnostic deployment across GCP and AWS — ultimately fulfilling our commitment to decentralization. At the same time, our focus on security and preventing slashing has ensured the protection of our customers' staking rewards. As we continue to develop new technologies and refine our infrastructure, we remain dedicated to providing a secure, reliable, and seamless staking experience for our customers.