Building a notification platform at Coinbase

Coinbase’s mission is to increase economic freedom in the world and the team works towards this mission by developing products that make it easier for customers to engage with the crypto economy. To deliver a superior experience to Coinbase users,  there is a cross-functional need to personalize how the Coinbase team communicates with its users beyond traditional channels such as email or push notifications. A notification platform that can communicate with Coinbase users while they’re actively using the Coinbase web or mobile app can serve this need by abstracting the complexity involved in message generation, delivery, and feedback. 

This article explains our journey towards building a notification platform — an ecosystem built upon the core notification service that makes the Coinbase user experience more dynamic, personalized, and real-time. This notification platform serves a wide variety of user needs including notifications like buys and sells, upcoming feature announcements, or incentive offers.

Principles

After understanding existing use cases, we identified a set of principles to guide our thinking about the features and evaluate trade-offs during the development process.

• Easy to integrate:

  • Inputs: Build systems with clear and simple APIs to make it easy for our teams to integrate. This allows other teams at Coinbase to automate their use cases while enabling developers to be early adopters. We create and publish developer documentation in line with the code repository along with examples they can use to onboard quickly.

  • Outputs: Enable other systems to listen and consume interesting events. This allows them to layer additional processing and execute custom actions for their use cases.

• Product agnostic:

  • The platform should be easy to extend to support notifications for existing and future Coinbase products. As an example, the concept of a user-id and the notification endpoints is different across Coinbase Mobile, Coinbase Wallet, and Coinbase NFT. 

• Experimentative:

  • Make it easy to experiment with different variations of the notification template by integrating with our in-house experimentation platform CIFER. 

• Reliable and scalable:

  • Build a reliable and scalable platform and provide guarantees appropriate to the use case. For example, critical notifications need a stricter guarantee of deliverability and latency than low-priority notifications. Notification load can change quickly and the system should be able to scale up or down seamlessly.

Architecture

The notification platform is built out of multiple building blocks as shown below and will be discussed in depth in the following sections. 

Each component is scoped to deliver well-defined functionality and exposes an API that can be used by other components or external services that need just that specific functionality. For example, services that need to send transactional notifications can call the notification service API directly.

Notification Service

The notification service is the workhorse of the Coinbase platform and provides a foundation for enhancing real-time communications between Coinbase and Coinbase customers across the board.  This solution needs to be precise enough to send customized content to each user, but also capable of scaling up to handle millions of notifications per minute. 

Since Coinbase has multiple products, the notification service needs to be product agnostic in how it targets to whom it sends messages or alerts. It also needs to handle sending notifications to everyone, from an institutional customer to a user on Coinbase Wallet. It is responsible for looking up user-specific information, enforcing rate limits and preference checks, substituting placeholders in the template to render the final message payload, and communicating with service providers to ensure message delivery. It is also responsible for collecting feedback from the downstream service providers on successful delivery, errors, and user interaction with the notifications when appropriate. 

Every notification serves a different purpose, so there needs to be the ability to customize and adjust delivery requirements. Some notifications are very time-sensitive, such as security alerts or transaction notifications, which need to be delivered immediately. Whereas other notifications like product announcements or weekly digests may not be as timely. To ensure time-sensitive notifications always have priority over less time sensitive notifications, the system has different queues — ranging from critically time sensitive to low priority. This allows us to control the scale for each priority and thus the throughput of each priority queue.

The preferences service is responsible for storing and retrieving user notification preferences. It is capable of supporting multiple preference models that vary across Coinbase products. The notification worker checks with the preferences service and adapts to permit or stop further processing of the request based on user’s updates to their notification preferences.

The communication storage service serves all pull-based notifications to our users. Some examples of these notifications are in-app notifications shown in the notification hub, banners and announcements on pull out drawers, and more. We call these “pull-based” notifications since users actively pull this content when using our apps on mobile or web. The core notification service routes all pull-based notification content to this service for storage. 

This integration allows us to seamlessly use the rest of the notification primitives like user segmentation and targeting to personalize content for our users. This service is built to be highly reliable and scale seamlessly to handle both write-heavy loads targeting millions of users and read-heavy traffic as users navigate the app’s surfaces. 

Targeting and Workflow

A common use case with notifications is to send a message to a set of users who are selected based on some criteria. The targeting component of the notification platform provides the tooling to identify users and send communications without the need to write code. It allows non-engineers to specify the targeting criteria, author a notification template, configure experiments, and launch them. 

The targeting criteria are run continuously to identify newly eligible users and send them notifications. This eliminates the need for teams to develop custom services or embed notification logic that is not core to their services. The decoupling and separation of concerns enable product teams to iterate faster while developing more maintainable services.

The workflow engine enables the creation of custom workflows that react to user events, execute actions, and transition across states. This enables use cases like omnichannel marketing like notifying a user about an incentive via an in-app notification, sending a confirmation email of a completed transaction, or reminding them via a popup before an incentive expires. Workflows are expressed as a DAG (Directed Acyclic Graph) in configuration files with support for experimentation and branching based on conditions (experiment membership, user attributes, or result of a prior action). Workflows are executed on temporal and triggered per user.

Content Platform and Consensus

The content platform makes it easy to author content included in the notification sent to a user. This is specific to a delivery channel and varies from simple textual content for SMS and Push notifications to complex formatted content like in an email or on-screen cards on the Coinbase mobile app. 

The content platform makes it simple for users to author multiple variants of the content and experiment with them. It is built on top of a headless content management system with a rich UI editor and allows users to preview before publishing their content.

The content platform integrates with the consensus service which blocks unvetted content from inadvertently being published. The consensus system requires approvals from at least one other reviewer and integrates with an internal role-based authorization system. We built this system to be extensible and capable of supporting consensus/approval workflow requirements of other internal systems. 

The content platform and consensus service together provide an easy to use authoring and publishing workflow that allows non-engineers to efficiently iterate on content and publish to production without engineering involvement.

Bringing it All Together - The Iris UI

Iris is our web application which ties together all services that make up the notification platform into a coherent experience for our internal teams to build notifications. Notification content is authored using a CMS application with a rich UI editor that publishes the template to the notification service. 

We have seen a significant increase in the adoption and usage of the notification platform after launching Iris. Users define SQL queries that select customers to target and tie that to templates, notification channels, and experiment configuration. The consensus workflow on Iris is used to review the targeting and notification configuration before the final launch. Iris reduces the time to build and launch notifications for internal teams while opening up this ability to non-technical users. 

Learnings

We built the various components of the notification platform over the past year but sequenced them in a manner that allowed our product teams to use their functionality as they shipped. This slow rollout has provided us with valuable feedback and informed requirements and design for other components. We identified some key learnings through this journey.

• API first Design: Design services with clear APIs that have a single purpose and don't make assumptions about how they are used. For example, all of our services come with command line tools that use these APIs to enable testing and debugging. The same APIs make it easy to connect internal admin user interfaces or integrate other applications in the overall system.

• Iterative Delivery: Ship an MVP with minimal dependencies so customers can start using it and provide feedback. For example, we shipped our first versions with configuration files checked into code for notification templates and targeting definitions. Though this limited usage to engineers, the deployments were simpler and iterations more efficient. We subsequently added functionality to improve the usability of the system by building a user interface and storing templates in a datastore. 

• Design for Reliability: Choose your SLOs in consultation with your customers and design your systems to easily scale up/down while ensuring the SLA guarantees promised to them. Studying the cost structure of your legacy components can help with identifying the top opportunities for cost savings which can factor into the design and technology choices made with the new components. For example, we were able to achieve significant cost savings by switching from a managed notification channel provider to using the underlying channels directly. We were able to reduce our costs by more than 50% while handling 10x more workload.

Future Work

By reducing the friction to build and experiment with notifications while building a scalable platform, we have seen a significant increase in adoption of our new notification platform. We continue to work to improve the notification experience and analytics and find new ways to make our services more efficient and improve reliability while reducing operational costs. 

Acknowledgements

We wouldn’t be able to build this platform without the contributions of our amazing engineers from the Growth Foundations team. We would like to thank the Growth engineering leadership for encouraging the development of the notification platform and providing ideas along the way to include use cases and build adoption. Finally, we would like to thank all the product engineering and marketing teams at Coinbase for adopting the platform and providing valuable feedback.