Automating Risk Model Retrain Loop with Agentic Skills

Financial fraud is a persistent and fast-evolving threat. To counter it, our risk models evaluate activity in real time and apply friction — such as step-up authentication or manual review — to high-risk events.

Static models fail as adversaries evolve their tactics to bypass defenses. Because new attack patterns often resemble legitimate behavior, sustainable defense requires constant retraining on the latest signals.

The challenge was not model complexity but manual effort. A single retraining cycle could take engineers several days to extract data from our data warehouse, analyze signals, and manage training jobs. In this post, we explain how we used Agentic Skills to automate this loop and what we learned along the way. Here, "Agentic Skills" refers to our internal, reusable workflows powered by coding agents, rather than any specific external product or framework.

Figure 1 — Before / After comparison 

We split the model enhancement process into three stages, each supported by a specialized automated skill:

We designed this structure so each stage produces a concrete artifact for human review before the next phase starts. This keeps humans in control: the agent cannot independently deploy models to production, and experts can adjust or stop the process at each gate. Our primary goal is to automate the labor-intensive coordination between stages while keeping expert judgment at the critical decision points.

Figure 2 — Three-step pipeline with human-review gates

Confirmed fraud incidents are reported on an ongoing basis. The first step is to separate incidents the model correctly identified — where we applied downstream friction — from those it did not. The latter being the more valuable signal for improving the model.

Technically, this means joining multiple analytics tables that include model outputs, incident labels, and user-friction outcomes. The logic is conceptually simple, but implementation is tricky due to large schemas and subtle column definitions. Initial queries generated by large language models (LLMs) often fail because of incorrect join keys or filters.

We developed this capability iteratively. In the first session, engineers provided coding agents with relevant table names and schemas to draft the query. Through manual checking and refinement, we converged on a robust query that reliably identifies a representative emerging trend cohort. We then formalized and persisted this process as an agentic skill.

The automated workflow is now triggered with a single command:

/emerging-trend-analysis start_date=... end_date=...

The skill handles its subtasks:

The summary document describes cohort characteristics, the estimated impact of the cohort, and candidate signals. It serves both as the main artifact for human review and as the input to the next phase.

Figure 3 — Step 1: Automated Gap Analysis Flow

Once the analysis is ready, we look for features in the feature store that could help the model catch similar cases in the future.

Although this task is open-ended, we followed the same skill-building pattern. The feature enrichment skill takes two main inputs: a link to the analysis and the current feature service definition. It then:

The output is a prioritized recommendation list, grouped into Tiers 1 through 4, with Tier 1 representing the strongest candidates for inclusion.

The core idea is simple: any model cannot learn from signals that are completely absent from the active feature set. Architecture tweaks and transformations help, but the fastest win is often to integrate underutilized features that target known gaps.

 Figure 4 — Step 2: Feature Enrichment Flow

Today this skill focuses on features already defined in the feature store; we are continually expanding our feature discovery capabilities. In a future post, we plan to describe an AI-driven feature discovery tool that builds on this workflow.

The third phase addresses the most resource-intensive part of the loop: running the training itself. Historically, this required manual work to kick off baseline training, monitor progress, collect metrics, and run hyperparameter sweeps.

The model retraining skill takes two primary inputs: a model configuration and dataset splits (training, validation, and testing). It then:

This automation allows the process to run to completion without manual oversight. Engineers can later review the tracker, identify the best-performing configurations based on selected metrics, and retrieve the associated artifacts.

Figure 5 — Step 3: Automated Model Retraining, Hyperparameter Sweep, and Metric Harvesting Flow

While building these three skills, a few patterns emerged.

A key practice is to stay engaged when the agent hits technical limits instead of taking the task back and doing it manually. We guide the agent through the hard parts while we’re still in the loop, then turn the outcome into a reusable skill. This upfront “torture test” is more expensive the first time, but it pays off in all future retraining cycles.

Figure 6 — Iterative Refinement to Reusable Skill

Each phase requires human verification before the next phase runs. This is not because the agent can’t handle individual tasks, but because the judgment required at these boundaries is complex and high-impact. The transitions—from gap analysis to feature enrichment, and then to model retraining—are natural points for humans to redirect, adjust inputs, or stop the workflow entirely.

For an agent to manage long-running tasks, it must be able to interpret execution results. Without this feedback loop, humans end up manually moving status and metrics between tools. Adding log-read capabilities for training jobs turned the model retraining phase from a supervised task into a truly autonomous sequence, without sacrificing observability.

By using specialized agentic skills, the Risk ML team turned our model retraining cycle into an automated three-stage loop: Emerging Trend Analysis, Feature Enrichment, and Model Retraining. The design removes manual coordination work, adds formal human-review checkpoints between automated steps, and uses log access to let agents manage long-running jobs end-to-end.

Converting complex workflows into persistent, reusable skills significantly increases the pace at which we can improve our defenses. This acceleration strengthens our security posture and supports Coinbase’s goal of being the most trusted cryptocurrency exchange.