RSS Feeds

Samartha Chandrashekar & Nick Parker

You click play on a Friday night, and in the second that follows, dozens of services answer in milliseconds. Most of them run on Titus, our internal container platform. Titus is, in essence, a Kubernetes-based container hosting platform: we keep both the control plane and the data plane (the container hosts that actually run containers) patched and up to date, so application teams get a consistent and reliable container experience. They hand us a container, and we run it, with no host, cluster, or kernel to own. Titus launches millions of containers every week across tens of thousands of EC2 instances in multiple AWS regions. It is multi-tenant by default: many containers from many teams share each host. That density is what makes container compute efficient, and it is why Titus has been the default home for so much of Netflix’s compute for years.

For most workloads, sharing a host works well. For a few, it does not, and those few include some of the most demanding real-time systems we run. This post is about them, and about single-tenant Titus, a mode that gives a task its own virtual machine while keeping everything else about Titus the same: the API, the deployment path, and the managed experience where the Compute/Platform team, not the application team, owns the host.

We will walk through why shared-kernel isolation runs out of road for certain workloads, how we measure isolation rather than asserting it, the options we weighed (microVMs, gVisor, EC2, and more shared-kernel tuning) and why a dedicated VM inside Titus won, how the mode is built on the Titus federation layer and autoscaling, and what we have seen along three axes: isolation, fleet efficiency, and the operational cost of keeping the fleet patched.

The cost of good neighbors

Multi-tenancy is a tradeoff. Packing many containers onto a large bare-metal instance keeps utilization high, but it also means those containers share a single Linux kernel and the underlying hardware. Titus isolates them with the tools the kernel provides: cgroups cap CPU, memory, and disk; namespaces give each container its own view of the system; seccomp and AppArmor restrict the system calls a container can make; the Completely Fair Scheduler divides CPU time; the iocost controller meters disk IO; and traffic shaping meters each container’s slice of the network. Each container also gets its own identity, IAM role, and security groups, so tenants are isolated for security and resource management.

These mechanisms provide isolation for what the kernel can partition. They cannot fully partition what is shared in hardware. A group of cores shares a last-level cache and a path to memory, so a container that suddenly pulls a lot of data from RAM evicts a neighbor’s cache lines and slows it down, even when both stay inside their CPU limits. There is no cgroup for memory bandwidth or cache residency, as there is for CPU shares. The interference is real, it is hardware, and it does not show up as either container exceeding its quota.

A noisy neighbor and a co-located app each stay within their cgroup CPU and memory limits, but both draw on a shared last-level cache and memory bandwidth that lack cgroup fences. The noisy neighbor takes most of it, so the co-located app stalls and its tail latency rises even though no one exceeded a quota.

We have written about this before. Predictive CPU isolation of containers at Netflix moved some placement decisions out of the kernel and into a service that forecasts each container’s CPU demand and re-pins containers to cores to reduce cache interference, formulated as a combinatorial optimization solved every minute. Noisy Neighbor Detection with eBPF instruments the kernel scheduler to surface per-container run-queue latency, so we can see a victim being starved in close to real time. And Mount Mayhem traced container-startup slowness to a global kernel lock amplified by modern CPU cache topology. Each of these raised the floor for everyone on the platform, and we are continuing that work.

A decade of shared-kernel isolation, at Netflix and across the industry, has not made co-tenancy invisible, because a noisy neighbor is not one problem but a family of them. Almost every physical resource a host shares is its own axis of interference, and a cgroup can fence very few of them:

• CPU performance indeterminacy. Co-located threads contend for the run queue, so scheduling latency rises even while a container remains within its CPU quota. Separately, a neighbor with a large working set evicts a tenant’s last-level cache lines and saturates memory bandwidth, neither of which a CPU share can partition.
• Network bandwidth and packets per second. Per-container shaping competes for link bandwidth, and receives processing runs in softirq context, charging one container’s packet work to whatever CPU a neighbor is running on. A flood of small packets exhausts a core’s softirq budget long before any bandwidth cap is reached.
• Storage bandwidth and IOPS. A neighbor saturating a device’s queue serializes everyone’s I/O, and on cloud block storage, the per-volume and per-instance throughput ceilings are a shared budget in their own right.
• Long tail. Page-cache reclaim, NUMA and cross-socket interconnect traffic, TLB-shootdown interrupts, PCIe and GPU contention, package-level power and thermal throttling, and contention on kernel global locks (the class of problem behind Mount Mayhem) each degrade neighbors in their own way.

What unites them is that they are physical, shared, and below the level a quota can reach. A platform can cap how much CPU time or memory a container is granted; it cannot hand each tenant its own slice of the memory bus, its own softirq budget, or its own portion of the last-level cache.

Of these, the network dimension has bitten hardest in our experience. Per-container bandwidth shaping uses a hierarchical token bucket (HTB) queueing discipline that serializes traffic with a global lock to account for each container’s bytes and packets. Under a flood of small packets from a single container, contention for that lock can burn kernel CPU and starve co-located containers of the time they need to process their own traffic, resulting in TCP retransmits and tail-latency spikes on services that did nothing wrong. The mechanism meant to prevent noisy neighbors becomes one.

The workloads most exposed to this are stateful, high-throughput streaming jobs: real-time data pipelines and stream processors that continuously move large volumes of data and maintain in-memory state. When one of these is perturbed, recovery is not cheap. A stream processor that falls behind or loses its place may have to restore from a checkpoint and pause processing for many minutes to catch up. For these jobs, intermittent interference from a neighbor is not a small slowdown that averages out; it is a stop-the-world event that pages an on-call engineer. The same teams report, accurately, that they never saw this on a plain EC2 instance.

Why we kept coming back to the VM

Full VM isolation, where a workload gets its own host and kernel, is the standard our customers are already happy with. The question was not whether VM isolation works; it was how to offer it without giving up what makes Titus worth using. We weighed four paths.

1. Keep improving shared-kernel isolation. This helps every workload on the platform, needs no opt-in, and we are still investing in it (more on the roadmap below). Its limitation is structural. Each round of tuning narrows the gap without closing it, because the underlying hardware remains shared. Cache, memory bandwidth, and the kernel’s own shaping and scheduling locks are not things a container boundary can fully contain. Shared-kernel isolation is the right default and the wrong promise for a workload that cannot tolerate a bad minute.
2. microVMs (Firecracker, Kata Containers). Lightweight VMs boot in about 150 milliseconds by emulating far less hardware than traditional VMs, which is why they power serverless platforms. They were the most appealing alternative on paper, and we carefully reviewed them. Three things made the cost hard to justify. First, the hardware virtualization they rely on is available only on bare-metal instances or through nested virtualization, and nested virtualization was not ready on the instance types we run at Netflix at the time, so adopting microVMs broadly would have constrained us to a narrow slice of the fleet. Second, there is no managed microVM service we could build on, which means we would have to run and maintain a virtualization layer ourselves: patching the hypervisor, live-migrating guests, and qualifying emulated devices. That is exactly the kind of operational surface we prefer not to own when we can avoid it. Third, even setting those aside, microVMs add virtualization overhead on CPU- and IO-bound work and do not support GPUs. Excellent isolation, but at a price in flexibility and operational load we were not willing to pay.
3. gVisor. A user-space kernel that intercepts system calls provides sandbox-grade isolation without a hypervisor, thereby avoiding the bare-metal constraint. The catch is performance and dependency. Intercepting syscalls in user space adds overhead on IO-heavy paths, which are precisely the workloads we most need to protect, and the open-source project lags behind its internal counterpart in fixes and is effectively maintained by a single vendor. The isolation we would gain came with a throughput tax on the exact jobs we were trying to help.
4. Move the workload to EC2. It is possible to run containers directly on EC2 instances. The problem is that the application team takes back the operational load that Titus exists to carry: choosing instance types, patching hosts, and sizing and scaling the fleet. For a team that came to Titus precisely to avoid those things, this is a regression in the experience, and the move between Titus and EC2 has historically been a one-way door that teams are reluctant to cross.

Spectrum of isolation options from lighter and more shared to stronger isolation: shared-kernel containers, then gVisor, then microVMs (Firecracker or Kata), then a full VM as used by single-tenant Titus, which is highlighted. Each option is annotated with its launch latency, overhead, and the entity that operates it; single-tenant Titus is a dedicated EC2 instance with no additional virtualization overhead.

There is also a question of what kind of isolation we actually need. microVMs and gVisor are built for security isolation between mutually distrusting tenants, the case where one tenant might actively try to attack or escape into another. Among Netflix teams, that threat model is more than the situation warrants. What these workloads need is resource and performance isolation: a guarantee that a neighbor cannot steal their CPU, cache, memory bandwidth, or network, not protection from a hostile co-tenant. A dedicated EC2 VM delivers exactly that, with hardware we do not have to emulate and a kernel we do not have to sandbox. We pay for the isolation we need rather than for a stronger security boundary.

microVMs and gVisor asked for a large, bespoke investment with real risk in order to approximate something we could get directly: a VM. Using EC2 to run containers provides the VM, but takes away the managed experience. So we took the direct route and brought single-tenant VM isolation into Titus itself, under the existing API and the existing shared-responsibility model.

How single-tenant Titus works

The idea is simple to state: a single-tenant task runs on its own EC2 instance, sized to fit it, with nothing else scheduled alongside. That instance is still a Titus container host that the Compute/Platform team owns and patches. The team running the workload sees the same Titus API and the same deployment path. Nothing about how they package or ship the job changes.

Side-by-side anatomy of a multi-tenant Titus, where multiple teams’ containers share one Linux kernel and the hardware beneath it and contend with one another, versus a single-tenant node where one task plus its platform sidecars owns the whole machine with no co-tenants.

Most of the work happens in the federation layer, the part of Titus that decides where a job runs (which cell, which capacity) before it is scheduled onto a node.

When a job is eligible for single tenancy, three things happen.

1. Federation decides eligibility and the envelope. Eligibility is governed by an allowlist maintained by the Compute/Platform team; a team opts in by targeting a single-tenant workload group rather than flipping a flag in the public API. For an eligible task, federation tags it as single-tenant and records the set of instance families it may run on, ensuring consistent performance across the workload. Consolidating this decision in federation follows the same principle we use for routing in general: placement intelligence belongs in one place, above the individual cells, where it can account for capacity and availability across the region.
2. Autoscaling provisions a dedicated node. Given a pending single-tenant task, our autoscaling implementation launches the smallest instance that satisfies the task’s requested CPU, memory, and GPU from the allowed families for the task just in time. Taking the tightest fit keeps the unused remainder of each dedicated node small. The next section covers this in detail.
3. The scheduler enforces one task per node. Single-tenant pods are configured with pod anti-affinity to prevent colocation with other pods, along with a priority class that exempts the task from preemption. Single-tenant nodes are also tainted to prevent unrelated workloads from being placed there, with only single-tenant workloads tolerating this taint. We also disable defragmentation/repacking for these nodes, so a single-tenant task is never relocated to repack the fleet. Together, these mean a single-tenant task has its host to itself for its entire life, undisturbed by a neighbor or by the platform’s own bin-packing.

Two design choices are worth dwelling on. First, single tenancy is allowlisted on purpose. If it were a free checkbox in the API, most teams would tick it, because who wouldn’t want a quieter host? And the efficiency that multi-tenancy exists to provide would erode across the fleet. Keeping the Compute/Platform team in control lets us direct dedicated isolation to workloads that need it and steer the rest toward shared capacity, while keeping the public API surface small as we learn. Second, we do not silently resize anyone’s job to fill its node. The job keeps the resources it requested; we recommend sizing it close to a node shape so the dedicated instance is well utilized, while keeping the workload portable and running identically whether in single-tenant mode or not, regardless of the host’s dimensions. Moving in and out of the mode is a configuration change, not a rewrite.

A few perpetrators, a lot of collateral damage

Noisy-neighbor pain is not spread evenly. In practice, a small number of workloads cause most of the disruption: a job that floods the host with small packets, churns the page cache, or saturates memory bandwidth degrades the performance of every other container unlucky enough to share its host. Some of these are also among the workloads most affected by contention, because a job can be both a source of interference and sensitive to it. The shape of the problem is a few perpetrators and many victims.

That shape is what makes single tenancy a precise lever rather than a blunt one. We do not need to give every workload its own machine. We need to pull the handful of perpetrators and the most disruption-sensitive victims onto dedicated nodes. Doing so protects their former neighbors directly and removes the reason the platform was spreading sensitive workloads apart with anti-affinity placement in the first place. The hosts left behind are quieter, and because they no longer have to be kept partly empty to honor spread rules, they pack more densely.

Before-and-after view of the Titus fleet. Before: a few red perpetrators are scattered across shared hosts, the amber workloads beside them are disrupted, and anti-affinity spreading leaves dashed stranded capacity. After: each perpetrator runs on its own dedicated single-tenant node, the remaining shared hosts are fully loaded with healthy workloads, and one host’s worth of capacity is reclaimed.

On the left is the fleet before: perpetrators sit on shared hosts, the workloads next to them suffer, and anti-affinity rules leave gaps that cannot be filled. On the right is the same fleet after: each perpetrator has its own right-sized node where it can be as noisy as it likes, with no one to disturb; the remaining hosts run full of undisturbed workloads; and the capacity that used to be stranded comes back. A few dedicated nodes buy isolation for the perpetrators and a calmer, denser fleet for everyone else.

Provisioning many small nodes

Single tenancy trades fewer large, shared instances for more small, dedicated ones, and that shift has consequences across the stack.

The first is the control-plane load. More nodes mean more for the Titus control plane to track and more VM launches against the EC2 control plane, the latter mattering most during a regional failover, when we scale up the fastest and can least afford a throttle. Titus is already a federated system: each region is divided into independent cells, which limit how large any single control plane can get, and let us add capacity by adding cells rather than growing a single stack without limit. We migrate workloads in stages rather than all at once, and we exercise the launch-rate path under controlled load on a schedule rather than discovering its limits during an incident.

The second consequence is capacity management, where the design changed the most. With one task per node, the fleet now spans many instance shapes whose demand shifts continuously as jobs come and go. Pre-staging static pools of every shape would be slow to manage and wasteful, so we instead provision single-tenant nodes on demand. It handles provisioning end-to-end, and several of its features line up with exactly what single tenancy needs:

• Just-in-time, right-sized provisioning. Single-tenant jobs are configured with a list of supported instance families based on their performance requirements. At launch, they are mapped to the best-fit node size by comparing requested resources against the selected instance families. New nodes of that instance type are brought up by the autoscaler to meet the pending demand. Single-tenant pods are configured with anti-affinity to all other pods, so the scheduler naturally lands one pod per autoscaled node. Newly vacated nodes may be reused by any pending deploys before they’re automatically torn down. The node pools themselves are shared across single-tenant workloads and describe a catalog of available instance families and sizes.
• Single tenant node pools. Single tenant may run in separate clusters or be shared with multitenant clusters. The pools are configured with a list of supported instance-type permutations and are pinned to a Titus container host AMI. This is where the managed experience lives: the host image and its configuration are ours, declared once and applied to every node created.
• Consolidation management. In multi-tenant environments, we default to a repack/defrag behavior that periodically relocates workloads. This doesn’t apply in single tenant, where nodes are explicitly each running at most one workload. We therefore reclaim only nodes vacated by workloads that have scaled in or terminated. Node termination is performed in two stages, allowing nodes to be reused for pending workloads before they are fully terminated.
• Drift-based patching. When we publish a new container host AMI, we support a “mutable” deploy to existing nodes without affecting any hosted containers, and an “immutable” deploy that updates the target AMI for newly created nodes. This means that many node upgrades don’t require disrupting workloads, but we reserve the right to roll the fleet with our users if needed.

The launch-latency tax, and why warm pools are harder here

Provisioning on demand does not make a node appear instantly, and launch latency is the tradeoff single tenancy asks us to manage most carefully. Bringing a single-tenant task to the point where it can serve traffic has two parts: standing the node up (booting a fresh EC2 instance, then pulling and extracting the container image, which is the largest part of host setup for most jobs) and the application’s own startup (a JVM loading and JIT-compiling its classes, a cache filling, and so on). A fresh instance takes tens of seconds before the image pull even begins.

In multi-tenant Titus, most of this is hidden. A new task usually lands on a Titus container host that is already running and already has the image layers cached, so starting it takes only a few seconds rather than minutes; the host boot was paid for once, long ago, and amortized across every task that has run there since. Single tenancy gives up that hiding place. Every single-tenant task waits for its own machine to boot and pull its image.

The usual remedy is a warm pool: keep some instances booted and idle so a pending task lands on one immediately. This works well for the multi-tenant fleet precisely because that fleet is close to homogeneous. A handful of large shapes cover almost everything, since any task can be packed onto a big shared host. A small amount of warm capacity in a few shapes gives a high hit rate, and its cost is spread across the many tasks that will use it.

Single tenancy inverts those economics, and this is the part worth being explicit about.

Press enter or click to view image in full size

Because every task gets a right-sized node, the fleet spans many instance shapes (several families across many sizes), and a warm instance only helps a pending task whose shape matches it. A warm r7i.2xlarge does nothing for a task that needs an m7a.8xlarge. To achieve a good hit rate, we would need to hold idle capacity in every family-and-size bucket, and idle capacity in a bucket with no task to fill is pure cost. The very property that makes single tenancy efficient at steady state, tight best-fit sizing, works against warm-pool reuse: more distinct shapes mean more buckets and a lower hit rate per dollar of standing capacity. And the pressure peaks exactly when it is least welcome. During a regional failover or a live event, when many tasks need fresh nodes at once, a thin warm pool spread across many shapes drains immediately, and the rest of the demand pays the full cold-boot cost.

So we treat warm capacity as a small, targeted lever rather than a default. We keep a little pre-warmed capacity for the most latency-sensitive single-tenant starts, and we deliberately narrow the menu of shapes to a curated set of sizes rather than the full continuum, so warm capacity concentrates into fewer buckets and trades a little best-fit efficiency for a better hit rate. Everything else provisions just-in-time and accepts the boot.

The more durable wins are in cutting the launch cost itself rather than pre-paying for it, and they help both single- and multi-tenant nodes, which is the better place to spend the effort:

• Faster image distribution. The image pull dominates the host setup, so newer OCI image formats and lazy-pulling snapshotters such as stargz and SOCI, which fetch only the bytes a container actually reads at startup rather than the whole image up front, reduce the largest component of the gap.
• Checkpoint and restore. For processes that are slow to initialize, snapshotting a warmed-up application with a tool like CRIU and restoring it, rather than cold-starting it, removes most of the second half of launch latency.
• Instance type selection. In multitenant configurations, we have historically used metal instances, which offer slightly better performance but come with a slower startup time. In a single-tenant configuration, node startup latency is more important, so we have moved away from these instances.
• Scheduling optimization. Through regular stress testing of fleet scaling, we found a scheduling bottleneck caused by Kubernetes preemption logic, leading to nodes being scaled up quickly but then left idle while waiting for workloads to be scheduled on them. This was ultimately unnecessary, as our single-tenant pools do not run preemptible workloads. Disabling preemption greatly improved scheduling performance when under strain.

None of this makes the tradeoff disappear, and it is worth stating plainly: a single-tenant node is slower to stand up than a slot on an already-warm shared host, and hiding that with warm pools costs more per node here than it does on the multi-tenant fleet, because the warm pool cannot be as homogeneous. But for the long-running services that single tenancy targets, paying that cost once at launch in exchange for a quiet life afterward is a trade these teams are glad to make.

Measuring the difference

Isolation claims are easy to assert and hard to trust, so we measure them with a repeatable benchmark and a single score.

The method is a differential one. We run a reference container, a healthy and representative application, and record its throughput and tail latency while it runs alone. Then we run it again next to a havoc container engineered to be a bad neighbor: a UDP flood through XDP to stress the network-shaping path, pointer-chasing and STREAM to pollute the last-level cache and saturate memory bandwidth, fsync storms against the disk, stress-ng to abuse the scheduler, and GPU-burn to monopolize an accelerator. We only trust a run once the host has reached a steady state, which we confirm with the kernel’s Pressure Stall Information: we wait until cpu_some or mem_some shows that tasks are actually stalling on a contended resource, not merely that the resource is busy. Each scenario is run many times with fixed seeds, and we report the median with a bootstrapped confidence interval, because noisy-neighbor distributions are heavy-tailed and a simple average can mask the spikes that matter.

From the reference container’s numbers, we derive a simple measure of how well isolation held. Throughput efficiency is the fraction of operations per second the app retains under attack; latency efficiency is the ratio of its undisturbed P99 to its P99 under pressure. A neighbor that doubles the reference app’s P99 drops the latency efficiency. As an illustration, a reference app whose P99 latency goes down next to a havoc container achieves a certain latency efficiency, and one whose throughput falls from a to b operations per second achieves a certain throughput efficiency. The composite score is a weighted blend of the two, with a latency-sensitive preset that gives greater weight to tail latency for interactive and streaming-path services. A perfectly isolated system scores 100%, and a single hard failure invalidates the run regardless of the average.

Noisy-neighbor isolation

cgroups container isolation isn’t perfect, and applications may experience worse performance due to sharing a machine with a determined neighbor. This could extend to disk and/or network IO where the same volumes or interfaces may be shared across workloads. For a distributed workload, this can lead to measurable inconsistencies in performance depending on where the replicas are placed. For more sensitive workloads, this variance can mean the difference between stability and getting paged.

The single-tenant option makes it possible to better isolate workloads that cause problems for their neighbors and/or are more sensitive to such problems. This can be used as a stopgap until the specific regression problem is resolved and consistency is restored, or kept permanent for load patterns that aren’t worth supporting in a shared environment.

Efficiency

The second effect is fleet efficiency, and it is the one people expect to go the wrong way. Dedicating a node to a single task consumes that node’s slack, so the naive expectation is that utilization falls. In aggregate, it can move the other way, for a reason related to how we dealt with poor isolation before single tenancy existed.

To protect sensitive workloads on shared hosts, the platform used anti-affinity placement, scattering them across the fleet to reduce colocation of the same workload on the same nodes. Spreading fights bin-packing: it strains capacity, because a host kept partly empty to honor an anti-affinity rule cannot be filled by a workload that would otherwise have fit there. When isolation-sensitive workloads move to dedicated nodes, the need for spreading and fragmentation goes away, and the remaining multi-tenant fleet can pack more densely.

Another source of fragmentation is services that are sensitive to disruption. These are typically stateful services where recovery is expensive. As these services become more common in a given population, they can lead to nodes being held up by pods with strict disruption requirements. This can lead to an inability to defrag and recompact the pool during off-peak times. The symptoms and resulting drop in fleet efficiency caused by disruption-sensitive workloads mirror what’s observed for workloads with anti-affinity rules.

Depending on the magnitude of the anti-affinity and/or disruption budget problem, moving such workloads to single-tenant pools can result in equivalent or even improved placement when all costs have been accounted for. It also becomes easier to quantify the cost to end-users. Rather than trying to attribute fleet-wide fragmentation to individual workloads, we can directly quantify waste as single-tenant node overhead.

The theory and the practice diverge here, and it was the practice that decided it. In principle, the efficiency ceiling is higher for multi-tenant bin-packing: pack enough tasks together, and a host fills almost completely. We do not reach that ceiling. AWS offers a discrete set of instance sizes, so a large shared host rarely matches the exact sum of the requests we place on it, and the gap between a host’s capacity and what we can actually pack onto it is a bin-packing hole we cannot fill. At the granularity of large instances, those holes are sizable. Right-sizing an instance to a single workload, picking the closest fit from many shapes instead of draping many workloads over one big host, captured more of the available efficiency in our experience than the higher theoretical ceiling ever delivered. The clean theory favored multi-tenancy; our measured fleet favored right-sizing.

Container-host maintenance

The third effect is operational, and it shows up in how we keep the fleet patched. A multi-tenant host is hard to take down for maintenance precisely because it is full: draining it means relocating many tenants at once, several of which carry disruption budgets that limit how fast they can move, so a routine patch becomes a careful, coordinated evacuation. A single-tenant host runs exactly one task, with consolidation and defragmentation turned off, so draining it means relocating only the one workload, and we can automatically handle that when a new AMI ships. The remaining multi-tenant hosts are easier to maintain, too, because the workloads that were most sensitive to disruption are no longer stranded on them.

What it changes, and what it doesn’t

Single-tenant mode is a configuration, not a migration. A team moves a workload onto a dedicated node, and back, with a labeling change, without rewriting anything and without leaving Titus. That turns the isolation decision into a reversible setting, which is the opposite of the one-way door between Titus and EC2 that teams used to face. A team can try single tenancy for a quarter, measure it, and step back if the trade-off isn’t worth it for them.

What does not change is the contract. The Compute/Platform team still selects the instance type, patches the host, and sizes and scales the fleet underneath. The application team still uses the Titus API, the same delivery pipeline, and the same observability. Single tenancy is a new point on the isolation axis offered within the existing managed experience, not a separate product that the team has to learn and operate.

When a dedicated node is the wrong call

A dedicated node is a real cost, and single tenancy is not free isolation. Two things are worth weighing before reaching for it.

First, a single-tenant task does not get the whole machine; it gets the machine minus overhead. Every Titus container host, dedicated or shared, runs the platform sidecars, system services, the kubelet, and the host environment itself, and that overhead is a roughly fixed slice of CPU and a few gigabytes of memory. On a shared host, that slice is paid once and amortized across the many tasks running there. On a dedicated node, a single task handles it. Because the cost is fixed rather than proportional, it is punishing as a fraction of a small instance and negligible on a large one: a small dedicated node can lose tens of percent of its resources to overhead, while on the largest shapes it rounds to a couple of percent.

Second, because a task is placed on the smallest instance that fits it, there is usually a gap between what the task asked for and the next available shape, and on a dedicated node, that gap is pure waste, with no neighbors to soak it up. The two effects compound: fixed overhead plus the best-fit remainder set a ceiling on how much of a dedicated node a workload can actually use, and that ceiling is lowest exactly where people are tempted to reach for isolation casually, on small instances. However, this sizing waste can be reduced through coordination with the service owner as needed.

The guidance follows from the arithmetic. Single tenancy pays off for large, long-running, performance-sensitive, disruption-sensitive workloads, sized close to a node shape and run as fewer, larger replicas rather than many small ones. It is the wrong call for small, bursty, or highly elastic workloads, where the overhead fraction is high, the dedicated node sits mostly idle, and multi-tenant packing is both cheaper and faster to scale. This is also why the mode is allowlisted rather than a self-serve checkbox: the platform can keep workloads on the side of that line where they belong, and leave the ones that would gain little from a node of their own on the shared fleet, where they are cheaper to run.

Who’s using it

The first adopters are the workloads that motivated the work: real-time data infrastructure. Managed Apache Flink stream processing, streaming data connectors, and telemetry and logging pipelines were both the most frequent sources of network contention and the most damaged by it, because their recovery is expensive and their disruption budgets are tight. These are stateful, high-throughput jobs where a few seconds of starvation can cascade into minutes of catch-up, and where the platform’s own repacking used to be a source of disruption in its own right.

Moving them onto dedicated nodes removed the class of problem rather than mitigating it. The teams running them report stable operation, having previously seen intermittent, hard-to-attribute interference, and their jobs are no longer relocated by fleet repacking, removing a second, self-inflicted source of disruption. Latency-sensitive serving paths and a handful of GraphQL and edge services have followed for the same reason: a predictable tail matters more to them than a few points of bin-packing efficiency.

This is an ongoing rollout, not a finished one. We are widening the set of eligible workloads as we learn, using the largest production jobs as the stress test for the platform’s scaling assumptions, and watching the control-plane and EC2 launch-rate metrics as the count of small nodes grows. We have also been running periodic stress tests to exercise scaling ahead of real user workload, and making improvements to autoscaling and scheduling logic along the way.

What’s next

We are not done on either side of the tradeoff. On single tenancy, we are reducing launch latency with smarter pre-warming of common shapes, broadening the allowlist as scaling data holds up, and extending the model beyond long-running services as cost attribution and capacity stories mature. On the multi-tenant side, which remains the right default for most workloads, we are addressing the dimensions that hurt the most. The largest is the network: replacing HTB’s global-lock shaping with an eBPF-based Expected Departure Time approach that rate-limits by timestamping packets rather than contending on a lock, and pairing it with a lower-overhead container network device. We are also evaluating hardware cache and memory-bandwidth partitioning (Intel RDT and AMD PQoS) to give the kernel a real lever on the last-level cache, hardware GPU partitioning (NVIDIA MIG) for accelerator isolation, and continued work on the cgroup v2 IO controller for disk. This isolation benchmark lets us rank these by return on effort rather than by intuition, and show customers, in the numbers, that the shared fleet is getting quieter over time.

Single-tenant Titus is also a step toward something larger. Netflix’s Platform teams offer two ways to get compute: virtual machines on EC2 & containers on Titus. We want to replace that with a single Compute API that takes intent rather than mechanism into account: how much isolation a workload needs, how quickly it must launch, how long it can run without disruption, and what business domain it belongs to. The platform then makes placement and runtime choices, and can make globally optimal ones because it sees the whole picture.

Conclusion

Multi-tenancy is the right default for a container platform, and we will keep making it better for everyone. But the right default is not right for everything. Single-tenant Titus gives the workloads that need it the isolation of a dedicated VM, provisioned and patched by our existing node management, under the same API and the same managed experience, without taking away the freedom to change their mind. We can now point to the difference rather than describe it: a tail that stops moving under a noisy neighbor, a fleet that packs more densely once spreading is retired, and hosts that are faster and safer to patch. For the streaming and data systems at the heart of Netflix’s real-time platform, a room of one’s own turned out to be the feature that mattered.

Single-tenant Titus: VM-level isolation on a multi-tenant container platform was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Samartha Chandrashekar & Nick Parker

You click play on a Friday night, and in the second that follows, dozens of services answer in milliseconds. Most of them run on Titus, our internal container platform. Titus is, in essence, a Kubernetes-based container hosting platform: we keep both the control plane and the data plane (the container hosts that actually run containers) patched and up to date, so application teams get a consistent and reliable container experience. They hand us a container, and we run it, with no host, cluster, or kernel to own. Titus launches millions of containers every week across tens of thousands of EC2 instances in multiple AWS regions. It is multi-tenant by default: many containers from many teams share each host. That density is what makes container compute efficient, and it is why Titus has been the default home for so much of Netflix’s compute for years.

For most workloads, sharing a host works well. For a few, it does not, and those few include some of the most demanding real-time systems we run. This post is about them, and about single-tenant Titus, a mode that gives a task its own virtual machine while keeping everything else about Titus the same: the API, the deployment path, and the managed experience where the Compute/Platform team, not the application team, owns the host.

We will walk through why shared-kernel isolation runs out of road for certain workloads, how we measure isolation rather than asserting it, the options we weighed (microVMs, gVisor, EC2, and more shared-kernel tuning) and why a dedicated VM inside Titus won, how the mode is built on the Titus federation layer and autoscaling, and what we have seen along three axes: isolation, fleet efficiency, and the operational cost of keeping the fleet patched.

The cost of good neighbors

Multi-tenancy is a tradeoff. Packing many containers onto a large bare-metal instance keeps utilization high, but it also means those containers share a single Linux kernel and the underlying hardware. Titus isolates them with the tools the kernel provides: cgroups cap CPU, memory, and disk; namespaces give each container its own view of the system; seccomp and AppArmor restrict the system calls a container can make; the Completely Fair Scheduler divides CPU time; the iocost controller meters disk IO; and traffic shaping meters each container’s slice of the network. Each container also gets its own identity, IAM role, and security groups, so tenants are isolated for security and resource management.

These mechanisms provide isolation for what the kernel can partition. They cannot fully partition what is shared in hardware. A group of cores shares a last-level cache and a path to memory, so a container that suddenly pulls a lot of data from RAM evicts a neighbor’s cache lines and slows it down, even when both stay inside their CPU limits. There is no cgroup for memory bandwidth or cache residency, as there is for CPU shares. The interference is real, it is hardware, and it does not show up as either container exceeding its quota.

We have written about this before. Predictive CPU isolation of containers at Netflix moved some placement decisions out of the kernel and into a service that forecasts each container’s CPU demand and re-pins containers to cores to reduce cache interference, formulated as a combinatorial optimization solved every minute. Noisy Neighbor Detection with eBPF instruments the kernel scheduler to surface per-container run-queue latency, so we can see a victim being starved in close to real time. And Mount Mayhem traced container-startup slowness to a global kernel lock amplified by modern CPU cache topology. Each of these raised the floor for everyone on the platform, and we are continuing that work.

A decade of shared-kernel isolation, at Netflix and across the industry, has not made co-tenancy invisible, because a noisy neighbor is not one problem but a family of them. Almost every physical resource a host shares is its own axis of interference, and a cgroup can fence very few of them:

• CPU performance indeterminacy. Co-located threads contend for the run queue, so scheduling latency rises even while a container remains within its CPU quota. Separately, a neighbor with a large working set evicts a tenant’s last-level cache lines and saturates memory bandwidth, neither of which a CPU share can partition.
• Network bandwidth and packets per second. Per-container shaping competes for link bandwidth, and receives processing runs in softirq context, charging one container’s packet work to whatever CPU a neighbor is running on. A flood of small packets exhausts a core’s softirq budget long before any bandwidth cap is reached.
• Storage bandwidth and IOPS. A neighbor saturating a device’s queue serializes everyone’s I/O, and on cloud block storage, the per-volume and per-instance throughput ceilings are a shared budget in their own right.
• Long tail. Page-cache reclaim, NUMA and cross-socket interconnect traffic, TLB-shootdown interrupts, PCIe and GPU contention, package-level power and thermal throttling, and contention on kernel global locks (the class of problem behind Mount Mayhem) each degrade neighbors in their own way.

What unites them is that they are physical, shared, and below the level a quota can reach. A platform can cap how much CPU time or memory a container is granted; it cannot hand each tenant its own slice of the memory bus, its own softirq budget, or its own portion of the last-level cache.

Of these, the network dimension has bitten hardest in our experience. Per-container bandwidth shaping uses a hierarchical token bucket (HTB) queueing discipline that serializes traffic with a global lock to account for each container’s bytes and packets. Under a flood of small packets from a single container, contention for that lock can burn kernel CPU and starve co-located containers of the time they need to process their own traffic, resulting in TCP retransmits and tail-latency spikes on services that did nothing wrong. The mechanism meant to prevent noisy neighbors becomes one.

The workloads most exposed to this are stateful, high-throughput streaming jobs: real-time data pipelines and stream processors that continuously move large volumes of data and maintain in-memory state. When one of these is perturbed, recovery is not cheap. A stream processor that falls behind or loses its place may have to restore from a checkpoint and pause processing for many minutes to catch up. For these jobs, intermittent interference from a neighbor is not a small slowdown that averages out; it is a stop-the-world event that pages an on-call engineer. The same teams report, accurately, that they never saw this on a plain EC2 instance.

Why we kept coming back to the VM

Full VM isolation, where a workload gets its own host and kernel, is the standard our customers are already happy with. The question was not whether VM isolation works; it was how to offer it without giving up what makes Titus worth using. We weighed four paths.

1. Keep improving shared-kernel isolation. This helps every workload on the platform, needs no opt-in, and we are still investing in it (more on the roadmap below). Its limitation is structural. Each round of tuning narrows the gap without closing it, because the underlying hardware remains shared. Cache, memory bandwidth, and the kernel’s own shaping and scheduling locks are not things a container boundary can fully contain. Shared-kernel isolation is the right default and the wrong promise for a workload that cannot tolerate a bad minute.
2. microVMs (Firecracker, Kata Containers). Lightweight VMs boot in about 150 milliseconds by emulating far less hardware than traditional VMs, which is why they power serverless platforms. They were the most appealing alternative on paper, and we carefully reviewed them. Three things made the cost hard to justify. First, the hardware virtualization they rely on is available only on bare-metal instances or through nested virtualization, and nested virtualization was not ready on the instance types we run at Netflix at the time, so adopting microVMs broadly would have constrained us to a narrow slice of the fleet. Second, there is no managed microVM service we could build on, which means we would have to run and maintain a virtualization layer ourselves: patching the hypervisor, live-migrating guests, and qualifying emulated devices. That is exactly the kind of operational surface we prefer not to own when we can avoid it. Third, even setting those aside, microVMs add virtualization overhead on CPU- and IO-bound work and do not support GPUs. Excellent isolation, but at a price in flexibility and operational load we were not willing to pay.
3. gVisor. A user-space kernel that intercepts system calls provides sandbox-grade isolation without a hypervisor, thereby avoiding the bare-metal constraint. The catch is performance and dependency. Intercepting syscalls in user space adds overhead on IO-heavy paths, which are precisely the workloads we most need to protect, and the open-source project lags behind its internal counterpart in fixes and is effectively maintained by a single vendor. The isolation we would gain came with a throughput tax on the exact jobs we were trying to help.
4. Move the workload to EC2. It is possible to run containers directly on EC2 instances. The problem is that the application team takes back the operational load that Titus exists to carry: choosing instance types, patching hosts, and sizing and scaling the fleet. For a team that came to Titus precisely to avoid those things, this is a regression in the experience, and the move between Titus and EC2 has historically been a one-way door that teams are reluctant to cross.

There is also a question of what kind of isolation we actually need. microVMs and gVisor are built for security isolation between mutually distrusting tenants, the case where one tenant might actively try to attack or escape into another. Among Netflix teams, that threat model is more than the situation warrants. What these workloads need is resource and performance isolation: a guarantee that a neighbor cannot steal their CPU, cache, memory bandwidth, or network, not protection from a hostile co-tenant. A dedicated EC2 VM delivers exactly that, with hardware we do not have to emulate and a kernel we do not have to sandbox. We pay for the isolation we need rather than for a stronger security boundary.

microVMs and gVisor asked for a large, bespoke investment with real risk in order to approximate something we could get directly: a VM. Using EC2 to run containers provides the VM, but takes away the managed experience. So we took the direct route and brought single-tenant VM isolation into Titus itself, under the existing API and the existing shared-responsibility model.

How single-tenant Titus works

The idea is simple to state: a single-tenant task runs on its own EC2 instance, sized to fit it, with nothing else scheduled alongside. That instance is still a Titus container host that the Compute/Platform team owns and patches. The team running the workload sees the same Titus API and the same deployment path. Nothing about how they package or ship the job changes.

Most of the work happens in the federation layer, the part of Titus that decides where a job runs (which cell, which capacity) before it is scheduled onto a node.

When a job is eligible for single tenancy, three things happen.

1. Federation decides eligibility and the envelope. Eligibility is governed by an allowlist maintained by the Compute/Platform team; a team opts in by targeting a single-tenant workload group rather than flipping a flag in the public API. For an eligible task, federation tags it as single-tenant and records the set of instance families it may run on, ensuring consistent performance across the workload. Consolidating this decision in federation follows the same principle we use for routing in general: placement intelligence belongs in one place, above the individual cells, where it can account for capacity and availability across the region.
2. Autoscaling provisions a dedicated node. Given a pending single-tenant task, our autoscaling implementation launches the smallest instance that satisfies the task’s requested CPU, memory, and GPU from the allowed families for the task just in time. Taking the tightest fit keeps the unused remainder of each dedicated node small. The next section covers this in detail.
3. The scheduler enforces one task per node. Single-tenant pods are configured with pod anti-affinity to prevent colocation with other pods, along with a priority class that exempts the task from preemption. Single-tenant nodes are also tainted to prevent unrelated workloads from being placed there, with only single-tenant workloads tolerating this taint. We also disable defragmentation/repacking for these nodes, so a single-tenant task is never relocated to repack the fleet. Together, these mean a single-tenant task has its host to itself for its entire life, undisturbed by a neighbor or by the platform’s own bin-packing.

Two design choices are worth dwelling on. First, single tenancy is allowlisted on purpose. If it were a free checkbox in the API, most teams would tick it, because who wouldn’t want a quieter host? And the efficiency that multi-tenancy exists to provide would erode across the fleet. Keeping the Compute/Platform team in control lets us direct dedicated isolation to workloads that need it and steer the rest toward shared capacity, while keeping the public API surface small as we learn. Second, we do not silently resize anyone’s job to fill its node. The job keeps the resources it requested; we recommend sizing it close to a node shape so the dedicated instance is well utilized, while keeping the workload portable and running identically whether in single-tenant mode or not, regardless of the host’s dimensions. Moving in and out of the mode is a configuration change, not a rewrite.

A few perpetrators, a lot of collateral damage

Noisy-neighbor pain is not spread evenly. In practice, a small number of workloads cause most of the disruption: a job that floods the host with small packets, churns the page cache, or saturates memory bandwidth degrades the performance of every other container unlucky enough to share its host. Some of these are also among the workloads most affected by contention, because a job can be both a source of interference and sensitive to it. The shape of the problem is a few perpetrators and many victims.

That shape is what makes single tenancy a precise lever rather than a blunt one. We do not need to give every workload its own machine. We need to pull the handful of perpetrators and the most disruption-sensitive victims onto dedicated nodes. Doing so protects their former neighbors directly and removes the reason the platform was spreading sensitive workloads apart with anti-affinity placement in the first place. The hosts left behind are quieter, and because they no longer have to be kept partly empty to honor spread rules, they pack more densely.

On the left is the fleet before: perpetrators sit on shared hosts, the workloads next to them suffer, and anti-affinity rules leave gaps that cannot be filled. On the right is the same fleet after: each perpetrator has its own right-sized node where it can be as noisy as it likes, with no one to disturb; the remaining hosts run full of undisturbed workloads; and the capacity that used to be stranded comes back. A few dedicated nodes buy isolation for the perpetrators and a calmer, denser fleet for everyone else.

Provisioning many small nodes

Single tenancy trades fewer large, shared instances for more small, dedicated ones, and that shift has consequences across the stack.

The first is the control-plane load. More nodes mean more for the Titus control plane to track and more VM launches against the EC2 control plane, the latter mattering most during a regional failover, when we scale up the fastest and can least afford a throttle. Titus is already a federated system: each region is divided into independent cells, which limit how large any single control plane can get, and let us add capacity by adding cells rather than growing a single stack without limit. We migrate workloads in stages rather than all at once, and we exercise the launch-rate path under controlled load on a schedule rather than discovering its limits during an incident.

The second consequence is capacity management, where the design changed the most. With one task per node, the fleet now spans many instance shapes whose demand shifts continuously as jobs come and go. Pre-staging static pools of every shape would be slow to manage and wasteful, so we instead provision single-tenant nodes on demand. It handles provisioning end-to-end, and several of its features line up with exactly what single tenancy needs:

• Just-in-time, right-sized provisioning. Single-tenant jobs are configured with a list of supported instance families based on their performance requirements. At launch, they are mapped to the best-fit node size by comparing requested resources against the selected instance families. New nodes of that instance type are brought up by the autoscaler to meet the pending demand. Single-tenant pods are configured with anti-affinity to all other pods, so the scheduler naturally lands one pod per autoscaled node. Newly vacated nodes may be reused by any pending deploys before they’re automatically torn down. The node pools themselves are shared across single-tenant workloads and describe a catalog of available instance families and sizes.
• Single tenant node pools. Single tenant may run in separate clusters or be shared with multitenant clusters. The pools are configured with a list of supported instance-type permutations and are pinned to a Titus container host AMI. This is where the managed experience lives: the host image and its configuration are ours, declared once and applied to every node created.
• Consolidation management. In multi-tenant environments, we default to a repack/defrag behavior that periodically relocates workloads. This doesn’t apply in single tenant, where nodes are explicitly each running at most one workload. We therefore reclaim only nodes vacated by workloads that have scaled in or terminated. Node termination is performed in two stages, allowing nodes to be reused for pending workloads before they are fully terminated.
• Drift-based patching. When we publish a new container host AMI, we support a “mutable” deploy to existing nodes without affecting any hosted containers, and an “immutable” deploy that updates the target AMI for newly created nodes. This means that many node upgrades don’t require disrupting workloads, but we reserve the right to roll the fleet with our users if needed.

The launch-latency tax, and why warm pools are harder here

Provisioning on demand does not make a node appear instantly, and launch latency is the tradeoff single tenancy asks us to manage most carefully. Bringing a single-tenant task to the point where it can serve traffic has two parts: standing the node up (booting a fresh EC2 instance, then pulling and extracting the container image, which is the largest part of host setup for most jobs) and the application’s own startup (a JVM loading and JIT-compiling its classes, a cache filling, and so on). A fresh instance takes tens of seconds before the image pull even begins.

In multi-tenant Titus, most of this is hidden. A new task usually lands on a Titus container host that is already running and already has the image layers cached, so starting it takes only a few seconds rather than minutes; the host boot was paid for once, long ago, and amortized across every task that has run there since. Single tenancy gives up that hiding place. Every single-tenant task waits for its own machine to boot and pull its image.

The usual remedy is a warm pool: keep some instances booted and idle so a pending task lands on one immediately. This works well for the multi-tenant fleet precisely because that fleet is close to homogeneous. A handful of large shapes cover almost everything, since any task can be packed onto a big shared host. A small amount of warm capacity in a few shapes gives a high hit rate, and its cost is spread across the many tasks that will use it.

Single tenancy inverts those economics, and this is the part worth being explicit about.

Because every task gets a right-sized node, the fleet spans many instance shapes (several families across many sizes), and a warm instance only helps a pending task whose shape matches it. A warm r7i.2xlarge does nothing for a task that needs an m7a.8xlarge. To achieve a good hit rate, we would need to hold idle capacity in every family-and-size bucket, and idle capacity in a bucket with no task to fill is pure cost. The very property that makes single tenancy efficient at steady state, tight best-fit sizing, works against warm-pool reuse: more distinct shapes mean more buckets and a lower hit rate per dollar of standing capacity. And the pressure peaks exactly when it is least welcome. During a regional failover or a live event, when many tasks need fresh nodes at once, a thin warm pool spread across many shapes drains immediately, and the rest of the demand pays the full cold-boot cost.

So we treat warm capacity as a small, targeted lever rather than a default. We keep a little pre-warmed capacity for the most latency-sensitive single-tenant starts, and we deliberately narrow the menu of shapes to a curated set of sizes rather than the full continuum, so warm capacity concentrates into fewer buckets and trades a little best-fit efficiency for a better hit rate. Everything else provisions just-in-time and accepts the boot.

The more durable wins are in cutting the launch cost itself rather than pre-paying for it, and they help both single- and multi-tenant nodes, which is the better place to spend the effort:

• Faster image distribution. The image pull dominates the host setup, so newer OCI image formats and lazy-pulling snapshotters such as stargz and SOCI, which fetch only the bytes a container actually reads at startup rather than the whole image up front, reduce the largest component of the gap.
• Checkpoint and restore. For processes that are slow to initialize, snapshotting a warmed-up application with a tool like CRIU and restoring it, rather than cold-starting it, removes most of the second half of launch latency.
• Instance type selection. In multitenant configurations, we have historically used metal instances, which offer slightly better performance but come with a slower startup time. In a single-tenant configuration, node startup latency is more important, so we have moved away from these instances.
• Scheduling optimization. Through regular stress testing of fleet scaling, we found a scheduling bottleneck caused by Kubernetes preemption logic, leading to nodes being scaled up quickly but then left idle while waiting for workloads to be scheduled on them. This was ultimately unnecessary, as our single-tenant pools do not run preemptible workloads. Disabling preemption greatly improved scheduling performance when under strain.

None of this makes the tradeoff disappear, and it is worth stating plainly: a single-tenant node is slower to stand up than a slot on an already-warm shared host, and hiding that with warm pools costs more per node here than it does on the multi-tenant fleet, because the warm pool cannot be as homogeneous. But for the long-running services that single tenancy targets, paying that cost once at launch in exchange for a quiet life afterward is a trade these teams are glad to make.

Measuring the difference

Isolation claims are easy to assert and hard to trust, so we measure them with a repeatable benchmark and a single score.

The method is a differential one. We run a reference container, a healthy and representative application, and record its throughput and tail latency while it runs alone. Then we run it again next to a havoc container engineered to be a bad neighbor: a UDP flood through XDP to stress the network-shaping path, pointer-chasing and STREAM to pollute the last-level cache and saturate memory bandwidth, fsync storms against the disk, stress-ng to abuse the scheduler, and GPU-burn to monopolize an accelerator. We only trust a run once the host has reached a steady state, which we confirm with the kernel’s Pressure Stall Information: we wait until cpu_some or mem_some shows that tasks are actually stalling on a contended resource, not merely that the resource is busy. Each scenario is run many times with fixed seeds, and we report the median with a bootstrapped confidence interval, because noisy-neighbor distributions are heavy-tailed and a simple average can mask the spikes that matter.

From the reference container’s numbers, we derive a simple measure of how well isolation held. Throughput efficiency is the fraction of operations per second the app retains under attack; latency efficiency is the ratio of its undisturbed P99 to its P99 under pressure. A neighbor that doubles the reference app’s P99 drops the latency efficiency. As an illustration, a reference app whose P99 latency goes down next to a havoc container achieves a certain latency efficiency, and one whose throughput falls from a to b operations per second achieves a certain throughput efficiency. The composite score is a weighted blend of the two, with a latency-sensitive preset that gives greater weight to tail latency for interactive and streaming-path services. A perfectly isolated system scores 100%, and a single hard failure invalidates the run regardless of the average.

Noisy-neighbor isolation

cgroups container isolation isn’t perfect, and applications may experience worse performance due to sharing a machine with a determined neighbor. This could extend to disk and/or network IO where the same volumes or interfaces may be shared across workloads. For a distributed workload, this can lead to measurable inconsistencies in performance depending on where the replicas are placed. For more sensitive workloads, this variance can mean the difference between stability and getting paged.

The single-tenant option makes it possible to better isolate workloads that cause problems for their neighbors and/or are more sensitive to such problems. This can be used as a stopgap until the specific regression problem is resolved and consistency is restored, or kept permanent for load patterns that aren’t worth supporting in a shared environment.

Efficiency

The second effect is fleet efficiency, and it is the one people expect to go the wrong way. Dedicating a node to a single task consumes that node’s slack, so the naive expectation is that utilization falls. In aggregate, it can move the other way, for a reason related to how we dealt with poor isolation before single tenancy existed.

To protect sensitive workloads on shared hosts, the platform used anti-affinity placement, scattering them across the fleet to reduce colocation of the same workload on the same nodes. Spreading fights bin-packing: it strains capacity, because a host kept partly empty to honor an anti-affinity rule cannot be filled by a workload that would otherwise have fit there. When isolation-sensitive workloads move to dedicated nodes, the need for spreading and fragmentation goes away, and the remaining multi-tenant fleet can pack more densely.

Another source of fragmentation is services that are sensitive to disruption. These are typically stateful services where recovery is expensive. As these services become more common in a given population, they can lead to nodes being held up by pods with strict disruption requirements. This can lead to an inability to defrag and recompact the pool during off-peak times. The symptoms and resulting drop in fleet efficiency caused by disruption-sensitive workloads mirror what’s observed for workloads with anti-affinity rules.

Depending on the magnitude of the anti-affinity and/or disruption budget problem, moving such workloads to single-tenant pools can result in equivalent or even improved placement when all costs have been accounted for. It also becomes easier to quantify the cost to end-users. Rather than trying to attribute fleet-wide fragmentation to individual workloads, we can directly quantify waste as single-tenant node overhead.

The theory and the practice diverge here, and it was the practice that decided it. In principle, the efficiency ceiling is higher for multi-tenant bin-packing: pack enough tasks together, and a host fills almost completely. We do not reach that ceiling. AWS offers a discrete set of instance sizes, so a large shared host rarely matches the exact sum of the requests we place on it, and the gap between a host’s capacity and what we can actually pack onto it is a bin-packing hole we cannot fill. At the granularity of large instances, those holes are sizable. Right-sizing an instance to a single workload, picking the closest fit from many shapes instead of draping many workloads over one big host, captured more of the available efficiency in our experience than the higher theoretical ceiling ever delivered. The clean theory favored multi-tenancy; our measured fleet favored right-sizing.

Container-host maintenance

The third effect is operational, and it shows up in how we keep the fleet patched. A multi-tenant host is hard to take down for maintenance precisely because it is full: draining it means relocating many tenants at once, several of which carry disruption budgets that limit how fast they can move, so a routine patch becomes a careful, coordinated evacuation. A single-tenant host runs exactly one task, with consolidation and defragmentation turned off, so draining it means relocating only the one workload, and we can automatically handle that when a new AMI ships. The remaining multi-tenant hosts are easier to maintain, too, because the workloads that were most sensitive to disruption are no longer stranded on them.

What it changes, and what it doesn’t

Single-tenant mode is a configuration, not a migration. A team moves a workload onto a dedicated node, and back, with a labeling change, without rewriting anything and without leaving Titus. That turns the isolation decision into a reversible setting, which is the opposite of the one-way door between Titus and EC2 that teams used to face. A team can try single tenancy for a quarter, measure it, and step back if the trade-off isn’t worth it for them.

What does not change is the contract. The Compute/Platform team still selects the instance type, patches the host, and sizes and scales the fleet underneath. The application team still uses the Titus API, the same delivery pipeline, and the same observability. Single tenancy is a new point on the isolation axis offered within the existing managed experience, not a separate product that the team has to learn and operate.

When a dedicated node is the wrong call

A dedicated node is a real cost, and single tenancy is not free isolation. Two things are worth weighing before reaching for it.

First, a single-tenant task does not get the whole machine; it gets the machine minus overhead. Every Titus container host, dedicated or shared, runs the platform sidecars, system services, the kubelet, and the host environment itself, and that overhead is a roughly fixed slice of CPU and a few gigabytes of memory. On a shared host, that slice is paid once and amortized across the many tasks running there. On a dedicated node, a single task handles it. Because the cost is fixed rather than proportional, it is punishing as a fraction of a small instance and negligible on a large one: a small dedicated node can lose tens of percent of its resources to overhead, while on the largest shapes it rounds to a couple of percent.

Second, because a task is placed on the smallest instance that fits it, there is usually a gap between what the task asked for and the next available shape, and on a dedicated node, that gap is pure waste, with no neighbors to soak it up. The two effects compound: fixed overhead plus the best-fit remainder set a ceiling on how much of a dedicated node a workload can actually use, and that ceiling is lowest exactly where people are tempted to reach for isolation casually, on small instances. However, this sizing waste can be reduced through coordination with the service owner as needed.

The guidance follows from the arithmetic. Single tenancy pays off for large, long-running, performance-sensitive, disruption-sensitive workloads, sized close to a node shape and run as fewer, larger replicas rather than many small ones. It is the wrong call for small, bursty, or highly elastic workloads, where the overhead fraction is high, the dedicated node sits mostly idle, and multi-tenant packing is both cheaper and faster to scale. This is also why the mode is allowlisted rather than a self-serve checkbox: the platform can keep workloads on the side of that line where they belong, and leave the ones that would gain little from a node of their own on the shared fleet, where they are cheaper to run.

Who’s using it

The first adopters are the workloads that motivated the work: real-time data infrastructure. Managed Apache Flink stream processing, streaming data connectors, and telemetry and logging pipelines were both the most frequent sources of network contention and the most damaged by it, because their recovery is expensive and their disruption budgets are tight. These are stateful, high-throughput jobs where a few seconds of starvation can cascade into minutes of catch-up, and where the platform’s own repacking used to be a source of disruption in its own right.

Moving them onto dedicated nodes removed the class of problem rather than mitigating it. The teams running them report stable operation, having previously seen intermittent, hard-to-attribute interference, and their jobs are no longer relocated by fleet repacking, removing a second, self-inflicted source of disruption. Latency-sensitive serving paths and a handful of GraphQL and edge services have followed for the same reason: a predictable tail matters more to them than a few points of bin-packing efficiency.

This is an ongoing rollout, not a finished one. We are widening the set of eligible workloads as we learn, using the largest production jobs as the stress test for the platform’s scaling assumptions, and watching the control-plane and EC2 launch-rate metrics as the count of small nodes grows. We have also been running periodic stress tests to exercise scaling ahead of real user workload, and making improvements to autoscaling and scheduling logic along the way.

What’s next

We are not done on either side of the tradeoff. On single tenancy, we are reducing launch latency with smarter pre-warming of common shapes, broadening the allowlist as scaling data holds up, and extending the model beyond long-running services as cost attribution and capacity stories mature. On the multi-tenant side, which remains the right default for most workloads, we are addressing the dimensions that hurt the most. The largest is the network: replacing HTB’s global-lock shaping with an eBPF-based Expected Departure Time approach that rate-limits by timestamping packets rather than contending on a lock, and pairing it with a lower-overhead container network device. We are also evaluating hardware cache and memory-bandwidth partitioning (Intel RDT and AMD PQoS) to give the kernel a real lever on the last-level cache, hardware GPU partitioning (NVIDIA MIG) for accelerator isolation, and continued work on the cgroup v2 IO controller for disk. This isolation benchmark lets us rank these by return on effort rather than by intuition, and show customers, in the numbers, that the shared fleet is getting quieter over time.

Single-tenant Titus is also a step toward something larger. Netflix’s Platform teams offer two ways to get compute: virtual machines on EC2 & containers on Titus. We want to replace that with a single Compute API that takes intent rather than mechanism into account: how much isolation a workload needs, how quickly it must launch, how long it can run without disruption, and what business domain it belongs to. The platform then makes placement and runtime choices, and can make globally optimal ones because it sees the whole picture.

Conclusion

Multi-tenancy is the right default for a container platform, and we will keep making it better for everyone. But the right default is not right for everything. Single-tenant Titus gives the workloads that need it the isolation of a dedicated VM, provisioned and patched by our existing node management, under the same API and the same managed experience, without taking away the freedom to change their mind. We can now point to the difference rather than describe it: a tail that stops moving under a noisy neighbor, a fleet that packs more densely once spreading is retired, and hosts that are faster and safer to patch. For the streaming and data systems at the heart of Netflix’s real-time platform, a room of one’s own turned out to be the feature that mattered.

Single-tenant Titus: VM-level isolation on a multi-tenant container platform was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Prasanna Vijayanathan, Renzo Sanchez-Silva

When Jake Paul and Mike Tyson stepped into the ring in November 2024, we delivered the fight to roughly 65 million concurrent streams. On an ordinary day our infrastructure takes in more than 2 billion requests and handles upwards of 38 million real-time logging events every second. At that scale, the whole picture doesn’t fit into a single engineering team’s head or onto a single dashboard. With more Live content, Podcasts and real-time streams, our need to understand and update our systems as quickly as possible becomes imperative to provide a good user experience. Understanding the system at this scale and within the time constraints we have, often becomes a daunting exercise in finding a needle in a haystack, in a dark jungle, blindfolded.

This post is about that search for the needle.

The Four hours before a fix

Consider a regression of the kind we see in a given week. The home screen where you see the list of lists of movies, starts taking about 1.5 seconds longer to render. An alert fires. And then we go looking for what caused this issue and how do we fix it?

As it so happens, a regression on the application could be related to anywhere in the system — a client app regression from a new version deployment or A/B test rollout, a load shedding in the infrastructure, a GraphQL query responding with bad data because of a service rollout 6 hops downstream, are all suspects at this point. So, answering “what caused the issue?” needs insights from across the system, and this involves combing through layers of data in a meticulous manner.

The MELT Layer

“What caused the issue?”

Metrics, events, logs and traces, or MELT, in the industry’s shorthand, are the right raw materials for such an investigation, and each of them answers roughly a quarter of the question. We collect observability data across these four pillars, unified into a single observability data layer. This lets us observe our systems in real time, across what our users are experiencing, how our applications are doing and what our backend infrastructure and services are processing and responding to.

The investigation that follows may look like this in each layer:

The timeline of an investigation

So how long does it take to investigate an issue, figure out the root cause and fix it, so our users stop running into this issue again?

In our chosen example, four hours. Very little of it went into computing anything difficult. Most of it went into reconstructing relationships that our observability systems already knew but we had never written down anywhere a human or machine could simply read and discover.

The disagreement underneath

The instinct here could have been, around 2023, to reach for more data or better dashboards, and while both help at the margins, neither addresses what makes these investigations slow.

It’s 2026 now, so we might say, “hey, agents”, but without the right context delivered to the agents, AI systems hallucinate, and especially when teams are on-call and need a fix “now”, we cannot let inaccurate agentic analysis lead us down the wrong path. We need the right needle, not any needle.

Each of our systems is internally coherent and externally isolated. Atlas, our telemetry platform, knows applications and their time series. Radar, our alerting system, knows thresholds and firing history. incident.io knows severities and responders. PagerDuty knows rotations. Spinnaker knows what shipped and when. And our Slack history knows, in a form no query engine can reach, which engineer solved something very similar last quarter and what they found.

The difficulty is that they do not agree on what they are talking about. A thing that is an application in one system is a service in a second and a cluster in a third, with names differing by a suffix, a casing convention, or a rename only a couple of people remember (SMEs, ring a bell?). Joining across them requires a human carrying the mapping in their head, and that person becomes the bottleneck in such cross-domain investigations. The hard part of end-to-end observability is probably better understood as a data modelling problem.

What we lacked was an explicit, machine-readable agreement about the entities in our operational world.

The Ontology

This agreement has a name: an ontology. Ontology is a formal specification of types, the properties they carry, and the relationships they may have. Its unit is the triple, one fact expressed as subject, predicate, object. Consider the following example triples.

api-gateway   rdf:type       ops:Application .
api-gateway   ops:ownedBy    "Team Bedrock" .
api-gateway   ops:deployedIn "us-east-1" .

INC-5377      rdf:type       ops:Incident .
INC-5377      ops:affects    api-gateway .
INC-5377      ops:severity   "SEV 2" .

In the example here, we have already encoded six facts, and can answer in one query, a question that used to span three tools.

“Which incidents affected applications owned by Team Bedrock in us-east-1?” is a single query in place of looking up incidents, Spinnaker and Radar.

We call our ontology layer, The Observability Ontology. It acts as the schema, rule set and shared agreement that standardizes how different systems understand entities like applications, services, incidents, deployments, etc.

We built it on the W3C stack for this: RDF for the data model, OWL for the ontology, SHACL for validation, and SPARQL for querying. These standards are well specified and widely implemented, which keeps the model portable across graph stores.

The data from the ontology itself, is stored as a knowledge graph, persisted to QuipuDB, our homegrown RDF graph database, a distributed quad store based on redb, a Rust-based embedded key value db. Data here is generated by parsing raw telemetry, logs, and incident discussions into triples (subject-predicate-object facts).

Every harvest also produces a full Turtle serialization for archival, making it human-readable and diff-friendly, which has proven more useful for debugging than we expected.

Our ontology is organized into twelve namespaces, each owning one operational domain:

There are two decisions about this ontology that mattered while we built it, so they’re worth sharing.

The first is that we refused a generic Service type. “Service” means four different things across our tooling, so applications resolve to exactly one type, ops:Application, and every one of them is validated against our software catalog of roughly 49,000 entries before the entity is created. Unvalidated strings like usernames, team names, dependency names scraped from network topology, never become applications. We also cap dependency edges in both directions, because one popular service can carry thousands of reverse dependencies and would otherwise dominate the graph.

The second decision concerns what we chose to model. Incident chat carries the reasoning behind an incident. The hypotheses tried and discarded, the mitigation that worked, who knew why, is all context in the incident’s Slack channel and that knowledge evaporates when the channel goes quiet. So a 200-message, 50-person, 15-service incident channel becomes a structure, converting the operational chaos of an incident into ordered triples:

Creating determinism from non-determinism

Building this graph means using tools that are inherently stochastic. LLMs summarize discussions. Fuzzy matching resolves service names. A probabilistic engine proposes causal chains. None of these produce the same output twice with certainty, but what comes out at the end of our system is a deterministic graph that answers a SPARQL query the same way every time.

The ontology is the mechanism that creates this determinism through its pipelined approach. We think of the pipeline as a ladder that drives entropy down, where entropy means the proportion of information still trapped in unstructured form. For example, at the start of this pipeline, an LLM could say anything, but it cannot emit a triple that violates the type system. We constrain the output space tightly enough and a stochastic process yields a deterministic contract:

# The model may produce any prose. Its output MUST fit these predicates:
category  IN ["incident", "question", "discussion", "decision"]
sentiment IN ["positive", "neutral", "negative", "urgent"]

# Result — a deterministic triple:
thread:123  slack:category          "incident" .
thread:123  slack:isIncidentRelated "true"^^xsd:boolean .

There are five layers that get us down this ladder. Any single step of this ladder is non-deterministic. But in aggregate, the chain becomes deterministic. That inversion is the whole trick. We describe them below:

Enrichers: Agents as entropy reducers

We use what we call, enrichers, to reduce entropy across the knowledge graph. Dozens of enrichers run over the graph. Each one of them takes noisy input and emits typed triples, and each reads what earlier enrichers wrote. The contract for an Enricher is deliberately narrow:

class Enricher:
    async def _enrich_async(self, ...):
        raw    = await self.fetch(...)      # 1. read noisy input
        result = self.analyze(raw)          # 2. process (may be stochastic)
        graph.add((uri, RDF.type, OPS.X))   # 3. emit TYPED triples
        graph.add((uri, OPS.pred, Literal(result.value)))
        return EnrichmentResult(success=True, triples_added=count)

Three invariants that are standard across the enrichers:

• every entity carries an rdf:type,
• every URI follows a single naming scheme, and
• the catalog gates every ops:Application.

Two operational choices are worth naming here, as well. The enrichment pipeline runs sequentially — load the graph, add triples, and save.

This sequencing removes an entire category of race conditions at a cost in wall-clock time we are happy to pay. And calls out to external systems like metrics, logs, events and the rest could be best-effort, so a slow downstream API doesn’t block the graph harvest, albeit some degradation. To that end, we are also working to translate these separate systems into the ontology, such as the Metrics Ontology and the Alerts Ontology, so everything could be queried from the Ontology itself.

The Correlation Engine: finding the correlation chains

The correlation engine assimilates the vast number of signals from diverse input sources and condenses them into causal or correlatory chains of events. Today, the engine reads thirteen signal types, including log errors, metric anomalies, slow and failed traces, alerts, incidents, deployments, config changes, experiments, autoscaling, commits, end-to-end bottlenecks, and error patterns. It then searches for paths among them. Some ordered pairs are known to be causal and carry a high prior weight. For example, deployment before log error, config change before metric anomaly, failed trace before incident, and so on. Others, which show high correlation are considered correlation chains. Such chains are scored on temporal proximity and severity, inside a ten-minute window, with anything under 0.6 confidence getting discarded.

The role of the engine here is to propose with evidence, not to make conclusions. Hence, the engine stores its chains along with the graph records that were used to create them.

A 0.87 chain in the above example, is a strong hypothesis for a human or an agent to check, and the graph records the confidence alongside the claim so that consumers can understand why a chain is what it is and weigh their options to make decisions.

The knowledge flywheel

Each harvest cycle turns the same four steps, and each rotation leaves the graph richer for the next. This is the Knowledge Flywheel, leading to continuous enrichment and improvement of the context about our system, stored in the knowledge graph.

Observe collects the signals. Enrich adds typed triples. Infer finds patterns invisible in the raw data. Adapt feeds knowledge back, so that established patterns become detection rules. The compounding result of the four phases leads to a self-improving system. A causal chain discovered in one incident becomes a known pattern for the next, and the ontology gradually stops being a schema and starts being a memory.

Consider the example above. When we first see an incident in api-gateway caused due to a timeout and resolve it, we convert this knowledge into entries to the ontology. This becomes encoded knowledge. As we go through multiple cycles of this encoding, the graph starts learning patterns. After N cycles, patterns become wisdom — “api-gateway deploys cause 40% of SEV-2s” and “30% of deployment issues in api-gateway come from Connection Timeouts”. Surface this as an insight and we proactively start reducing incidents, not just react quickly to new ones.

Putting the graph to work

On top of the graph sits AutoRCA, our internal agentic AI solution, built with a set of specialist agents coordinated around one question. Walking the home page regression from the top of this post, the agentic system’s reasoning chain might look something like this:

Each specialization to the inputs is built by people who understand their respective domain deeply. That division of labour only works if the specialists can pass findings to one another without having to agree on the terms of communication or even have knowledge of each other. The ontology acts as the interface. Each specialist learns one vocabulary — the graph’s. A team writing specialized use cases for how they use traces record their findings as triples about an ops:Application, the logs specialization reads triples about that same ops:Application, and neither needs to know the other exists. A new specialist learns the same vocabulary once and can immediately read everything the other six have written.

Set against the investigation we opened with, the change is narrower than it might sound. The alert still fires at 3:30pm, an engineer still has to weigh the evidence, and the fix still has to be written by someone who understands the code. The hour or two that used to go into assembling context — finding the owning team, reconstructing what shipped, correlating information from different parts of the system, recalling that something similar happened last quarter — all that work is the part the ontology and the graph absorbs. We are early enough that the ranked hypotheses are sometimes wrong, which is why each one carries the evidence that produced it. But when the engineer now starts the investigation, they are already holding the picture that used to take half an afternoon to assemble.

If you want to build one of these

A fair amount of what we have built, we learned by getting it wrong first. Based on this experience, these are the principles we would carry into a second attempt.

• Model identity before anything else. Choose one authoritative catalog per entity kind and validate against it before creating an entity.
• Treat the ontology as code. Version it, review it, test it, and let changes arrive as pull requests. Treating the ontology as code lends itself to agentic development and lets teams scale and contribute to it.
• Separate concepts from instances. Concepts change on a human timescale and instances change continuously, conflating them makes both harder to evolve.
• Build on open standards. RDF, OWL, SHACL and SPARQL are specified, implemented many times over, and portable. A bespoke graph model ties a future consumer to one implementation.
• Constrain stochastic tools with a type system. This is the principle we would defend hardest. Models are useful precisely where determinism is impossible, and a sufficiently narrow output contract converts their output into something a query engine can trust.
• Make enrichment best-effort and ordered. A slow dependency should utmost degrade the graph, never block it. And ordering enrichers so each builds on the last buys more accuracy and that’s worth the tradeoff with parallelism that buys speed.
• Model human reasoning alongside machine telemetry. The most valuable edges in our graph came from discussion. Telemetry records what the system did, discussion records what people understood and interpreted. The reasoning is more valuable to store and serve as wisdom when we need it.
• Keep humans as the approvers. Let agents propose fixes, enrichers, and ontology changes, and route every proposal through a review process with humans in the loop.

What comes next

The graph makes three things possible. First, automatic root cause analysis, where a regression arrives with a ranked set of graph-backed hypotheses attached. Next, automated remediation, where a sufficiently confident diagnosis carries a proposed action like a targeted rollback, a traffic shift, a code change. And eventually infrastructure that heals itself, acting on a predicted failure before members feel it.

Alongside this, we plan to integrate the Observability Ontology with Upper, the domain-modeling language at the heart of Netflix’s Unified Data Architecture (UDA). Upper lets teams describe business and system domains as interconnected, machine-readable models and preserve their meaning and constraints as those models are projected into concrete schemas and systems. This integration would connect operational concepts such as applications, alerts, deployments, and incidents to the broader set of domains modeled across Netflix, giving engineers and agents a shared semantic layer for querying, validation, and reasoning. We will begin with a proof of concept that models incidents in Upper and use what we learn to determine where the two systems should converge.

Acknowledgements

None of this would exist without the teams who share their data, expertise, and patience with an early system. We want to specially acknowledge the efforts of our own team members who made this possible from very early on in this lifecycle, Kaela Marsheck and Aditya Gaur. We are also especially grateful to our collaborators in Netflix’s ontology working group, Peter Royal, Alex Hutter, and to Alexandre Bertails and members of the Unified Data Architecture (UDA)/Knowledge Foundations team for sharing Upper and their domain-modeling expertise, and for helping us explore how the Observability Ontology can become part of a broader semantic foundation at Netflix.

Ontology-Driven Observability: Building an End-to-End Knowledge Graph at Netflix Scale was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

How we detect failing GPUs and network fabric, remediate machines automatically, and let long-running jobs survive hardware loss.

Authors: Nick Rutigliano, Argha C, Samartha Chandrashekar

Open Netflix, and the home screen is assembled for you: which rows appear, the order they appear in, the titles within them, and the artwork for each title are chosen by machine learning models trained on how hundreds of millions of members watch. Personalization is one of the most widely used things we build, and members feel it every time they open the app. Like a growing share of Netflix, it runs on GPUs, which train those models and increasingly serve them live.

Those GPUs have to stay healthy for that experience to hold up, and keeping them healthy is harder than it sounds. A single large training job can run for weeks across hundreds of GPUs spread over many machines. It runs as a gang: the ranks move in lockstep, exchanging data every step, so the job advances only as fast as its slowest participant and stops entirely if one participant dies. Every GPU, every high-speed network adapter, and every interconnect must keep working throughout the run. At the scale at which modern machine learning runs, individual GPUs and network links can fail during a run, so the underlying platform must be prepared for this and handle it without a person in the loop.

This post describes how we keep our GPU infrastructure healthy for training and inference. We cover how we detect failing hardware before and during jobs, how we automatically remediate a bad machine, and how we let a long job survive the loss of a machine. These training workloads run on our modern Kubernetes-native Compute platform.

Why GPU reliability matters

Netflix aims to entertain the world, and machine learning has been part of how we do that for years, from the recommendations that help members find something to watch to the systems that help produce and promote titles. As generative AI advances, we are investing in it on both sides of the business: for members, through better recommendations and higher-quality promotional assets; and for storytelling, through creative tools for our production partners.

All of this runs on GPUs, for both training and inference, and GPU capacity is among the fastest-growing and most expensive parts of our infrastructure. That makes reliability a business concern on two fronts. — -

• Availability: a training run that loses a machine, or an inference service that cannot get healthy capacity, delays work that members and creators depend on.
• Cost efficiency: a GPU sitting idle behind a slow neighbor, or a whole machine pulled from the pool because one device is degraded, becomes capacity we paid for and are not using. Higher reliability and higher utilization mean more useful work per GPU, which is equivalent to spending less to do the same work.

Inference is a growing share of that GPU footprint as more of these AI features reach members, so the reliability and efficiency that matter for multi-week training runs increasingly matter for always-on serving too.

Why basic health checks are not enough

We start from the baseline provided by the cloud provider. Cloud providers run instance status checks on virtual machines: a system status check for the underlying host’s power, network, and hardware reachability; an instance status check to verify that the operating system is up and reachable; and a check on attached storage. It emits scheduled maintenance events when it plans to retire or reboot a host, and it can automatically recover an instance that becomes impaired. We use all of these, and the maintenance events feed one of our own checks. They catch a machine that is down, unreachable, or slated for maintenance, and they stop at the instance level. They confirm the box is up and reachable. They do not verify that each GPU, its on-chip memory, its interconnect, and its network fabric are healthy enough to support distributed training.

The next step most teams take is a GPU-level check to confirm that the driver responds and that the GPUs are visible. That catches a dead or missing GPU, but misses the cases that take the most time.

• A GPU can answer a status query and still be faulty once a workload touches it. A fault in on-chip memory or in the firmware that manages the device can remain invisible until a job allocates memory or runs sustained compute, at which point the job fails or slows.
• A high-speed network adapter may be present but show fewer links than expected. Collective operations still run, at a fraction of the bandwidth, so the job does not crash. It crawls, and the cause is easy to miss.
• A check can fail in ways that leave a faulty machine looking healthy. The diagnostic tool can hang when the driver hangs, and a naive check that waits on it hangs too. A binary that the check depends on can go missing after a driver upgrade, causing the check to stop running and report nothing. A check that can silently stop running is worse than no check, because it gives false confidence while the machine keeps working.

Because a job runs as a gang, any one of these on any one machine can degrade or stop the whole job, and none of them shows up in an instance-level check or a driver-responds check. The health system has to assume the hardware can lie and the tools that inspect it can break.

Our approach

We treat reliability as four layers, each catching what the layer below cannot:

1/ Node and hardware health: find bad GPUs, network adapters, interconnects, memory errors, and thermal issues, then taint or recycle the machine.

2/ Scheduler and platform: retry at the task & job level, validate a machine before it takes work, and drain & recycle machines safely.

3/ Training loop: checkpoint and use elastic workers so a job survives losing a machine.

4/ Observability: metrics, service-level objectives, and failure attribution across the other three.

The first two layers keep hardware problems away from jobs. The third accepts that some loss is unavoidable and makes it cheap. The fourth tells us whether the first three are working.

How the health checks work

Checks run in two tiers. Device-level checks look at each GPU. A node-level agent, built on the open-source Kubernetes node-problem-detector, runs on every GPU machine every 30 seconds and turns check results into conditions on the Kubernetes node, the same mechanism Kubernetes uses to mark a node NotReady. Controllers watch those conditions and decide whether the machine can take on work.

GPU device checks are built on NVIDIA’s DCGM for driver-reported error codes, uncorrectable memory errors, interconnect integrity, and thermal state, and networking adapter checks inspect the host state for the count and link state of the Elastic Fabric Adapter interfaces that carry cross-machine traffic. A few of these checks deserve detail.

• Failing closed: if the driver or the diagnostic tool is unresponsive, the check times out and marks the machine unhealthy. A check that cannot reach the hardware reports a problem rather than passing, so a wedged driver takes the machine out of rotation rather than letting it hide behind a hung tool.
• A fallback path: driver error codes are the strongest signal that a GPU has faulted, but the tool that reads them can itself be broken in certain failure modes! When it does, we read the kernel log dmesg directly for those codes, so no Xid or SXid errors are missed.
• Our own fatal-error map: vendor driver error catalogs lag newer GPU generations and do not always match what we see in production, so we keep our own mapping of which codes are fatal for each GPU family. A code that means reset-the-GPU in one generation can be benign in another, and one GPU generation emits a burst of normally-fatal code on every healthy boot, which we learned to treat as noise rather than drain fresh machines.
• No silent no-ops: a check fails when a required dependency is missing. This one came from an upgrade that dropped a binary the network-adapter check relied on. The check quietly stopped running, and machines with the wrong number of adapters shipped for days before the slow jobs were traced back to them. Now a missing dependency fails the check.

Turning a bad machine into healthy capacity, automatically

When a check fails, the machine’s node condition changes, and controllers take over. We distinguish two levels. A problem condition blocks new work from landing while jobs are running, because pulling a job off a machine is disruptive, and many problems clear on their own. If the condition persists beyond a tolerance window, it escalates to a failure condition, which evacuates the machine, drains its tasks (overriding the normal disruption limits to move quickly), terminates it, and replaces it.

Karpenter node auto-repair is used on our compute stack. We have customized it with a policy and a controller that maps each node condition to the duration we tolerate it before acting, so a fast-fatal error is actioned within minutes. At the same time, a flaky signal gets a longer window, and a cap limits how much of a cluster we will disrupt at once, so a bad rollout of the checks themselves cannot cascade into tearing down a whole cluster. We plan to upstream this feature to all Karpenter users.

Not every fault requires a full replacement, and replacement is the slowest way to restore capacity. We match the action to the fault. Killing the offending job clears a software wedge. Resetting a single GPU clears many transient device faults. Rebooting the machine clears the firmware state that has drifted over a long uptime. Terminating and replacing is reserved for hardware that fails a test we trust. For that last case, we record the GPU’s immutable hardware identifier, which stays with the physical device across reboots and relaunches even when the instance changes, report it to the cloud provider, and keep our own list, so a device that fails repeatedly does not quietly rejoin the pool under a new instance.

One subtlety sits between detection and scheduling. A freshly checked machine was handed to the scheduler just as its checks were still settling, so a job could land on it a moment before a slow check flagged it. We now hold capacity until the checks have run and settled, which closed that window.

Validating a machine before it takes work

A machine that just booted is not trusted until it proves itself. Before we release its capacity, it passes system checks, confirms the GPUs and their topology are present, and runs a round of GPU diagnostics. We then wait for the health checks to run twice, because the longer checks, such as counting network adapters, take more than one cycle to fire, and releasing after a single pass would let work land before those checks are reported. Only then do we remove the startup taint. On teardown, we run deeper diagnostics and record the GPU identifiers, which feed the same bad-hardware list, so hardware that failed once does not silently cycle back into service.

Letting long jobs survive machine loss

Even with all of the above, some machines are lost mid-job, so we make that cheap. Jobs checkpoint their state, and on a lost machine, they resume from the last healthy checkpoint rather than restarting from the beginning of a multi-week run. Workers are elastic and retryable, and the scheduler automatically retries platform-caused failures.

Attribution is what makes automatic retries safe. If we retried every failure, we would mask genuine bugs in a job and burn GPU hours looping on an error the platform cannot fix, so we classify each failure as platform or application before retrying. One example of why this matters: a class of timeouts in NCCL, the collective communication library GPUs use to exchange data, appeared to be a network fault and was traced to data-loading stalls at high worker counts. The GPUs and the network were fine, and the fix was in the data pipeline. Treated as hardware failures, those timeouts would have cycled healthy machines without fixing the cause.

Measuring whether it works

We track GPU reliability along three service-level objectives, each split by cause, so a miss points at the layer responsible:

• Availability: the amount of GPU capacity available for scheduling at any given moment.
• Job-start success: whether a job starts promptly once its capacity is available.
• Job success: whether a job finishes without a platform-caused failure.

Each objective is split into hardware, driver, platform, and user causes. Hence, an availability dip from bad hardware reads differently from one caused by a scheduler change or a user misconfiguration. We optimize for goodput, the useful work the fleet completes, because a job held back by one slow GPU can look healthy on an uptime chart while getting almost nothing done. Specific targets for each objective are provided here for illustrative purposes.

A closed remediation loop

These pieces form a loop. Hardware is validated at allocation, before a job starts, with a short collective communication run and pairwise network tests. Hence, any machine that fails within the first minute fails before the job is admitted. During the job, we watch passively for driver errors, memory errors, thermal events, and more. We view each job as a distributed system, which lets us identify a single slow rank dragging the rest, rather than only whole-machine failures. When something is wrong, remediation is automatic and graded. The loop is the same whether a job runs through our batch scheduler, Kueue, or Ray.

Sharing GPUs safely

Not every workload needs a whole GPU, and leaving small inference jobs on dedicated GPUs wastes capacity. We partition GPUs with MIG for memory-level isolation between tenants, so a fault or a memory hog in one partition does not affect another. We also reconfigure those partitions without restarting the machine, so that a pool can shift between many small slices and a few large ones as demand changes. GPU and network adapter allocation is device-plugin-based today, to move to Kubernetes Dynamic Resource Allocation once it is supported in Karpenter. This matters because a GPU talking to a network adapter or working with a CPU on the wrong PCIE root of the machine incurs a bandwidth penalty.

Device-level allocation also changes how we handle a single bad GPU. With DRA support, instead of losing the whole machine, we could isolate the one faulty device and keep the others serving, which matters most for inference, where many small jobs share a machine. And because allocation is expressive, a job can request the capabilities it needs, such as memory or tensor-core support, and the platform matches a suitable GPU type, so users are not pinned to a model they had to choose in advance.

Results

Moving from basic checks and manual draining to layered checks and automatic remediation changed two things users feel. More of the GPU fleet is available for scheduling at any time because bad machines are caught and cycled quickly, rather than sitting in the pool or waiting on an on-call engineer. And fewer jobs fail due to platform issues because most faults are caught before a job lands, and the rest are automatically remediated and retried. Graded remediation also returns capacity in minutes for faults that only require a job kill, a GPU reset, or a reboot, rather than the longer wait for a replacement machine.

Since the above improvements have landed, we’ve seen:

• Consistently achieving >99.95% availability of our GPU nodes
• Job success rate has gone from ~98% to above 99.5%

What is next

We are extending active checks to cover additional failure modes under load, widening burn-in so more classes of failures can be cleared before a user encounters them, and making remediation more granular so more error types are cleared in place rather than requiring a full node replacement loop for certain Xid errors. Additionally, we want to do more periodic diagnostics between jobs.

We are working to make placement more topology-aware using Kubernetes Dynamic Resource Allocation (DRA) once it lands in Karpenter, for example, by aligning on Compute/IMEX Domains, PCIe roots, and NUMA nodes.

More on the performance side for multi-GPU training, we plan to run a custom Linux scheduler built on sched_ext. Firstly, we can prioritize processes during collective coordination, such as when GPUs exchange results before the next round of compute, thereby reducing overall GPU idle time by establishing a fleet-wide priority for these tasks. The effect is small on one machine but grows across hundreds of machines, where a few percent of throughput amounts to a large amount of capacity. Secondly, we can prioritize data-loading processes to minimize the bottleneck that sometimes occurs when data is not loaded onto GPUs quickly enough to maintain steady progress.

While we already monitor throttling events, GPUs do not have to report throttling for it to hurt a job. One running ten to fifteen degrees hotter than its peers, even within healthy limits, sets the pace for the gang, so we can treat a large temperature gap as a fault, stop the job, and move it to cooler GPUs.

Additionally, we plan to expand our NCCL RAS (reliability, availability, serviceability) monitoring to identify NCCL issues sooner, such as mismatched collectives or hangs, to more quickly resume work.

Closing

Hardware fails, and at this scale, it fails often enough that the platform has to treat failure as normal. Detecting problems before and during jobs, automatically remediating machines, and allowing long jobs to resume after a loss lets teams run large-scale training and inference without babysitting the hardware. If you like these kinds of problems, we are hiring.

Reliable GPU compute for large training & inference jobs was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

By Violetta Pidvolotska and Naveen Mareddy

Stratum is Netflix’s internal serverless media processing platform, shaped by over a decade of experience developing large-scale distributed systems. It powers a wide range of media workloads — including encoding, packaging, inspection, and quality scoring — behind playback, advertising, studio workflows, AI embeddings, and more.

Engineers develop Stratum functions by packaging application code and OS dependencies into OCI images. Stratum then schedules these containers as batch jobs on Netflix’s Titus compute platform, with upstream services triggering function executions.

Every day, Stratum manages hundreds of thousands of containers across bursty, high-throughput, and latency-sensitive workloads. As Netflix’s core media processing platform, this load needs to scale as new content formats enter the catalog, such as podcasts, live events, sports, ads, or regional ingests like TF1 in France.

To request resources for these functions, owners specify their CPU, memory, disk, and network limits as YAML configuration, as shown below:

functionName: VideoEncoder.encodeV1
resources:
  numCpus: 8
  memoryInMB: 18000
  diskSizeInMB: 50000
  networkInMbps: 1000

Maintaining optimal values manually is challenging amidst evolving Netflix infrastructure, EC2 instance upgrades, and application-level changes. Consequently, configs are often over-provisioned upfront or slowly drift away from their ideal values over time. Mis-sized configurations are a universal pain of large-scale serverless platforms, causing additional cognitive load for function builders and operational costs for platform owners.

In this post, we share how we built the Stratum Resource Tuner (SRT) — an automated system that safely right-sizes running containers in production without human intervention. We will walk through real-world examples, our safety architecture, and what we learned when an automated change went wrong.

The Challenge: Over-Provisioning

When engineers write a new Stratum function, they typically copy a resource YAML from an existing function or size it for the absolute worst-case scenario. Once running in production, engineers rarely revisit these configurations until a failure occurs, prompting them to over-provision resources out of an abundance of caution.

Manual resource tuning does not scale for the following reasons:

• High Operational Friction: Downsizing requires days of manual metric monitoring and trial-and-error testing, diverting valuable engineering cycles away from other high-impact work. If a cut goes too far, engineers face production incidents — making manual tuning a high-risk, low-reward task.
• Continuous Infrastructure & Codec Evolution: Cloud providers regularly upgrade their hardware. A resource allocation that was optimal on an older hardware generation quietly becomes inefficient on newer hardware. At the same time, Netflix continuously advances media standards and improves encoding algorithms (such as AV1, AV2, and spatial audio). An allocation tuned for one codec might not be optimal when an improved algorithm is released.
• Dynamic Workload Evolution: As Netflix expands into new media types — such as streaming video podcasts, sports, interactive AI-adapted ad formats, and major broadcaster integrations like TF1 in France — media processing footprints can change unpredictably, rendering static container resource limits imprecise.
• Rising Industry Hardware Costs: Driven by massive global AI infrastructure demand, server hardware and DRAM prices continue to climb, making idle container memory significantly more expensive to waste.
• Compounding Platform Scale: At Netflix scale, even minor over-provisioning multiplies exponentially across hundreds of thousands of daily container executions, compounding wasted capacity across the entire Stratum platform.

To understand the scale of the challenge, we analyzed historical usage across Stratum’s workloads. The telemetry confirmed our hypothesis: over-provisioning was frequent, wasted compute was compounding, and the sustainable path forward required a fundamental re-evaluation of how resource configurations are managed.

Meet the Stratum Resource Tuner

To address the challenges listed above, we built the Stratum Resource Tuner (SRT) to automatically right-size Stratum containers. Our north star is to ensure that users never have to manually specify the resource configurations. We started with a handful of high-throughput workloads with relaxed latency requirements, which account for the majority (70%–80%) of resource usage.

As a quick note on terminology, we use “right-sizing” and “tuning” interchangeably in this post. “Downsizing” means decreasing resource limits, while “upsizing” means increasing them. Since our goal is to optimize for cost, this post focuses on downsizing.

Automatic right-sizing is not enabled by default. Function owners must manually opt in on a per resource type basis (e.g., CPU, memory, network, or disk). Once enrolled, SRT operates without a human in the loop across five steps:

1. Collect resource usage metrics across all executions of that function.
2. Generate a recommendation for each resource type (e.g., CPU or memory).
3. Verify the recommendation through experiments.
4. Apply or reject the recommendation based on the results of experiments.
5. Monitor post-apply metrics, watching for anomalies after applying the recommendation.

“Recommendation” is a key construct in SRT and serves as the primary unit of operation. As SRT executes its steps, it promotes the recommendation through several stages:

• CREATED: A scheduled workflow generated the recommendation.
• EVALUATING: Recommendation verification is running, and SRT is analyzing the results.
• APPLIED: All experiments passed, and SRT automatically rolled out the recommended configuration.
• REJECTED: At least one experiment failed, leaving the function configuration untouched.
• MONITORED: SRT monitors the recommendation during a post-apply soak period to verify stability.

The sections below examine each of these stages in detail. To illustrate the flow, we will follow the real-world lifecycle of the VideoEncoder.encodeV1 function — a high-throughput video encoding function whose over-provisioned memory and CPU footprints were successfully tuned.

Note that all operational thresholds and safety limits discussed below are fully configurable per function to accommodate varying workload characteristics and risk tolerances.

Metrics Collection

To make these recommendations possible, we need a solid metrics collection mechanism. Every Stratum container reports its function execution metadata, infrastructure dimensions, and resource utilization (CPU, memory, disk, and network) at sub-minute resolution.

These metrics flow through Keystone, Netflix’s real-time event streaming platform, and land in two separate datastores:

• Execution summaries store: Stores execution-level summary metrics indexed in Elasticsearch for fast point lookups, serving as our short-term datastore for querying and inspecting individual runs.
• High-resolution long-term store: Stores raw, high-resolution usage metrics in Iceberg tables. While slower to query, it serves as our long-term datastore, holding months of history for resource utilization trend analysis.

One record is written per execution, which is represented as a span. Here is how these records look for a single span of our VideoEncoder.encodeV1 function.

In Elasticsearch, the record captures the high-level summary of the completed span, identified by tags.stratum.spanId:

{
  "tags.stratum.spanId": "38536313f65f060a",
  "tags.functionName": "VideoEncoder.encodeV1",
  "tags.stratum.phase": "run",
  "tags.stratum.success": true,
  "tags.stratum.retriedWithMoreResources": false,
  "tags.stratum.function.execDurationSecs": 26,
  "tags.cpuCount": 8,
  "tags.memoryInMB": 18000,
  "tags.diskInMB": 50000,
  "tags.networkInMbps": 1000,
  "tags.titusTaskId": "06e1a7c0-2502-4a54-905f-b13a1e786107",
  "tags.instanceType": "r7a.24xlarge",
  "tags.nf.region": "us-east-1",
  "ts": 1780743028000
}

In Iceberg, the record contains the same metadata, but adds the sub-minute resource utilization timeseries tracked during that span:

{
  "span_id": "38536313f65f060a",
  "function_name": "VideoEncoder.encodeV1",
  "phase": "run",
  "success": true,
  "start_epoch_secs": 1780743002,
  "end_epoch_secs": 1780743028,
  "runtime_secs": 26,
  "cpu_requested": 8,
  "memory_mb_requested": 18000,
  "disk_mb_requested": 50000,
  "network_mbps_requested": 1000,
  "titusTaskId": "06e1a7c0-2502-4a54-905f-b13a1e786107",
  "instance_type": "r7a.24xlarge",
  "tags.stratum.retriedWithMoreResources": false,
  "env": "prod",
  "resource_usage": [ 
   { "cpu_usage": 1.13, "memory_mb": 3701.37, "disk_mb": 20120.4, "net_in_mb": 221.09, "net_out_mb": 0.12 },
   { "cpu_usage": 2.25, "memory_mb": 3584.09, "disk_mb": 25410.1, "net_in_mb": 0.02, "net_out_mb": 0.05 },
   { "cpu_usage": 4.16, "memory_mb": 6750.79, "disk_mb": 30400.5, "net_in_mb": 165.90, "net_out_mb": 0.38 },
   ... 
  ]
}

Recommendation Generation

A scheduled Conductor workflow runs daily, reading the past month of usage data from Iceberg and generating recommendations for every function and resource type. The workflow also generates recommendations for functions that have not opted into automatic right-sizing to preview potential savings.

Each recommendation is saved in the recommendations datastore, enabling us to query and filter recommendations by function, resource type, or status on an ad hoc basis.

For each function and resource type, the workflow applies the following generation logic:

• Filter by workload type: We only tune high-throughput workloads. This focuses our efforts on right-sizing, which yields the most meaningful fleet-wide savings.
• Aggressively tune CPU and network: We target p99 usage with no extra headroom. CPU and network are compressible resources; if a container occasionally hits these limits, the function might execute slightly slower.
• Conservatively tune memory and disk: We target p99.9 usage and include a safety buffer. Running out of physical memory or disk space causes hard failures — such as Out of Memory (OOM) or Out of Disk (OOD) crashes — making a protective margin essential.
• Apply safeguards to changes: We discard recommendations if the absolute reduction is too small to justify operational effort. For memory and disk, we clamp the maximum reduction allowed in a single cycle to 15% — a safety limit introduced after learning from early production runs.

The workflow creates new recommendations in the CREATED state. If a recommendation already exists in this state for a given function and resource type, SRT overwrites it with updated calculations. Once a recommendation advances past the CREATED state, its recommended_value and stats fields become immutable. Subsequent daily runs generate a new recommendation with a unique ID instead of modifying the existing record.

Here is how these recommendations look across resource types in their initial CREATED state for our VideoEncoder.encodeV1 function:

• CPU recommendation

{

  "function_name": "VideoEncoder.encodeV1",
  "env": "prod", "resource_type": "CPU",
  "current_value": 8, "recommended_value": 4,
  "status": "CREATED",
  "resource_cost_usd_per_hour": ***,
  "stats": {
    "usage": { "p99": 4.16, "p999": 4.57 },
    "execution": {
      "total_fn_executions": 748762182,
      "success_fn_executions_pct": 99.75,
      "total_fn_exec_runtime_hr": 88102723
    },
    "window": { "start": "2026-05-05T00:00:00Z", "end": "2026-06-05T00:00:00Z" }
  },
  "experiment_inputs": null,
  "experiment_workflow_execution_id": null,
  "outcome_message": null,
  "finalized_at": null
}

• Memory recommendation (15% reduction in one step)

{
  "id": "2edb734a-6184-4f5e-8ffa-23004bd665d7",
  "function_name": "VideoEncoder.encodeV1",
  "env": "prod", "resource_type": "MEMORY_MB",
  "current_value": 18000, "recommended_value": 15300,
  "status": "CREATED",
  "resource_cost_usd_per_hour": ***,

  "stats": {
    "usage": { "avg": 5296.43, "p99": 6750.8, "p999": 7820.3 },
    "execution": {
      "total_fn_executions": 748762182,
      "success_fn_executions_pct": 99.75,
      "total_fn_exec_runtime_hr": 88102723
    },
    "window": { "start": "2026-05-05T00:00:00Z", "end": "2026-06-05T00:00:00Z" }
  },
  "experiment_inputs": null,
  "experiment_workflow_execution_id": null,
  "outcome_message": null,
  "finalized_at": null
}

As the recommendation moves through its lifecycle, SRT populates the remaining parameters.

When proceeding to the verification stage, SRT transitions the recommendation status to EVALUATING, records the target experiment settings, and links the active workflow execution tracking the experiment.

Finally, once verification runs complete, the status changes to APPLIED or REJECTED, storing the outcome details and completion timestamp for analysis.

Retry with More Resources

We take a conservative approach to generating recommendations, prioritizing system stability over extreme efficiency. Because recommendations rely on historical usage data, they cannot always anticipate sudden spikes or novel workload profiles when functions encounter heavy, unseen requests.

To handle these unexpected spikes, we developed a fail-safe mechanism. A watcher process running in each tuned container tracks memory and disk usage during function execution. If the container exceeds the pre-set thresholds, 85% for non-cached memory usage or 95% for disk usage, the watcher flags the execution as high risk, cancels execution, and re-queues the task onto a larger container:

new_container_size = original_container_size * 1.5

This 1.5 multiplier only applies to memory and disk dimensions.

This “retry with more resources” mechanism serves as our safety net: larger container configurations are applied only when retrying tasks to self-heal from edge cases. Although these infrequent retries incur additional compute costs and latency, they allow us to keep day-to-day configurations highly optimized without having to size for rare, costly outliers up front.

For our VideoEncoder.encodeV1 function, if a heavy run causes memory usage to exceed the 85% threshold (15,300 MB) within its current 18,000 MB container, the system aborts the execution and reruns it on a 27,000 MB container.

While we prefer to trigger retries based on actual OOM or OOD events rather than pre-set thresholds, technical constraints in our current architecture make true event detection challenging. Using pre-set thresholds serves as a reliable stopgap while we develop improved detection capabilities.

SRT automatically enables the retry mechanism for any function opted into memory or disk right-sizing. Each Stratum container reports retry events, which flow into Elasticsearch and Iceberg alongside standard performance telemetry.

Recommendation Verification

Every recommendation must pass a two-stage verification process managed by a Conductor workflow before rollout. During this EVALUATING stage, SRT executes two distinct experiment types:

1. Synthetic experiment: A low-cost, low-risk test to catch obvious regressions before production rollout.
2. Production experiment: A live evaluation that validates changes across diverse production workloads to increase confidence.

A recommendation advances to production testing only after passing the synthetic experiment. When an opted-in function has multiple pending recommendations, SRT prioritizes and evaluates the candidates with the highest estimated savings first.

Synthetic experiment:

• Runs in a test environment isolated from production traffic.
• Sets up two clusters: one for the current and one for the recommended config.
• Replays identical request sets across both clusters simultaneously without splitting traffic, exposing both configurations to 100% of test requests.
• Typically runs for 1–2 hours.
• Runs to completion without early termination checks.

Production experiment:

• Runs in production on real user traffic, split three ways:
  ◦ ~90% remains on the main cluster (current production config).
  ◦ ~5% goes to a baseline cluster (current production config).
  ◦ ~5% goes to an experimental cluster (recommendation config).
• Evaluates recommendations against a similarly sized baseline cluster rather than the main cluster to preserve comparable traffic profiles.
• Typically runs from one day to one week, depending on traffic volume.
• Fails fast: cancels the experiment early if regressions are detected.

The Stratum platform executes other experiments alongside SRT experiments. To avoid over-splitting traffic, SRT experiments run at lower priority and yield resources to active, higher-priority experiments.

Traffic arrives in irregular bursts separated by multi-day or multi-week lulls. The verification pipeline accommodates these variable arrival patterns rather than expecting steady traffic.

Back to our VideoEncoder.encodeV1 memory recommendation (18,000 MB → 15,300 MB), verification is triggered with:

{
  "recommendationId": "2edb734a-6184-4f5e-8ffa-23004bd665d7",
  "syntheticExperiment": { "numberOfExecutions": 3000 },
  "productionExperiment": {
    "enabled": true,
    "maxContainers": 20, // max containers shift to baseline & experimental clusters
    "upGateTimeoutMinutes": 10080, // time to wait for traffic to start flowing
    "maxRunDurationMinutes": 1240, // max time to run the experiment once traffic starts
    "minSamples": 500 // min executions needed on both clusters before evaluating
  }
}

Both verification stages evaluate recommendations against two sequential checks, terminating immediately if either fails:

• Retries with more resources: Restricts memory and disk downsizes to a maximum 1% retry threshold over the experiment duration.
• Execution time: Compares median runtimes to prevent outlier distortion, permitting up to a 10–15% slowdown relative to baseline, depending on workload, and is configurable by function.

If a recommendation passes both verification stages, SRT updates its status to APPLIED and rolls out the configuration to the main production cluster. If a check fails at any point, SRT marks the recommendation as REJECTED and preserves the existing configuration. SRT sends notifications for both outcomes.

For VideoEncoder.encodeV1, both stages passed cleanly:

{
  "outcome": "APPLIED",
  "syntheticExperiment": {
    "result": "SUCCESS",
    "samples": { "baseline": 3000, "experimental": 3000 },
    "medianExecutionSecs": { "baseline": 59.0, "experimental": 59.0 },
    "slownessIncreasePct": 0.0,
    "retryRatePct": { "baseline": 0.0, "experimental": 0.0 }
  },
  "productionExperiment": {
    "result": "SUCCESS",
    "samples": { "baseline": 50361, "experimental": 49790 },
    "medianExecutionSecs": { "baseline": 196.0, "experimental": 198.0 },
    "slownessIncreasePct": 1.0,
    "retryRatePct": { "baseline": 0.0, "experimental": 0.0 }
  }
}

The recommendation moved to the APPLIED state, permanently setting the memory allocation for VideoEncoder.encodeV1 to 15,300 MB. We observed similarly clean results for the CPU recommendation, which reduced CPU allocation from 8 cores to 4.

Post-Apply Monitoring

Passing verification experiments does not mark the end of the recommendation lifecycle. Updated function code, modified input parameters, or underlying changes in the compute environment (Titus/EC2) can alter production behavior over time. To ensure configuration safety and foster feature adoption, we instituted continuous post-apply monitoring for applied changes.

We monitor metrics for 90 days across two rolling time windows: a 30-minute fast window and a 24-hour slow window. Each window maintains specific alert thresholds: a 3% retry rate over 30 minutes or a 2% rate over 24 hours triggers an automated notification.

At steady state, workloads exhibit baseline retry rates around 0.1%, with rates up to 2%–3% falling within acceptable operational variances. Most applied recommendations remained stable over the past three months.

VideoEncoder.encodeV1 has operated on its tuned configuration for months without breaching thresholds. However, VideoEncoder.encodeV2 presented a challenging edge case: a 50% memory reduction passed all verification checks and ran stably for two weeks before an unexpected traffic shift drove retry rates above 60%.

Because early versions lacked automated rollback capabilities, engineers manually triaged exception cases. This incident highlighted the need for automated mitigation, prompting us to construct self-healing rollback functionality.

Today, the two auto-revert windows look like this:

• Last 30 minutes: 3% retries warn us; 10% triggers an auto-revert.
• Last 24 hours: A 2% retry rate triggers both an alert and an automatic revert.

Breaching an auto-revert threshold immediately restores the function configuration to its pre-recommendation baseline.

We intentionally maintain straightforward revert logic — restoring the last known stable configuration rather than attempting dynamic resource adjustments during an incident.

The VideoEncoder.encodeV2 incident also reshaped our recommendation generation rules for memory and disk allocations. Rather than applying full target reductions immediately, SRT now applies smaller, incremental 15% step changes, advancing only after confirming stability at each step. Additionally, we introduced a global kill switch to disable automated right-sizing across all functions in case of unexpected issues or side effects.

Additionally, we enforce a mandatory cooldown period — typically one week — before initiating new experiments on a function. Pausing adjustments allows us to observe performance across diverse production traffic patterns, identifying potential risks before applying subsequent recommendations.

Where Are We Today

Oversized containers and config drift don’t have to be an accepted tax of running serverless workloads at scale. By building the Stratum Resource Tuner, we transformed manual, risk-averse resource sizing into an automated, self-healing pipeline. Through real-world workloads and hard lessons from unexpected spikes, we’ve proven that continuous right-sizing can safely deliver significant gains in infrastructure efficiency.

We initially focused on right-sizing high-throughput video encoding workloads, aiming for an initial 10% reduction in overall Stratum compute costs. For instance, VideoEncoder.encodeV1 achieved a 50% reduction in CPU usage and 40% in memory usage without compromising performance or retry rates.

However, edge cases like VideoEncoder.encodeV2 highlighted key lessons: large single-step reductions can introduce hidden risks, and manual incident response slows recovery. These insights led us to enforce stricter safety guardrails in the Stratum Resource Tuner, including capping memory and disk reductions at 15% per cycle and adding automated rollbacks when retry thresholds are breached. Guided by these safeguards, re-tuning VideoEncoder.encodeV2 yielded a stable 20% memory reduction with zero subsequent incidents.

Looking ahead, our key priorities include:

• Expand SRT beyond high-throughput encoding workloads to support a wider variety of functions and workload types.
• Further tune the resources in regions with constrained resource availability.
• Move from reactive alerts to proactive, automated upsizing for under-provisioned functions.
• Experiment with intentional upsizing for latency-sensitive workloads to evaluate performance gains and reduce function execution time.
• Upgrade our failure signals for OOM/OOD events from lagging indicators to leading indicators to improve tuning precision.

If solving problems like this sounds exciting, we are hiring in our Warsaw office. Check out open roles here.

Credits

Special thanks to Olof Johansson, Jakub Karczewski, Adam Mazur, Ethan Mathieu, Rosanna Lee, Ameya Vasani, Jean Barkhuysen, Paul Hieromnimon, and Adithya Prakash.

Netflix Stratum Media Processing: Automated Container Right-Sizing was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

By Emma Yanyang Kong, JJ Tan, Ishan Gupta, Lars Olds, Claire Campbell, David Fagnan, Ratna Kavuri, Rohan Gosain, Veli Balin, Minsu Jang, Louis Garcia

Introduction

Any team generating text with an LLM at scale hits the same problem. You cannot evaluate everything manually. You don't know which outputs are great, which are mediocre, and which are harmful, because you simply cannot read them all. The standard solution is LLM-as-a-Judge, a second model that scores the first model’s output as a proxy for human raters. It is fast, cheap, and easy to deploy. However, it introduces a new and critical question: Who evaluates the evaluator? Furthermore, if the judge drifts, quietly degrading and starting to approve bad output, nothing downstream will flag it.

This post is about how we solve that problem, by building, aligning, and continuously monitoring LLM judges so they stay trustworthy over time. Our core principle is simple: An LLM judge must never operate alone. It has to be anchored in human judgment. We keep a human-in-the-loop (HITL) at every phase of the lifecycle. Humans define what “good” means, create the labeled benchmark datasets to train and validate the judge, monitor its behavior while it runs in the system, and review and approve any updates before they’re rolled out.

We believe:

A reliable LLM judge is not a static artifact. It has a lifecycle.

It needs to be built against trusted human labels, aligned with human preference and reasoning, run under human-set policy, and maintained as the data and usage pattern shift. Each of these stages has human owners. In the sections below, we walk through these four phases, the key design decisions in each, and the lessons we’d share with a team building their own LLM judge, using one concrete task as our running example: recommendation explanations.

A recommendation explanation is a short natural-language message that accompanies a recommended item, and it shapes how users perceive and trust a recommender system. In our work, an explanation is human-readable evidence for why a title was recommended based on what the member has previously watched, shown on the title’s detail page, which is the screen a member reaches by selecting a title from the homepage. We focus on similarity-based explanations, which connect a recommended title to one reference title the member has interacted with via shared attributes such as genre and tone. For example,

A funny, heartfelt holiday romance about love and new beginnings, much like “My Secret Santa.”

Quality matters here because a misleading or poorly grounded explanation erodes the very trust the explanation is meant to build.

Everything that follows comes out of a series of controlled, user-facing online experiments we ran on the explanation feature, including a month-long experiment on the Netflix mobile app. In these tests we used an LLM to generate explanations, hundreds of thousands a week across a constantly changing catalog. Manually reviewing them at this scale isn’t feasible, so we deployed a second LLM to evaluate them.

What counts as a good explanation is specific to our task. However, the way we build, align, and monitor the LLM judge for evaluating explanations should be reusable and generalizable across different tasks.

The Lifecycle at a Glance

Figure 1 summarizes the four phases of our LLM-judge lifecycle and the data flow connecting them.

We start with Birth: Benchmark Data Creation, where human experts craft hard or near-boundary examples and human raters label them with pass/fail decisions and written rationales. We augment this with LLM-generated gray-area examples to build a richer benchmark dataset.

Next, in the second phase of Training: LLM Judge Development, we tune the LLM judge to align with human labels and rationales from the Birth phase, using self-reflection to identify mismatches between the judge and humans, and updating the rubrics based on those mismatches and anchor examples. We iterate in this phase until our metrics show that the judge is sufficiently aligned with human raters.

In the third phase, Deployment, we integrate the judge into the explanation generation system as both a gate and a critic. The LLM generator writes explanations, and the LLM judge scores them and either passes them through or blocks them and supplies a rationale that the generator can use for self-refinement and retries. Passed explanations are served to members, and low-quality explanations are gated.

Finally, in the last phase, HITL Alignment & Drift Monitoring, we keep the judge aligned with humans over time. “Human-in-the-loop” here means a person is directly involved in the decision, not just reviewing outcomes after the fact. We benchmark the judge against trained raters, and a human approves every change before it is rolled out to the system. During each test we sampled a set of explanations each week (both passed and gated, with a bias toward newer items), had human raters review them with labels and rationales, and compared the judge against these human ratings. If drift exceeded a threshold, we triggered retraining and benchmark data augmentation. This closes the loop by feeding new human-labeled data and updated signals back into the benchmark and development phases.

System components

Four automated components appear across all phases, alongside the human roles that define, validate, and supervise them.

• Generator: produces the explanation text, optionally using prior judge rationales and traces as part of its prompt for revision.
• Judge: evaluates an explanation and returns a pass/fail label together with a written rationale. In the deployed pipeline, it acts as both a guardrail (gating bad outputs) and a critic (providing feedback for self-refinement).
• Reflector: revises the Judge’s rubric when the Judge makes mistakes. It updates the rubric based on mismatched examples, human rationales, and anchor examples. It tunes the rubric and does not directly evaluate explanations.
• Meta-Judge: checks whether the Judge failed an explanation for the right reason by comparing the Judge’s rationale against a human rater’s rationale.
• Human experts and raters: the experts who define the quality criteria; the raters whose labels and written rationales the Judge is tuned and measured against; the reviewers who monitor the Judge while it is running and explicitly approve any changes before they were rolled out.

Phase I, Birth: Establishing Ground Truth

Everything downstream depends on knowing what “good” means, so this is the most human-intensive phase. Our internal writing experts define the must-have criteria an explanation must satisfy, author detailed labeling guidelines, and hand-craft tricky adversarial examples. Those same guidelines do double duty. They’re both the rulebook for human raters and the seed rubric for the LLM judges we tune later.

Why we built our own benchmark

Off-the-shelf LLM-judge benchmarks focus on general-purpose text evaluation and miss what makes our task specific. Explanations are short, item-specific, and context-dependent (each one is tied to a target title and one or two reference titles the member has watched). We also care about alignment to our own editorial bar, not just generic language quality. As a result, we built our own custom benchmark from three complementary sources:

• Expert-crafted adversarial cases: hand-authored pass/fail examples with known failure modes, the kind of edge cases nothing else reliably produces.
• Boundary examples: LLM-synthesized explanations that are deliberately pushed near the pass/fail boundary, then rated by humans. These capture the ambiguous cases that sampling from live traffic rarely surfaces on its own.
• Real sampled explanations: explanations sampled from the output of our LLM generator during test runs.

Two details that mattered

• The benchmark is not frozen. A monitoring pipeline (Phase IV) appends a fresh batch of human-rated explanations on a weekly cadence, so over the course of a test the benchmark tracked the current catalog and new usage patterns instead of going stale.
• Every “fail” label carries a free-text rationale explaining why the explanation failed. This looks like a small bookkeeping detail, but it is actually critical for the next phase, since those rationales are what we use to tune and audit the judge’s reasoning.

Phase II, Training: Aligning the Judge to Humans

The failure mode: right answer, wrong reason

There is a failure mode that does not show up in label-agreement metrics at all. A judge can agree with humans on the label, passing when they would pass and failing when they would fail, while reaching the right verdict for the wrong reason. On a dashboard it looks perfectly aligned. But as soon as you ask it to explain a rejection, which our pipeline does to steer self-revision, it can return misleading guidance.

We therefore care about not just label agreement but also reasoning alignment. In our setup, the Judge is always asked to output:

• a label (e.g., PASS/FAIL), and
• a short natural-language rationale for that label.

Here is an illustrative example. The member watched a musical, a family fantasy with a young heroine and a well-known ensemble cast. We recommend an adventure movie that is also a family fantasy with a young heroine, which is why the recommender paired them. But it shares neither the songs nor the ensemble cast.

The label (FAIL) is correct, and the stated reason is reasonable at first glance. But it turns out to be incomplete in a way that simple label agreement cannot detect, as we show below.

Reasoning-Aligned Rubric Tuning

We tune each criterion’s LLM Judge with a procedure we call Reasoning-Aligned Rubric Tuning (RART). As introduced in the system components above, the Judge, Meta-Judge, and Reflector are distinct components. The Judge scores an explanation against a rubric and gives a rationale, the Meta-Judge compares that rationale against a human-written one, and the Reflector edits the rubric text itself. RART is how the three are combined into a training loop.

Every Judge is a fixed prompt template parameterized by a rubric, so tuning it does not mean retraining the underlying model. Instead we iteratively refine the rubric text until the Judge’s behavior matches human labels and human rationales. At iteration i, the Judge is conditioned on the current rubric text and maps an explanation to a pair, a label and a rationale.

At a high level, each RART iteration proceeds as follows.

1. Score the training data with the current rubric.
2. Evaluate weighted alignment metrics on a held-out validation set. If all per-criterion metrics exceed their thresholds, stop.
3. Otherwise, construct an error-driven focus set of misaligned examples and hand it to the Reflector, which proposes a revised rubric.

The key difference from a standard loop is how we define a “mistake.” Most self-improvement loops look only at label disagreements. RART adds a second class of error. The Meta-Judge examines the cases where the Judge and a human both said fail and compares the Judge’s stated reason against the human’s rationale. When the two diverge, that is a right answer, wrong reason error, and it is also fed to the Reflector. Concretely, the focus set handed to the Reflector is the union of two error classes:

• Label mismatches: examples where the Judge’s label differs from the human’s.
• Right label, wrong reason: examples where the Judge and the human both said FAIL, but the Meta-Judge finds their two rationales don’t match.

So the Reflector revises the rubric from both ordinary label errors and right label, wrong reason errors. That second term is the core of RART. The Judge is pushed not only to match human decisions but to justify them in a way the Meta-Judge recognizes as aligned with human reasoning.

Returning to the earlier example, the LLM Judge said FAIL and a human said FAIL, so a label-only check would register agreement. Placing the two reasons side by side shows what that check misses.

RART routes it into the focus set, so the next rubric revision has to account for it.

Since the Meta-Judge is an LLM too, we checked it against humans first. Trained human raters labeled a sample of agreed-fail rationale pairs, and the Meta-Judge matched their verdicts on nearly all of them.

Alignment metrics

We run one Judge per criterion, each with its own rubric, tuned and scored on its own. So everything below is measured per criterion on held-out data. For each one we track three metrics.

• Specificity (fail-recall): how often the Judge correctly rejects bad explanations. This is the one we care about most, because a bad explanation that slips through the gate is a trust hazard.
• Recall (pass-recall): how often it correctly passes good explanations, which keeps coverage high and revision cost low.
• Reasoning agreement: on the explanations both reject, how often they reject for the same reason.

Each criterion’s Judge gets its own three, and one can be healthy while another slips. That is also why the gate is strict. An explanation has to pass every must-have criterion to be eligible to serve.

Ablation: does the reasoning signal help?

To isolate the impact of the reasoning signal, we compare RART to an otherwise identical loop that uses only label disagreements. We call this baseline vanilla: same Reflector, same Judge, same data, but no Meta-Judge and no use of human rationales.

Figure 2 shows the result. On two of the three must-have criteria, vanilla and RART land in the same place, because the starting rubric was already near ceiling and the metrics had very little room to gain. On the third criterion, the vanilla (label-only) method degrades. The Reflector in the vanilla loop rewrites the rubric into degenerate forms, and specificity and reasoning agreement fall with every iteration, ending up worse than the rubric it started from. RART stays stable on that same criterion and lifts both, with a small recall gain as well. The reason is that a vanilla (label-only) optimizer can reach the right label through unsound reasoning, and unsound reasoning doesn’t generalize to new data. Anchoring the signal on the rationale prevents that. Reasoning alignment helps where there is headroom and doesn’t hurt where there isn’t.

Phase III, Deployment: Putting the Judge to Work

Once deployed, the judge sat inside the generation pipeline, gating and critiquing explanations as they were produced during our tests.

Two roles for one judge

Once we align the LLM Judge with humans, we reuse it in two distinct roles for every generated explanation:

• Role 1, the gate. The Judge scores each explanation against the must-have criteria and rejects anything that fails.
• Role 2, the critic. When an explanation fails, we don’t just throw it away. The Judge’s free-text reason gets appended to the Generator’s prompt, and the Generator tries again.

The second role turns the pipeline into a generate → judge → revise loop, with the Judge’s critique steering each revision.

Bounded retries, and why we drop rather than serve

We cap the retries. Each revision costs another round of generation and judging, and the gains flatten quickly. Figure 3 shows what the loop actually buys, on 1,000 generated explanations run through as many as 12 revisions with four generators. Every curve climbs, which is the Judge’s critique doing its job, but the steep part is early, and the gap between generators is obvious. Revision amplifies a capable generator rather than rescuing a weak one. In our test runs we used a budget of 3 retries with our strongest generator, and an explanation still failing after that was dropped rather than served.

That “drop rather than serve” rule is deliberate. The two outcomes aren’t symmetric. A bad explanation is a trust hazard, while a missing explanation just forgoes an opportunity. When in doubt we’d rather show nothing, and most explanations cleared the bar within the retry budget.

Do judge-approved explanations help members?

Passing the Judge proves an explanation is good by our standards, but it doesn’t prove that judge-approved explanations are useful to members. To test that connection, we ran a one-time controlled experiment on the Netflix mobile app, explanations vs. no explanations on the title detail page, over roughly a month, with the pipeline generating and judging hundreds of thousands of explanations a week during the period.

Members who saw explanations shifted their viewing toward titles they hadn’t watched before, and more of their browse sessions ended in a meaningful play. Both effects were statistically significant, and we saw no member-initiated takedowns or escalations about explanation quality during the test. The result worth carrying over isn’t the size of that effect. It’s that offline agreement with our raters was necessary for these explanations to work but never sufficient to show they help members. Only the online A/B test could establish that judge-aligned explanations are not just quality-compliant but genuinely useful.

Phase IV, Monitoring: Keeping the Judge Aligned

A judge that’s well aligned at launch won’t stay that way. The catalog shifts, and so does what counts as high quality. The lifecycle has to close the loop.

The weekly human review

For each week a test was running, we pulled roughly 300 generated explanations for human review, stratified across the Judge’s decisions (served, revised, dropped) and weighted toward newer catalog items, where drift is most likely. The stratification stayed fixed week to week, so successive weeks were comparable. A panel of at least three raters reviewed each explanation, and the majority label was our ground truth. Those labels did two jobs. They told us whether judge-human agreement was slipping, and they grew the benchmark.

Setting the bar with human disagreement

The panel also tells us how much the raters disagreed with each other, which turns out to be the right yardstick for the Judge. Rather than hold it to a fixed threshold, we score the Judge and each rater against the same majority label and ask that the Judge fall within the raters’ own spread.

Take one criterion and one of its metrics, M. That week’s sample gives one value M(J) for the Judge and a set of per-rater values M(H) for the three or more raters on the panel. We require

so the Judge must score no worse than two standard deviations below the average rater. A harder week widens human disagreement and widens the band with it, so the Judge isn’t penalized for difficulty the humans also felt.

Falling below the band on any metric-criterion pair is a drift event, which triggers re-tuning (back to Phase II) on the augmented benchmark. A re-tuned rubric never goes live automatically. Someone on the team reads the proposed changes and signs off before it reaches the running pipeline, and we keep the previous rubric for instant rollback.

What human review catches that metrics can’t

Beyond asking whether the Judge still agrees with human raters, the weekly review answers a second question. Are the current guidelines still asking for the right things?

In the weekly review process, reviewers surfaced issues that did not show up in Judge–rater agreement. An explanation can use accurate attributes and still overstate how similar two titles really are. A single vague adjective (“exciting”, “fun”, “emotional”) can technically satisfy the rubric while saying almost nothing. Sometimes the reference itself is the problem, something the member wouldn’t think of as a comparable title at all, so the comparison lands oddly no matter how the attributes line up. Some genres show the same problem at scale. Stand-up specials can share similar metadata tags but differ completely in tone and cultural context, so a comparison may look justified on paper, not be bad enough for a rater to fail, and still land poorly with members. And some explanations meet every must-have criterion while still reading like generic “word soup” that does not help the user much.

When recurring patterns kept appearing, we updated the Phase I guidelines, which seed both the raters’ instructions and the judges’ rubrics, so one revision updates both sides. Other times we added product rules outside the Judge, for example by blocking certain title types from being used as references.

Drift detection catches a judge that has stopped agreeing with people. Only the continued human review tells us when our definition of a “good explanation” is too narrow or incomplete. We need both.

• Drift detection keeps the Judge aligned with the current rubric.
• Continuous human review keeps the rubric aligned with what actually works for members.

What We Learned

Five lessons we think transfer to any team shipping an LLM judge:

1. Invest in the benchmark before the judge. A modest, rationale-annotated benchmark beats a large label-only one. Those written rationales are exactly what make reasoning-aligned tuning possible. You can’t correct a judge’s reasoning if you never wrote down the human’s.
2. Tune on reasoning, not just labels. A judge can reach the right verdict for the wrong reason, which looks fine on a dashboard but gives misleading feedback and doesn’t hold up on new data. In our ablation, the label-only loop degraded a rubric that the reasoning-aware loop kept stable.
3. One aligned judge can serve many roles. Reusing the same judge as both gate and critic amortizes the alignment cost and keeps production behavior consistent across roles. Alignment is the expensive part, so get one judge right and use it more than once.
4. Plan for monitoring from day one. A recommender platform’s items and members never stop changing, so an LLM judge, like any ML model, has to be watched and re-tuned if needed, not shipped and forgotten.
5. Keep people in the loop by design, not by exception. Automating evaluation is not the same as delegating responsibility for it. In our lifecycle, humans set the standard (Phase I), supply the rationales the judge is aligned to (Phase II), decide the conservative default of dropping rather than serving a doubtful output (Phase III), and review a standing sample of live decisions and approve every rubric change before it ships (Phase IV). These checkpoints are much harder to retrofit once a judge is already making decisions on its own. The standing review also tells us what our rubric is missing, not just when our judge has slipped.

Taken together, the judge extends the reach of our reviewers, and those reviewers are the basis for trusting the judge. That relationship is what makes it responsible to run this system at scale. An automated evaluator that no one audits provides no assurance about the decisions it is making.

Acknowledgements

This post summarizes our research paper, “The Lifecycle of LLM-as-a-Judge for Large-Scale Recommendation Explanations,” which has the full methodology, metrics, and ablations.

Our thanks to the internal human writers and review experts who defined what a good explanation is and keep those guidelines current; to our Human Data Operations team and the raters they train and calibrate, whose labels and written rationales are what the judges are aligned to; and to the engineers across the team who built and operate the pipeline; and to our UI engineering partners, who built the member-facing surfaces that bring these explanations to the homepage.

The Lifecycle of LLM-as-a-Judge: Building, Aligning, and Monitoring at scale was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

By Prashanth Gedde Narayanaswamy

As one of Netflix’s largest and critical data pipelines, member-sessionizer consolidates tens of billions of hourly client events, encompassing user taps, playbacks, and engagement signals, into individual viewing sessions per profile.

A quick two-week tuning effort spiraled into two months when standard approaches hit a wall. Because the failures only happened at full production scale, each test took hours, limiting me to just three or four attempts daily. The turning point was changing the goal: instead of trying to speed up a single run, I built a way to run and monitor multiple tests at once and track every result.

That mindset led to an unconventional setup: kicking off five Spark jobs at once and monitoring them over spotty inflight Wi-Fi during a day-long trip home, while cloud servers handled the actual heavy lifting. This post covers how I set up that loop, along with the Spark bottlenecks and memory quirks I uncovered along the way.

The Scale Dilemma: Small Samples Don’t Catch Scale Bugs

Standard engineering wisdom says that you should isolate bugs on small datasets to iterate quickly. That works for standard software bugs, but it fails completely for big data performance bottlenecks.

While upgrading our sessionization pipeline from a legacy logging architecture to a modern platform, the new job hit critical failures, including executor Out-Of-Memory (OOM) errors, severe shuffle spillage, and thousands of stage retries. None of these issues could be reproduced using sampled data. The failures only triggered at full production scale because of data skew: an idle Netflix tab left streaming overnight generates tens of thousands of telemetry events, while a typical session only produces a handful. A standard 1% sample misses these extreme long-tail sessions entirely.

To catch the real bugs, every test run had to execute against a full hour of unsampled production data, along with all the deployment overhead that came with it.

The Strategy: Branch-Driven Isolation

The biggest win came from a feature in Dataflow, Netflix’s command-line tool for building, testing, and deploying data pipelines. It supports branch-driven development: our workflows run on Maestro (Netflix’s workflow orchestrator), which schedules and executes the jobs, and they already use a sanitized branch name (the ${dataflow.sanitizedbranch} placeholder). So if I push a branch like pgn/cap-experiment, then the deployed workflow, its tables, and its run namespace all get tagged with that branch name. This eliminates collisions with team members or between my own concurrent runs.

With that isolation in place, the model was simple: one branch is one independent experiment. I parameterized the scheduler (cap size, shuffle partitions, heavy-device threshold, feature flags) and gave each variant its own branch and output table. Instead of queuing five sequential tests over 10 hours, I fired off five variants simultaneously against the same time window. Total execution time dropped from the sum of all runs to just the length of the single slowest job.

Managing per-branch tables added zero overhead. Dataflow provisions and updates tables automatically from declared schemas without manual DDL. Whenever I modified the event bag structure between experiments, the table schema updated seamlessly, no CREATE TABLE or ALTER TABLE required.

The only real constraint was cluster contention. Because these heavy Spark jobs shared resources with the rest of the company, peak hours sometimes limited me to two concurrent runs instead of five. But even running two or three jobs in parallel was a massive improvement over single-threaded debugging.

The key takeaway: don’t focus on making a single multi-hour run faster, build a setup where you can run many experiments simultaneously without them stepping on each other.

While the previous chart shows the time savings, the architecture that enables this isolation looks like below:

How an AI Agent Helped

Running five parallel experiments gets messy fast. Tracking instance IDs, configs, memory spills, and stage failures across variants makes it easy to lose track.

This is where an AI agent stepped in, not to write clever code, but to handle operational grunt work. I described variants in plain English, and the agent updated scheduler configs, deployed to the branch, triggered jobs, and polled metrics. Once complete, it compared spill and failure counts across variants in a summary table, allowing me to ask, “Which run is closest to OOM-free at peak load?” and get an immediate answer.

The value wasn’t deep Spark expertise, but operational memory. It turned tedious Spark UI cross-checks into a single prompt.

Eventually, I packaged this workflow into a reusable agent skill (.agents/skills/sandbox-deploy-and-run/) run via /sandbox-deploy-and-run <module> <scheduler-file> [PARAM=value]. It handles the full lifecycle—preflight, deploy, background polling, auto-diagnostics, and re-entrant resume <instance_id> support—turning multi-hour experiments into a single command.

Why tmux Mattered

All of this ran on Workbench, Netflix’s cloud dev environment, using my laptop as a thin terminal. Without session persistence, an SSH drop or sleeping laptop would kill multi-hour Spark runs mid-flight.

tmux running directly on Workbench solved this. Every experiment lived in a named cloud session: I could close my laptop, reattach hours later, and pick up where I left off. Combined with branch isolation, this made "start five and walk away" possible.

This is now my default setup: a local terminal (Ghostty), dedicated git worktrees per branch, and persistent tmux sessions on Workbench. The local laptop is disposable—closing it or losing network never interrupts the jobs.

That setup turned a long travel day into active iteration time. I kicked off batches at the gate, checked results via AI agent over spotty inflight Wi-Fi, and squeezed 4–5 full experiment cycles into a single 24-hour flight.

What All Experiments Uncovered

The root cause came down to a single operator: collect_list. Because Spark buffers each full event bag in memory, long-session OOMs were driven by individual bag size, not total dataset volume. The legacy pipeline handled identical row counts without crashing, proving that each event in the new bag had grown significantly heavier.

Two factors inflated event weight:

• Field types and Option boxing: The session ID changed from an 8-byte Long to a UUID string. Worse, wrapping fields in Scala Option forced Spark to deserialize typed Datasets into JVM objects (Some wrappers), driving up object allocation and GC pressure across tens of thousands of events per session.
• Bloated string attributes: Low-value fields (e.g., referrer URLs, test allocations) added several hundred bytes per event. Multiplying these strings across collect_list wasted terabytes per hour, even though downstream tasks rarely used them.

The Fixes:

1. Unboxed primitives: Removing Option wrappers from non-nullable fields dropped peak executor memory from 112% to 58% of the slot with zero disk spill.
2. Slimmed event bags: Dropped unused per-event attributes and hoisted session-invariant metadata (app version, country) up to the session header.

Takeaway: Reducing individual event weight delivered far better stability than throwing more cluster memory at the problem.

Two Experiments That Changed My Mind

Out of dozens of runs, two findings completely overturned my assumptions about Spark memory management:

• Finding 1: Narrowing rows can trigger faster OOMs. Slimming our table dropped its size below Spark’s auto-broadcast threshold, prompting the query planner to switch from shuffle joins to broadcast joins. Because broadcast joins pin build-side tables directly into executor memory, narrower rows actually increased total memory pressure.
• Finding 2: Broadcast hash joins were silently draining memory. Adaptive Query Execution (AQE) rewrote nine sort-merge joins into broadcast hash joins at runtime. While each join stayed under our 400 MB broadcast limit individually, AQE pinned ~3.1 GB of cumulative data simultaneously in a single executor, pushing containers over the edge.

The Surgical Fix: Instead of disabling broadcast joins globally, I lowered only AQE’s runtime threshold (spark.sql.adaptive.autoBroadcastJoinThreshold) to 10 MB while leaving the compile-time threshold untouched. That single change freed up ~3.1 GB of executor memory without impacting initial query planning.

Takeaway: Broadcast joins speed up queries, but pinned memory isn’t free. Watch accumulated broadcasts in your execution plan, not just dataset sizes.

Keeping a Single Experiment Log

When running dozens of parallel tests, the biggest risk isn’t job failure, it’s losing track of what you’ve already tried or confusing which JAR and config produced a result.

To solve this, I scaffolded the AI agent to append a row to a repository Markdown table after every run:

• Run # | Jar ver | Change vs. Previous Run | Config | Key Metrics (Spill, Failed Tasks, Peak Memory, Duration)

If a run yielded a key finding, the agent added a short note explaining what the result proved or disproved.

Key Takeaways:

• Single source of truth: It created an auditable history for the entire investigation.
• Forced discipline: Requiring “Change vs. Previous Run” forced me to isolate variables. Whenever I bundled multiple changes, I couldn’t isolate the root cause and had to rerun the test — a mistake the log format immediately exposed.

A secondary doc (events-join-spark-tuning.md) captured the distilled findings as a quick "symptom-to-knob" lookup, so teammates didn't have to wade through the full chronological history.

A Mistake I Learned From: The Fix That Only Worked Off-Peak.

To cap session size, I added row_number().over(Window...) and filtered for the first N events. It passed off-peak tests cleanly, but crashed with OOMs at peak load.

The issue was structural: Spark’s Window operator buffers an entire session in memory before evaluating row numbers and filtering. The cap trimmed output downstream, but it didn't limit memory usage during buffering, so a single skewed session overwhelmed executor memory.

The Fix & Lesson: We moved truncation ahead of the Window stage to cap events before buffering occurs.

Takeaway: Test fixes against worst-case input skew. Off-peak runs hide structural bottlenecks that peak scale will immediately trigger.

The final fix

Pulling it together, four changes shipped, each isolated to a single experiment before it landed:

• Unboxed the never-null fields from Option to plain primitives. Peak executor memory dropped from about 112% of the slot to about 58%, with zero spill.
• Slimmed the event bag. Dropped per-event fields nobody used, and hoisted session-invariant fields (versions, country, connection type) to the session level so they are stored once per session, not once per event.
• Capped session size before the Window stage. Dropped oversized sessions ahead of the buffering, and truncated the bag to the cap rather than dropping whole sessions.
• Lowered AQE’s runtime broadcast threshold. Set spark.sql.adaptive.autoBroadcastJoinThreshold to 10 MB, leaving the compile-time threshold untouched.

The Unresolved Bottleneck

While these fixes made the pipeline run cleanly, they didn’t solve the underlying scaling issue: aggregating an entire session’s event bag into a single row has a hard ceiling. As our upstream telemetry source evolves and adds new attributes, each event in the bag becomes heavier, meaning this memory pressure will eventually return without a single line of code changing on our end.

Slimming and unboxing bought us headroom, but they didn’t change the fundamental data shape. The biggest takeaway from this effort is that tuning buys time, not infinite scale. If your underlying data model allows unbounded row growth, tuning alone won’t save you , eventually, the architecture itself has to change.

Summary

• If the bug only appears at scale, stop trying to sample. Build a way to run many full-scale runs in parallel instead.
• If your scheduler can use the branch name, then one branch becomes one isolated experiment, and running many at once becomes easy.
• Use an AI agent to run the operation, not only to write code. Deploy, poll, pull metrics, compare variants: this is where most of the time goes.
• Once a loop repeats, package it as a skill. The deploy-run-watch-diagnose cycle becomes one command.
• Use tmux for long runs. The run must survive you closing the laptop.
• Keep one single experiment log, and make every entry state the one change from the previous run. The format itself forces you to change one variable at a time.
• For memory problems, look at the per-row width first, not only at the total data volume. Bigger field types, Option boxing, and fat unused strings all add up inside a collect_list bag.
• Tuning buys headroom, not a new ceiling. If the data shape can grow without a bound, the design has to change eventually, tuning alone will not save you.

Running Apache Spark experiments in my sleep (and on a plane) was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Netflix Conductor : The Next Chapter

by Aravindan Ramkumar on behalf of the Conductor team

Conductor is the workflow orchestration engine Netflix uses to stitch micro-services into reliable, observable business processes.

If you’ve followed Conductor from the outside, the last thing you heard was probably the note on its GitHub repository: Netflix discontinued maintenance of Conductor OSS to refocus its resources on the internal fork.

That announcement marked the end of the open-source chapter, and from the outside it may have looked like the end of the story. It wasn’t. Internally, Conductor never slowed down. If anything, it grew faster, and the demands of that growth pushed us to rebuild it from the ground up. This post is the answer to the natural question that announcement leaves open: what happened to Conductor after that?

A lot, it turns out. The clearest way to put it is in terms of growth. Not long ago, Conductor crossing a billion workflow executions in a single year felt like a milestone. Today it clears that bar roughly every quarter, and the monthly volume is still climbing. Sustaining that kind of growth meant rethinking nearly every layer of the system. The data plane it runs on was modernized piece by piece. The evaluation engine was rewritten. The way it allocates work, controls concurrency, and runs compute all changes. The through-line is simple: scale forced a rethink of the evaluation engine and the layers beneath it, and everything else followed.

The rest of this post walks through that evolution: how the architecture changed, the performance and scale work at the heart of it, the leap forward in developer experience, and the newer capabilities for controlling how workflows and tasks execute.

How Netflix uses Conductor

When we last wrote about Conductor, it ran a few hundred workflow definitions for a few dozen teams. That footprint has grown by orders of magnitude. Conductor now backs roughly 200,000 workflow definitions owned by around 150 applications, spanning Content and Studio Engineering, Ads, Games, and more. The runtime side has grown to match. In a typical month it executes on the order of 420 million workflows, and on a given day it runs anywhere from 5 to 20 million, with teams adding hundreds of new workflow types every month.

Conductor and the Media Production Suite

One of the clearest ways to see what Conductor does is Netflix’s Media Production Suite (MPS). It’s a set of applications that move production data into the cloud. An average Netflix title generates around 200 terabytes of original camera files, and MPS is what gets that media ingested, validated, transcoded, and delivered to the right people around the world.

The multi-step pipelines behind MPS’s workflows (Footage Ingest, Dailies, VFX Pulls, and Conform Pulls) run as Conductor workflows. A dedicated orchestration layer called Studio Orchestrator stitches together the platform services these pipelines depend on, such as asset management, media encoding, and storage, using Conductor under the hood to ensure reliable execution and the durability to recover from the inevitable failures of long-running, distributed work.

Beyond the studio: Ads and Games

Content engineering is where Conductor started, but it’s no longer the whole story. Two of Netflix’s newer businesses lean on it just as heavily.

Netflix Ads has made Conductor a backbone of its programmatic pipelines. Creative ingestion, advertiser-certificate management, and privacy-preserving Data Clean Rooms all run as Conductor workflows.

Netflix Games uses Conductor to orchestrate the path a game takes from a partner’s submission to the app stores. When a studio uploads a build, a Conductor pipeline ingests the binary, runs the required validations (for example, iOS privacy-manifest checks), and drives App Store publishing, coordinating steps that span multiple platform services and external stores with the same reliability and visibility the studio pipelines rely on.

How the architecture evolved

Our earlier post described a Conductor built on a particular set of building blocks: DynoQueues for queueing, Dynomite for metadata, Cassandra (partially) for execution data, and Elasticsearch for indexing. That stack carried Conductor a long way, but as adoption climbed, almost every layer of it was eventually rethought. The data plane Conductor runs on today looks very little like the earlier one.

1.0: the original stack, and where it strained

In its first generation, Conductor leaned heavily on Dynomite (with DynoQueues as the queueing recipe layered on top) backed by Elasticsearch for indexing. It worked, but two classes of problems showed up as scale grew. The first was the datastore. Dynomite is an in-memory store: a clustering layer that adds sharding and replication on top of Redis. You scale it by adding memory and nodes, with each replica rack holding a full copy of the data, which suits a cache but not the durable record for millions of workflows a day. Dyno Queues, built on the same store, inherited those limits, and its zone-sharded design made task scheduling uneven. The second was the engine’s design: evaluation ran synchronously and mutated workflow state in several places, so competing updates could leave a workflow inconsistent with no automated way to repair it.

The 2.x line was a sustained campaign to address these, and two replacements did most of the heavy lifting.

2.0: Cassandra as the backbone

The most consequential change was making Cassandra the primary datastore for execution data, displacing Dynomite. Cassandra gave Conductor the elastic, horizontal scaling it needed, and removed the scaling ceiling and zone-affinity constraints that the old datastore imposed. Around the same time, large task inputs and outputs were offloaded to S3, keeping bulky data off the primary datastore and further easing the pressure on it.

Timestone: a purpose-built queueing layer

The second replacement was the queue itself. Timestone, Netflix’s high-throughput, low-latency priority queueing system, took over from DynoQueues. Beyond shedding the Dynomite dependency, it brought two capabilities that mattered for reliability: a delayed-queuing mechanism, and a background repair process that could detect and recover workflows. Timestone also turned out to be foundational for work that came later. The async evaluation queue and the queue-depth-driven dynamic thread allocation described later in this post both build on it.

Alongside these, the 2.x series also decoupled indexing from the critical path, moving to a Kafka-buffered, independently scalable indexer writing to Elasticsearch.

3.0: consolidating the modern data plane

By Conductor 3.0, these threads came together into a coherent platform. Dynomite was fully sunset. Timestone was the queue and Cassandra the datastore. A Kafka-fed indexer pushed to Elasticsearch, with Iceberg added for long-term storage. The release also shipped automated workflow repair and multi-cluster deployment, the last of which lets workflows run in fully isolated contexts (for example, a dedicated context for the Media Production Suite, kept separate from the shared default and scaled on its own).

The net effect is that by 3.0 the foundational reliability and scaling problems of the early era were largely solved, and the data plane reached roughly the shape it has today.

Rebuilding the engine: Conductor 4.0

By 3.0, everything around the evaluation engine had been modernized, but the engine and its data model were largely the original, and at this scale that became the bottleneck. The engine evaluated a workflow by loading its entire state into memory to decide what to do next, which meant heavy memory pressure and slow evaluations as workflows grew toward tens of thousands of tasks. It also stored a workflow’s metadata and its task payloads in the same Cassandra partition, producing oversized “wide rows” and leaving room for race conditions when several updates to the same task competed. Evaluation overhead, vertical-scaling pressure, and occasional inconsistent state needed to be addressed.

A partitioned task model

The old data model co-located everything about a workflow in one place. The rewrite separates workflow metadata from task and user data and gives each task its own record. The engine no longer loads the whole workflow graph to make a decision. It evaluates against a lightweight blueprint of the workflow and loads only the specific task data a decision needs. The practical result is a dramatically smaller memory footprint per evaluation and far less work to advance a workflow by one step.

The 3.0 engine topped out around 2,500 tasks per workflow. The rewritten engine scales to 30,000 tasks per workflow, more than a 10x increase. Evaluation got faster too: in production, the rewrite delivered roughly a 40% reduction in p99 workflow-evaluation latency.

Making competing updates safe using partitions

Separating the data is only half the story. The engine also has to make competing updates to the same task safe, because a task can be written from more than one place at once. Picture a long-running task whose worker sends periodic updates to keep it alive, each one marking the task in-progress, before finally reporting it COMPLETED. If a stray in-progress update arrives just after that completion, reordered by the network, it could quietly revert a COMPLETED task back to IN_PROGRESS.

The first version leaned on Cassandra lightweight transactions (LWT) to serialize these writes, but mixing LWT with ordinary writes is a known Cassandra anti-pattern. The two use different timestamping mechanisms, and combining them breaks the strict consistency Conductor needs.

So the data model evolved away from locking entirely. Task state is split across two partitions: a pending partition for tasks still in flight, and a terminal partition that a task is written to exactly once, when it reaches a terminal state (COMPLETED, FAILED, etc). When the engine evaluates a workflow, it reads both and reconciles them in the application layer with one simple rule: a terminal state always wins over a non-terminal one, and between two writes of the same kind (two in-progress, or two terminal) ordinary last-write-wins applies. So in the race above, when the task is marked COMPLETED it lands in the terminal partition, and the late in-progress update loses on read.

The net of the rewrite is reliable task lifecycle management, a lower memory footprint, faster evaluation, and far higher concurrent throughput.

Removing the last synchronous bottleneck: async workflow evaluation

A faster engine still left one structural problem. Workflow evaluation ran synchronously. When an update arrived, a new workflow or a task completing, the request itself evaluated the workflow, and to do that safely it had to take a lock on that workflow. Under load, concurrent updates to the same workflow raced for that lock. Only one won, and the rest were deferred to a background repair process that, in extreme cases, could take several minutes to catch up. The result was exactly the kind of unpredictable tail latency that hurts most under peak load.

The fix was to decouple the trigger from the evaluation. Updates no longer evaluate inline. Instead they enqueue the workflow onto a Timestone exclusive queue, and a central asynchronous evaluator processes each workflow’s pending work in order. Because the queue is exclusive, only one worker ever evaluates a given workflow at a time, so the lock contention simply goes away.

The results are most dramatic at the tail (p99 and p99.9), which is the point, since the tail is what users feel under load:

Unsuccessful lock-acquire attempts, which had spiked to around 2,700 per interval under contention, drop to essentially zero with async evaluation.

Controlling how workflows execute

Run Strategies: native control over workflow concurrency

For most of its life, Conductor would happily run as many instances of a workflow as you started, all at once. That’s the right default, but it left a gap. When several workflow instances act on the same entity, the same movie id, the same creative, the same script, there is no built-in way to serialize them or guarantee only one runs at a time. Teams that needed this had to build it themselves: custom locks, custom queues, custom coordination logic bolted onto their workflows. That’s extra complexity in every workflow that needs it, and a real source of operational risk.

Run Strategies are a way to declare, on a workflow definition, how concurrent executions should be handled. There are two strategies today. Parallel, the default, applies no restrictions and behaves exactly as workflows always have. Sequential runs at most one workflow at a time per sequence key, with subsequent requests enqueued and started automatically, in order, as each running workflow finishes. A run strategy is declared on the workflow definition, and can be set directly in the Workflow SDK as a @RunStrategy annotation argument on the @WorkflowMethod.

The key idea is the sequence key: two executions that share a sequence key run one after another, and executions with different keys still run in parallel. So a content-ingestion workflow keyed on movie id processes each title strictly serially while happily running many different titles concurrently, with no custom locking required.

Beyond polled workers: managed, serverless tasks

A task meant a polled worker: your application implements the worker, polls the Conductor server for work, runs it, and reports the result back. That model is simple and language-agnostic, and it’s still the backbone for most workloads. But it has a cost. You operate the worker fleet. You stand it up, keep it running, and scale it, even for work that’s bursty or runs rarely.

So Conductor added managed tasks: built-in task types where Conductor drives the compute directly, with no long-lived worker for you to host or scale. Two integrations carry most of this today: Titus for containers and Stratum for serverless media-processing functions.

A TITUS task launches a Titus job to run a script, whether shell, Python, or whatever your image packages, using Titus, Netflix’s container management platform, as the compute plane. You point the task at a Docker image and an entry point, and specify the shape of the run, such as cpu and memory. The lifecycle is fully managed. The task starts IN_PROGRESS, Conductor monitors the Titus job, and it transitions on the container’s exit behavior. An exit code of 0 marks the task COMPLETED, and a non-zero code, a timeout, or a missing job marks it FAILED. The upshot is that you can spin up containerized compute as a step in a workflow without running a worker that sits around polling.

A STRATUM task invokes a Stratum function and collects its output. Stratum is the serverless function layer of Netflix’s Cosmos platform, purpose-built for media-processing workloads, and a Conductor STRATUM task is the natural way to call one from a workflow. You give it a function name and an ordered list of arguments. The task stays IN_PROGRESS until the function finishes, then resolves to COMPLETED, FAILED, or TIMED_OUT. Because Stratum exposes functions to Conductor directly, teams can fold resource-intensive media operations into a workflow without owning the compute that runs them.

Polled workers, Titus jobs, and Stratum functions are three points on a spectrum, and a single workflow can mix all three: a long-lived worker for steady high-volume work, a Titus container for an occasional heavy batch step, and a Stratum function for media processing. Managed tasks shift the operational burden from the team to the platform.

Scaling task polling: dynamic thread allocation

The engine work in the previous section made Conductor evaluate workflows fast. But there’s a second half to throughput: how quickly workers pick up the tasks that evaluation schedules. That’s a polling problem, and at scale it has its own failure mode, one that showed up most sharply for Studio Orchestrator, the heavy Conductor user behind the Media Production Suite pipelines from earlier in this post.

Conductor clients poll for work using worker threads, and historically the number of threads was fixed per task type. That’s predictable, and it prevents any one task from starving the others, but it’s static. During burst traffic, when thousands of instances of a single task suddenly pile into a queue, a fixed thread count can’t ramp up to drain them. The symptom was distinctive: CPU on the instances stayed flat while queues backed up, and idle threads sat stranded on quiet queues unable to help the overloaded one.

Allocating threads by real-time queue depth

The fix is a dynamic allocation strategy that rebalances threads across task queues continuously, based on how much work each queue actually has, while still guaranteeing every queue a floor so nothing starves.

Each instance periodically polls Timestone for the current depth of every task queue it manages. It then splits its thread pool into two parts. A minimum allocation gives every queue a baseline number of threads, so no queue is ever fully starved. A floating pool holds the remaining threads, distributed across queues in proportion to each queue’s share of the total load. A queue with a deep backlog draws a large slice of the floating pool. An idle queue falls back toward its minimum, freeing threads for whoever needs them.

One guardrail keeps this from becoming its own problem: a multiplier ceiling. A queue’s thread count can grow by at most a configurable factor per cycle, so even a dramatic load spike ramps up in controlled steps rather than all at once, giving downstream and dependent services time to absorb the increase instead of being bombarded. Allocations are recomputed every polling interval, so the system tracks shifting load in near real time while always honoring the per-queue minimums.

Two things come out of this. First, because threads are allocated where they’re actually needed, each instance can safely run a much larger pool. Second, CPU utilization becomes an accurate signal of how busy an instance is, which means autoscaling on CPU makes the cluster scale out in response to real computational demand.

Developer experience: Workflow SDK

For most of Conductor’s life, authoring a workflow meant writing JSON. You declared each task, then wired one task’s output into the next with ${taskRef.output.field} string templates, and on the worker side you pulled inputs back out of a Map<String, Object> and cast them to the types you expected. It worked, but it pushed an entire class of errors to runtime, and made large workflows genuinely hard to read.

The Workflow SDK is our answer: a type-safe Java DSL for authoring Conductor workflows. You write Java code. The compiler and IDE catch mistakes before anything deploys, and a Gradle plugin translates your Java code into the JSON workflow definition the Conductor server understands.

Crucially, the SDK gives you the best of both worlds. One of Conductor’s defining strengths has always been visibility. Because a workflow is a declared graph rather than imperative code, during execution the exact control flow can be inspected, every branch, loop, and parallel fork, and light up live as an execution moves through it. Writing workflows in imperative code gives you none of that. A timeline is not a workflow diagram. The SDK keeps both halves. You write in idiomatic Java and at execution time you still get the full visual graph: the same task-by-task flow, inputs and outputs, and at-a-glance “where is this stuck and why” that Conductor has always offered.

The authoring pains it removes

No type safety. In the JSON model, a task input like “movieId”: “${awesomeTask2.output.movieId}” is just a string template. The worker reads it back with inputData.get(“movieId”) and casts. If the upstream value is actually an Integer and you cast to String, you get a ClassCastException at runtime. Fix that, and a subtle typo bites next: the JSON declares an input variable movieId but the worker reads inputData.get(movieid), which returns null and you get a NullPointerException downstream. None of this is visible until the workflow runs.

public class GetMovieWorker implements Worker {
  @Override
  public TaskResult execute(Task task) {
    Map<String, Object> inputData = task.getInputData();
    long movieId = (Integer) inputData.get("movieId"); // cast, fingers crossed
    // more logic here
  }
}

Verbosity. Defining a couple of tasks is already wordy. A real workflow’s JSON balloons into hundreds of lines that are difficult to review or refactor. A complex branching workflow becomes a single large JSON blob that no one wants to touch.

{
 "name": "wrap_movie_workflow",
 "version": 1,
 "semver": "0.0.1", 
 "schemaVersion": 2,
 "restartable": true,
 "workflowStatusListenerEnabled": false,
 "ownerEmail": "team@netflix.com",
 "timeoutPolicy": "ALERT_ONLY",
 "timeoutSeconds": 0,
 "tasks": [
   {
     "name": "get_movie",
     "taskReferenceName": "movie",
     "type": "SIMPLE",
     "inputParameters": {
       "movieId": "${workflow.input.movieId}"
     }     
   },
   {     
     "name": "wrap_movie",
     "taskReferenceName": "wrapMovie",
     "type": "SIMPLE",
     "inputParameters": {
       "movie": "${movie.output}"
     }
   }
 ]
}

How the Java DSL works

@TaskMethod
public Movie getMovie(long movieId) { /* real worker logic */ }

@TaskMethod
public WrappedMovie wrapMovie(Movie movie) { /* real worker logic */ }

@WorkflowStub(taskNames = {"movieapp.getMovie", "movieapp.wrapMovie"})
public interface MovieWorkflows {
  @WorkflowGenerated(name = "movieapp.getMovie")
  Movie getMovie(int movieId);

  @WorkflowGenerated(name = "movieapp.wrapMovie")
  WrappedMovie wrapMovie(Movie movie);

  @WorkflowMethod(name = "wrap_movie_workflow", ownerEmail = "team@netflix.com")
  default WrappedMovie wrapMovieWorkflow(MovieInput input) {
    Movie movie = getMovie(input.getMovieId());  // long in, Movie out (checked)
    return wrapMovie(movie);                     // Movie in, WrappedMovie out (checked)
  }
}

The DSL has two core building blocks. The first, @TaskMethod, is a method that is executed at runtime and can contain arbitrary code. Crucially, it takes and returns real types: a getMovie task can take a long movieId and return a Movie object directly, with no maps and no casts. The second, @WorkflowMethod, is a method that is not executed and cannot contain arbitrary code. It expresses the workflow as ordinary method calls (call getMovie(…), then pass the result to wrapMovie(…)), and a Gradle build plugin parses it into a Conductor JSON definition at build time. The server does not see the Java code. It gets the same JSON it always has.

Code generation is central to how this works. In the @WorkflowStub annotation you declare the task and workflow names you want to use, and at build time the Gradle plugin generates their input and output types along with typed stub methods that you call inside a @WorkflowMethod. Those generated stubs are what let the method body compile against real signatures with full type safety. Because the stubs are generated from names rather than shared code, teams can reuse each other’s tasks and workflows without sharing libraries.

Real control flow, expressed in Java

The examples above are linear, but most production workflows branch, loop, and fan out. The DSL expresses these as the Java constructs you’d reach for anyway, and the SDK parses them into the corresponding Conductor operators. A useful mental model: you’re describing a workflow graph in Java, not running Java, so the parser only supports the constructs that map cleanly to Conductor (if/else, switch, do-while, parallel forks, sub-workflow calls) and asks you to express a few things its way. Iteration, for instance, is always a do-while, since there are no for or while tasks in Conductor.

A common shape is to kick off work, poll until it settles, then branch on the outcome, which is a do-while loop wrapped around an if/else:

@WorkflowMethod(name = "process_asset_workflow", ownerEmail = "team@netflix.com")
default void processAssetWorkflow(AssetInput input) {
  JobStatus status = submitJob(input);

  do {                                        // loops are do-while
    status = checkStatus(input.getJobId());
  } while (status.getState() == "PENDING");

  if (status.getState() == "SUCCEEDED") {     // branch on the result
    publish(input);
  } else {
    flagForReview(input);
  }
}

Independent work runs concurrently with ParallelExecution.fork(…) plus allOf(…) (a FORK and JOIN), and any other @WorkflowMethod can be called as a sub-workflow just by listing it in workflowNames:

@WorkflowStub(
    taskNames = {"movieapp.getMovie", "movieapp.enrichMovie", "movieapp.getArtwork",
                 "movieapp.refreshCache", "movieapp.publishPage"},
    workflowNames = {"movieapp.localize_page_workflow"})
public interface MoviePageWorkflows {
  // stubs not shown for brevity
  @WorkflowMethod(name = "prepare_movie_page", ownerEmail = "team@netflix.com")
  default void prepareMoviePage(MoviePageInput input) {
    // three independent branches, run in parallel
    Fork<WrappedMovie> details = ParallelExecution.fork(() -> {
      Movie movie = getMovie(input.getMovieId());
      return enrichMovie(movie);
    });
    Fork<Artwork> art = ParallelExecution.fork(() -> getArtwork(input.getMovieId()));
    Fork<Void> cache = ParallelExecution.fork(this::refreshCache);

    ParallelExecution.allOf(details, art, cache);   // FORK_JOIN and JOIN

    WrappedMovie enriched = details.get();
    Artwork artwork = art.get();
    LocalizedPage localized = localizePageWorkflow(input);  // sub-workflow call

    publishPage(enriched, artwork, localized);
  }
}

Sharing tasks and workflows across applications

Teams shouldn’t re-implement common tasks. Listing another team’s task or workflow in your @WorkflowStub is enough to call it like a local method, or invoke it as a sub-workflow, and the shared POJO types (a Movie, say) come along directly, with no copy-pasting between codebases.

Feature parity and rapid adoption

The SDK is gaining rapid adoption and has become the primary choice for authoring new workflows at Netflix. We continue to invest in the developer experience, with intelligent IDE plugins, better debugging, and an integrated testing framework so that testing a workflow in code feels like testing any other code.

In production: Netflix Ads

Netflix Ads runs complex, multi-step processes on Conductor with the Workflow SDK, and three examples show the range. Creative ingestion is a heavily branching pipeline where the SDK turned a sprawling JSON blob into type-safe, testable code, and a sequential run strategy eliminated a duplicate-processing race by serializing work per creative. Advertiser certificate management automates a previously manual, hard-to-scale process across several system integrations, with downstream automation when certificates expire. And Data Clean Rooms are privacy-preserving environments where Netflix and a partner run joint analysis on shared data, built on the SDK from day one. Provisioning a clean room is a multi-day sequence (create at a vendor, wait for the partner to join, activate, and eventually archive) that previously relied on hand-rolled async polling. Re-authored as Conductor workflows, each phase became a durable, observable state machine with automatic retries, state that survives restarts, and long partner waits that yield rather than hold a thread.

The common thread is that the SDK made multi-process, multi-integration workflows easy to author and maintain, and let teams lean on Conductor’s reliability and visibility instead of hand-rolling retries and distributed locking.

What’s next

Like the last time we wrote about Conductor, the road ahead is less a fixed roadmap than a wish-list, a sense of where we’re headed, shaped by what teams keep asking for.

On scale, the demands keep growing. As Netflix expands into new content types, like live, games, and podcasts, we anticipate our workflow load could grow 5x in the coming year, and the engine work described above is what gives us confidence we can absorb it without re-architecting again.

On developer experience, the Workflow SDK now covers all task types, so the focus shifts to the experience around writing workflows: a richer integrated testing story so a workflow can be validated against real worker behavior as easily as any other code, deeper agent tooling such as skills, plugins, and MCPs, and safe schema evolution: a schema registry for shared tasks and workflows, with guardrails that block non-backward-compatible changes so they can evolve without breaking the teams that depend on it.

On execution control, sequential is only the first run strategy. A family of richer policies is on the roadmap, including running N at a time per key, idempotent de-duplication, keep-first and keep-latest, and combinations of these, to cover the coordination patterns teams currently hand-build. And managed tasks will keep expanding the ways Conductor can run compute directly, beyond today’s container and serverless function integrations.

None of this is exhaustive, and priorities shift with the workloads we take on. But the direction is consistent: keep the overhead low, keep authoring safe and simple, and keep widening the range of work a single workflow can reliably coordinate.

Acknowledgments

Netflix Conductor is built and operated by the Enterprise Workflow Platform team, with contributions from many partner teams across the company.

Andrew Seier, Anoop Panicker, Christian Johnson, Christine Yu, Jasmine Lee, Jiaofen Xu, Michael Torku, Peter Lau, Prashant Rastogi, Ryan Kastilani Surafel Korse and Victor Lee have contributed to Conductor. We thank Charles Zhao, Sumukh Shivaprakash and the teams in the Content and Business Products and Ads organizations for their guidance and constructive feedback while developing and operating Conductor.

Netflix Conductor: The Next Chapter was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

By Yogesh Nagarur

Introduction

In Part 1 of this series, we shared how Netflix built the Ads Event pipeline and approached it as a system design problem: replacing fragmented pipelines with a centralized collection, enrichment, and a standard data contract. The result was Ads Event Publisher, the system that collects ad telemetry from client devices, enriches it with the context of the ad that was actually served, and publishes a single, unified stream of ad events to every downstream consumer: frequency capping, metrics, billing, measurement partners, and reporting. Any member viewing or account details that move through this pipeline end-to-end are encrypted and handled in accordance with Netflix’s strict privacy and security practices to safeguard member PII.

If ad serving is the “brain” and ad events are the “heartbeats” that feed it, then enrichment is what gives each heartbeat its meaning, turning a bare “[Somebody] watched an ad to completion” signal into “this campaign, this creative, at this price, for this break.” Part 1 described that enrichment step as a single box in the architecture. This post is about what was inside that box, and why we rebuilt it.

Part 1 ended with a look at what was on the horizon: ads on live streams, de-duplication, and richer data signals. A single project delivered all three by rebuilding the Ads Event Publisher around stateful stream processing. We replaced a database lookup with a real-time streaming join in Apache Flink, which let the system scale for Live, took a dependency off the critical path, and replaced the engine of a high-traffic pipeline without downtime.

Let’s dive in.

A quick recap: enrichment and the Metadata Registry

When an ad is served, the ad server knows everything about it: the campaign, the creative, the price of the impression, the targeting context, and the tracking URLs that measurement partners expect us to fire. The client device, by contrast, knows almost nothing: just an opaque token and when to send each event. Enrichment is the act of reuniting the two.

In the architecture from Part 1, that reunion happened through the Ads Metadata Registry, a Key-Value store:

• When the ad server made a decision, it wrote the metadata for that impression into the registry.
• It encoded only reference identifiers into the opaque token returned to the device, keeping the token small.
• Later, when the client fired an event carrying that token, the Ads Event Publisher decoded the token, looked up the metadata in the registry, and stitched the two together into an enriched ad event.

This design served us well. The registry was globally replicated, which gave us a useful property almost for free: an event arriving in any region could find its metadata regardless of where the ad was decided. It future-proofed the token against ever-growing metadata, and it carried us through the first phase of Netflix’s advertising growth.

But as we looked ahead, especially to ads on live streams, the limits of this design became clear.

Why we needed to evolve

The forcing function was Live. Netflix’s live slate has grown rapidly, from NFL games to blockbuster fight nights to the FIFA Women’s World Cup 2027, and viewership keeps growing. That traffic looks nothing like Subscription Video on Demand (SVOD): a single event can surge toward tens of millions of concurrent viewers within minutes, then drain just as fast when it ends. Building the ad pipeline for Live meant building for that trajectory.

To see why the old design could not get there, let’s put some numbers on it. When Netflix streamed the Jake Paul vs. Mike Tyson fight to more than 100 million viewers, a single ad break at that scale would generate an enormous volume of ad events. Serving those ads in the old design would have meant writing metadata for each of them to the Metadata Registry and waiting for acknowledgment before the ads could serve: millions of must-succeed database writes per second on the ad-serving critical path. That pattern created three core problems at Live scale:

1. It was a scaling bottleneck. Every ad and every ad break is a write. Multiply that by tens of millions of concurrent viewers, keep it non-lossy (a dropped write is a dropped ad, and lost revenue), add Live’s spiky on/off pattern, and the write load becomes enormous. But you cannot safely scale the registry up once the event starts. Adding nodes reshuffles data across the cluster to rebalance it, and doing that in the middle of a live event could slow or even fail the very writes you are adding capacity for. That leaves one safe option: pre-scaling for the peak in advance and hoping the forecast is accurate.
2. Fewer dependencies on the critical path. Serving an ad meant a synchronous, must-succeed write to the registry, followed by a wait for its acknowledgment. Every system on the critical path is one more thing that can stop an ad from serving, and the metadata registry had no fallback, so an unhealthy registry meant serving stopped. Taking hard dependencies off that path was the goal, and the registry was the biggest one we could remove.
3. It put event freshness at risk. If writing slowed down under load, or lookups timed out, events spilled into retries and dead-letter queues, and enriched ad events were delivered late. Some systems might be fine, but others, such as frequency capping, cannot. It needs events under a strict SLA to decide whether the next selected ad can run without breaking policies.

So we asked a simple question: the ad server already knows and logs everything about the ad it served. Does it really need to write that to a database on the hot path? Can we match that against the downstream Kafka stream of client events, which is always there, after the fact?

The write leaves the critical path entirely; no shared datastore outage can stop serving, and enrichment becomes something we scale elastically with Live rather than a cluster we babysit.

Matching two streams after the fact is a streaming join, and that is where Flink comes in.

Why Flink

We chose Apache Flink because near-real-time, stateful stream processing is exactly what it is built for, and because it has deep, production-tested platform support at Netflix. Just as important, we already knew how to run it. As Part 1 noted, our Ads Sessionizer is a large stateful Flink job, so operating one at scale, tuning state, checkpoints, and recovery under production load, was familiar ground for the team. Enrichment on Flink was built on that experience rather than starting from scratch.

The new architecture

At its core, the new Ads Event Publisher is built as an Apache Flink streaming job. Each event moves through a handful of stages:

1. Normalize. Client events arrive from several sources, including SVOD, live, playback, and embedded ad breaks, each with a slightly different shape. We first normalize them into a common form.
2. Re-route. Client events and response events may appear in different regions; we inspect and reroute them to the correct job-processing region for a successful join.
3. Join. The heart of the job: we match each client event to the ad-serving response for the same impression, enriching it with the full serving context using a KeyedCoProcessFunction
4. Deduplicate. Revenue-bearing events are de-duplicated, so nothing is counted twice.
5. Publish. We extract measurement beacons for our partners and route the enriched ad event to its downstream consumers.

The join is the interesting part. Client events and ad serving responses arrive on separate streams at different times, so the job holds recent responses in memory and matches events against them over a 60-minute window. Three things can happen:

• The response is already there when an event arrives → we enrich it on the spot.
• The event arrives first → we hold it for a configuration duration (currently set to 60 minutes) until the matching response shows up, then enrich it.
• No response arrives within the window → we mark the event as unmatched for later recovery.

We built the join as a custom Flink operator (a KeyedCoProcessFunction) rather than a built-in windowed join. We do not need a windowed join ordering and time-window semantics, since downstream consumers handle ordering; we simply match each client event to its response by key. Owning the operator at its lowest API form lets us control the matching state directly, including how long we hold it and what we do with an event that never matches. This allows us to tune retention per stream or persist unmatched events to a durable store later without rewriting the join. Our goal is always to build a strong base to reap the benefits later as we scale.

To make that concrete, picture a member watching a live game. When the ad server decides which ad to show, it logs its response, and a few seconds later, the member’s device fires a “first quartile” event carrying the opaque token. Both arrive at the job keyed to the same impression, and because the response is already waiting, the event is enriched on the spot with its campaign, creative, price, and tracking URLs, then published downstream in near real-time.

Why 60 minutes? Almost all events, roughly 99.999% of them, reach the job well before that, so extending the window further would only grow the state store and lengthen checkpoints for a vanishingly small gain. The remaining 0.001% still matters for precise billing and reporting, and we recover those events via a separate path described later on.

Cross-region: matching events where their data lives

The Metadata Registry handled cross-region enrichment for us because it was built on a globally replicated key-value store: a client event could land in any region and still find its ad’s metadata. Global replication was a feature the platform provided (see Netflix’s Key-Value Data Abstraction Layer), so serving an ad in one region and enriching its events in another was simply not something we had to think about.

A streaming join does not come with that convenience, and here the two sides of the join live in different places. The ad serving response stays in the region where the ad was served. The client event, though, is fired by the device, and the device can be anywhere: a member might start an ad on home wifi and switch to mobile data partway through, and that network change can route the event to a different region than the one that served the ad. So a client event is routed to a region that does not have the matching serving response.

The tempting fix is to stream every region’s serving responses into every other region so that each region can match any client event locally. We rejected it: that would mean every region storing every serving response worldwide, the vast majority of which will never be used in the join, which is wasteful in both cost and compute.

Instead, we treat this as a routing problem and route the client event rather than replicating the response. We encode the serving region into the ad token, so when a client event arrives in the wrong region, we forward it to the region that owns its response. For example, an ad is served in us-east-2, the member’s device switches from wifi to mobile data mid-ad, and the resulting event arrives in us-east-1. That region reads the token, sees the ad belongs to eu-west-1, and forwards the event back there, where its serving response is waiting. Only the small trickle of genuinely cross-region events pays for the extra hop.

Catching the long tail of late-arriving events

Sixty minutes covers the vast majority of events, but not all of them. A device may have been offline, an ad paused, or an engagement event delayed by hours. Rather than drop these stragglers, we set them aside and recover them through a companion path: an hourly Spark batch job, orchestrated by Maestro, Netflix’s workflow orchestrator, re-pairs each straggler with its serving response from our data warehouse (which has a far longer retention than our in-stream window) and publishes it through the same downstream sinks. This keeps the streaming job’s memory bounded while still catching the long tail.

This also lets us support event types that arrive well after the 60-minute window, including certain tracking-related events, while keeping the door open to extending the system to additional event types down the line and improving our match rate as needed. We also designed this offline job with reuse in mind: it can replay dead-letter queues and backfill historical data, giving us a single mechanism for both long-tail recovery and operational recovery.

What was hard

Beyond the architecture itself, three things took real work, and together they are what made the pipeline stable, correct, and safe to switch on.

Keeping the core lean.

Operating a stateful streaming join at live-event scale took real tuning, and most of our hard-won lessons come back to one idea: the job’s only job is matching events. Anything else we were tempted to bolt on, such as external lookups or incidental side-processing, was better moved off the hot path, or it eventually showed up as instability under load. Getting this right was the difference between a job that comfortably kept pace with live traffic and one that fell behind. It is unglamorous work, and it is most of the work.

De-duplicating Events

Duplicates are a fact of life here in an event processing pipeline. A client may retry an event, and our upstream delivery is at-least-once so that the same event can arrive more than once. For revenue-bearing events, the ones that feed billing, nothing should be counted twice.

The solution here is a simple and idempotent key. For each event, we compute a stable ID by hashing a few fields that uniquely identify it, such as the ad ID, the event type, and, for repeatable events, an offset or timestamp. The same underlying event always hashes to the same ID, so if we have seen an ID before, we drop the repeat.

What changed with Flink is where we remember those IDs. The previous design kept them in an external key-value store and did a check-and-write over the network for every event; Flink lets us keep that state local to the job instead. We key the stream by event ID, which routes every copy of a given event to the same place, and a local state store with a short time-to-live remembers the IDs we have recently seen. Keying by ID also removes the race the external store had to guard against, since a given ID is handled in only one place at a time, with no network hop on the hot path. Multiple improvements here alone.

We are deliberate about the guarantee: at-least-once delivery with best-effort de-duplication, not exactly-once. In-stream dedup removes the vast majority of duplicates within its window. However, a few can still slip past, for instance, through a cross-region reroute, a late-arriving recovery, or a restart that replays recent events. So we also carry the event ID through to the enriched ad event itself, which lets consumers who need exact counts, such as billing, run their own idempotent de-duplication over their own, longer windows.

Production Rollout

This was the most challenging part of the project. For a system that feeds billing and frequency capping, “trust me, the new one is correct” is not good enough, so we ran the Flink pipeline alongside the legacy one and made it earn its traffic.

Both pipelines continuously processed the full event stream, allowing us to compare them in real time: we published their outputs side by side. We ran automated audits that alerted us when the two drifted apart. Those audits earned their keep, surfacing subtle discrepancies we would never have caught from dashboards alone.

How do we know the enrichment logic between the two systems was equal for the audits to make sense? Early in development, we extracted that logic into a shared module that both the legacy service and the Flink job depend on, so any change to how an event is enriched is applied to both at the same time. A failure in the audits, therefore, pointed to a real difference in the pipeline mechanics.

Cutover was a separate question from comparison, and this is the part that warrants precision. We could not simply send a random percentage of events to each pipeline, because the same event landing in both would be double-counted downstream. Instead, the split was deterministic: a stable key on each event decided which pipeline was responsible for publishing it, so every event was published exactly once, even though both pipelines saw all of them. Raising the threshold moved events from the legacy pipeline to Flink a slice at a time, and we could dial it back instantly if anything looked off.

By the time we fully cut over, the two pipelines agreed on well over 99.99% of events, and we rolled it out region by region rather than all at once. Then came the satisfying part: we turned off the ad server’s writes to the Metadata Registry, and decommissioned the legacy service and the registry itself. The single point of failure on the ad-serving critical path is simply gone. One of the oldest ad serving components at Netflix was deleted.

Operating the Flink platform

Operating a large stateful streaming job in production is its own job, and we did not build that muscle alone. We leaned heavily on Netflix’s Flink platform team. Early on, the job’s checkpoints were slow and heavy. The first checkpoint in particular took a long time, and we were holding more state in memory than we needed. Working with the platform team, we trimmed the state, reduced checkpoint durations, and validated the gains in load tests. That is what keeps the job from stalling during Live bursts.

The same team also gave us operational scaffolding we did not have to invest in, including continuous delivery and canary deployments, so we could ship and roll out the Flink job with the same guardrails and confidence as the rest of our production systems. Netflix operates more than 25,000 Flink jobs (see Building a Scalable Flink Platform at Netflix), and that maturity is what we have to build on. We layered on custom canary metrics that catch issues before a release, CI/CD that lets us stagger rollouts, and a deploy flow that keeps the old job running until the new one has detected healthy checkpoints, so we never destroy a working job before the new job gives us the green light.

The results

• Enrichment is off the critical path for ad serving. The ad server no longer performs a synchronous, must-succeed database write to serve an ad. It logs and moves on, and an entire class of “the lookup store is down, so ad serving is down” failures no longer exists.
• Fewer systems, fewer schemas, fewer hops. We retired both the legacy service and the Metadata Registry.
• Built for Live, and continuously tested for it. An ad break is what this service is designed to handle, so during our journey, we also ran large-scale load tests to simulate real-life traffic. In our latest load test, a single AWS region handled one ad break for a large concurrent audience, processing hundreds of thousands of events per second, joining the large majority of them against their serving responses in real time, and outputting enriched ad events at a comparable rate, all with negligible timeouts and errors. We test this throughout development and keep doing so as live audiences grow, and we are not at the ceiling yet.
• We delivered the Part 1 roadmap. Live-stream ad events, in-stream de-duplication, and richer real-time enrichment all shipped on this foundation.

Key takeaways

• Take the database off the hot path when you can. Our biggest win wasn’t raw performance — it was blast radius. Trading a synchronous write-and-wait for a fire-and-forget log plus a downstream join removed an entire failure mode from ad serving.
• Match events where their data lives. Real-time joins beat global replication by routing events to their data rather than copying the data everywhere.
• Earn the cutover. Running both systems in parallel and diffing their output on live traffic was slower than a flag-day switch — and it’s the only reason we trust the result.

What’s next

We have an exciting list of projects on the horizon: scaling the live path for the next tier of concurrent audiences, bringing the remaining delayed engagement events fully into the streaming world, reducing the Flink state size we need to manage, working on bringing even more resilience to the eventing processing pipeline, and continuing to add new event types and richer signals for our downstream consumers.

Re-architecting the engine of a running, revenue-critical pipeline in place was a real multi-team effort. If building real-time systems at this kind of scale sounds like your thing, Ads Engineering is hiring.

Acknowledgements

This was a multi-team effort across Netflix. Thank you to everyone who designed, built, validated, and operated the new pipeline:

Ads Engineering: Yogesh Nagarur, Kinesh Satiya, Jeet Mehta, Nitin Saini, Persis Shah, Guangyu Chen, Jimmy Liu, Riley Berton, Simon Spencer, Aadesh Patel

Ads Data Engineering: Eugene Yusim, Brandon Nicklas, Indu Narayan

Ads SRE: Max Michaels, Kevin Chung, Carlos Avalos, Brian Christie

Data Movement: Mark Cho, Herbert Wang, Shraddha Ladda, Nikhil Sulegaon, James Lubin, Daniel Trager

Behind the Scenes: Evolving Netflix’s Ads Event Pipeline for Live — Part II was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

By Oleksii Tkachuk, Rajiv Shringi, Chris Lohfink, Abe Ratnofsky, Kartik Sathyanarayanan

Introduction

In an earlier post, we introduced the TimeSeries Abstraction, our high-throughput, low-latency platform designed to store and query temporal event data at scale. It has since become the operational backbone for Netflix’s largest datasets, which now span multiple petabytes.

However, managing such large datasets in hot storage, tuned for low-millisecond latency access, is not cheap. At Netflix, cost-efficiency is a continuous discipline rather than a temporary initiative. This post shares two solutions that significantly reduce storage costs by continuously moving older, less frequently accessed data into cheaper object storage:

• Solution 1: Uses Apache Kafka and Flink to stream data directly to S3
• Solution 2: Leverages Cassandra backups in S3 and pluggable Memtables to read data directly from S3

We’ll start with the Kafka/Flink-based design, then describe how we evolved to a native Cassandra-based cold tier.

Challenges

As Netflix’s TimeSeries datasets have expanded, we’ve encountered several compounding challenges:

• Growing retention demands: Users often push for increasing data retention to serve a broader range of use cases, which in turn increases the storage costs.
• Escalating costs: Maintaining massive, dense clusters for infrequently accessed historical data is prohibitively expensive.
• Sparse historical access: Most queries target recent events, but older data must remain readily available for occasional audits and investigations.
• Operational strain: High storage density on hot nodes complicates scaling, deployment, and cluster stability.
• Latency requirements: Even historical data sometimes requires sub-second query performance for internal tooling and analysis.

This creates a clear opportunity: keep the most recent data on fast storage and migrate older, rarely read data to cheaper storage while preserving a consistent API. Done well, this can meet user and operational needs while greatly reducing our infrastructure spend.

TimeSeries Abstraction

Before we dive into the cold storage solutions, here is a quick recap of how TimeSeries partitions its dataset into multiple tables, each covering a window of time we call Time Slices.

This enables several key optimizations, most notably allowing a table to become immutable once it crosses the write boundary, which lets us safely apply further optimizations. For more details, refer to our previous post.

This slice information is critical and is maintained in a dedicated metadata table:

Creating new Time Slices and maintaining their metadata is a primary responsibility of the TimeSeries control plane. The TimeSeries data plane uses this information to correctly route the read and write traffic to the provisioned tables.

With that background, let’s look at the different solutions we built for data tiering.

Solution 1: Streaming with Apache Kafka and Flink

Our first solution followed a familiar streaming pattern using Apache Kafka and Flink, where we streamed data to S3 in realtime while simultaneously persisting it to Cassandra.

Getting Data Into S3 Efficiently

To optimize data streaming to S3, the system hash-partitions Time Series IDs into buckets. This partitions the stream and lets dedicated Flink tasks handle each bucket, ultimately batching and reducing the write throughput to S3. Flink workers batch events into Parquet files grouped by event time within the hour. These files are great for high-throughput streaming writes, but not for low-latency reads.

Enter the Compaction job.

Compaction Job to Optimize Reads

The compaction job runs daily and merges small Parquet files into a single, optimized daily Parquet file per namespace.

The job further optimizes the Parquet files in the following ways:

• Small Row Groups: Creates small (4–8 MB) row groups to minimize lookup cost. We can then use the Parquet Java library to perform range reads of row groups from S3. This is intentionally well below Parquet’s recommended row group floor — smaller groups were better because our reads are dominated by point/range lookups rather than full scans.
• Event Sorting: Sorts events within a row group by time, eliminating brute-force scans when querying for a time range.
• Bloom Filters: Caches Bloom filters and file metadata in EVCache, allowing the TimeSeries query planner to efficiently skip unnecessary reads from S3.

Once the compaction job completes, all compacted files per partition are recorded in the TimeSeries control plane, and the hot storage slices can now be truncated!

On some of our largest datasets, this job typically takes about 12 hours, prioritizing lower compute costs over execution speed.

Streaming Cold Tier Read

Once compacted files are optimized in cold storage, they are ready to be read. The TimeSeries data plane fans out a given query across both the hot and cold tiers and merges their result. The users still use the same query interface:

The Cold Tier Data Plane uses compaction metadata for aggressive planning and pruning, which dramatically reduces the S3 query fan-out. As a result, retrieving a full day of data requires at most three sequential S3 reads, even in the uncommon case where an ID spans two row groups. This yields a p50 latency of ~150 ms and a p99 latency of ~500 ms per read.

At the time of writing, the streaming cold tier implementation was processing up to 1,250 reads per second from cold storage globally:

Why not stop here?

The streaming cold tier has delivered real savings, shrinking both Cassandra’s EBS footprint and the Cassandra backup footprint in S3. But the streaming infrastructure itself is expensive and adds moving pieces that widen the team’s on-call scope:

We wanted to improve this further and asked ourselves:

• Can we avoid streaming data into the object store and the associated data transfer costs?
• Can we reduce the compute and operational overhead of new components?

That led to a more radical question: Can we read directly from existing Cassandra backups in S3?

Solution 2: Cassandra Cold Storage for Immutable Data

At Netflix, we back up Cassandra data into S3 at regular intervals for disaster recovery. The idea for tiering is simple:

Read directly from these S3 backups instead of paying a separate data transfer cost to S3.

Once TimeSeries marks a hot Time Slice as ready for cold storage, we can customize Cassandra to read directly from its backups, allowing us to delete the on-disk data copy. This would also incur minimal changes to the TimeSeries read path!

To make this viable in production, we had to answer a series of “How…?” questions.

How Can We Read the Backups Efficiently?

We introduced a new compaction strategy in Cassandra, which we call ReadOnlyCompactionStrategy (ROCS). Here is how it works:

Once a time slice crosses the user-configured cold-tier boundary, TimeSeries stops accepting writes for that time slice making it immutable. This immutability unlocks significant optimization opportunities:

• SSTable overlap: Standard strategies (LCS, TWCS) often produce SSTables that overlap in the token space, and so a single partition key may exist across multiple files.
• Read fan‑out: Because a read must touch every SSTable covering the key’s token range, this overlap turns a single point-read into multiple lookups, increasing latency across all tiers.

ROCS re-compacts all SSTables once, aligning them precisely with token ranges:

ROCS applies more broadly to all TimeSeries use cases, driving tail latency down on all storage tiers by reducing the SSTable fan-out during read operations:

Once the immutable Time Slices have been optimized, it is time to trigger backups.

How Would We Leverage Backups?

Cassandra backs up every node in the cluster, organizing each backup on S3 as a declarative manifest, along with the SSTables backing the keyspace’s tables:

 {bucket}/
  └── {prefix}/                             ← cluster identity (env_backup/hash_app)
      └── {token}/                          ← node identity (ring token)
          ├── META_V2/                      ← manifests, one per snapshot
          │   ├── {snapshotInstantMs}.json
          │   ├── {snapshotInstantMs}.json
          │   └── ...
          └── SST_V2/                       ← actual SSTable files
              └── {backupTimestamp}/        ← snapshot timestamp
                  └── {keyspace}/
                      └── {table-with-uuid}/ 
                          └── {compression}/
                              └── {encryption}/
                                  ├── {component-file}
                                  └── ...

The backup manifest orchestrates the backup, listing the paths to all its SSTables:

manifest.json
  ├── info
  │   ├── appName              "cass-cluster"
  │   ├── region               "us-east-1"
  │   ├── snapshotInstantMs    1748908800000
  │   ├── backupPathPrefix     "prod_backup/1234_cass/token"
  └── data[]
      └── [0]
          ├── keyspaceName     "ks"
          ├── columnfamily     "tbl-a1b2..."
          └── sstables[]
              └── [0]
                  └── components[]
                      ├── na-1-big-Data.db
                      ├── na-1-big-Index.db
                      ├── na-1-big-Filter.db
                      ├── na-1-big-Statistics.db
                      ├── na-1-big-Summary.db
                      ├── na-1-big-CompressionInfo.db

Alongside the large Data.db file, backups include a compact set of index, metadata, and integrity files. This structure allows us to quickly locate, validate, and restore specific partitions or data ranges without scanning the entire dataset. Furthermore, since the manifest catalogs every S3 path, we can download most of these components directly to the Cassandra node (excluding the Data.db file) where they occupy negligible space. These local components enable efficient query planning before any data is read from S3.

How Would Cassandra Read From Backups?

Cassandra reads typically pull data from two sources: the Memtable, which holds recent unflushed writes, and flushed SSTables on durable storage. As of Cassandra 4.1, we can leverage pluggable Memtables to redirect reads from “cold” tables directly to the backup storage in S3.

-- create table for a given Time Slice
CREATE TABLE data_20260616_00_04 (
    ...
    ...,
    PRIMARY KEY ((time_series_id, time_bucket, event_bucket), event_time, event_id, key)
) WITH CLUSTERING ORDER BY (event_time DESC, event_id DESC, key DESC)
    ...
    AND compaction = ...
    AND compression = ...
    AND memtable = 'backupmemtable:timestamp=1781953200000' ← Custom memtable
    ...

The timestamp marks the point at which the backup is guaranteed to cover the full table, and the table is guaranteed to be immutable.

Opening a backup Memtable kicks off a state-machine initialization that pulls the index data from S3 onto the node once:

This process ensures all SSTable components are downloaded reliably through several resilience mechanisms:

• Retries and backoff: All S3 operations and file-system cache writes use retries with exponential backoff.
• Resource isolation: Initialization runs on a dedicated thread pool, isolated from normal Cassandra operations.
• Partial download protection: If a downloaded component’s size doesn’t match the manifest, it is deleted and re-downloaded.
• DDL change protection: A .timestamp file marks completed initialization and ties it to the backup timestamp. If the table’s backup timestamp changes, this file is removed and initialization restarts for the new timestamp.
• Read protection: Memtable initialization is asynchronous; reads to an initializing memtable block for at most half the read RPC timeout.

Once initialization has pulled every SSTable component (except Data.db) from S3 onto the local disk, the Memtable is ready to serve reads. Here is what the read process looks like:

To minimize latency, the system parallelizes S3 reads across requests, SSTables, and individual chunks. A shared cache deduplicates fetches and prefetches data, so chunks are ready before they are needed.

We further optimize cluster and table settings to reduce the number of S3 calls:

• Larger compression chunks: Reduce S3 calls per partition by reading fewer, larger chunks (accepting some over-fetch on small reads).
• Read repair disabled: Avoid expensive, unnecessary repair traffic against immutable, pre‑repaired backup data.
• Speculative retries disabled: Prevent S3’s higher latency from triggering redundant, overlapping reads.

How Would We Know When It Is Safe To Read From Backups?

However, before the Memtable can point to a backup, we must confirm two things across the whole ring, and we use Cassandra Virtual Tables to do that.

First, that ROCS re-compaction succeeded:

cqlsh> SELECT * FROM netflix_views.cluster_view                                                                                                                                                          
       WHERE keyspace_name = 'system_views' AND table_name = 'sstable_tasks';

keyspace_name | table_name    | host     | value                                                                                                                                                        
--------------+---------------+----------+------------------------------------------------
system_views  | sstable_tasks | 10.0.0.1 | [{"table_name":"<.>","kind":"compaction", ...}]
system_views  | sstable_tasks | 10.0.0.2 | [] <---- no active compactions
system_views  | sstable_tasks | 10.0.0.3 | []

And second, that backups were fully captured (denoted by uploaded=True):

cqlsh> SELECT * FROM netflix_views.complete_backups                                                                                                                                                      
       WHERE keyspace_name = 'my_ks' AND table_name = 'my_table';

keyspace_name | table_name | timestamp     | num_tokens | num_uploaded | uploaded                                                                                                                       
--------------+------------+---------------+------------+--------------+----------
my_ks         | my_table   | 1765211400000 |          3 |            3 |     True
my_ks         | my_table   | 1765213200000 |          3 |            2 |    False

How Would We Automate This Entire Flow?

The TimeSeries control plane automates the entire transition as slices cross the immutable boundary:

Each slice can be in any lifecycle stage at any time without impacting system availability, and the system remains resilient to delayed backups or lagging compactions.

How Would Cassandra Handle The Additional Load?

Serving data from S3 is much more CPU-intensive than reading from local storage. A sudden spike in read requests can overwhelm the Cassandra cluster and affect the hot path, turning a cost-saving feature into a reliability risk.

To avoid this, cold reads can optionally run on a separate Cassandra cluster that points its backup Memtables at the hot cluster’s backups:

The cold Cassandra cluster must match the original cluster in topology but need not match it in hardware! We can run the cold cluster on much smaller, cheaper, storage-light instances.

Real World Performance

Cassandra Cold Tier was rolled out to the most storage‑intensive TimeSeries Abstraction clusters. It currently stores over 15 PB of compressed data in Cassandra cold storage and has allowed us to cut attached storage significantly on all affected clusters:

At the same time, it serves thousands of reads per second from cold tier storage in any given region:

Cassandra Cold Tier improves p90 latency by ~30% over the previous cold storage implementation, thanks to more efficient S3 access patterns and smaller read sizes, while keeping p99 latency under ~500 ms. Cassandra already handles bursts of cold reads gracefully via its chunk cache, so this design preserves that same resilience for the S3-backed cold tier:

Acknowledgements: Special thanks to our stunning colleagues who contributed to the TimeSeries Abstraction’s success and collaborated with us on the cold storage solutions: Kaidan Fullerton, Matt Lehman, Sumanth Pasupuleti, Carlos Matias Herrero, Nick Pavlakis, Bryan Keller, Francesco Di Chiara

TimeSeries Tiered Storage Journey: Kafka/Flink Streams to Native Cassandra Cold Reads was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

By Kevin Zielnicki, Guy Aridor, Aurélien Bibaut, Allen Tran, Winston Chou, and Nathan Kallus

Introduction

Personalization is everywhere on the internet. It shapes the products we discover, the stories we read, the music we hear, and the movies and TV shows we watch. At Netflix, personalization has been central to the member experience for decades — from the Netflix Prize to our modern recommender system that helps members navigate a large and constantly changing catalog.

But personalization is not only an algorithmic problem, it is also a measurement problem. How do we know how much value personalization creates? More precisely, how do we separate the appeal of a title from the value created by recommending that title to the right member at the right time?

In a new article in the International Journal of Industrial Organization — which we are also presenting at this year’s ACM Conference on Economics & Computation (EC ‘26) — we develop a model of member behavior, title choice, and personalized recommendations. The model lets us estimate how recommendations affect both whether members choose to watch and, if so, which titles they choose. In this post, we summarize our approach and main findings.

The Measurement Challenge

A natural first attempt at valuing recommendations is to ask: what share of viewing comes from titles that were recommended? This is an intuitive metric, but it fails to answer the causal question at hand.

The problem is that recommendations are not randomly assigned. A title appears on a member’s homepage precisely because the recommender system predicts that the member might want to watch it. If a member watches that title, was it because the recommendation caused the member to watch it? Or would she have watched it anyway because of her underlying preference for that kind of title? This is the core measurement challenge: recommendations are targeted to demand, so viewing a recommended title combines the effect of personalization with the member’s underlying preferences.

One way to solve this would be to run a large experiment that replaces Netflix’s personalized experience with an unpersonalized one. That would create clean variation — but it would also be a poor member experience. Personalization is a central part of how members locate valuable titles in a large catalog, so turning it off broadly is not an acceptable measurement strategy.

Instead, we take a hybrid approach. We build a behavioral model that separates members’ underlying preferences from the incremental effect of recommendations. The model uses the natural variation in the recommendation system, for example due to algorithmic exploration, to estimate how recommendations affect choice. Then, to validate the model predictions about title substitutability, we use small-scale A/B tests that “nudge” the salience of a handful of titles exogenously, generating ground-truth substitution patterns for a subset of titles. This lets us combine the scale and richness of modeling with the credibility of experimental evidence.

A Model of Member Choice

To measure the value of personalization, we need a model of what members would watch under different recommendation systems. Our model treats Netflix viewing as a day-by-day choice process. On each day, a member chooses among the titles available in the catalog, or she chooses not to watch anything. This gives us a concrete target to estimate: the probability that a member watches each title on a given day, as a function of their preferences, their viewing history, and the recommendations they see.

A simple version of this model would be too restrictive. It might assume, for example, that removing one title from the catalog shifts viewing toward all other titles in roughly proportional ways. But that isn’t how member choice works. If a member is considering a romantic comedy, the closest substitutes are likely other romantic comedies, not every title on Netflix equally. For similar reasons, two members may react very differently to the same recommendation.

To capture this complexity, we use a low-rank choice model. The model learns a representation of each title and a corresponding representation of each member’s current preferences. These representations are not limited to manually specified features like genre, cast, or language. Instead, they are learned from viewing behavior itself, allowing the model to capture nuanced similarities between titles that may be difficult to describe with coarse features, such as genre or release date, alone.

Member preferences are dynamic. What someone wants to watch today depends on what they have watched recently. A member who just finished a crime thriller may be in a different preference state than the same member after watching a stand-up special or a kids’ movie. Rather than estimate a fixed preference vector for each member, the model summarizes each member’s past viewing history into a current preference state. This lets us capture heterogeneous and time-varying tastes while also scaling to millions of members.

The key component for our purposes is that the model explicitly parameterizes the effect of recommendations. When a title is recommended, the model gives it an additional recommendation effect, or “bonus,” that increases its probability of being chosen. This bonus is a reduced-form way to capture two related mechanisms. First, recommendations can create awareness by helping members discover titles they may not have known about. Second, recommendations can provide information by signaling that a title is likely to be a good match.

Once estimated, the model becomes an engine for predicting counterfactuals. We can ask what would happen if a title were not recommended, if a different set of titles were recommended, or if the current recommender system were replaced with simpler alternatives. Because the model separately represents member preferences, title appeal, and recommendation effects, it lets us estimate not only how recommendations affect total viewing, but also how they change which titles are watched.

The model is only useful, however, if it captures real substitution patterns. The next step is therefore validation: comparing the model’s predictions to randomized experiments that directly perturb, on a small scale, the visibility that each title receives.

Validating the Model with Randomized Experiments

The model relies on an important assumption. After conditioning on the member, title, and state information available to the recommender system, we treat the remaining variation in whether a title is recommended as if it were randomly assigned. In our setting, this assumption is reasonable: we observe all of the information available to the recommender system, so we should be able to model its behavior accurately. But it is still an assumption. A useful model should not only fit historical viewing patterns; it should also predict what happens when recommendations change for external reasons.

Thus, to validate the model, we used randomized experiments that introduce variation in title visibility exogenously. These experiments mirror the kinds of variation that occur routinely when recommendation algorithms are updated: some titles receive more impressions, while others receive fewer. The key difference is that, in the validation experiment, this variation was prespecified so that we could estimate clean experimental benchmarks.

Specifically, we randomly assigned members to treatment arms that slightly nudged recommendations towards and away from different categories of titles. This let us measure diversion ratios: when a set of titles becomes less visible, how much viewing shifts to other titles, and how much shifts out of Netflix viewing altogether?

We then asked whether the model could reproduce those experimentally measured substitution patterns. We estimated the model on the control group, simulated the same recommendation changes, and compared the model-implied diversion ratios to the experimental ones. The agreement was strong: the model’s predicted diversion ratios had a correlation of 0.86 with the experimental estimates and an R-squared of 0.73.

This validation gives us confidence that the model captures counterfactual substitution patterns accurately. Subsequently, we ask what would happen under more dramatic changes, such as different recommender systems.

Counterfactuals: Aggregate Consumption and Diversity

With a validated model in hand, we can move from measurement to counterfactuals. Instead of asking only what happened under the current Netflix experience, we can ask what would have happened if members had seen recommendations generated by a different system.

We compare the current recommender system to three baselines:

1. The first is random recommendations: every member receives titles selected uniformly at random from the catalog. This is not intended as a realistic product experience, but it helps isolate the value of having recommendations that are relevant.
2. The second is popularity-based recommendations: every member receives broadly popular titles. This is a natural non-personalized benchmark.
3. The third is a basic matrix factorization model, a classic personalized recommendation approach and a useful baseline for older generations of recommender systems.

In each counterfactual, we replace the recommended titles while keeping the rest of the choice environment fixed. This lets us estimate how much viewing would change if the recommendation policy changed, holding constant the catalog, members, and model of member choice.

The results show that the current recommender system generates substantially more engagement than each benchmark. Relative to the current system, random recommendations reduce engagement by 16%, popularity-based recommendations reduce engagement by 12%, and matrix factorization reduces engagement by 4%. These are large differences at Netflix scale. Even the comparison to matrix factorization — a personalized baseline — suggests that continued improvements in personalization translate into meaningful gains in how effectively members find titles they want to watch.

Engagement is only part of the story. A recommender system could increase viewing while concentrating attention on a narrow set of already-popular titles. That would be a worse experience for members whose tastes are not well represented by those titles, and it would make poorer use of a broad catalog.

We therefore also examine consumption diversity: how broadly viewing is distributed across titles. The current system performs well on this dimension too. Popularity-based and matrix-factorization recommendations both substantially increase concentration, meaning that fewer titles receive meaningful viewership. Random recommendations produce slightly more diversity, as expected, but at a large cost to engagement.

Taken together, these counterfactuals show that the value of personalization is not just that members watch more. It is also that members are matched to a broader set of titles. The current recommender system increases engagement while preserving a wide distribution of viewing, reflecting the diversity of member tastes across the Netflix catalog.

Decomposing the Impact on Titles

The aggregate counterfactuals show that personalization increases engagement while maintaining broad consumption across the catalog. We can also use the model to ask a more granular question: why does a recommended title get watched?

There are three conceptually distinct channels:

1. The first is exposure: simply showing a title to a member can increase the probability that the member watches it. This captures the basic value of surfacing a title that may otherwise have been hard to find.
2. The second is selection: recommended titles are shown to members who already have a relatively high baseline probability of watching them. This channel doesn’t lead to incremental engagement, but it can smooth the discovery experience and help users feel that the recommender system understands their preferences.
3. The third is targeting: the recommender system may identify members for whom the recommendation itself is especially effective, meaning members who are much more likely to watch the title when it is recommended than when it is not.

The distinction matters because not all recommendation-driven viewing should be interpreted the same way. If recommendations worked only through exposure, then any reasonably visible surface could generate much of the same value. If they worked mostly through selection, then recommendations would mainly be finding members who were already likely to watch a title. Targeting is different: it captures whether the system is placing titles in front of members for whom the recommendation changes the decision.

Our model lets us estimate these channels title by title. For each title, we can compare the predicted probability that a targeted member watches it when recommended to the probability that an average member watches it when not recommended. We then decompose that difference into selection, exposure, and targeting.

We first show that exposure is meaningful: showing a title to an average member roughly triples its consumption probability relative to the baseline. Exposure is not the main driver of differences in consumption, as mechanical exposure explains only about 7% of the gap under the current recommender system, while selection and targeting account for the vast majority of the difference and the targeting effect is nearly seven times larger than the exposure effect.

This pattern is especially informative when compared with simpler recommendation systems. Random recommendations are dominated by exposure because they do not personalize: differences in consumption mostly reflect differences in what titles are shown. Matrix factorization changes this by making recommendations personal, reducing the role of exposure and leaving exposure, selection, and targeting to contribute nearly evenly to consumption differences. The current recommender system goes further: its gains come less from mechanical exposure, but more from targeting the right titles to the right viewers.

Targeting is also not uniform across the catalog. It is strongest for mid-popularity titles: titles with enough potential audience to be broadly relevant, but not so much mass appeal that everyone already knows about them. Very popular titles need less help from targeting, while very niche titles may have too small an audience to target effectively. The largest gains come from the middle of the catalog, where personalization helps connect specific titles with specific member tastes.

Conclusion

Personalized recommendations are valuable because they help members navigate a large catalog and find titles they want to watch. They are also valuable to Netflix because they make the service more useful, more engaging, and better able to serve a wide range of tastes.

But measuring that value is not straightforward. Recommendations are targeted by design, so we cannot simply compare recommended and non-recommended viewing. At the same time, turning off personalization at scale would create a worse member experience.

Our approach combines a large-scale model of member choice with true experimental validation. The model lets us separate underlying title preferences from the incremental effect of recommendations, and the experiments verify that the model captures real substitution patterns.

Using this framework, we find that personalization affects both how much members watch and which titles they watch. The value of recommendations comes not just from exposure, but from matching the right titles to the right members.

Measuring the Impact of Personalized Recommendations was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

by Olek Gorajek, Samuel Redai, Yian Shang

Intro

Our Netflix journey from data to dashboards, particularly within the space of the experimentation platform, always presented a unique set of challenges. This blog post aims to illuminate these hurdles and explain how an innovative open-source metric platform became instrumental in overcoming some of them. We’ll showcase the critical role that DataJunction played in streamlining our data workflows and enhancing our ability to derive meaningful insights from our vast datasets.

Metrics are the foundation of data-driven decision-making, but at many organizations, they are scattered, inconsistently defined, and difficult to discover or trust. At Netflix, this challenge first became apparent in our experimentation platform, where reliable metrics are essential for interpreting test results. But the complexity of the original tooling made it very difficult to understand and introspect these metrics both in terms of their definitions as well as their values.

Initial Problem

Scattered metric definitions, redundant datasets, and pipelines are often a natural effect of a fast-moving company with a large and powerful analytical workforce. This is a common problem in big data-driven firms, and it is not easy to solve across multiple departments, which are often governed by somewhat different rules and regulations. An easy-to-use, well-governed, and centralized semantic layer can help with these issues. Our initial problem was most evident in our Experimentation Platform team, where the previous system required:

• extensive business knowledge
• familiarity with experimentation conventions
• and complex engineering practices

This led to metric authoring processes that would take weeks, which was a prohibitive requirement for Data Scientists and Analytics Engineers who wanted to onboard new metrics to the Experimentation Platform.

Enter DataJunction

To address this problem and to set ourselves on the path to a successful semantic layer solution in the future, we looked at several candidates. Among other solutions we evaluated Mando (an internal framework), Minerva (an open source semantic layer from AirBnB), and DataJunction (another open source solution, “DJ” for short). All of them had some strength, but DJ offered one thing that the others did not at the time: a solid SQL parsing and SQL generation engine. At the time of the POC, DJ was also the only open-source solution that stored all its definitions in a connected graph, similar to how relational databases store metadata on their views and queries. Easy “SQL wrangling” with rich metadata was the main feature that convinced us to try it for our immediate use case.

DataJunction originated from a similar internal Facebook project created (around 2015) by the data science and infrastructure (DSI) team, mainly as an answer to the need for metrics portability. During several iterations of the Deltoid project — which was the main experimentation platform at Facebook at the time — users were required to rebuild and repopulate their metrics when they wanted to move to the next version of Deltoid. But after implementing DJ and abstracting all metric definitions the team offered seamless migrations by separating the concerns between the metrics and the stats computation platform. That project was subsequently renamed to MDF (Metrics and Dimensions Factory) and promoted as a generic semantic layer used widely by various departments. In 2021, one of the Facebook engineers (Beto Dealmeida), started the DJ Github project and built out some basic functionality. Then together with some folks at Netflix expanded the scope further, and soon after that the project was introduced at Netflix. Following that introduction, three Netflix engineers (authors of this blog) together with a couple of external contributors, most notably Nick Ouellet, have been building out the remaining features of the platform.

Architecture Decisions

A critical design constraint for building DJ at Netflix was resource scarcity. We refused to compromise on product quality, choosing instead to limit the feature set. Although DJ was clearly capable of addressing many simultaneous problems across diverse use cases, our lack of adequate resources and official support meant we developed it as a side project for over two years. Our hope was that demonstrating its value would eventually secure management attention.

API First, UI and Other Interfaces Later

A metrics platform doesn’t need a flashy front-end to deliver value — its core is the semantic layer, which stores a consistent set of dimensions and metrics and serves this information through a rich API service. You can see below how we structured the main components of DataJunction in the Netflix implementation. Our DJ services (in the middle) interact with the underlying data (on the left) and supply our access layer with the business definitions in the form of SQL w/ metadata or directly with data depending on the client request.

Consistent Graph First, Other Features Later

A complete semantic layer system can seem like an octopus, with its tentacles reaching out into many functional areas and can expand from a basic semantic metadata stored for some datasets to full dashboard functionality. Let’s call out some of them:

• semantic metadata authoring and management and monitoring
• metric and dimension query / definition generation
• metric and dimension data generation (requires a query engine)
• data visualization and data tracking
• performance optimization (aka caching or materialization)
• integration layer with other tools and
• any / all the above functionality served as: UI, API, client libraries, MCP servers and more.

Of course it is extremely hard to provide the best in class functionality across all these layers, so with our small “unofficial” team we decided to focus on the most important piece: solid semantic layer engine (and API) with the ability to parse SQL as the input to the graph and generate SQL as the serving layer. That is the core of the system and everything else can be added later with less effort and w/o the need to rebuild the core.

Cube as The Perfect Consumption Protocol

Analytical multi-dimensional cubes have been a fundamental concept for decades, with their representation integrated across various data processing and analysis platforms. While definitions and features may vary between tools, they all serve the common purpose of encapsulating, or abstracting, a subset of semantic objects.

In DataJunction, we adopted cubes as first-class features for two main reasons: their familiarity to users and their suitability as a serving interface. They efficiently deliver both the semantic definitions (of metrics and dimensions) and fast data access. See an example of data “flow” graph (dotted lines) and dimensional relationships graph (solid lines) below.

We simplified the creation of basic cubes, allowing them to easily aggregate arbitrary collections of shared metrics and dimensions. Furthermore, accessing, managing, and visualizing these cubes is straightforward for any client integrating with DJ. Custom integrations, such as for Netflix’s internal fast Druid data analysis tool (Data Explorer → see below), became trivial to build, and we also made it easy to materialize these cubes directly in our Druid clusters.

One of DJ’s most powerful client integrations is the ability to link any DJ-originating metric definition back to the DJ ecosystem. This creates a valuable feedback loop: troubleshooting and introspecting business dashboards becomes simple if they display DJ-defined metrics that include links back to the definition source. This link then makes inspecting and running ad-hoc analysis, by modifying existing metrics, a trivial process. This integration is not only possible with our inhouse tools, but also with other well known open-source products, like Superset.

Experimentation at Netflix

Metrics are the backbone of experimentation. They translate data into signals that tell us whether a change is meaningful, safe, and worth scaling. Without trustworthy metrics and dimensions, experiments risk becoming inconclusive or misleading, so a robust metrics layer becomes crucial for data-driven decisions.

At Netflix, the original metrics layer integrated into the experimentation platform was overly complicated. Creating new metrics not only required tribal knowledge of experimentation data artifacts, but also familiarity with complex engineering workflows. This unnecessary friction meant that onboarding a single new metric could take weeks, slowing adoption of new metrics and hindering experimentation.

This is where DataJunction comes in: it provides a central hub for defining metrics and dimensions that can easily be shared and explored via the semantic graph. It also provides flexible, user-friendly interfaces (including a UI, API, and various clients), making it easy for users to author and manage their metrics and other semantic models.

Conclusions

With experimentation domain well supported by DataJunction, we would like to take on next steps to unifying of the general analytics at Netflix and to make it easier for Netflix users to both:

• find and use metric definitions across verticals (defined in DJ)
• easily access, modify and experiment with new metrics across Netflix data stack

That will require building out strong integrations between DJ and other tools, but that has been underway and it is one of the most enjoyable parts of this project, where most of the feature exists and we are simply enabling existing systems to exchange information in a structured and reliable way. With a system like DataJunction our internal LLMs can deliver accountable information about the business metrics as well as explanation of how they are defined and used across Netflix.

Related resources:

• 2023 : DataJunction at Netflix (slide deck)
• 2025 : DataJunction talk at Data Engineering Open Forum 2025 (video)
• 2025 : DataJunction: Unifying Experimentation and Analytics (blog)

DataJunction as Netflix’s answer to the missing piece of the modern data stack was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Authors: Divya Gadde, Ko-Jen Hsiao, Dhaval Patel and Moumita Bhattacharya

The motivation for building the Netflix Foundation Model for personalization was laid out in this previous blog post. The Netflix homepage is powered by several specialized models and it requires significant time and resources to maintain and continue innovating on these individual models. Therefore, we decided to centralize member preference learning and have one powerful model learned from comprehensive user interaction histories and content data at a large scale and have this model distribute its learnings to other models.

The previous two blog posts(1, 2) focus on the development of the Foundation Model and here we talk about how we integrate the Foundation Model into various applications. We found that there is extensive literature focused on the training and inference of large-scale transformer-based models. However, there remains a significant gap in practical guidance and research on how to effectively and efficiently integrate large models into existing production systems. At Netflix, we experimented with 3 distinct integration approaches — embeddings, subgraph and fine-tuning. The motivation for trying different approaches stems from varying application needs. Specifically, different teams have different latency requirements, tech stack constraints and different appetites for leveraging the full power of a large foundation model.

In this blog, we go over the 3 approaches we experimented with and which are now used in production for different use cases. Each integration approach has its own set of motivations, tradeoffs, complexities and learnings. The following sections go into the details of each of these integration approaches.

Embeddings from Foundation model

Transformer style foundation models naturally lend themselves to generating comprehensive user profile and item representations. Our approach for generating embeddings includes getting the hidden state of the last user event as the profile embedding and weights from the item tower as item embeddings.

Netflix has an elaborate set up for refreshing the embeddings every day. First, the Foundation Model is pre-trained from scratch every month. Then, the monthly pre-trained model is fine-tuned everyday based on the latest data. The daily fine-tuning also expands the entity id space to include the newly launching titles. After the daily fine-tuned model is ready, we run batch inference to refresh the profile and item embeddings and publish them to the Embedding Store.

An important part of this setup is the embedding stabilization technique (paper). When the pretraining job retrains the model from scratch, the embedding spaces between different runs are completely different due to the random initialization in the model. In addition, the embeddings start to drift when the daily job fine-tunes on the latest data, despite the daily fine-tuning job warm-starting from the previous day’s model. Therefore, it is important to map the embeddings generated everyday into the same embedding space, so that downstream models can consume pre-computed embeddings as features.

This embedding approach relies heavily on the Embedding Store, which is a feature store specializing in embeddings built by the platform team at Netflix. The embedding store makes producing and consuming embeddings easy by taking care of versioning and timestamping the embeddings. The Embedding Store also provides various interfaces to access the embeddings offline and online.

This approach is often the initial and most straightforward way for applications to interact with the Foundation Model. Embeddings can serve as powerful features for other models or for candidate generation, helping to retrieve appealing titles for a user or facilitate title-to-title recommendations. Integrating embeddings as features into existing pipelines is generally well-supported and offers a low barrier to entry. Compared to other integration approaches, using embeddings from the Embedding Store is expected to have a relatively smaller impact on training and inference costs.

However, this approach also has some cons. The time gap between embedding computation and downstream model inference can lead to staleness, impacting the freshness of recommendations. This may prevent applications from fully unlocking the benefit of the Foundation Model, especially for use cases requiring real-time adaptability.

Overall, the default starting point for leveraging the Foundation Model is through its profile and item embeddings via the Embedding Store, provided that the embeddings’ freshness and stability meet application-specific needs. While embeddings may not leverage the full power of the Foundation Model, they offer a good initial step.

Our biggest learning from working on this approach is that embeddings are a low cost and high leverage way of using the Foundation Model. A resilient embedding generation framework and embedding store are essential for updating the embeddings daily and serving them to application models. We found this infrastructure to be so valuable that we stepped up our investment here to build a near-realtime embedding generation framework. This new framework also enables us to update the embeddings based on user actions during the session, making the embeddings and our downstream models more adaptive. Although the near-real time framework cannot be used for very large models, it represents an important direction for addressing embedding staleness and adaptiveness of recommendations.

Foundation Model as a Subgraph

This approach involves using the Foundation Model as an input to the downstream model i.e. the Foundation Model decoder stack is a subgraph of the application model’s full model graph and hence the name “subgraph integration”. The Foundation Model subgraph processes raw user interaction sequences and outputs representations that are then fed into the downstream model graph.

Applications can fine-tune the Foundation Model subgraph as part of their own training process, potentially leading to better performance than just using static embeddings. In this approach, there is no time gap or staleness between the Foundation Model inference and the application model’s inference, ensuring that the most up-to-date learnings are used. This approach also allows applications to leverage specific layers from the Foundation Model that may not be exposed through the Embedding Store, uncovering more application-specific value.

However, the subgraph approach comes with some complexities and trade-offs. To use the Foundation Model as a subgraph, application models need to generate all the features necessary for the subgraph as part of their feature generation process. This can add time, compute and complexity to their feature gen jobs. Merging the Foundation Model as a subgraph also increases the application model size and inference time. To alleviate some of these complexities, the Foundation Model team provides some reusable code and jobs that make the feature generation more compute efficient. To optimize the inference performance, we split the subgraph and ensure it only runs once per profile per request and is shared by all items in the request.

Overall, the subgraph approach allows for a deeper integration and enables the applications to harness the full power of the Foundation Model. But it can also make the application’s training code more complex, requiring a trade-off between metric improvement, compute cost, and time. Therefore, we would reserve this approach for high impact use cases where the increase in metric wins compensate for the cost and complexity.

Fine-tuning Foundation Model and use directly

This approach is similar to fine-tuning LLMs with domain specific data. In the previous blog post, we described how the Foundation Model is trained on a next-token prediction objective, with the tokens being different user interactions. Different interactions are of different importance to various surfaces on the Netflix website. For example, for the ‘Trending now’ row, recent interactions on trending titles might be more important than old interactions. Hence, the Foundation Model can be fine-tuned on more product specific data and then the fine-tuned model can be used to directly power the product. During fine-tuning, application teams can choose to do full parameter fine-tuning or freeze some layers. They can also choose to add a different output head with a different objective. We built a fine-tuning framework to make it easy for application teams to develop their own custom fine-tuned versions of the Foundation Model.

This approach offers the ability to adapt the Foundation Model to application’s specific data and objectives, making it optimized for their use case. A side benefit of this integration approach is that it provides a de facto baseline for new models/applications. Instead of coming up with a new model stack and spending months on feature engineering, new applications can directly utilize fine-tuned foundation models. However, this approach can lead to more models and pipelines to maintain across the organization.The latency and Service Level Agreements (SLAs) of the fine-tuned model must be carefully optimized for the specific application use case.

Conclusion

We are continuing to innovate on these integration approaches over time. For example, the Machine Learning Platform team is working on near-real time embedding inference capabilities to address the staleness issues with the embedding approach. We are also working on a smaller distilled version of the Foundation Model to reduce the inference latency of the subgraph approach. We also provide a fine-tuning framework to make it easy to fine-tune and leverage the Foundation Model. We also continue to refine and standardize the APIs used in these approaches to make it easy for application teams to use them. Overall, these three integration patterns provide a comprehensive framework for how Netflix applications can interact with and extract value from the Foundation Model.

Acknowledgements

We thank our stunning colleagues in the Foundation Model team for developing and continuing to innovate on the Foundation Model. We thank our colleagues in the AIMS applications teams for driving the integrations and their valuable feedback and discussions. We also thank our Machine Learning Platform team for building and supporting the infra components to get these approaches up and running in production.

Contributors to this work (name in alphabetical order): Davis Shepherd Inbar Naor Jin Wang Jonathan Indig Kevin Zielnicki Luke Wang Mario Garcia-Armas Maryam Esmaeli Michael Tu Nicholas Hortiguera Qiuling Xu Robbie Greuner Roger Menezes Sejoon Oh Trung Nguyen Vito Ostuni Yesu Feng.

Integrating Netflix’s Foundation Model into Personalization applications was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

By Dhruv Pratap

Introduction

Organizations that have been around for a while usually run two identity systems side by side. One belongs to the cloud provider: IAM roles, instance profiles, execution roles. The other is your own, and it is the one your internal services actually check when they decide whether to answer a request.

On infrastructure you build yourself, you can bootstrap your own identity however you like. On managed compute you cannot. The provider hands your process a cloud identity and nothing else.

This post describes how we close that gap for Apache Spark workloads running on Amazon EMR. A workload that starts with only an AWS identity has to end up holding a first-class internal identity. Doing the exchange is straightforward. Making it trustworthy is the part that took the design work. Very little of what follows is specific to Spark or to EMR.

Two identity systems

Internal service-to-service authentication at Netflix runs on a private PKI called Metatron. Every workload gets a short-lived X.509 certificate, and services authenticate each other with mutual TLS. A workload cannot simply ask for a certificate. It is issued only after the identity service has satisfied itself that the requester really is the workload it claims to be. That verification step is called attestation. In each environment we support (VMs, containers, functions) attestation rests on some environment-specific proof that the platform can check on its own.

The second concept is the Data Project, which is our unit of ownership for data. A Data Project owns tables, has a set of authorized users, and has an identity of its own. Jobs run as the Data Project rather than as the person who launched them. That is what keeps table-level access control and audit consistent across scheduled and interactive use.

So the problem is this. A Spark job on managed compute starts with an AWS execution role and no internal identity. Everything it needs from inside the company (reading an encrypted column, evaluating table ACLs, holding an interactive session open longer than a short-lived token lasts) requires the internal one.

One identity, one role

The design rests on a decision made outside the Spark stack entirely. Each Data Project identity maps 1:1 to a dedicated IAM role, and the service that owns Data Project metadata records that mapping.

The mapping is what makes translation possible. It lets a statement in the provider’s vocabulary (“this process is running as role R”) become a statement in ours (“this process is workload W”). Without it there is nothing to translate into, and no amount of cryptography helps. The hard part of bridging two identity systems is rarely the protocol. It is committing to a mapping and then keeping it authoritative.

One practical objection arrives immediately. A single AWS account cannot hold tens of thousands of IAM roles, and we expect data projects in the order of ten thousands. We handled this by sharding data project roles deterministically across a small pool of dedicated accounts, which scales well past the projected number of projects and has the useful side effect of keeping workload roles on the far side of an account boundary from the control planes that launch them.

Components

Five components take part.

The control plane is the only service allowed to launch Spark jobs. It resolves the Data Project’s IAM role, builds and signs a workload metadata payload, and submits the job with that role as the execution role. It holds a signing key issued to it for this purpose alone.

The Data Project service is the authority for the identity-to-role mapping.

The Identity service receives attestation requests, verifies them, and issues certificates.

A Spark plugin provides the hook. Spark 3.0+ added a plugin interface with driver-side and executor-side components that are initialized while those processes are bootstrapping.

AWS STS acts as a notary, which is not its usual job.

Step one: the control plane makes a signed claim

When a job is submitted, the control plane validates the caller, looks up the Data Project’s IAM role, and builds a small payload describing the workload it is about to launch. The payload names the application identity, the mapped role, and the environment, stack and detail that the resulting credentials should be scoped to. The control plane signs the payload and passes it and the signature through as ordinary job configuration.

Two properties of this step matter. First, only one service can produce a valid signature, so the set of things that can assert “this workload is legitimate” is small and auditable. Second, the payload is a claim rather than an authority. On its own it proves nothing, because job configuration travels through infrastructure we do not fully control and anything that can read it can replay it.

Step two: the workload proves it holds the cloud identity

When the managed service launches the driver process, it does so under an OS user that has the execution role’s credentials available. The driver-side plugin initializes and uses those credentials to sign, but not send, a request to the provider’s identity endpoint (sts:GetCallerIdentity). The result is a short-lived pre-signed URL.

That URL is a transferable proof of possession. Anyone can fetch it. Only the holder of the role’s credentials could have produced it. And the response comes from AWS rather than from the workload, stating which role signed the request. The workload cannot lie about the answer because the workload does not supply the answer.

The pattern is not new. It is the same idea behind AWS IAM authentication in HashiCorp Vault and in AWS’s own function attestation flows. It generalizes well: any environment that gives a process cloud credentials and nothing else can still produce a verifiable statement about its own identity.

The plugin sends one attestation request carrying both artifacts, the pre-signed URL and the signed metadata.

Step three: corroboration

The identity service does four things.

1. It fetches the pre-signed URL against AWS STS and reads back the role that signed it. This is the provider’s statement.
2. It verifies the metadata signature against the key issued to the control plane. This is the platform’s statement.
3. It corroborates the two. The role AWS reports must be the role the control plane said it dispatched, for the identity the metadata names.
4. It issues certificates for that identity, scoped to the environment, stack and detail from the signed metadata.

Step 3 is the point of the whole design. Neither statement is sufficient by itself, and they fail in complementary ways. The provider’s statement is unforgeable but under-specified: it gives you a role, and a role is not a workload. It says nothing about why this process exists or what it should be allowed to be. The platform’s statement is well-specified but unverifiable on its own, for the reasons above. Put together, the signature says the workload was legitimately dispatched and the pre-signed URL says this really is it. Neither party has to trust the workload’s own account of itself.

There is a tension here that we settled deliberately, and other teams should expect to meet it. The provider’s answer identifies a role, not an application. Deriving an application name from a role name is fragile string matching, and the session identifiers the platform assigns are not under our control. We chose to take the application identity from the signed metadata and use the provider’s answer only for corroboration. That costs a network round trip and a stricter signing requirement, and it buys a much clearer trust story.

The fan-out problem

A Spark application is one driver and up to thousands of executors, created and destroyed throughout the life of a job. Attestation designed around one process per host does not survive that.

There are two defensible answers.

In the first, each executor attests independently. This is uniform, adds no new trust boundary, and grounds every process’s identity in the same provider-verified proof. The cost is amplification. One large job can produce thousands of STS calls and thousands of attestation requests in a burst, which looks a lot like an attack and creates a hard dependency on provider-side rate limits.

In the second, executors inherit credentials from the driver. The driver attests once and distributes credentials to executors over Spark’s internal RPC, which we configure with authentication and AES-GCM encryption so that only processes in the same application can take part. Load on the identity service stays constant. The cost is a second trust boundary and a driver that is now a credential distribution point.

Considering our scale at Netflix here we went with the second approach. The general lesson is that identity amplification is a capacity question, and it is better to answer it on purpose than to have it answered for you during an incident.

Lifecycle

Certificates are short-lived by design, so bootstrapping is only half the work. On self-managed hosts a system service handles renewal. Inside managed compute there is no equivalent hook, so the driver JVM runs a timer that re-attests on a fixed interval, well ahead of expiry. Credential material is written to a temporary directory readable only by the process owner and removed when the process exits.

Executors do not refresh. They are usually short-lived, and an executor that somehow outlives its credentials can exit and be replaced, which is cheaper than making every executor a renewal client.

The rule worth carrying into any similar system is that attestation has to be a repeatable operation rather than a bootstrap step. Anything that can only happen once at process start will eventually be the reason a long-running job dies nine hours in.

Why this generalizes

If you run your own identity system and are moving workloads onto managed compute, the specific services here matter less than the shape of the solution.

1. Anchor on one exchangeable primitive. A single authoritative 1:1 mapping between your identity and a provider identity is what makes translation possible. The rest is plumbing.
2. Require two independent claims and corroborate them. One from the provider, unforgeable and under-specified. One from your control plane, well-specified and replayable. Trust the intersection and never either alone.
3. Keep the signer scarce. If exactly one service can make the platform’s claim, your trust story fits in a sentence.
4. Hook the layer you still own. On managed compute you rarely control the host or its init system, but you almost always control the runtime: a plugin, an agent, an entrypoint. Attestation belongs there.
5. Decide the amplification policy on purpose. Distributed engines multiply every per-process operation by their parallelism.
6. Make attestation repeatable and credentials short-lived. Renewal is a requirement, not a follow-up.

What makes the result trustworthy is that no single participant issues an identity by itself. Not the workload, not the control plane, not the provider. And the workload, which is the party in the weakest position to be trusted, is never asked to vouch for itself.

Acknowledgements

Thanks to Amer Hesson for the Data Project abstraction, Nick Siow for the Data Project IAM role sharding, and Doug Clark for Metatron attestation.

Trading a Cloud Identity for Your Own: Workload Attestation on Managed Compute was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introducing composable, module system native and agent friendly command line tools for modern Java development

By Danny Thomas, JVM Ecosystem Team

Recent work on the Java language to pave the on-ramp has made it easier than ever to start a Java program and evolve it using the full language and platform. At the end of that on-ramp lies Java’s mature build and dependency management ecosystem, capable of carrying software to enormous scale and complexity.

That ecosystem reached its maturity by developing strong models for projects, dependencies, and builds. When the Java Module System arrived, those models were already serving developers exceptionally well. The module descriptor consequently became just another description of the project to keep in agreement.

We’re excited to announce a preview of ja and its family of composable tools, that build on the capabilities of the Java Module System to provide a modern command line development experience for Java. We take the module descriptor and make it a complete description of a project, with dependency versions sitting naturally beside its requires directives and module metadata provided through documentation tags:

/**
 * @mainClass com.example.application.Main
 */
module com.example.application {
    requires com.example.framework; // @1.2.3
}

Combined with command line ergonomics you’re used to in other languages, creating and consuming Java modules has never been easier.

Composable Tools

Java developers have long been exceptionally well served by graphical tools. An IDE formats source, navigates between declarations and usages, presents API documentation, and maintains a compiled view of the project. That experience has been so complete that Java has had less need to expose the same capabilities through small, composable command line tools. Those gaps become quickly apparent when coding agents work with the Java language, with agents frequently struggling to locate dependencies, documentation and sources.

ja only provides command line ergonomics and tool orchestration, each feature is underpinned by a standalone tool. You don’t need to adopt ja to get the benefit of these tools, you can compose them in any way you choose:

• jig performs module version resolution, compilation and assembly, outputting standard module system arguments for use with other tools. It is also the bridge to and from Maven repositories providing a standalone module proxy and publishing commands
• jfmt formats source using the Code Conventions for the Java Programming Language, adapted for the modern Java language. Avoids the very common whitespace, indentation, import ordering and qualified class references introduced in agent written code
• jist provides source aware symbol search, providing a grep style interface for understanding class files and their associated sources. Gives coding agents access to symbols and sources without indexing, LSPs or MCPs while interoperating with other build tools via an argument file contract
• jdocserver serves locally browsable API documentation

These projects use the tool discovery and execution capabilities of the platform, and are intended to be installed in your JDK along with the standard tools. They all implement Tool or ToolProvider, allowing them to be run in process.

This is also the tool discovery and execution model for ja. There we use OptionChecker and optional custom metadata to discover which module system options are supported so it can resolve the arguments on behalf of the tool. This provides a seamless transition from your source path modules to the standard JDK tooling such as jdeps, jlink and jshell.

Maven as a foundation

In a recent survey of the 1,000 most popular artifacts on Maven Central, just 232 had explicit module definitions and another 248 declared automatic module names. The remaining 520 expressed no Java module name opinion. The module system also makes no distinction between namespace and module name, so module-first tooling requires a solution to module naming and location in existing repositories.

Fortunately, Maven Central already gives published artifacts a verified namespace. Publishers prove control of reverse domain group IDs, reflecting Sonatype’s long standing case for namespaces in public repositories.

We use these conventions to establish a canonical Maven module coordinate, paring a verifiable DNS namespace with the complete module name, for example pkg:maven/com.netflix/com.netflix.tools.ja. For existing modules, authors choose to publish a single Maven relocation pom at the canonical coordinate, to allow for discovery of the original coordinate.

When neither are available, candidates are walked from the root of the namespace using common Maven artifact conventions inferring coordinates from module names. We also bundle a short list of aliases for the most popular modules that don’t use a reverse DNS module name, but we suggest authors should always namespace their modules. The module proxy in jig presents resolved modules using the filename based conventions for module naming, making even automatic modules without stable names safe when used with these tools.

These conventions and location strategies allow the majority of existing artifacts to be discovered using only the module name and version.

Integrity by default

ALL-UNNAMED has become unfortunately common in Java access options, because of the heavy use of the class path. It hides the source of the technical debt that applications are incurring by allowing such access and becomes increasingly consequential as Java moves toward Integrity by Default. For example, Preparing to Make Final Mean Final asks applications to explicitly authorize the modules allowed to mutate final fields.

We allow runtime access requirements to bedeclared as module metadata and carried with the module descriptor throughout the module’s lifecycle. For example a library may record the access it requires:

/**
* @enableFinalFieldMutation com.example.framework
*/
module com.example.framework {
}

However, the consuming application remains in control and must explicitly authorize the framework, for it to be available at runtime:

/**
* @mainClass com.example.application.Main
* @enableFinalFieldMutation com.example.framework
*/
module com.example.application {
    requires com.example.framework; // @1.2.3
}

The command line interface for ja allows the dependency and authorization to be added together:

ja require com.example.framework@1.2.3 \
 --enable-final-field-mutation com.example.framework

Without that authorization, dependency resolution fails with an unsatisfied access requirement. Native access follows the same model through @enableNativeAccess and qualified exports and opens are also supported.

Module integrity is ensured by persistent hashes of resolved binary dependencies in a module-info.hash file, sequent resolution verifies those hashes and rejects an artifact that has changed.

We also take a step further than the recent improvements to annotation processor security by treating annotation processing as an explicit code generation step. The resulting sources are alongside regular module source, making them visible in code review and allowing a module to be assembled without executing generator code.

Make modules your default

We think every Java project should be modular, regardless of the build tool you’re using. If you’re a library author producing automatic modules, we’d encourage you to avoid split packages and produce explicit modules.

You can get started with our tools today with our installation guide.

Leave the Class Path in the Rearview Mirror was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

By Emma Yanyang Kong, Aditya Deshpande, Asad Abbasi, Bowei Yan, David Fagnan, Ashish Rastogi, Dhaval Patel, Ray Zhang

Introduction

The Netflix experience is a journey of discovery. Every visual cue, from the artwork on a title to the video previews that autoplay while you browse, is there to connect you with a story you will love. We call these visual cues assets, and choosing the right one for each member is a personalization problem of its own. But which image or video preview of Squid Game should we show you? And what do we do right after a title launches, when there’s far too little interaction data to know which asset we should recommend to each member?

For years, our models answered the first question well and the second poorly. They learned which assets members interacted with, but treated every asset as an opaque ID, blind to what was actually in the artwork or video preview. Right after a title launched, its assets had no history, so we dialed up exploration on its assets to gather interaction data, and otherwise fell back to popularity heuristics that ignore your taste. Only once enough interactions had piled up could personalization take over. This is the classic cold-start problem.

This post shares how multimodal embeddings let our models see and hear the assets they recommend, so personalization can kick in far sooner, close to a title’s launch. Because a new asset arrives with its embedding the model already understands, that embedding carries member taste signals from related assets immediately. Consequently, the model needs far less interaction history before it can personalize. We cover three production systems, artwork personalization, query-aware artwork ranking, and video preview personalization, plus a cheap trick for choosing new embeddings before committing to full end-to-end integration and A/B testing.

Artwork Personalization

A single image is often a member’s first touchpoint with a title, so we create a diverse set of artworks for each title to appeal to different member tastes. We already use personalized artwork based on members’ interaction histories, but this approach breaks down for newer titles and their assets, where there is little or no behavioral data to learn from.

Making the Model See the Artwork

Our solution is to let the model “look” at the picture. We encode each artwork with CLIP, a pretrained image-text embedding model, and fold the result into how the model represents that asset, concatenating the per-asset CLIP image embedding, a 768-dimensional vector, with the asset’s learned ID embedding to give an asset representation:

This single change transforms how the model handles a brand-new artwork. Instead of treating it as an unseen ID, the model now receives a CLIP embedding the moment the asset is created. That allows a member’s preferences over visual themes, talent, and color palettes to be applied immediately, long before the asset accumulates any interactions of its own. Because those preferences are expressed in image-embedding space rather than tied to specific asset IDs, they transfer seamlessly across titles. If you consistently engage with artwork featuring a particular comedian, the model can carry that signal to their new title and prioritize the asset that places them front and center, even if it has never shown you that exact image before, as in the figure below. In this way, cold-start shifts from being a blind spot to something the embedding space already has an informed opinion about.

From Five Models to One

That shift, from scoring an asset by the ID it happens to carry to scoring it by what the image actually contains, powers a second big win, model consolidation. Each title’s artwork spans multiple canvases with different croppings (billboard, vertical-box, horizontal-panel, short-panel, landscape-panel), and historically we trained a separate model per canvas, since an ID-based model has no way to know that the cropped and resized renderings of one scene are related, so signal could not flow between canvases and each faced its own cold-start.

CLIP embeddings break that barrier. Because they are largely invariant to crop, resize, and aspect ratio, those near-identical renderings map to nearly the same vector, as the figure further below shows. A single unified model can therefore pool interaction signal across every canvas, so a member’s affinity learned on a high-traffic canvas immediately informs the artwork we pick on a sparse one. The result is one model in place of five, with the largest gains on the canvases that have the least interaction data.

Mixing Five Canvases of Training Data

Consolidation introduced a challenge that the per-canvas models never faced: how to effectively mix data across disparate canvases? The canvases differ widely in impression volume, and the interactions they log are not all worth the same to a member’s long-term experience. Training on pooled raw counts would let the highest-volume canvas and the most frequent interaction types dominate, so the low-data canvases we were trying to help would benefit least. Hand-tuning a weight per canvas would just trade that problem for a set of arbitrary hyperparameters and endless online sweeps to tune them.

Instead we use reward-based weighting, building on Netflix’s long-term reward modeling. Each training example is weighted by the long-term reward score attached to its interaction type:

where e(·) is the type of the observed positive interaction and ρ is that type’s long-term reward score. Because interaction types are not distributed evenly across canvases, weighting by long-term value rebalances the canvas mixture on its own, with no weight set by hand. A canvas contributes in proportion to the long-term value of the interactions it drives rather than to how many impressions it happens to get. Consolidation becomes feasible, and the unified model optimizes for long-term member satisfaction instead of whichever short-term action is most frequent.

A Note on Offline Evaluation

Every result presented here must clear two bars: an offline metric evaluation followed by a large-scale online A/B test. The offline metric is the subtle one. Judging a new model on logs from the current production policy is biased, because that policy shows some assets far more often than others. The logged rewards describe what the policy preferred, not what members would have chosen from the full candidate set, so a new model that disagrees with the logging policy looks worse than it is, because the impressions it would have picked are barely represented in the data.

We handle this with inverse propensity scoring (IPS) computed on a dedicated slice of exploration traffic. A small fraction of traffic is served by a randomized policy that samples among a title’s candidate assets from a known distribution, so the propensity of showing a given asset in a given context is logged exactly at serving time rather than estimated after the fact. Reweighting every observation by the inverse of its logged propensity gives:

where D is the exploration slice and r(x, a) is the observed reward, such as a play. Impressions that exploration made rare are upweighted accordingly, and the estimator becomes an unbiased estimate of the reward a candidate policy would have earned had we actually deployed it. Having propensities that are known by construction, rather than modeled after the fact, is in our experience the single biggest reason our offline numbers track online outcomes. We report IPS as a ratio against the production baseline, and a candidate has to win there before it gets any A/B traffic.

Combining Both Ideas Works Better

Two ideas are bundled together here, so we ablated them separately against the old five-model production system.

• V1, image embeddings only. The five per-canvas models kept as they were, each one augmented with image embeddings.
• V2, unified model only. A single model trained over all five canvases, but with learned ID embeddings alone and no image content.
• V3, both together. One unified model over all five canvases, with image embeddings in its asset representation.

As the chart below shows, each idea helped exactly where we expected: on the data-starved short-panel canvas and landscape-panel canvas. V3 was the clear winner. A change inside ±1% is not significant for this offline metric, and those bars are hatched in the chart. Most of what V1 and V2 do on their own sits inside that band.

In the online A/B test across all device platforms, which ran for at least four weeks, the results drew a much clearer line: Neither idea moved our online core member metrics on its own. V1 and V2 were both flat and non-significant, and only V3 won a statistically significant lift. It is what runs in production today.

The two ingredients need each other. V1 tells a per-canvas model what an asset looks like, but one sparse canvas has too few examples to teach it how to use that. V2 supplies plenty of data, but only ID-based data, which a new asset lacks. V3 has both, so mature canvases teach the shared model how CLIP embeddings map to member preference and that mapping transfers straight to the sparse ones. The effects compound rather than add, since the V3 short-panel lift (5.691%) exceeds V1 and V2 combined. The lesson is to look for a second blocking factor before concluding that content features do not help.

Cold-Start Challenge from a New UI Launch

The real test came from the product change that motivated the work. Netflix was preparing its largest TV home-screen redesign in a decade, which would make short-panel the dominant artwork canvas effectively overnight. This was a cold-start problem in its sharpest form. The canvas about to receive the most impressions had the least historical data, and waiting for short-panel interactions to accumulate would have degraded the user experience. Consolidation lets short-panel selection draw on signal pooled from every canvas, and CLIP embeddings let the unified model personalize a short-panel asset that has gathered very few interactions of its own.

We shipped V3 ahead of the launch and measured it with a month-long holdback A/B test, keeping a small control group on the prior per-canvas model. V3 absorbed the shift immediately, with statistically significant gains on both our core discovery metric and streaming hours, and larger gains than in the steady-state ablation. That stronger result is what we expected, since a sudden shift in which canvas dominates is exactly where V3 should help most.

Query-Aware Artwork Personalization

Your general taste is the right signal when browsing, but not when searching. For example, when searching for a specific actor, you want artwork that features them, even if your broader taste says otherwise. On the Netflix Search Page, the member’s intent is explicit and stated in the query, and the displayed artwork should reflect it.

The same CLIP embeddings we added for cold-start hand us this almost for free. Because CLIP projects text and images into one shared embedding space, we can measure how well a query matches a candidate artwork directly by the cosine similarity between the CLIP text embedding of the query and the CLIP image embedding of the asset. We blend that alignment term with the usual personalization score:

Here the personalization term is the score the artwork model above already produces for a member and asset, the second term compares the text embedding of the query against the image embedding of the asset, and the mixing weight α between 0 and 1 is tuned through online A/B testing. The first term is “what we think you like”; the second is “what you just asked for,” and α sets how much each matters.

Crucially, this took no extra modeling effort. The CLIP embeddings already sit in the asset representation from the artwork work above, so they carry the text-image alignment for free, and we get a query-aware ranker by adding a single similarity term at scoring time. The effect is visible in the search results themselves.

Personalizing Video Previews via MediaFM

Video previews raise the bar over still artwork. A video preview unfolds over time, and its appeal comes as much from motion, pacing, dialogue, and soundtrack as from any single frame. Our older video preview personalization models saw none of that. Like the early artwork models, they treated each preview as an opaque ID. Our first content-aware attempt, SeqCLIP, described a video preview by its frames, encoding each with a CLIP embedding and then averaging them into one vector. That captured what a video preview looked like, but a mean of still frames still misses what it sounds like, the dialogue and music that carry so much of a preview’s tone.

To capture the rest, we turned to MediaFM, Netflix’s first in-house multimodal foundation model. Trained on 80 million shots, MediaFM fuses the following three signals per shot into a single embedding:

• Visual: SeqCLIP
• Audio: A pretrained speech and audio embedding model
• Text: Captions encoded via a large-scale text model

Adopting MediaFM required no new infrastructure, since we simply integrate its shot embeddings into the asset representation, exactly as we did with CLIP embeddings for artwork.

The added modalities paid off. We evaluated both embeddings against the ID-only baseline offline with IPS and then in a five-week online A/B test across all device platforms, and both signals gave the same ordering, MediaFM > SeqCLIP > ID-only, and each step of added content awareness helped, with the gains largest on TV. Offline, both content-aware embeddings beat the ID-only baseline on IPS and MediaFM beat SeqCLIP, as the chart below shows. Online, MediaFM came out on top too, delivering a statistically significant lift in our core streaming metric over the ID-only baseline and outperforming SeqCLIP. This shows that the audio and timed-text signals, which a visual-only encoder like SeqCLIP cannot capture, add real value. We have since shipped MediaFM as the default video preview embedding across all platforms.

Choosing Embeddings Cheaply with a Proxy Task

New embeddings arrive constantly, but end-to-end trials are expensive, which cost data engineering, model retraining, and weeks of A/B test traffic. We couldn’t afford to run the full pipeline for every candidate, so we gated the funnel with a cheap question:

From the content embedding alone, can you predict which asset wins under a plain, unpersonalized policy?

We first select a fixed set of titles. For each title we use exploration data to find its debiased popularity winner, the asset with the highest interaction rate after we adjust for how often it was shown using its propensity score. We mark this winner with a binary label, 1 for the winner and 0 otherwise. We then train a linear probe to recover that label from the asset embedding alone, with no title, cast, or metadata, by minimizing the standard binary cross-entropy loss:

Keeping the probe linear and embedding-only is intentional, since it isolates how much of an asset’s popularity is actually encoded in the embedding. If the embedding captures the semantic drivers of popularity, a simple linear classifier should be able to identify likely winners. If it does not, the probe performs no better than random guessing, which is the baseline we score it against.

We first used the linear probe to screen and prune a broad set of candidate embeddings before modifying any production pipeline, narrowing the field to two finalists, SeqCLIP and the leading MediaFM variant. We then carried both through full offline evaluation and online A/B testing. All three signals, the linear probe accuracies, the offline IPS lifts, and the online A/B results, ranked MediaFM ahead of SeqCLIP, as the chart below shows. That alignment is why the linear probe now gates every new MediaFM version before release.

The Netflix Embedding Store

None of this would be practical without shared infrastructure. Every embedding in this post, CLIP for artwork, SeqCLIP and MediaFM for video previews, lives in the Netflix Embedding Store, a component of Netflix’s AI Platform that hosts dense embeddings for titles, games, member profiles and multimedia assets. A foundation model encodes raw asset content into a dense vector once, and the Embedding Store serves that vector to every downstream system, the artwork model, the query-aware ranker, the video preview model, and others, through the same interface. Crucially, it serves the exact same embeddings at training time and at online inference time, so there is no skew between what a model learns from and what it sees in production.

Its key property is that it decouples foundation-model updates from personalization-model deployments. A new embedding, or a new version of an existing one, can be registered, backfilled across the catalog, and validated entirely on its own, without touching the training or serving code of any model that consumes it. Once it is in the Embedding Store, it becomes available to every ranking and personalization model through configuration alone, no downstream code changes, no coordinated release. This is what let us swap CLIP into the artwork model, stand up the query-aware ranker on the same vectors, and roll MediaFM through the video preview model, each as an independent change rather than a cross-team migration.

What We Learned, and What’s Next

Three lessons stood out.

1. Pretrained CLIP embeddings let us consolidate five artwork models into one while boosting performance on data-starved canvases. This benefit became especially clear when the redesigned TV home screen rolled out.
2. For video, multimodality wins decisively. The audio and text signals that a purely visual encoder cannot access pushed MediaFM past SeqCLIP.
3. A cheap proxy task yields big savings, efficiently pruning the candidate set before running full end-to-end experiments and online A/B tests.

Next, we aim to extend the Embedding Store toward a single shared semantic space for image, text, and video. Such a unified representation would enable cross-modal retrieval, such as matching a video preview to a search query, or a static artwork to the video preview it was derived from, as well as unified asset ranking across surface types and a more cohesive, intuitive discovery experience for members everywhere.

Acknowledgements

We thank Aneesh Vartakavi, Santiago Castro, and Avneesh Saluja for the CLIP embedding and MediaFM work that made the content-aware models described here possible, and Ratna Kavuri for the backend systems that serve multimedia personalization in production.

MAPS: Netflix’s Multimodal Asset Personalization at Scale was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Samuel Yeboah, Francesco Di Chiara and Mingliang Liu

Today, Netflix runs two Flink autoscalers. That is exactly one more than we want. We built the first one in-house years ago, when there was no mature option suited to our platform. The second came from the Apache Flink community, and it can scale workloads our homegrown system was never designed for. We now run both in production and are steadily converging on the open-source one. Along the way we learned some hard lessons about metrics, cost, and the real price of maintaining infrastructure you could instead adopt, and we hope they are useful whether you run a handful of Flink jobs or tens of thousands.

Why autoscaling is not optional at our scale

Netflix has run stream processing on Apache Flink since 2017. As of 2026 we operate more than 30,000 Flink jobs across multiple AWS regions. Most are not deployed by hand; they are generated by our managed platform Data Mesh, so the majority of users never touch a Flink job directly. A smaller but growing set are custom jobs, built and operated by teams across the company for use cases like personalization, Ads, and Live events. They range from single-operator jobs that shuttle records between Kafka topics to stateful pipelines with branches, joins, and terabytes of state, and their load swings with daily cycles, launches, and regional failovers.

Provisioning every one of those jobs for its peak is wasteful; provisioning for the average causes lag during surges. And in our platform a scaling action is not free: by default it means taking a savepoint, stopping the job gracefully, and restarting it at the new size, which for a large stateful job can take minutes. That leaves a genuinely hard question: how do you give each job the resources it needs, when it needs them, without a human in the loop and without breaking anything?

The first autoscaler: watching from outside

Our first answer, built around 2019, was an autoscaler shaped like a stream-processing job. It ran on Mantis, consuming a live feed of cluster-level metrics from Atlas, our telemetry platform, including CPU, network, Kafka lag, input-rate, and consume-rate signals for every job. The scaler combined lag-derived catch-up time, CPU/network utilization thresholds, observed performance history, and regression over recent input rate to decide when to scale up or whether a smaller cluster could handle the lookahead window. Because the autoscaler operates independently of the Flink platform, it remains unaffected by issues within Flink itself. Building it as a streaming job also made it easy to scale. Each autoscaler node handled the metrics for a subset of Flink jobs, and we never had to write custom sharding or coordination logic to keep up with a growing Flink fleet. It reliably cut resource usage by 25–45% across thousands of managed pipelines. Check our previous talk at Flink Forward 2020.

But watching from outside has a ceiling. The system reasoned about a whole cluster through coarse container metrics, and it scaled a single knob, the total TaskManager count, so every operator in a job moved together. That fit the simple, single-operator pipelines it was built for, but not the multi-operator, stateful DAGs that teams were increasingly bringing to us for Ads, recommendations, and games. Those were exactly the jobs it could not reason about, and supporting each new case meant more custom logic rather than any general capability.

The autoscaler is only as good as the metrics served by external systems beneath it. Those metrics could miss real trouble: a job could be completely busy without any of it showing up as CPU utilization, leaving the job stuck in a degraded state the scaler had no way to see. Recently a networking migration quietly changed how some traffic was reported, and a subset of the Atlas metrics the scaler relied on stopped capturing everything accurately. The gap stayed invisible until it surfaced in production much later.

It was time to reconsider build versus buy.

The second autoscaler: reasoning from inside

When we started, the Flink community had no mature autoscaler to offer. By the time we re-evaluated, it did: the Apache Flink Autoscaler. Instead of watching containers from outside, it reasons from inside the job.

Its key idea is to estimate each operator’s true processing rate (TPR): the throughput it could sustain if it were fully busy. Flink reports, per subtask, the fraction of each second spent doing actual work, separate from time spent backpressured or idle. Dividing observed throughput by that busy fraction extrapolates capacity to full utilization: an operator handling 700 records/sec while busy 70% of the time has a TPR of 700 / 0.7 = 1,000 records/sec. Starting from the sources, the autoscaler walks the job graph and uses each operator’s TPR, its input/output ratios, and a target utilization to compute the parallelism every vertex needs so that no operator becomes the bottleneck, rather than resizing the whole cluster as a unit.

The two approaches make a different contract, summarized below.

The decisive difference for us is the last two rows: the OSS autoscaler can scale exactly the stateful, multi-operator jobs our homegrown system could not, and it lets each job carry its own configuration — stabilization periods, thresholds, and other scaling behavior tuned to the workload. That made it the natural fit for the custom jobs teams had been scaling by hand.

Making it work at Netflix scale

Adopting the algorithm was straightforward; the community had done the hard part. The work for us was running it reliably across our own jobs, and this is where our system differs most from the stock open-source deployment.

Firstly, the OSS autoscaler was originally architected to reside within the Kubernetes Operator for Flink, but our Flink platform runs on its own control plane, not that operator (see our previous talk at Current Conference 2024). The community later made a fantastic decision to keep the core logic as a standalone library. They refactored four generic interfaces that made it easy to plug directly into our internal ecosystem: a context carrying job metadata and REST API info, a state store, an event handler, and a realizer that applies scaling decisions.

That service is a Spring Boot application whose orchestration runs on Temporal, the durable workflow engine. An orchestrator workflow polls our Flink control plane about once a minute for the jobs with autoscaling enabled, and starts one long-running workflow per job. Each per-job workflow pulls that job’s per-vertex metrics from its Flink JobManager, runs the OSS evaluation algorithm, and, when a scaling decision results, hands it to a realizer that actuates the change through our Flink control plane.

The workflow-per-job design was a direct response to pain. We first ran evaluations in a single batch loop over the whole set of jobs, and it was fragile: one slow or misbehaving job could stall metric collection and scaling for every job behind it. Giving each job its own durable workflow isolated that blast radius, so a single problematic job now fails and retries on its own, and the runtime scales out as we onboard more jobs.

Secondly, three engineering gaps stood between “works in community” and “works at Netflix scale”:

• Metric collection at high parallelism. On big jobs, pulling metrics from the JobManager became a bottleneck, and part of the cause was in Flink’s runtime. To address that, we changed the JobManager to cache transient metric names and clean them up once instead of rescanning on every fetch, and we added server-side filtering so the autoscaler asks only for the metrics it needs. This let the autoscaler work on jobs up to 3,000 Flink subtasks, where it had previously struggled above roughly 1,000. Those are in our internal fork of Flink release, while some are contributed upstream such as FLINK-36172.
• Preserving forward chaining. Two separate vertices joined by a forward connection must run at the same parallelism, because records are handed over in memory on a fixed local channel. Scale one of them alone and Flink does not fail; it silently converts that edge into a network shuffle. Our fork detects forward-connected subgraphs and scales each as a unit.
• Respecting sink limits. Some sinks have finite write capacity, so we added detection for async-sink backpressure (also a fork change) to keep the autoscaler from scaling a job up into a sink that cannot absorb more.

Before it actuates anything, the realizer runs a set of safety checks. For example, it refuses to scale a job down in a region being evacuated during a company-wide region failover. It also verifies there is enough disk for the new cluster to hold the job’s checkpoint state, and it adds a small standby buffer for larger clusters.

The road to one autoscaler

Last year, the OSS-based autoscaler achieved general availability for custom jobs at Netflix, yielding promising initial outcomes. For instance, our client telemetry and logging team achieved a 58% reduction in its annualized Flink compute expenditures, saving approximately $1.1 million annually. This efficiency is driven by three key factors. First, whereas static provisioning must always account for peak loads, autoscaling dynamically adapts to daily cycles, capturing the drop in traffic during nights and weekends compared to weekday peaks. Second, rather than relying on teams to manually optimize resources following performance improvements or post-holiday slowdowns, the autoscaler continually adjusts capacity. Finally, adopting uniform container dimensions enables superior bin-packing and more granular scaling increments.

Additionally, scaling down too eagerly is its own trap. Cut too deep and CPU saturates, lag spikes, and the system cannot react instantly because its metric window and stabilization period have to rebuild after each restart. We now run a target utilization of 0.45, below the community default of 0.7, deliberately trading a little efficiency for stability. Fewer and calmer rescales are worth the marginal cost for large stateful jobs.

While our scaler provides fine-grained signals and vertex-level decision units for stateful DAGs, fast rescaling still heavily depends on Flink Core’s state restoration performance. Today, the biggest remaining cost in scaling a stateful job isn’t the scaler’s logic — it’s the restart and state recovery process itself. Flink 2 addresses this through its disaggregated state architecture, keeping state in external storage rather than on local disk, which can sharply reduce how much a rescale or recovery depends on total state size. Having started supporting Flink 2.2 at Netflix, we plan on experimenting with this new state backend to see if it can help eliminate state recovery bottlenecks when scaling large stateful jobs.

Looking ahead, we aim to migrate all internal scaler use cases onto the new one based on OSS autoscaler to simplify our operational surface area.

Key Takeaways

Along the way, three lessons that generalize beyond Flink:

• Metric choice matters more than algorithm sophistication. Our most useful debugging was rarely about the scaling math; it was about which signal to trust most. Understand your metrics before you tune your algorithm.
• Set sensible defaults, but leave room to tune. Our managed jobs are similar enough that one good default covers most of them untouched, which is the point of a platform. But forcing a single configuration on every job punishes the ones that do not fit, so we pair defaults with per-job overrides and deliberately hide the knobs that need deep expertise. Most teams should never have to think about the autoscaler.
• Adopt, then extend. We built in-house because in 2019 nothing mature fit our platform. When a strong community project appeared, the right move was neither to defend our investment forever nor to rip it out overnight, but to adopt it for new workloads, contribute fixes back, and plan a deliberate migration.

Thanks to the Flink and Data Mesh teams for the control-plane changes this work depended on, to the Temporal team and our early pilot teams, and to the Apache Flink autoscaler maintainers whose foundation we built on. Special thanks to Andy Zhang, Calvin Cheung, Daniel Trager, Guil Pires, Mark Cho, Matthew Kornitsky, Nikhil Sulegaon, Sujay Jain, and Tom Lee.

A Tale of Two Flink Autoscalers was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

How and Why Netflix Built a Real-Time Distributed Graph: Part 3 — Querying the graph with gRPC execution API

Authors: Nilesh Mishra and Ajit Koti

This is the third entry of a multi-part blog series describing how we built a Real-Time Distributed Graph (RDG). In Part 1, we discussed the motivation for creating the RDG and the architecture of the data processing pipeline that populates it. In Part 2, we discussed how we designed the storage layer to handle billions of nodes and edges while maintaining single-digit-millisecond latency. In Part 3, we will explore how we designed a fast, flexible serving layer to efficiently query the graph.

Introduction

In Part 1 of this series, we described why Netflix needed a Real-Time Distributed Graph (RDG) and how we used Apache Flink to build an ingestion and processing pipeline that turns streaming events into graph primitives. In Part 2, we explored how we designed a storage layer capable of handling billions of nodes and edges while still delivering single-digit-millisecond latency.

In this post, we focus on the next challenge: querying the graph efficiently to power real-time insights for our internal partners. All of the work on ingestion and storage only matters if we can actually ask complex questions and get answers back quickly. As we optimized for lower latency, we found that the serving layer posed its own set of challenges, distinct from those of ingestion and storage. How do we turn a constantly evolving, billion-edge graph into sub-100ms responses across a wide variety of workloads? This is the problem we tackle in this post.

The Real World Needs

As we integrated the RDG into Netflix’s ecosystem, we realized that “querying the graph” is not a one-size-fits-all operation. We needed to handle a wide range of access patterns: from high-volume security lookups to deep, exploratory personalization traces.

Let’s revisit our example from Part 1 and expand on it slightly. In the earlier posts, we focused on accounts, devices and content. In practice, the graph is richer: each account has multiple profiles.

A member journey often looks like this:

1. Alex logs in to their Netflix profile on a smartphone and starts watching Stranger Things.
2. They later switch to a smart TV in the living room to continue the episode.
3. The next morning, they use a tablet to play the game Stranger Things: 1984.

In the RDG, this journey creates the following graph structure:

Graph queries vary along two axes: how wide they fan out at each hop, and how deep they chain across hops. To see this range, let’s look at two scenarios from opposite ends:

1. Shallow and wide: “Which devices has this account used?”

Consider a “shallow, wide” query: “Which devices has this account used to stream in the last 30 days?”

Using the graph structure above, this translates to:

• Starting Point: A specific Account Node.
• Hop 1 Edge Traversed: The streamed_from edge.
• Hop 1 Destination: Device Nodes.

While this is only a “single hop,” it presents a significant scaling challenge. For a highly active account, the fan-out can be massive. The query layer must fetch hundreds of streamed_from edges, apply temporal filters on each edge’s last_watch_timestamp property to capture only those within the last 30 days, and aggregate the results, all while maintaining sub-100ms latency.

2. Deep & Narrow: What has this profile watched?

Consider a scenario where personalization teams need to understand a member’s viewing journey. They might ask: “For Account X, show me the Stranger Things viewing history across all profiles: which profiles watched it, what they watched, and when”.

This path unfolds as follows:

• Starting Point: A specific Account Node.
• Hop 1 Edge Traversed: has_profile
• Hop 1 Destination: Profile Nodes
• Hop 2 Edge Traversed: started_watching (filtered for title_name = “Stranger Things”)
• Hop 2 Destination: Content Nodes

The core challenge in this scenario is sequential dependency: we cannot fetch a profile’s viewing history until Hop 1 has identified which profiles exist. In a distributed environment, the client has to wait for Hop 1 to finish before sending Hop 2. If each hop takes 10ms of network time, that’s 20ms of overhead before we’ve processed a single byte. To hit our sub-100ms goal, we needed a way to package this multi-step logic into a single request.

This example is a 2-hop traversal, but queries can chain 3–4 hops across different entity types, and the latency penalty of sequential execution only grows with depth.

Balancing Depth and Breadth

These two scenarios pull the system in opposite directions. Shallow-wide queries stress I/O throughput: can we handle massive fan-out without slowing down? Deep-narrow queries stress execution efficiency: can we chain multiple hops without the network overhead adding up? Supporting both on the same system is what shaped the design that follows.

Design Constraints and Key Choices

The two scenarios above sit at opposite ends of the spectrum, but they are not unusual. In practice, the RDG serves tens of thousands of queries per second, each potentially different, all needing sub-100ms responses while the underlying graph continues to grow. Scale, latency, query diversity, and the need for extensibility pulled the design in different directions at once, and every choice came with a trade-off we had to live with.

Why breadth-first, not depth-first? The most intuitive way to traverse a graph is depth-first: pick a path, follow it to the end, backtrack, try another path. But in a distributed system where every hop is a network call, depth-first can lead to high latency. If Account X has 5 profiles and each profile has watched hundreds of titles, depth-first would trace all of one profile’s watched titles before moving to the next, missing the opportunity to batch lookups across profiles. Breadth-first flips this by working one level at a time across all nodes, rather than one path at a time through each node. We fetch all profiles for the account at once, then fetch the started_watching edges for all profiles, and finally fetch content details for all matching titles. Three rounds of parallel calls instead of sequential chains. With breadth-first, there is a clear trade-off in memory, because we hold each level of the graph in memory at once, so the cost scales with how wide a level fans out rather than how deep the query goes. We keep this comfortable by bounding each hop with the per-edge-type limits described in Step 5 below, so even a high fan-out level stays a manageable frontier. We’ll walk through how this works, level by level, in Step 3 below.

Why async-first, not thread-per-request? Latency in the RDG is dominated by I/O, reading from the storage layer, calling enrichment services, and waiting on caches. A traditional thread-per-request model would pin a thread to each in-flight query, and most of the time, the thread would be idle, waiting for a network response. With thousands of concurrent queries, we’d need thousands of threads, most of which would be doing nothing. Instead, we decided to build the entire execution pipeline around asynchronous composition. A small set of dedicated thread pools (16–24 threads total) handles thousands of concurrent requests because no thread ever blocks on I/O. While a storage call is in flight, the thread continues with other work and picks up the result when it arrives. This is the foundational design decision on which everything else rests. We’ll see this in action in Step 4 below, where we cover parallel execution.

Why cache selectively, not everything? Not all data in the graph changes at the same rate. Some properties, such as account plan type and content metadata, are relatively stable: they change on the order of hours or days. Edges like who watched what and when change constantly. For stable data that many queries touch, we use a distributed cache (EVCache) with TTLs tuned to data volatility. Getting the caching strategy right took iteration. We started by caching aggressively and measured the impact: tracking hit rates, monitoring stale-data incidents, and adjusting TTLs based on how quickly different node types actually changed in production. The result: 70–80% hit rates on node lookups, achieved by narrowing the cache to nodes that are both frequently accessed and slow to change, while skipping data that would expire before the TTL ran out. Step 6 below covers how this works in practice.

Why opt-in enrichments, not automatic? Clients know what they need. A query checking account relationships doesn’t care about title artwork; a personalization service building a viewing timeline does. Rather than fetching metadata from external services by default and penalizing every query, we make enrichments opt-in: clients specify exactly which external data they want per request. Also, enrichment is fail-open: if a service is slow or unavailable, we return the graph data without it.

Why eventual consistency, not strong? Most of our queries ask “What has this member done recently?”, not “What happened in the last millisecond?” By defaulting to eventual consistency, we read from the nearest replica and avoid coordination overhead. While the RDG is used to power in-the-moment experiences, it is not set up as the source of truth for the data it holds.

Architecture Overview

The above choices lead to the following three-layer architecture:

The Graph Query Service is the entry point. It accepts gRPC requests, validates the traversal specification, and hands it to the query execution engine. The execution engine orchestrates breadth-first traversal: expanding one level at a time, applying filters and limits at each hop, and composing all I/O asynchronously.

The Storage Abstraction Layer sits between the execution engine and the underlying KVDAL storage. It provides a clean interface for node lookups and edge retrieval, handles streaming for large adjacency lists, and manages node caching (EVCache).

The Enrichment Layer fetches additional metadata from external Netflix services on demand. It batches requests, runs them in parallel with graph data assembly, and degrades gracefully when an enrichment source is unavailable.

When a client sends a query, the request flows through these layers in sequence: the Query Service parses the request into an execution plan, the execution engine walks the graph level by level through the Storage Abstraction Layer, and if enrichments are requested, the Enrichment Layer fetches and merges external data before the response is serialized back to the client.

Now, with that mental model in place, let’s follow a query through this system and see how these choices play out in practice.

Executing Queries Efficiently: Following a Query’s Journey

To see how the RDG query layer works in practice, let’s follow a single query end-to-end and focus on one question: how do we make every step fast?

We’ll reuse the deep-narrow example from above:

For Account X, show me the Stranger Things viewing history across all profiles: which profiles watched it, what they watched, and when.

In graph terms, this becomes a 2‑hop traversal:

1. Account X → has_profile → Profiles
2. Profiles → started_watching → Content (filtered for “Stranger Things”)

We’ll walk through how this query moves through the layers we described above:

1. Reading and interpreting the request
2. Reading from storage efficiently
3. Executing traversal with breadth‑first levels
4. Running many operations in parallel, but safely
5. Filtering smartly to keep only what matters
6. Making repeat queries faster with caching

By the end, we’ll see how a 2-hop query like our Stranger Things example, with streaming, filtering, and parallel execution, can complete in under 100ms.

Step 1: Reading the Request: Deciding What the Query Really Wants

Every query starts as a gRPC request. Before we touch storage or walk a single edge, the engine needs to understand what the caller actually wants.

For our running example below:

For Account X, show me the Stranger Things viewing history across all profiles

The engine creates a traversal plan with a set of levers: how many hops, how many edges per hop, how much history to consider, and whether to favor recent activity.

We resolve these upfront by merging a hierarchy of filters and limits, from application-level defaults down to per-edge-type overrides, into a concrete execution plan. By the time we read from storage, every hop has clear rules. We’ll see how this hierarchy works in detail in Step 5, but the key insight is simple: interpreting the request up front prevents over-fetching from the downstream storage layer.

Step 2: Reading from Storage: Direct Lookups and Streaming Fan‑Out

Once we’ve parsed the request and decided what the query should do, the next step is to actually touch the graph. For our running example:

For Account X, show me the Stranger Things viewing history across all profiles…

The first concrete question the engine has to answer is very simple:

Which profiles does Account X have?

Under the covers, that really means: how do we find all relevant edges for Account X without scanning the entire graph every time?

Finding Edges Fast with Adjacency Lists

If we stored every edge in one massive table, the naive approach would be to scan for rows where source = Account X. Even with indexing, doing that across billions of edges for every request would be slow.

Instead, we organize edges as adjacency lists. For each node, we keep a compact list of “who it’s connected to” by edge type. For Account X, a simplified view might look like:

Account_X: has_profile → [Profile_Alex, Profile_Kids, …,]

Now “get all profiles for Account X” is no longer a global search; it’s a direct lookup into Account X’s stored adjacency. The storage layer can usually pull that list back in a few milliseconds because it’s reading a small, well‑indexed slice of data instead of hunting through everything.

For our query, the first hop is quick: Account X has just two profiles. The engine fetches those edges with has_profile and moves on. For more information on Storage, refer to our previous post.

When One Node Has A Lot of Neighbors

The first hop was small, but the second is where things get interesting. Each profile can have a large number of started_watchingedges. Loading the entire adjacency list at once would spike latency and memory usage.

To avoid this, we treat adjacency lists as streams rather than blobs.

When the engine requests Profile_Alex’s started_watching edges, the storage layer streams them in batches of 100. As each batch arrives, we apply filters (e.g., “last 30 days”) and decide whether to continue.

If we’ve collected enough edges to satisfy the query’s limits ( max_edge_cnt, lookback window, etc.), we stop reading. Otherwise, we pull the next batch.

In our Stranger Things example:

• Storage streams the started_watching adjacency for Profile_Alex.
• There are about 500 edges total: months of viewing history
• As each batch arrives, we filter for Stranger Things and drop anything older than 30 days.
• After a few batches, we’ve found what we need: a handful of Stranger Things sessions.
• We never materialize more data than needed. Filtering happens at the source.

Why This Matters Later

These two choices, the adjacency‑list lookups and streaming fan‑out, enable everything that follows:

• Small fan‑outs (like Account → Profiles) yield predictable, low‑millisecond lookups.
• Large fan‑outs (like Profile → Content) stay efficient by reading only what’s needed.
• Traversal logic treats “neighbors of this node” as a cheap, bounded operation.

By Step 3, we’re working with concise frontiers like “Profile_Alex and Profile_Kids,” ready for the next hop into their viewing histories.

Step 3: Traversal Execution: Walking the Graph Level by Level

We’ve completed the first hop. From Account X, we pulled the has_profile edges and found two profiles: Profile_Alex and Profile_Kids.

But we’re not done. The query was:

For Account X, show me the Stranger Things viewing history across all profiles: which profiles watched it, what they watched, and when

So we still need to fetch each profile’s history and filter it down to Stranger Things sessions. As we covered in our design choices, we use breadth-first traversal: expanding all nodes at the current level in parallel before moving to the next.

Querying, Level by Level

Let’s walk through the Stranger Things query level by level.

Level 1: Account → Profiles

Starting at Account X, the engine pulls has_profile edges, discovering two profiles:

• Profile_Alex, Profile_Kids

These become the frontier for Level 2, a single small lookup that takes a few milliseconds.

Level 2: Profiles → Content (Stranger Things)

From those two profiles, we fetch started_watchingedges and filter for Stranger Things. Instead of exhausting Profile_Alex’s entire viewing history before touching Profile_Kids, we treat this as one logical step:

• For each profile, fetch started_watching edges in parallel.
• Filter for title_name = “Stranger Things” as edges stream in.
• Each profile might have hundreds of content edges, but filtering at the source keeps the result set small.

We discover that Profile_Alex watched Season 1 and Season 2, while Profile_Kids watched Season 4. Level 2 turns “2 profiles” into “a handful of Stranger Things sessions” in roughly one storage round trip.

The traversal completes: two levels, two frontiers.

Why This Matters for Latency

We parallelize within each phase, then regroup. This provides:

1. Predictable resource usage: known requests per level
2. Maximum parallelism: all frontier nodes processed together
3. Far fewer round trips: one per level, not per path

For a 2-hop query: two rounds of parallel lookups instead of hundreds of sequential ones. That’s why our Stranger Things query completes in under 100ms.

Step 4: Parallel Execution: Doing Many Things at Once, Safely

Breadth-first traversal enables parallel work at each level, which is the key to low latency.

At Level 2 of our Stranger Things query, we fetch started_watching edges for each profile. With two profiles, this is trivial, but in production queries fan out across many profiles, each with hundreds of edges to stream and filter. So do we process them sequentially or in parallel? Sequential means waiting for each profile before starting the next, and the delays stack up. Parallel finishes in the time of the single slowest profile, but hundreds of queries doing this at once could overwhelm storage with unbounded concurrency.

The goal: parallel speed without unbounded chaos.

A Kitchen, Not a Single Queue

We structured the query engine like a professional kitchen, with specialized stations for appetizers, mains, and desserts, each with its own capacity. If one station is slammed, the others keep flowing. In practice, that means dedicated thread pools for different work types: fetching nodes, reading adjacency lists, and performing enrichments. When the Stranger Things query reaches Level 2, calls route to the adjacency-list pool, where 8 workers stream and filter each profile’s edges in parallel.

Knowing When to Back Off

Thread pools give us local control, but we also need a global view of total capacity, so we use adaptive concurrency limiting. When things are healthy, we raise the limit gradually (100 in-flight, then 101, 102, and so on); when timeouts or errors spike, we back off by a larger step (say, 100 down to 70). Combined with per-pool limits, the engine constantly tunes parallelism, fanning out within each level while staying inside safe storage and network limits.

Fetching Extra Metadata Along the Way

If the client opted into enrichments (say, maturity ratings for the matched content), the Enrichment Layer fetches them in parallel on its own thread pool and merges them into the response. Enrichment is fail-open: a slow or unavailable source never blocks the query, and we just return the graph data without it.

​​Step 5: Smart Filtering: Keeping Only What Matters

We’ve traversed from Account X to profiles, then to their viewing histories. But raw edges aren’t what our partners need. They care about recent, relevant activity, not every started_watching edge accumulated over the years. This is where filtering decides which parts of the story make the final cut.

From “All Activity” to “The Last 30 Days”

Go back to the original question:

For Account X, show me the Stranger Things viewing history across all profiles: which profiles watched it, what they watched, and when.

The phrase “viewing history” is deceptively simple. Under the hood, it means we need to:

• Ignore older viewing activity, even if it exists in the graph
• Avoid pulling more edges than we actually need
• Let different teams choose their version of “recent enough.”

We handle this with a filtering hierarchy. The system starts with conservative defaults (e.g., 100-day lookback, 300 edges per hop), and requests can override them globally, per-hop, or down to specific edge types. In our query, the 100-day default applies broadly, but the caller sets 30 days for started_watching edges, and the narrower rule wins. Older sessions are discarded. The same engine can just as easily provide a tight recent window on one edge type and full history on another, all in a single query.

Choosing Which Edges to Keep: LATEST vs ANY

Sometimes there are still more edges than we want to return after time filtering. If Profile_Alex watched the same episode several times last month, pausing and resuming, we don’t want to send all those edges back. So we offer two selection modes.

LATEST sorts edges by timestamp and keeps the newest ones up to the limit, ideal for “what has this profile watched recently?” where teams want the current state, not every play event. ANY grabs whichever edges it encounters first, no sorting, which is faster and fine for “has this profile ever watched Stranger Things?” where timing doesn’t matter. Teams default to LATEST and switch specific edge types to ANY when “any proof” is enough.

Bringing It Back to Our Story

So what happens for our running query?

We start with all the started_watching edges for each profile. The time filter narrows this to 30 days. Edge-count limits prevent response flooding. LATEST mode selects the most recent viewing session per title. The result: a concise answer distilled from a verbose history:

• Profiles that watched Stranger Things in the last month.
• Which seasons and episodes they watched.
• The most recent session for each, tying it all together.

This filtering turns raw history into a focused answer.

Step 6: Making It Even Faster: Caching the Things We Keep Seeing

By now, we’ve walked the full path of our query: we’ve traversed from account to profiles, filtered viewing history by time, and focused on Stranger Things sessions.

Despite our optimizations, each storage call still costs a network round-trip. When the same nodes appear across thousands of queries per minute, those redundant calls add up: both in infrastructure cost and in tail latency at scale.

The key question: what can we avoid repeating?

The Things That Don’t Change Every Second

Look back at the entities in our Stranger Things journey:

• The Account node (plan type, region, etc.)
• The Profile nodes (“Alex”, “Kids”, whether it’s a kids profile)
• The Content nodes (Stranger Things seasons and episodes)

These rarely change. Profiles don’t flip between “kids” and “non-kids” every minute. Title metadata is stable.

To improve efficiency, we keep a distributed cache of hot nodes (accounts, profiles, content) that are likely to reappear. When the same entity appears again, we answer “What is this node?” from memory, skipping storage.

Result: for high-traffic entities, we eliminate storage calls and noticeably reduce infrastructure cost and tail latency at scale.

A Quick Replay of Our Query With Caching Turned On

The first time the Stranger Things query runs for Account X, the cache is cold, so we pay the full cost: we fetch the account and its profiles, then the started_watching edges and matching content nodes, caching each node as we go. Minutes later, a different query arrives:

Show me everything Account X’s profiles have watched in the last 7 days, and flag anything rated TV-MA on the kids profile.

This time, many of those nodes are already in the distributed cache. Storage still handles the adjacency lists and edges, but node lookups are lighter and latency drops. At scale, that reuse gives us comfortable headroom for traffic spikes.

Not Everything Deserves a Spot in Cache

We can’t cache everything. The RDG prunes old activity after a set retention window, so caching a node that’s about to be deleted is wasteful.

To avoid polluting the cache, we consider:

1. The node’s last activity timestamp
2. The graph’s retention period (e.g., 100 days)
3. The cache TTL (e.g., 30 days)

If a node was last active 99 days ago, it expires from the graph in a day, so a 30-day TTL makes no sense, and we skip it. We reserve cache space for active nodes like Account X. This “smart TTL” policy keeps the cache focused on live stories rather than archival ones, so repeat queries for the same part of the graph return faster.

Caching is integrated into the journey, not an afterthought. The engine reuses knowledge from previous queries, so repeated traversals over the same part of the graph keep getting cheaper

The Payoff

The serving layer sits in front of 8 billion nodes and 150 billion edges, serving mixed workloads, all of which need to feel interactive. Single-hop queries return at a P50 of 15–30ms with P99 under 100ms. Even 3-hop traversals, the kind that chain across accounts, profiles, and content, come back at P99 between 100–150ms. Breadth-first execution and parallelism within each level keep these numbers stable even as fan-out grows.

The async-first design is what enables the throughput. Thousands of concurrent requests flow through just 16–24 threads spread across dedicated pools because no thread ever blocks on I/O. When load spikes, our concurrency limiter lets work queue briefly: slowly increasing capacity when things are healthy, backing off aggressively when they’re not

Caching has the most visible impact on day-to-day efficiency. Popular entities like accounts, profiles, and content achieve 70–80% cache hit rates, resulting in roughly 3–4x fewer storage calls on common query paths. Smart TTLs keep the cache focused on active data, avoiding wasted memory on nodes that are near the end of their graph retention window.

These properties, together, make multi-hop graph queries over billions of entities feel, at query time, much closer to in-memory lookups than to remote calls.

What We Learned Along the Way

The biggest surprise wasn’t any single optimization: it was how much async composition changed the economics of our system. We expected it to help latency; we didn’t expect it to slash infrastructure cost. A serving layer that would have needed hundreds of threads per instance runs comfortably on 16–24, because no thread ever blocks on I/O. The tradeoff is debuggability: async stack traces are hard to read, and exceptions can get lost in future chains. We compensated with per-stage metrics, measuring each request at validation, storage, enrichment, and end-to-end, so when something is slow, we know exactly which stage to blame.

Caching took longer to get right than expected. Our first instinct was to cache everything in EVCache and let TTLs handle freshness, but that wastes memory on nodes about to expire from the graph anyway. The breakthrough was matching TTLs to data volatility: stable node properties get long TTLs, while nodes near the end of their retention window aren’t cached at all. The 70–80% hit rate we see today came from being selective, not aggressive.

The filtering hierarchy was born out of frustration. Early on, every new use case meant a code change: one team wanted a 7-day lookback, another 90 days, a third different limits at different depths. Instead of bespoke logic per team, we built a layered override system: application defaults, global overrides, per-depth limits, and per-edge-type limits. It took real effort, but it eliminated an entire class of feature requests and teams now tune their own queries without touching our code.

Closing: Principles for Distributed Systems

The lessons above are specific to the RDG, but the underlying principles apply to any distributed system built around I/O-heavy, fan-out workloads.

• Think in terms of frontiers, not features. Design your APIs so callers describe what frontier to explore, then let the system decide how to walk it efficiently.
• Filter early, not late. Every byte you fetch but don’t need is wasted I/O. Push filters and limits as close to the storage layer as possible: discard irrelevant data at each stage rather than fetching everything and trimming at the end.
• Parallelize deliberately, not by default. Unbounded concurrency feels fast until it overwhelms the systems you depend on. Set explicit limits, monitor them, and adjust dynamically: treat concurrency as a dial, not a switch.
• Treat caching as a first‑class design choice, not an afterthought. Decide what is worth remembering, for how long, and what should be allowed to fade out of memory. Match TTLs to data volatility, and don’t cache what’s about to expire.

—

Thanks for reading Part 3 of the RDG blog series. For us, getting these details right is what turns a constantly changing, billion-edge graph into something that, at query time, feels like a responsive, in-memory data structure.

How and Why Netflix Built a Real-Time Distributed Graph: Part 3 — Querying the graph with gRPC… was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

by Aarti Laddha, Richard Diaz-Cool, Rishika Idnani, Venkatesh Selveraj

Netflix supports a vast and evolving set of features and content types, ranging from 4K streaming and immersive audio to live streaming and cloud gaming, across a diverse ecosystem of devices. However, not all devices are created equal. Hardware limitations such as available RAM, CPU cores, display capabilities, or platform support mean that some features cannot be supported on certain device models. To ensure the best possible user experience, we rely on a deep understanding of device capabilities. We have invested in building a comprehensive device capability data model and integrating feature flags from internal systems, paving the way for smarter, more granular feature management across our global device landscape. This approach helps us identify bottlenecks in feature penetration and accelerates the pace of innovation.

We have designed our data storage and modeling strategies to efficiently support analytics at scale. We use a cumulative table to process information about the device’s capabilities. This table is structured to efficiently capture the latest state of each device and its associated capabilities (like Screen resolutions, Video Profiles Supported, Surround Sound, RAM size etc) making it ideal for analytics and reporting use cases.

{
"Screen Height": ["720"],
"Screen Width": ["1280"],
"Video Profiles": 
[
"playready",
"hevc",
],
}

For aggregate analytics, we leverage a histogram table that captures active device counts over the past 28 days, broken down by device model and software version. This table also records the number of devices supporting specific capabilities, enabling detailed distribution analysis. One use case for this histogram data is to analyze the distribution of external display capabilities attached to streaming sticks. For example, the histogram below shows that out of total X number of devices, all supported the HD profile (playready), while only 20% devices supported the UHD profile (hevc).

{
"Video Profiles": {
      "playready": 100%, # HD profile
      "hevc": 20% # UHD profile
}
}

We have built analytical products that leverage these datasets to provide a comprehensive view of feature reach such as 4K Ultra HD, Netflix Spatial Audio, Cloud Gaming and the latest UI. By relying on data-driven insights, we can make informed decisions about which features to enable on specific devices, ensuring both performance and reliability.

Modeling Device Capabilities for Analytics was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Authors: Ying Li, Arjun Rao, Shradha Sehgal

Introduction

Recommendations sit at the heart of the Netflix experience. Our current production models rely on thousands of hand‑crafted features over users, items, and interactions, along with specialized architectures for sequence modeling, feature interactions, and multi‑task objectives. This stack has evolved over many years to support diverse content types (movies, series, games, live, podcasts) and product surfaces, but its complexity makes it costly to onboard new use cases: adding a content type or surface can require significant feature engineering, architecture change, infrastructure work, and experimentation.

At the same time, large language models (LLMs) are changing how we think about recommendation, as shown by recent work such as PLUM, GLIDE, and OneRec-Think. Their broad world knowledge and strong language understanding make it possible to represent user histories and item metadata directly as text, capture rich relationships in a shared semantic space, and steer recommendations via natural‑language prompts. However, off‑the‑shelf LLMs are still far from production‑ready recommenders: they often over‑recommend globally popular content, hallucinate out‑of‑catalog items, ignore business constraints, and provide only limited personalization.

To address this, we built GenRec, an LLM‑backed recommendation ranker that post‑trains an internal foundation LLM on Netflix‑specific data and objectives. GenRec shows that an LLM‑based ranker can match or exceed a mature production system while relying on far fewer labeled examples and input signals.

At a high level, GenRec:

• Verbalizes user histories, item metadata, and context as text.
• Post‑trains a Netflix‑adapted foundation LLM for ranking.
• Adds a catalog‑aware scoring head over Netflix titles.
• Uses reward signals to align with long‑term member value and business goals.
• Runs in prefill‑only mode on Netflix’s LLM serving stack for cost efficiency.

In a large‑scale A/B test against a well‑tuned production ranker, GenRec achieves statistically significant improvements in both short‑term and long‑term online metrics, while using only a small fraction of the Phase‑2 labeled data and input signals. It reduces our reliance on hand‑engineered features and shifts the focus from feature engineering to context engineering. In this blog post, we will describe how GenRec works, how it performs, and why we believe it points toward a more LLM‑centric future for recommendation at Netflix.

Problem Setting

We focus on a full‑catalog ranking task (or top‑K ranking when a candidate set is provided).

Given a user 𝑢, their interaction history 𝐻, and the current context 𝜏 (device, surface, locale, time, etc.), GenRec scores each item and produces a personalized ranking that can directly power recommendations or serve as input for downstream personalization systems.

Formally, we map a request (u,τ,t,H) — user, context, time, and history — to a ranking 𝜋 over the catalog C, where π(i) is the position assigned to item i. We optimize π for expected long‑term member utility (a proxy for satisfaction and retention), not just short‑term engagements.

From Foundation LLM to Recommendation Ranker

GenRec follows a two‑phase training framework (Figure 2):

Phase 1 — Netflix-Adapted Foundation LLM.

We start from an open‑source LLM and adapt it on proprietary Netflix corpora, so it learns foundational capabilities such as

• Netflix content understanding
• Member behavior and preference patterns
• General language understanding and generation.

Phase 1 is updated relatively infrequently and serves as a shared, Netflix‑aware backbone for many applications.

Phase 2 — GenRec.

We then turn this foundation model into a high‑quality ranking model by post‑training on ranking‑specific data and objectives. Phase 2:

• Focuses on ranking quality and steering
• Incorporates multiple reward signals via reward‑weighted losses
• Is refreshed more frequently to track new content and evolving tastes
• Is explicitly optimized under serving cost constraints.

Training Data as Conversations

Netflix members generate hundreds of billions of interaction events spanning many surfaces (views, plays, durations, thumbs up/down, add to list, abandons, etc.). We convert this log data into single‑turn or multi‑turn “conversations” between a user and a recommender. Each turn contains:

• User message: verbalized context, profile, history, item metadata, and task (e.g., recommend what the user will watch or thumb next).
• Assistant message: the member’s actual engagement (e.g., which titles were played, for how long, what feedback they provided).

During Phase‑2 training, the LLM learns how assistant messages depend on user messages. This allows us to express rich recommendation signals as text, jointly supporting both the language-modeling (LM) and ranking objectives.

At inference time, we feed in the verbalized context and apply a catalog‑aware scoring head to rank items; we do not decode assistant messages. The conversational format is primarily used during training to support the LM objective and preserve strong language understanding over the verbalized text.

Verbalization and Context Engineering

Traditional recommenders operate on dense features and embeddings. GenRec takes a different approach: it verbalizes rich user histories and context as natural language, encoding raw interaction signals directly in the LLM’s semantic space. In doing so, it relies on the model to discover higher‑level patterns — such as item relationships and evolving user interests — rather than on manual feature engineering.

Naively verbalizing every interaction in a user’s history can quickly exceed the token budget and be too expensive at Netflix scale. The context window becomes our new “feature budget”, so we apply context engineering:

• Retain in full: high‑signal engagements (e.g., long plays, thumbs‑up) with richer details
• Omit: low‑signal events (e.g., very short plays or quick hovers)
• Summarize or compress: repetitive behaviors (e.g., binge‑watching )
• Elaborate selectively: important or cold‑start items (e.g., new releases)

Within a fixed token budget, we prioritize recent, high‑signal history and compress or drop older history. We also structure the prompt to maximize shared prefixes for better prefix caching. The goal is a compact, high‑information prompt that preserves ranking quality without prohibitive costs.

Objectives: Ranking, Language, and Rewards

The overall GenRec model is trained with a multi‑objective loss that combines a recommendation ranking objective, language modeling objectives, and alignment via reward‑weighted training.

1. Catalog‑Aware Ranking Objective

The primary task is a ranking objective that teaches the model to score items by engagement quality. We label positives using high‑value engagements (e.g., sufficiently long plays, strong explicit feedback), with thresholds and denoising logic, and train the model — via a cross‑entropy loss over the catalog or candidate set — to assign higher scores to these positives given a verbalized context.

2. Language Modeling Objective

We also retain a language modeling (LM) objective over the verbalized inputs and outputs. This helps preserve the model’s general language understanding, improves its ability to interpret rich natural‑language histories and item metadata, and keeps the door open for text‑generation use cases such as recommendation explanations.

3. Reward‑Weighted Loss for Alignment

Beyond raw ranking accuracy, GenRec must (1) respect business requirements — for example, balancing movies, series, games, live, and podcasts — and (2) optimize long‑term member satisfaction rather than just immediate clicks or plays.

Training only on raw interaction sequences can lead to undesirable behaviors, such as over‑favoring binge‑watching or over‑focusing on a single content type. To address this, we weight the ranking loss using signals from separate reward models. Each training example receives a scalar weight derived from two types of signals:

• Long‑term satisfaction proxies: estimate how much a short‑term engagement contributes to long‑term outcomes, such as return behavior, catalog exploration, or sustained engagement.
• Behavior rebalancing: adjust behaviors across content types and launch stages (for example, games vs. movies, new releases vs. evergreen titles) to better align with business goals.

The example’s ranking loss is then scaled by this weight: high‑value engagements receive larger weights, and low‑value ones are down‑weighted. This reward‑weighted approach is simpler and more cost-efficient than full reinforcement learning, yet provides effective alignment in practice. We have seen additional gains from RL‑style methods (e.g., GRPO), but leave them to future work due to their higher cost.

Model Architecture and Serving

Backbone and Scoring Head

GenRec’s architecture closely follows our foundational LLM: a decoder‑only Transformer trained with next‑token‑prediction style objectives, augmented with a catalog‑aware ranking head that scores only Netflix in-catalog items. The scoring pipeline works as follows:

1. Verbalization: A verbalizer V serializes user history H, context 𝜏 , and relevant item metadata into a single text sequence x.
2. Pooled representation: The LLM processes x, and we extract a pooled hidden state h that summarizes the user’s current preferences and context.
3. Catalog‑aware scoring: Each catalog item i has a learned embedding eᵢ. A scoring head ϕ combines h and eᵢ (e.g., via dot product or small MLP) to produce a score sᵢ. Applying a softmax over scores yields a probability distribution which we convert into a ranking π.

All parameters — the backbone, scoring head, and item embeddings — are trained jointly. For very large catalogs, we can use sampled softmax or candidate sets for efficient training and inference. This architecture constrains recommendations to the Netflix catalog while supporting efficient scoring over large candidate sets.

Serving and Cost Optimization

GenRec is served on Netflix’s internal LLM stack using vLLM. At Netflix scale, serving cost is driven primarily by 1) Model size; 2) Context length; 3) Inference mode (prefill vs. autoregressive decoding). We control cost through three strategies:

• Smaller / distilled models: We train GenRec on smaller or distilled foundation models, often with larger or more targeted datasets, to capture most of the quality of larger models at lower serving cost.
• Aggressive context compaction: Using the context engineering described earlier, we minimize tokens while preserving ranking quality.
• Prefill‑only inference: Autoregressive decoding over large candidate sets would be prohibitively expensive. Instead, we run in prefill‑only mode: the model consumes the prompt once and scores the entire candidate set in a single forward pass, with no token‑by‑token decoding.

Together, these choices make it feasible to serve GenRec on high‑volume workloads within compute budgets.

Offline and Online Experiments

We evaluated GenRec against a mature production ranker that has been tuned over many years. The baseline model relies on thousands of engineered dense and embedding features, as well as custom architectures for modeling feature interactions and sequences. We assessed performance using both offline evaluation metrics and a large‑scale online A/B test.

GenRec vs Production Baseline

Offline, GenRec outperformed the production ranker on ranking metrics despite using far fewer input signals and labeled examples. With roughly 40× fewer Phase‑2 labeled training examples, GenRec achieved about +1.6% improvement in Mean Reciprocal Rank (MRR). As we increased Phase‑2 training data and enriched the input signals, GenRec’s offline metrics continued to improve.

Online, we ran a large A/B test on batch‑compute recommendation surfaces, covering ~10% of Netflix traffic over ~4 weeks. In this low‑data, low‑signal configuration, GenRec delivered statistically significant gains over the production baseline on both short‑term and long‑term online metrics (Figure 3).

These results indicate that a properly post‑trained and aligned LLM‑backed ranker can be a strong alternative to traditional recommendation models, with substantial headroom as we further scale data and input signals.

Data, Model, and Phase Contributions

We ran ablations to understand where GenRec’s gains come from.

Data and Model Scaling

• Data scaling: For both ~1B and ~10B parameter backbones, offline MRR improves as we increase Phase‑2 post‑training data. Larger models reach higher absolute MRR but follow a similar scaling curve (see Figure 4).
• Model scaling: Under a fixed training budget, we post‑trained GenRec variants from ~1B to ~10B parameters. Within this budget, larger backbones consistently achieved higher offline MRR than smaller ones.

Phase-1 vs. OSS, Phase-2 vs. Phase-1

• Phase-1 vs. OSS: Using the Phase‑1 Netflix‑adapted foundation LLM as the base model improves offline ranking metrics by roughly 10–20% compared to starting directly from an off‑the‑shelf LLM.
• Phase-2 vs. Phase-1: Phase‑2 post‑training adds another 35–50% gain in offline ranking metrics when evaluated near the Phase‑1 training cutoff (i.e. when Phase‑1 model is the freshest). As time passes and Phase‑1 becomes stale with new content and shifting tastes, the relative benefit of Phase 2 grows to about 80% after 2 weeks.

Data efficiency vs. production ranker

• Starting from a strong Phase‑1 model, GenRec matches or exceeds the production ranker using 10–40× fewer Phase‑2 labeled examples, depending on configuration. This marginal data efficiency is especially valuable because Phase 2 is refreshed far more frequently than Phase 1.

Context Length Optimization

Context length drives both quality and cost: longer verbalizations expose more behavior and context but increase training and serving cost. To study this trade‑off, we varied context length and verbosity and optimized them in three steps:

1. Clean and compress events: drop low‑signal engagements and compress repetitive behavior to form a cleaned sequence of events.
2. Find the “elbow point”: vary how many historical events we include and plot MRR vs. number of events to identify an elbow beyond which additional context yields diminishing returns (see Figure 5).
3. Optimize verbosity: for the retained events, test different levels of details and simplified wordings, measuring MRR each time.

In our experiments, we can reduce the context tokens to roughly one-third of the original budget with negligible degradation in offline ranking metrics. Since serving cost is approximately proportional to context length, we observed a similar reduction in serving cost.

Towards LLM‑Native Recommendation

GenRec is more than “swapping in a Transformer” for an existing ranker. It hints at a broader shift toward LLM‑native recommendation at Netflix. A few notable changes:

From Feature Engineering to Context Engineering

Traditional RecSys stacks revolve around large feature sets and heavy feature infrastructure. LLM‑centric systems instead revolve around constructing rich textual contexts from raw logs, metadata, and tools. The “prompt” becomes the new feature vector.

Modeling effort shifts from designing features to deciding which signals to include, how far back in time to go, how to compress or summarize history within a token budget. Our experiments on verbalization compaction illustrate this shift: careful context design can preserve quality while dramatically reducing serving cost.

From Customized Architectures to Foundation Backbones

Historically, each recommendation task often had its own custom architecture (two‑tower models, DLRM‑style networks, bespoke attention blocks). In an LLM‑centric world, many tasks share a common foundation backbone, with differentiation coming from data and verbalization strategies, post‑training objectives and rewards, and inference optimization.

GenRec leverages the same backbone as our foundation LLM rather than introducing a new architecture built from scratch. This makes it easier to share learnings across applications, and opens the door to natural‑language steering for future experiences.

Scaling Laws as Design Guides

Traditional RecSys can hit diminishing returns due to sparse IDs, heavy engineering objectives, and task‑specific architectures. With an LLM‑backed backbone, recommendation inherits clearer data and model scaling behavior: within cost limits, more data and larger models consistently improve quality. This brings RecSys design closer to the broader LLM paradigm, where scaling laws help guide model and data investment.

From RecSys Infra to LLM Infra

LLM‑backed recommenders push us toward LLM‑style infrastructure: GPU‑accelerated, vLLM/Triton‑based, with careful batching and caching. Over time, recommendation serving infra starts to look more like general LLM infra than classic RecSys stacks built around MLPs or factorization models.

Conclusions

We have presented GenRec, an LLM‑backed recommendation ranker at Netflix that adapts an internal foundation LLM for large‑scale personalization. By verbalizing user histories, context, and item metadata, adding a catalog‑aware ranking head, using reward‑weighted objectives aligned to long‑term satisfaction and business goals, and serving efficiently on our LLM infrastructure, we obtain a model that improves on a strong production ranker while using far fewer Phase‑2 labels and input signals.

GenRec is an early but promising step toward a more LLM‑centric recommendation stack at Netflix. Our results suggest that, with careful attention to cost, infrastructure, and alignment, LLM‑backed recommenders can play a central role in large‑scale personalization.

Acknowledgments

GenRec is the result of close collaboration among multiple teams and organizations across Netflix. The contributors to this work (in alphabetical order):

AI for members: Arjun Rao, Ashish Rastogi, Baolin Li, Fernando Amat Gil, Grace Huang, Justin Basilico, Kamelia Aryafar, Linas Baltrunas, Moumita Bhattacharya, Ogheneovo Dibie, Rein Houthooft, Shradha Sehgal, Sejoon Oh, Sergi Perez, Sourabh Medapati, Thea Wang, Yaochen Zhu, Yesu Feng, Ying Li, Yun Li, Yucheng Shi, Yunan Hu

AI platform and serving: Abhishek Agrawal, Adam Singer, Binh Tang, Daneo Zhang, Derek Olejnik, Ed Maddox, Erik Osheim, Lingyi Liu, Liping Peng, Meghana Chilukuri, Nicolas Hortiguera, Shaojing Li, ZQ Zhang

Product: Ilke Kaya, Michelle Kislak, Scarlet Chen, Si Cheng

GenRec: Towards LLM-Native Recommendation at Netflix was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

by Matthew Wood, Ishan Gupta, Kevin Mercurio, Devon Bryant, and Claire Dorman

In his seminal book “Thinking, Fast and Slow,” Daniel Kahneman describes two systems that drive human cognition: System 1, which operates automatically and quickly with little effort, and System 2, which allocates attention to more challenging mental activities requiring deliberate focus. This dual-process theory has profound implications not just for understanding human behavior, but for designing intelligent systems that must balance immediate responsiveness with strategic foresight. Similar “plan vs. act” decompositions show up in other domains too — for example, robotics and autonomous driving often separate a slower planning layer (setting goals and constraints over longer horizons) from faster control and execution loops, and modern LLM agents frequently pair deliberate planning with rapid, step-by-step tool use and reaction.

At Netflix, our messaging platform faces a similar challenge every day. We send hundreds of millions of personalized notifications — push messages, emails, and in-app alerts — to help members discover content they’ll love. This creates a central tension: optimizing each notification for near-term engagement can conflict with what is best for the member over the long term. Higher message frequency can increase fatigue and opt-out risk, while lower frequency can reduce awareness of relevant titles and features the member would value.

This blog post introduces our framework for personalized notifications — a hierarchical system where a “slow” policy makes strategic, personalized decisions about a member’s weekly messaging plan (e.g., the intended frequency per channel and the resulting pacing over the week), while a “fast” policy handles the tactical, real-time decisions about which specific message to send when a send opportunity occurs. Together, they balance near-term engagement with longer-term member experience.

The Problem:

Before introducing our new framework, it is helpful to ground the discussion in a representative baseline for a personalized notification system. In our previous production system, we used a causal model to make send decisions by predicting the causal effect of a single message over a short time horizon. While this approach is effective as a baseline, it suffers from two fundamental limitations:

Short-Term Reward Horizons

The single-message outcome model is trained to optimize short-horizon metrics, such as immediate user actions occurring shortly after a notification is sent. While this is excellent for driving near-term engagement, it misses the cumulative, long-term effects of a messaging strategy. A message that drives an interaction today might also contribute to notification fatigue, reducing responsiveness in the weeks to follow. Because critical indicators of member satisfaction — like sustained viewing habits or gradual opt-out risk — only surface over extended timeframes, a short-term model will always miss the bigger picture.

Coupled Ranking and Pacing Decisions

When a single system evaluates daily incrementality to decide both whether to send something and, if so, which item to send, an individual member’s weekly message frequency becomes a by-product of those daily decisions rather than an explicit control variable. In our previous single-policy system, frequency was controlled implicitly through a relevance threshold on the model score calibrated to achieve a target aggregate send rate. While effective for managing overall frequency, this mechanism limited the system’s ability to personalize frequency based on individual engagement patterns. Moreover, because send eligibility and message selection were coupled in the same decision rule, adjusting the threshold to control frequency also changed the distribution and quality of selected messages, and vice versa.

To solve these challenges, we needed a system that could separate longer-term strategy from shorter-term decisions. What if we could determine an optimal, personalized message plan for each member, and then focus on selecting the most relevant content within those bounds? In the following sections, we detail how we realized this vision by decoupling our notification engine into a hierarchical ‘System 1’ and ‘System 2’ framework.

The Proposed Method: A Hierarchical Slow-Fast Architecture

The Slow policy’s primary role is to define a personalized pacing of messages over a defined time horizon. The decisions made by slow policy are consumed by the Fast Policy whose role is to maximize immediate relevance and select the optimal message for the member at any given moment.

To illustrate the Slow Policy in practice: For example, if optimized at a weekly cadence, the policy evaluates a member’s long-term engagement patterns to select a “Pacing Plan Action.” To keep the action space manageable yet expressive, we discretize the decision space into a set of actions that independently specify push and email frequencies. This provides approximately O(100) distinct combinations of cross-channel pacing strategies.

The Utility Function

The Slow policy selects actions by maximizing a personalized utility function. This function explicitly trades off positive engagement signals against the long-term “cost” of messaging.

U(member, action) = Σ wₖ·Reward_k(member,action) — Cost(action)

To capture a holistic view of member health, this utility is composed of:

• Positive Signals: Capturing the likelihood that a member will find value in and engage with the platform.
• Negative Signals: Capturing the likelihood of member fatigue or a propensity to opt out of a messaging channel.

Ideally, negative signals alone would naturally penalize over-messaging. In practice, however, explicit negative feedback is extremely sparse. Without an additional constraint, the predicted ‘cost’ of an incremental message appears negligible, causing the model to gravitate toward maximum frequency.

To address this, we introduce a universal message cost that is added to the personalized negative‑feedback prediction for every send. This additional cost term keeps the reward function concave and well‑behaved, preventing degenerate “always send” policies. The message cost parameter is empirically tuned using a combination of online experiments and offline evaluation metrics.

Pacing Strategy

The two-stage design naturally allows for optimizing both the average frequency as well as pacing of messages over time. The simplest pacing strategy is uniform random: we translate the frequency target into a per-opportunity send probability and, at each eligible opportunity, effectively flip a weighted coin to decide whether to send. This produces an organically randomized pattern whose expected send rate matches the target.

While uniform pacing provides a clean and robust baseline, the framework readily extends to richer, non-uniform pacing profiles (for example, day-of-week patterns, conditioning on user activity, or launch-aligned bursts) whenever product or user-experience considerations call for more structured temporal distributions.

Policy-to-Policy Communication

The true power of this hierarchy lies in decoupling. By splitting into “Slow” and “Fast” policies, we allow each to focus on what it does best.

To bridge these two worlds asynchronously, decisions are events and state is managed through a low-latency feature store:

• The Planner (Slow): The Slow policy calculates a member’s ideal pacing plan. It writes this strategic intent to a feature store
• The Executor (Fast): Every day, when a notification opportunity arises, the Fast Policy simply pulls that stored “plan” as a feature. It then executes the tactical send decision within those strategic guardrails.

This architecture provides two critical advantages:

1. “Stickiness”: It ensures a member receives a consistent experience. The Slow policy will be executed once at a defined cadence; the plan is stored and honored.
2. Independent Evolution: We can retrain, optimize, or A/B test our weekly pacing strategies (the “Slow” layer) without ever touching the real-time ranking logic (the “Fast” layer).

Key Results & Takeaways

The transition to a hierarchical architecture resulted in one of our largest production metric lifts to date. We observed several key breakthroughs:

• Empowering the “Casual Viewer”: Gains were most significant among members who watch less frequently — a critical cohort where timely, high-relevance awareness of new content is vital.
• The Power of Decoupling: Separating frequency planning from message selection was as transformative as the modeling itself. This new architecture unlocks incredible flexibility, allowing us to iterate on content ranking models and pacing strategies as two independent, clean variables.
• Respecting the Horizon: The impact of messaging is rarely an isolated event; its effects build up cumulatively based on ongoing interactions between our system and the member. By isolating pacing into a dedicated strategic layer, we now have the mechanism to explicitly manage long-term fatigue and opt-out risk.

Acknowledgments

We could not have delivered this project without the help of our outstanding colleagues, and we sincerely thank them for their contributions.

Feature Store Team: Aaron Lewis, Tom Switzer, Abby Whittier, Ray Zhang
Product: Fiona Li
AI for Member Systems (supporting contributor): Sergi Perez

Thinking Fast & Slow for a Personalized Notification System was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

By Guil Pires, Jennifer Prince, Jose Camacho, Ken Kurzweil, Phanindra Chunduru

Background

In a previous post, we introduced Data Bridge, a unified management plane for batch Data Movement at Netflix. Historically, several bespoke Data Movement connectors were developed across different engineering organizations to fulfill their specific requirements. Over the last few years, the Data Movement team has started centralizing these offerings through an abstraction that provides a catalog of connectors, along with simple UI and APIs to initiate Data Movement jobs.

One such case is the Cassandra to Iceberg connector. Apache Cassandra powers mission critical applications at Netflix, including Member, Billing, Recommendations, Subscriptions and many more. These use cases heavily leverage Data Movement to Apache Iceberg for many analytics and operational tasks, and central to this movement was a connector for Cassandra to Iceberg built in-house named Casspactor. As many Cassandra based Data Abstractions emerged, such as Key Value, Time Series and Graph — the need for larger and more complex Data Movement with transformations became more critical to the business.

Data movements are fundamentally fulfilled by leveraging the existing Cassandra backup infrastructure. Regularly scheduled backups are performed directly on the Apache Cassandra nodes, via a sidecar process managing the upload of all necessary SSTables and associated Metadata files directly into Amazon S3. When a Data Movement job is initiated, the job constructs the specific backup structure it needs by referencing the S3 based metadata, allowing it to precisely locate the SSTable files. The engine then downloads these files, performs the required mutation compaction and processing, and finally writes the fully transformed, compacted data directly into the target Apache Iceberg tables.

Casspactor: The Engine We Outgrew

Casspactor processed roughly 1,200 data movements per day, transferring approximately 3 PB of data from Apache Cassandra into Apache Iceberg tables. It served some of the most critical workloads at Netflix. For years, it worked. Then, two compounding challenges made it clear we needed a fundamentally different architecture.

Fragile Metadata Dependencies

Before Casspactor could move a single record, it needed to answer a deceptively simple question: which backup exists, is it complete, and what does it contain?

Casspactor assembled this answer from multiple independent systems:

Each system had its own failure modes, update cadences, and accuracy guarantees. Casspactor’s view of the world was a composite, and composites diverge from reality.

Metadata fell out of sync with actual backups, causing Casspactor to read stale or incorrect data silently. Routine maintenance on the Cassandra Clusters triggered uncoordinated snapshots, and because Casspactor required all nodes in a region to snapshot at the same clock second, a single node replacement could break data movement for an entire region.

The fix was hiding in plain sight. The answer to “which backup exists and is it complete?” already lived in the backup storage layer (Amazon S3) itself. By reading metadata directly from the backup files, we could replace the entire dependency chain with a single source of truth.

Every Connector Inherited Casspactor’s Limitations

Cassandra at Netflix does not just store raw tables. It backs higher level data abstractions, such as Key Value, Time Series, and others, each with its own data model, access patterns, and semantics. When any of these abstractions needed to move data to Iceberg, they all funneled through Casspactor.

Every abstraction inherited Casspactor’s constraints:

• Skewed partition failures: Casspactor could not handle tables with large partitions, a common pattern in Key Value and Time Series workloads. Jobs crashed with out-of-memory errors on some of Netflix’s largest datasets.
• No data model awareness: Casspactor moved raw Cassandra tables as is. Connectors for Key Value and other abstractions had to bolt on post processing to reconstruct their data models from the raw output — extra cost, extra complexity, and an extra surface for failures.
• Intermediate table bloat: Casspactor wrote to an intermediate Iceberg table before producing the final output. The Key Value connector added another intermediate table and a snapshots table. Connectors for abstractions on top of Key Value added even more. This compounded into significant storage cost overhead.
• Inability to Time Travel: by relying on multiple services to compose a backup unit, Casspactor was unable to restore prior backups in the event of cluster Topology or Keyspace schema changes.
• Monolithic design: Casspactor was built as a single connector, not as an engine. There was no way to build a family of purpose built connectors on a shared foundation.

We needed something fundamentally different: an engine that reads directly from backups in S3, produces standard Spark DataFrames, and lets each data abstraction build its own connector with full awareness of its data model. One foundation, many connectors.

The New Stack: A Layered Architecture

The new architecture, built upon the foundation of Apache Cassandra Analytics and the in-house Move Data framework, represents a fundamental shift toward a layered, purpose-built stack designed for reuse and maintainability. This new engine was conceived with clear separation of concerns, moving away from Casspactor’s monolithic design. The architecture is intentionally layered with the foundation being a core S3 reading capability: the Cassandra Analytics Wrapper, which is built on top of the Open Source Cassandra Analytics with Netflix’s internal backup representation and an S3 Client.

This layer handles the raw data retrieval from backups, translating it into standard Spark DataFrames. Sitting atop this foundation is a “Connector Factory” model, via both Java UDFs and transforms which allows individual data abstractions (Key Value, Time Series, others) to build highly optimized, data model aware connectors that process the generic Spark DataFrames, avoiding the need for complex, expensive, and failure-prone post-processing steps. This layered approach ensures that improvements to the core reading engine benefit all connectors, while the connectors themselves are focused solely on data transformation.

• Handles Skewed Partitions: By moving the mutation compaction and processing to the Executor level within Spark, the new engine can efficiently handle tables with highly skewed or wide partitions, a major pain point for Casspactor. Crucially, this processing occurs without excessive data shuffling, preventing out-of-memory errors and enabling reliable movement of Netflix’s largest datasets.
• Operates at Spark DataFrames (No Intermediary Tables): The new architecture directly generates standard Spark DataFrames from the Cassandra backups. This eliminates the need for Casspactor’s costly, multi-stage intermediate Iceberg tables, which led to storage bloat and operational complexity. This native DataFrame operation enables the “Connector Factory” by providing a universal, easily consumable interface for building diverse, model specific connectors.
• Jobs Auto Size: The engine integrates intelligent auto-sizing capabilities, allowing jobs to dynamically adjust resource consumption based on the source table’s characteristics. This removes the burden of manual tuning from engineering teams, ensuring optimal performance and cost efficiency without sacrificing reliability.
• Reduced Dependencies: By reading metadata directly from the backup files stored in S3, the new stack removes the fragile, multi-service dependency chain that plagued Casspactor. S3 becomes the single, authoritative source of truth for backup existence and completeness, vastly improving data movement reliability and consistency.
• Time Travel: A critical feature of the new stack is the ability to process the schema, cluster topology, and data as a cohesive unit at a specific point in time. This capability provides robust time travel functionality, essential for auditing, debugging, disaster recovery and reproducing past data states.
• Performance: Collectively, these architectural improvements, including native DataFrame processing, optimized partition handling, and streamlined metadata retrieval have resulted in notable performance gains, reducing overall data movement execution runtime and cost compared to the legacy Casspactor system.
• Cost: by eliminating intermediary Iceberg tables and efficient SSTable compaction on Executors, the new stack needs a significantly smaller storage and compute footprint leading to significant cost savings in the order of USD millions.

The Journey Towards a Safe Migration

The successful validation of the new stack was the critical first step, but it only marked the beginning of the most challenging phase: the migration. Large scale data migrations are inherently complex, high-risk undertakings that can be time consuming and often result in customer frustration and service disruption. To navigate the high stakes of decommissioning a mission-critical system like Casspactor and seamlessly replacing it, we needed a strategy that prioritized reliability and transparency above all else.

The migration was fundamentally enabled by a Like-for-Like strategy, which served as the cornerstone of our Platform Engineering philosophy, abstracting complexity. The core tenet was to maintain absolute consistency across the user-facing interface, the output contract, and the final data artifact. This meant ensuring that the data movement parameters defined via the Data Bridge abstraction remained unchanged, and, critically, the schema, metadata, and data within the destination Iceberg tables were identical to the legacy output. By preserving these external contracts, we eliminated the need for complex, time-consuming coordination with dozens of internal teams who relied on these data pipelines. This approach transformed the migration from a distributed, high-risk, multi-team effort into an internal platform implementation detail, allowing us to achieve a transparent, zero-impact transition and accelerate the retirement of the legacy system without requiring any code changes or validation from downstream users.

To navigate this migration, we developed a strategy anchored by three core pillars that serve as a blueprint for successful, large-scale data migrations:

1. Validation: Establishing and maintaining absolute confidence in data consistency through rigorous, ongoing validation.
2. Visibility: Instrumenting every part of the system to provide a clear, real-time understanding of migration progress and system health.
3. Safety: Ensuring user impact is minimized or eliminated, despite the inevitable system failures, by leveraging abstractions and robust fallbacks.

The next section will provide a detailed exploration of these key pillars.

Pillar 1: Validation

Trust is earned, and in data migration, it is earned one row at a time. The first pillar is the most critical: providing a measurable guarantee to users and partners that the data produced by the new system is an exact, row-by-row replica of the data produced by the old one.

Our foundational tactic was deploying the new Move Data connector in a “shadow” testing that ran in parallel with the production Casspactor jobs. This allowed us to validate the new system with real-world, production workloads without any customer impact.

• Let C be the set of rows in the legacy Casspactor output (Iceberg table).
• Let M be the set of rows in the new Move Data output (Iceberg table).

The test for trust: prove that C = M. This required continuously checking for two conditions:

1. Rows in C but not in M (C-M): The new system missed data.
2. Rows in M but not in C (M-C): The new system introduced phantom or erroneous data.

Any result where the cardinality of these difference sets (the number of differing rows) was greater than zero triggered an immediate, high-priority investigation. The target was 100% similarity.

Uncovering and Resolving Disparities

The shadow mode quickly became a powerful forensic tool, exposing “unknown unknowns”, subtle discrepancies that were not bugs in the new system but rather differences in behavior between the new and old systems. Resolving these was the core work of building trust. For each problem we initiated an investigation log where we captured the details, logs, queries that allowed us to diagnose. Based on the assessment the issues were categorized so that similar differences on other datasets were later resolved affecting many of the shadow pipelines.

Maintaining an investigation log was critical to organize the outstanding issues and effectively communicate to stakeholders the progress and confidence of the new connector so that we effectively measure the appropriate level of “confidence” to initiate the migration.

We observed differences in how connectors leverage reference timestamps for Time-to-Live, Consistency Levels, backup selection, and various internal business logic. This continuous, data-driven cycle of discovery and resolution was the mechanism by which we built confidence in the new architecture.

Pillar 2: Visibility

Trust is built in the background, but an active migration requires real-time insight: Visibility. The second pillar involves instrumenting the system to provide an unambiguous, clear understanding of operational health and migration progress.

We extended our instrumentation to the overall migration workflow and its dependencies:

• Dashboards: We created centralized dashboards to track migration status, visualizing the total number of data movements migrated versus those remaining. The dashboards tracked execution status, average runtime, and cost comparisons between the two connectors.
• Dependency Tracking: Since the new system relied on a new set of APIs to fetch backup metadata, we implemented detailed metrics for failures to keep track of the APIs or dependencies failed.
• Alerting: Proactive alerts were set up for job failures (Move Data or Casspactor), failures on Move Data that triggered a fallback to Casspactor or any data discrepancy being detected.

This comprehensive instrumentation allowed the team to be proactive, fix issues as they emerged during the migration, and gain the necessary confidence to accelerate the migration timeline.

Pillar 3: Safety

Even with perfect data correctness and enhanced visibility, the third pillar, Safety is required for a zero-impact migration. The challenge is ensuring that when a system inevitably fails, the user experience is uninterrupted. Our strategy centered on decoupling the user’s workflow from the underlying connector implementation.

Leveraging Abstraction: The Decider Pattern

To achieve a transparent swap, we leveraged the Maestro workflow orchestration platform to implement the Decider pattern:

1. Data Movement Abstraction: From a user’s perspective, their Data Movement job definition remained the same.
2. The Decider Step: Internally the workflow responsible to execute the job was modified to include a Decider step. This step took the data movement parameters (source cluster, table name, destination) and invoked a control plane: Connector Controller.
3. Connector Controller as the Registry: The control plane served as the dynamic registry. Based on the migration cohort and the data movement attributes, it determined and reported the appropriate connector to use either Casspactor (legacy) or Move Data (new).

This abstraction gave our team complete control. We could upgrade or rollback any connector for any data movement instantly by simply updating a configuration in the controller, with zero modification required to the thousands of downstream customer workflows. Crucially, this abstraction guaranteed the critical safety net: a conditional step in the Maestro workflow logic ensured that if the Move Data step fails, it would immediately execute the Casspactor step.

This pattern would increase the chances that the user’s data movement completes successfully, even if the new connector encountered a bug or transient failure during the initial rollout phases. User impact was completely eliminated; they might see a slightly longer runtime in the event of a failure and fallback, but they would never see a migration failure or suffer from stale data.

Beyond the workflow, the new system architecture itself was inherently more resilient. By building the new data movement connector on Cassandra Analytics and reading backups directly from S3, we removed fragile dependencies on deprecated internal services.

Conclusion

The migration from Casspactor to the new, layered architecture built on Cassandra Analytics and the Move Data connector was more than a typical “tech debt” project; it was a fundamental shift in our approach to data movement reliability and scalability at Netflix.

The legacy system, while serving us well for years, was ultimately constrained by monolithic design, fragile metadata dependencies, and an inability to handle the complexity of modern data abstractions. The new stack resolves these issues by delivering a robust, cost-efficient, and inherently more resilient solution that reads directly from S3, handles wide partitions gracefully, and eliminates costly intermediate tables.

Our blueprint for the migration, anchored by the three pillars of Validation, Visibility, and Safety, ensured a transparent and high-confidence transition. Through rigorous shadow testing and a data-driven audit framework, we achieved the desired data consistency. Enhanced dashboards and alerting provided the real-time operational insight necessary to manage risk. Most critically, the implementation of the Decider pattern within our workflow abstraction minimized the impact for all downstream users.

This successful migration validates a core philosophy: by abstracting complexity at the platform level, we can perform large system migrations without burdening our product engineering partners. The new foundation is now ready to support the next generation of Netflix’s data abstractions.

Looking ahead

This foundational work on the Cassandra Data Movement stack has done more than just replace a legacy system: it has become an accelerator for innovation across the entire Data Movement organization. By providing a reliable, performant engine that standardizes data retrieval into Spark DataFrames, we’ve enabled the rapid development of new, highly optimized connectors. This new “Connector Factory” approach has already delivered a dedicated Key-Value to Iceberg and Time Series connectors, both of which are fully aware of their respective data models, eliminating costly post-processing. This architecture is also paving the way for ambitious new initiatives, including the development of a solution for bulk loading data into Cassandra itself, effectively completing the data movement cycle, and enabling safer fleetwide connector rollout with canaries inspired by the Decider Pattern.

We are incredibly grateful for the extensive collaboration among the Data Movement, Data Bridge, Online Data Stores, Membership, Billing, Subscriber and Ads platform teams at Netflix; this work simply couldn’t have been accomplished without their partnership!

The Evolution of Cassandra Data Movement at Netflix was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

by Emily Gill

Each year, we bring the Analytics Engineering community together for an Analytics Summit — a multi-day internal conference to share analytical deliverables across Netflix, discuss analytic practice, and build relationships within the community. This post is one of several topics presented at the Summit highlighting the breadth and impact of Analytics work across different areas of the business.

Understanding Risk in Content Launches

Every title you see on Netflix goes through several key phases: Development, Pre-Production, Production/Principal Photography, Post-Production, and finally, Launch Preparation, all leading up to the Title Launch. Once Principal Photography wraps, the focus shifts in Post-Production from content creation to quality assurance and visual effects (if needed).

At the end of Post Production, Netflix receives the final audio and video files — often delivered as an IMF (Interoperable Master Format) — which triggers a flurry of Launch Preparation activities, focused on tasks such as the development of artwork and trailers, creation of subtitles, maturity ratings & quality control, that happen within a tight window and rely on having the finalized media assets in hand.

Some of this work can be kicked off earlier using a non-final version of the media called the Locked Cut, but since it’s not the absolute final deliverable, this presents a tradeoff: should our teams who prepare content for service wait for the more finalized IMF to begin their work, or start sooner with the unfinal Locked Cut? Waiting for the IMF risks a compressed timeline if it arrives late, while starting with the Locked Cut means teams may need to do additional conformance work if there are significant changes between the Locked Cut and the final IMF.

Identifying Gaps in Schedule Accuracy

To help navigate the decision of when to start launch preparation, our teams rely on estimated delivery dates for both the Locked Cut and IMF media assets, which are manually provided by content partners in production schedules. However, these schedules often have gaps in coverage and lack accuracy for both asset types (see Figure 1).

This isn’t unexpected — productions are dynamic, facing frequent changes, scheduling conflicts, and unforeseen obstacles that can shift timelines without warning. As a result, there’s a clear opportunity to leverage the wealth of production data we collect to predict the risk of schedule slips. By developing a predictive model, we aim to both fill in ETA gaps (providing asset delivery estimates when none exist) and improve the accuracy of existing ETAs compared to traditional manual schedules.

Correlation between Schedule Accuracy and Launch Misses

Our analysis reveals a strong correlation between scheduled inaccuracies and launch misses — instances where a title experiences delays. To quantify schedule inaccuracy, we created a metric called Accumulated Error Days (AED), which measures the cumulative deviation between estimated (scheduled or predicted) delivery dates and actual delivery dates over time. AED is calculated retrospectively as the area between the scheduled (grey line) or predicted (blue line) delivery dates and the actual delivery date (green line).

When we compare titles with at least one launch miss to those without, we find that mean AED is significantly higher in the group with launch misses. Notably, this effect is even more pronounced when we focus on the period closer to delivery — indicating that high AED (i.e., inaccurate schedules) in the final stretch before launch is especially correlated with launch misses, more so than AED accumulated over a longer timeline. These findings further motivate our efforts to improve schedule accuracy and reduce AED by leveraging rich production data and predictive modeling.

Modeling Time-to-Delivery

Our predictive models are designed as boosted tree regression models that predict the “days until” either media asset delivery for in-progress productions.

To power these models, we leverage a range of upstream data sources including production-level signals of progress, title metadata, and seasonal signals. We are able to predict the days until media asset delivery using daily update snapshots, allowing us to generate up-to-date predictions that reflect the latest state of each in-progress production. This means that we have each feature and what its value was as of each day of a production. Modeling with this snapshotted data enables us to generate up-to-date predictions as new information becomes available, build a flexible model that works across all production phases, and seamlessly incorporate dynamic features that evolve over time (Figure 2).

Evaluating Our Approach

Building a Comprehensive Metrics Suite

When evaluating the performance of the predictive models, we look across a suite of metrics to try to understand where and when predicted dates outperform scheduled dates. Among these are mean and median absolute error, relative to actual delivery, to understand the accuracy of our estimated dates. We also consider bias metrics, such as mean and median error, to understand if we are consistently over- or under-predicting the actual delivery. We calculate the standard deviation of our errors to understand if there are large shifts in the bulk of the distribution of errors. For the tails of our error distributions, we calculate the percentage of our absolute errors that are greater than x days to delivery.

For scheduled dates, we calculate coverage across various horizons to delivery. This is a value prop of the model; we’ve built the model in such a way that we can always provide a predicted date and recoup any coverage gaps that exist from scheduled dates alone.

Benchmarking Against Manual Scheduling

In a backtest, we observed significant improvements across all of our metrics and across most horizons from delivery. As an example, see Figure 3 which plots global mean absolute error (MAE) and shows large reductions in errors (greater accuracy) in predicted IMF and Locked dates as compared to scheduled dates. Additionally, we see large reductions in outliers from scheduled to predicted dates as well.

Since our teams use these dates over a period of time and not at a single point in time, there is an additional benefit that we’re describing as an Earlier Accuracy Signal. By leveraging predictive dates, our teams benefit from a level of accuracy that they would otherwise have to wait x amount of time for if using scheduled dates. As an example, 6 months out from Locked Cut delivery the predicted dates are better than scheduled dates on 76% of titles and have a level of accuracy (6.1 wks MAE) that scheduled dates don’t reach until 11 weeks later.

Circling back to AED, which we mentioned earlier is correlated to launch misses, we find that in our backtested titles globally, and across most buying orgs and content types (i.e., series versus standalones), predicted IMF and Locked Cut dates reduce AED from scheduled dates when calculated across the 6 months leading up to delivery. We see similar patterns when we repeat this for shorter horizons to delivery as well.

Streamlining Workflows with Improved Scheduling

A key advantage of this predictive model is that estimated delivery dates are already integral to our stakeholders’ workflows — meaning we can introduce predictive dates without overhauling existing processes. However, this creates a new challenge: with both scheduled and predicted dates available, teams need to determine which is more reliable. While predictive dates are often more accurate on average, there are situations where scheduled dates perform better. To address this, we’ve built serving logic that defaults to scheduled dates in buying orgs where the model underperforms. Elsewhere, teams can view both dates side by side in dashboards, allowing them to apply their own judgment. Additionally, our predictive models leverage features that are tied to scheduled dates, which has emphasized the need and impact of ensuring our upstream teams continue to input and update scheduled dates even in the presence of our predictions. We’re piloting these predictive signals in multiple ways, tailoring the approach to fit the diverse needs and tools of our various launch prep functions.

Predicting Risk in Content Launches: How Data-Driven Insights can Transform Launch Planning was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

By Amer Hesson, Marcelo Mayworm, James Mulcahy, and Brittany Truong

The Problem: Managing Assets at Netflix Scale

Netflix’s Data Platform is vast. We have millions of tables in our data warehouse and tens of thousands of scheduled workloads running across our orchestration systems. Behind each of these assets sits an engineer, a team, or an initiative — and behind each of those sits a set of decisions about who can access what, and how those workloads execute day after day.

For years, the tools we used to manage access and identity for these assets operated at the granularity of the individual asset. Every table had its own Access Control List (ACL). Every workflow ran under the identity of the engineer who authored it. In a workforce that is fluid, where people change teams, change roles, and occasionally leave the company, this fine-grained model broke down in two persistent, painful ways.

Problem 1: Permissions that can’t keep up with organizational changes

Imagine you’re on a team that owns a few hundred tables. Your org restructures, a neighboring team merges into yours, and you inherit another few hundred. Now you have to find every ACL on every table, figure out who should still have access, and update them one by one. Multiply that by every reorg across every team across the company. The result? Two failure modes:

1. The support team gets flooded. A significant and outsized share of support threads were requests to update table permissions en masse in response to org changes. While self-service tooling and best practices are in place to manage this, adherence is inconsistent. Data Projects addresses this by promoting the solution from optional tooling to a foundational part of the data platform.
2. Access gets granted far too broadly. Rather than maintain fine-grained ACLs, teams would often open up table access to the whole company. This defeated the purpose of having ACLs in the first place.

Problem 2: Workloads tied to human identities

Scheduled and asynchronous workloads — Maestro workflows, data movement jobs, Spark pipelines — need an identity to run as. Historically, that was a human: whoever authored the workflow.

Human identities are not durable. People change teams, get new responsibilities, and leave the company. When they do, their permissions change, and the workflows running under their identity start to fail. The only fix was to swap in a colleague’s identity, which inevitably had different permissions, kicking off a “permissions whack-a-mole” as each fix surfaced the next missing grant. And then, eventually, that colleague would also move on, and the cycle would repeat.

Enter Data Projects

We introduced Data Projects to tackle both problems head-on. At its core, a Data Project is two things:

1. A container to manage and view a set of related assets in aggregate: tables, workflows, and other data assets grouped under a single logical umbrella.
2. A synthetic, durable, and assumable identity: one that asynchronous and scheduled workloads can execute under, independent of any human’s lifecycle.

You can think of it as hoisting the granularity of management up from the individual asset to a meaningful container: the project. Instead of managing permissions on 500 tables, you manage them on one project that contains those 500 tables.

While the initial focus has been access and identity, the abstraction has applications well beyond those concerns. That broader potential is part of what makes it worth investing in.

Grants and Roles

Each Data Project has a set of grants managed by the owning team. Different identity types can be added as grants: users, groups, applications, and continuous integration (CI) jobs. Each grant has a role that determines what the grantee can do within the project. For example, a Contributor has read/write access to the project’s assets, while a Viewer has read-only access. These roles roll up neatly — instead of rewriting hundreds of ACLs when someone joins or leaves a team, you update a single project grant.

The Identity Umbrella: Netflix and IAM

Every Data Project is provisioned with a Netflix application identity, and optionally an AWS IAM role. This is the “identity umbrella” that makes workloads durable:

• The project’s Netflix identity is what executes the project’s async workloads (e.g. Maestro workflows). It belongs to the project, not to any person.
• The project’s IAM role supports specialized use cases in AWS like Spark jobs on Amazon EMR. Crucially, the IAM role can be exchanged for the project’s Netflix identity in a cryptographically secure way.

Members with privileged roles can also assume the project’s Netflix identity. This is enormously useful for testing and troubleshooting from a development context like a laptop or a notebook — you get to run commands as the project, exactly as the scheduled workload would.

Gravity

One of the more elegant properties of Data Projects is what we call gravity. When a workload running under a project’s identity creates a new asset — say a Maestro workflow creates three tables — those assets are automatically added to the project as contained assets. The project becomes the center of mass for everything produced under its identity. You get organization for free as a side effect of how the platform already works, eliminating future challenges of discovering relevant assets and gaining access to them.

Securing Data Workflows with Data Projects

Maestro is Netflix’s primary workflow orchestrator for batch analytics, covering scheduled ETL pipelines, data movement jobs, ML training, and much more. Because workflows can run on schedules without the original user present, Maestro is designated a Trusted Workload Manager (TWM), formally authorized to mint fresh identity tokens on behalf of the workloads it manages.

That identity matters everywhere. A single workflow execution may be checked against table ACLs in the Secure Data Warehouse, authorization policies for Netflix resources, and IAM policies for AWS — all in a single run. If the identity is fragile, the whole workflow is fragile.

The Problem with User-Tied Identity

The standard pattern was to run workflows under an On-Behalf-Of (OBO) credential — for example, maestro OBO alice@netflix.com. This gave the workflow the union of Maestro’s and the human’s permissions, but in doing so it also bound the workflow’s permissions to that person’s. When they changed teams or left Netflix, the workflow broke. A colleague might take over ownership, but they rarely had the same access as the previous owner, so the workflow would stay broken for days while permissions were sorted out. At Netflix’s scale, with tens of thousands of scheduled workloads, many of them business-critical, this was unsustainable.

Data Projects: Durable Identity

Data Projects solves this by replacing user-tied identity with a durable, team-owned Netflix application identity: one that doesn’t change teams, go on vacation, or leave the company. Each project groups related workflows, tables, secrets, and other assets under a single consistent identity, and Maestro validates the caller’s access to the project before executing any workflow under it.

The downstream improvements are as follows:

• Tables created during execution are automatically associated with the project’s identity through gravity, inheriting its access controls without additional configuration.
• Secrets are scoped to project policies, so ownership transfers no longer strand credentials.
• Access is managed once at the project level, replacing fragmented per-user grants across every asset the workflow touches.

The result is a workflow identity model that is stable, auditable, and built to survive the organizational changes inevitable at any company operating at this scale.

Success Stories

Many Data Projects have already grown to contain tens of thousands of assets in production. A couple examples are highlighted below:

• Streaming Quality of Experience: A core observability pipeline tracking quality of experience (QoE) metrics whose continuity used to depend on whichever engineer happened to own the underlying workflows. Now it runs under the project’s identity, stable regardless of team membership changes.
• Member Analytics: Analytical models and ETL workflows for member data products. A concentrated set of business-critical analytics whose access is managed at the project level rather than across hundreds of individual tables and workflows.

More broadly, we’ve seen Data Projects adopted as the organizing principle for entire analytics domains. Where teams previously maintained their own access policies, ad-hoc grant lists, and tribal knowledge about “who should have access to what,” the project is now the single answer.

Using Data Projects

Onboarding workflows onto Data Projects is a matter of:

1. Creating a project for the logical grouping of assets (or using an existing suitable one).
2. Granting the right people and groups the appropriate roles.
3. Configuring the workflow to run with the project’s identity.

Thanks to gravity, new assets produced by project workflows land in the project automatically. Migrating existing workflows can be a challenge as it requires setting up the Data Project with the appropriate permissions before changing its execution identity. We are actively working on infrastructure to track the access patterns of existing workflows so that we can recommend precise permission updates for the destination project. Our goal is to make the Data Project the de facto option for executing any kind of asynchronous workload.

What’s Next

Data Projects started as an Analytics Platform initiative, a response to specific pains in the data warehouse, but the underlying ideas are not unique to data. We see a potential future where Projects (not just Data Projects) are a first-class platform concept spanning data assets, software assets (GitHub repositories, Spinnaker applications, Docker images), and even studio assets (production content, pipelines, and transformations).

We’re also investing in:

• Rightsizing: we’re integrating a layer on top of our authorization policies that automatically rightsizes permissions based on actual usage patterns, proactively eliminating unnecessary access and preventing “permission creep”.
• Hoisting beyond access and identity: the project is a natural unit for surfacing other concerns at the aggregate level — cost attribution, health indicators, and more.
• Ad-hoc use case integrations: extending project identities beyond scheduled workloads to cover interactive, on-demand actions like running a query through the Data Portal.
• Activity logs and audits: a unified timeline of grant changes, asset changes, and workflow versions at the project level.

Conclusion

Data Projects is an answer to a simple observation: at Netflix’s scale, the unit of identity and access management can’t be the individual asset or the individual human. It has to be something larger, something durable, something that matches the way teams actually think about the work they own.

A project is that unit. And as we continue to generalize the concept beyond the data warehouse, we expect it to become one of the foundational primitives of how engineering at Netflix is organized, not just how data is organized.

Acknowledgments

We would like to express our gratitude to the following individuals for their contributions to this effort: Ryan Bordo, Doug Clark, Luke Fernandez, Sarrah Figueroa, Ankit Gupta, Brian Hoying, Ye Ji, Abhishek Kapatkar, Anmol Khurana, Matheus Leão, Hechao Li, Raymond Liu, Alice Naghshineh, David Noor, Anjali Norwood, Javier Garcia Palacios, Kunaal Parekh, Brandon Quan, Andrew Seier, Jason Seo, and Ethan Zhang.

If you are interested in helping us solve these types of problems and helping entertain the world, please take a look at some of our open positions on the Netflix jobs page.

Data Projects: Managing Data Assets at Netflix Scale was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

By Celina Amados

At Netflix, our catalog metadata is crucial to our member experience, and a single corrupted data state can impact millions of viewers immediately. To protect streaming reliability, we built an automated data canary system that validates data transformations using production traffic. This canary detects issues in under 10 minutes, and blocks bad data from reaching our members.

Intro

Catalog metadata is what makes Netflix functional. It defines what titles exist, where they’re available, whether they can be played, and more. This data gets transformed and distributed across our vast infrastructure near-continuously, powering everything that helps members find what they want to watch. Accurate catalog data delivers moments of joy. Corrupted catalog data breaks streaming.

What Went Wrong

A production incident revealed a critical gap in our resilience strategy. No code had been deployed. No configuration had changed. But, a manual mitigation action taken during a previous incident had inadvertently corrupted a data feed, rendering it empty for a subset of titles.

The impact was immediate: missing metadata prevented manifest generation, causing failures in our catalog service and playback issues.

Engineers were alerted immediately, but identifying the root cause took time. After intense triaging, responders pinpointed the corrupted data feed and pinned services back to a known-good state, restoring playback.

The problem? Our sophisticated code canary deployments had caught nothing. No code had changed — the data had.

This incident exposed a fundamental gap in our resiliency capabilities: we can validate code deployments, but we had no equivalent for our high-velocity data pipelines. Our catalog metadata, consisting of titles, artwork, availability, and more, was continuously transformed from multiple upstream sources and published at a regular cadence. Each upstream source had its own validation, but these checks didn’t catch corruption in the final transformed output.

We needed to treat data deployments with the same rigor as code deployments.

The Challenge: Validating Data at Short Intervals

Our catalog metadata service operates as a high-velocity data pipeline: it processes multiple input feeds, transforms them, and publishes the final catalog state that gets distributed across our infrastructure.

This creates unique validation challenges that our traditional canary analysis tools aren’t designed to handle:

Time Constraints: Our existing canary analysis tools require 30–60 minutes to reach statistical confidence. We had a much shorter window between data cycles; we needed to detect issues, make a decision, and block publishing all within a single cycle.

Emergent Issues: While each upstream data source has independent validation, problems often only manifest in the final transformed state. We needed to validate the actual output that clients would consume, not just the inputs, as close to the clients as possible.

Production Traffic is Essential: We initially considered shadow traffic, but quickly realized it was insufficient. Shadow traffic can only replay requests to our catalog metadata service; it can’t simulate the entire playback lifecycle across multiple services and domains. To detect real customer impact, we needed real production traffic.

Limit Blast Radius: Despite using production traffic for validation, we couldn’t allow customers to experience widespread issues during the validation process. Any regression needed to be detected and contained immediately.

Our Solution: The Data Canary Orchestrator Pattern

After evaluating several architectural approaches, we developed a solution built around three key innovations:

1. Dedicated Orchestrator Pattern

We created a dedicated cluster for the purposes of canarying new catalog metadata that separates concerns, avoids self-testing, and provides a pattern for extensibility. Here’s how it works:

Orchestrator Instance: A dedicated orchestrator instance of our catalog metadata service coordinates the data canary flow. When a new catalog version is published to the canary environment, the orchestrator validates that both baseline and canary clusters are healthy and version-synchronized, then triggers a chaos experiment.

Permanent Baseline & Canary Clusters: Two dedicated service clusters run continuously in our canary region. The baseline cluster always serves the latest production catalog version, while the canary cluster receives new versions for validation.

Generic Integration Point: Upon chaos experiment completion, the orchestrator reports results back to the transformer service via a REST endpoint. This generic interface means new data sources can implement their own orchestrator patterns without requiring transformer code changes.

This pattern can now be adopted by other teams at Netflix for validating different data sources, which is exactly the kind of extensibility we designed for.

2. Utilizing and Extending our Chaos Platform

Meeting the 10-minute constraint required not only leaning on our chaos platform, but also extending it to meet our needs:

Custom Threshold Tuning: We worked with our Resilience team to customize experiment thresholds for our use case. Standard chaos experiment thresholds were too conservative for our time constraints.

Multi-Tenant Testing: Our catalog service supports multiple client types with different traffic patterns and downstream dependencies. We ran separate experiments for major client types and discovered that running traffic through the tenant that handles playback requests consistently identified failures fastest.

Sticky Canaries: To isolate experiment traffic, sticky canaries use session affinity to guarantee that once a user’s traffic is routed to the baseline or canary clusters, it stays there for the duration of the experiment window. This prevents cross-contamination from concurrent chaos experiments, ensuring a clean apples-to-apples comparison between data versions.

Behavioral Metrics Over Technical Metrics: We focused on Starts Per Second (SPS), or actual customer playback attempts, as our primary signal. SPS proved more reliable than latency or error rates for detecting catalog corruption because it directly measures customer impact, and data errors may not always manifest as application errors to our catalog metadata service.

Immediate Abort on Regression: Instead of collecting data for post-hoc analysis, we stream metrics in real-time and abort experiments the moment we detect regression. This trades some statistical confidence for speed, but our tight thresholds and clear signal make this not only acceptable, but necessary.

3. Production-Hardened Edge Case Handling

Building a system that runs in production every 10 minutes taught us that the devil is in the details:

In-Flight Experiments During Redeployment: When the orchestrator restarts, it must detect and continue polling any ongoing experiments, as we can’t abandon a validation cycle mid-flight.

Leader Election: During orchestrator deployments, multiple instances might be running simultaneously. We implemented safeguards to ensure only one experiment is triggered per version announcement.

Version Synchronization: In a multi-tenant service where different clients consume data at different cadences, we track version state to ensure baseline and canary clusters are properly aligned before triggering experiments.

Validating the Validator: Controlled Failure Injection

To prove the system worked, we needed to break things on purpose. We ran a series of controlled experiments where we deliberately corrupted catalog data — denylisting high-profile titles and simulating real data corruption scenarios — to validate that the canary could detect issues and block publication.

These experiments were coordinated as proactive incidents during business hours, with product operations teams on standby. We routed approximately 0.2% of global traffic through the validation flow, minimizing blast radius while still generating meaningful signal.

Key Results:

• Detection Speed: Issues identified in 2.5–4 minutes depending on client type
• Clear Signal: 10x error differential between canary and baseline
• Automatic Blocking: Publishing workflow blocked as designed when regressions detected

The experiments validated our end-to-end workflow and revealed important operational insights: different client traffic patterns detect failures at different speeds, and threshold tuning requires careful refinement based on the magnitude of impact we want this system to detect. Most importantly, they proved that even with a 10-minute validation window, far shorter than traditional 30–60 minute canary analysis, we had sufficient signal to catch high-impact catalog corruption.

Bringing Code Validation Principles to Data

This effort wasn’t just about building a validation system, it was about recognizing that data deployments deserve the same rigor as code deployments. Just because something isn’t a binary doesn’t mean it can’t break production. The patterns we landed on aren’t specific to catalog metadata, and can be applied to systems with high-velocity data pipelines more broadly.

If you’re working with data that changes frequently and impacts customers directly, ask yourself:

• What’s your MTTD for data corruption?
• Can you validate with production traffic safely?
• How would you detect emergent issues in transformed data?
• What behavioral metric most closely indicates customer impact in your domain?

Today, the failure mode that caused the aforementioned incident would be caught and mitigated in under 10 minutes. We all know outages aren’t a question of if, but when. The next time you find yourself faced with bad data, how fast will you be able to respond?

Acknowledgments

This work was a collaborative effort across multiple teams at Netflix. Special thanks to Jongyoon Lee, David Su, and Zubeen Lalani of the Catalog Foundations & Distribution team for their contributions to the design, and to Ales Plsek of the Resilience team for their support in customizing our chaos platform for our unique use case.

The Data Canary: How Netflix Validates Catalog Metadata was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

By Alvin Bao, Alex Petrov, Jennifer Lai, Aidan Sherr, and Samartha Chandrashekar

As a part of the journey to transition Netflix’s compute infrastructure to be more Kubernetes-native, we have leaned into incorporating components from the Kubernetes ecosystem into our container platform Titus. One example of this is our use of Kueue, a cloud-native job queueing system for batch workloads, which has largely replaced the custom queuing and scheduling logic in our homegrown managed batch solution Compute Managed Batch (CMB). In this post, we’ll give an overview of what motivated the migration, how we migrated millions of batch jobs to use Kueue, and what Kueue allows us to offer as a Compute platform.

Brief Overview of CMB and Titus

CMB is a managed batch solution that allows users and applications to execute and manage workloads that run to completion. Using a tenant hierarchy, workloads are managed and queued with ordered execution through priorities, and capacity is managed on a per-tenant basis. Workloads that are submitted to CMB are then run on Titus. The features of Titus relevant to CMB are workload federation across multiple cells (Kubernetes clusters) and federated capacity reservations. This means CMB can talk to a single Titus endpoint to get/submit workloads and update capacity reservations without having to worry about the underlying cell/cluster topology.

CMB Tenant Hierarchy

Tenants provide a grouping mechanism for jobs submitted on behalf of certain organizations, platforms, or applications. Users can create and organize tenants however best suits their organization or use case. For example, an organization may use a single tenant across several applications or a complex hierarchical structure that matches its team and application ownership structure.

Tenants are associated with a capacity configuration. The capacity configuration defines the amount of compute capacity available to the tenant and provides certain guarantees around isolation from other tenants. The capacity configuration contains weight (used for fair sharing) and resource dimensions.

There are two types of tenants in CMB:

1. Internal Tenants — meant to facilitate the creation of a tree of tenants. Internal tenants’ children can be both internal and leaf tenants. Internal tenants themselves do not accept work and thus do not have associated queues.
2. Leaf Tenants — can accept work and have queues associated with them. Leaf tenants cannot have any children.

With regards to capacity configuration, tenants can use 2 types of capacity:

Reserved Capacity

For internal tenants, if a user specifies reserved capacity, it is fair-shared across the subtree and usable by the leaf tenants under that internal tenant.

For leaf tenants, if a user specifies reserved capacity, it partitions capacity within the hierarchy so that other tenants cannot reserve the same resources. Those reserved resources are not shared with any other tenant, ensuring throughput for a given leaf tenant.

Shared Capacity

The Compute team maintains a global pool of shared capacity that any tenant can burst into, in addition to its reserved capacity. Reservations are not required to use CMB, so a tenant can run out of shared capacity entirely. The pool is fair-shared across tenants, but in CMB, this applied only at admission: CMB had no preemption, so once a job was admitted, it ran to completion regardless of shifts in fair-share demand.

Kueue changes the semantics for both types of capacity, which the fair sharing and preemption section covers.

Here is an example of what a tenant hierarchy looks like:

CMB User/Application Workload Submission Flow

CMB User/Application Tenant Management Flow

Why Kueue?

CMB was created in 2018, before or alongside many of the open-source batch compute offerings available today. Over the years, as the Kubernetes ecosystem has evolved, many of the features that CMB offered or strived to offer have been included in these open source projects e.g., fair sharing, hierarchical tenants, capacity management, priority queuing. In addition, it became increasingly cumbersome to develop new features such as preemption when CMB was so far removed from the underlying Kubernetes cluster.

The team took a look at what it would take to modernize our batch abstraction and settled on Kueue for the following reasons:

1. Unlike other options such as YuniKorn or Volcano, Kueue does not replace pod scheduling by the kube-scheduler, allowing integration with existing Titus scheduling profiles. Replacing Titus scheduler profiles can fragment job placement, potentially harming efficiency.
2. Adoption momentum and pace of innovation.
3. Kueue supports multi-tenant quota management over heterogeneous hardware.
4. Kueue can operate on primitives such as v1.Pod and batch/v1.Job, and also supports higher-level abstractions such as RayJob / RayCluster for future extensibility.
5. Kueue has native features that the team would have liked to implement in CMB, such as preemption, all-or-nothing scheduling, topology aware scheduling.

Migrating to Kueue

This initiative of migrating CMB workloads to Kueue became known as Netflix Batch. The key tenets of our migration were the following:

1. Migration should require zero lift for CMB end users and be completely transparent to them
2. No regressions in container launch rate and overall max throughput
3. Replace CMB queuing and scheduling with Kueue

Netflix Batch User/Application Workload Submission Flow

The key difference between the old and new flows is that we defer queuing and scheduling to Kueue, which is enabled in each Kueue-enabled Titus cell. Titus federation routes the job to Kueue cells using our custom Kueue router.

Netflix Batch User/Application Tenant Management Flow

For us as operators, the migration was as simple as clicking a button on a tenant in our UI (as shown in the example above). This also allows us to easily rollback changes if there were issues.

Under the hood, this enrollment converts internal tenants to Cohorts and leaf tenants to a ClusterQueue + LocalQueue. The capacity configuration on a given tenant is converted into resource flavors and nominal quotas. The architecture for this looks as follows:

Lessons Learned

1. Maintaining API parity with the existing system (vs exposing a new API surface) and migrating the underlying components as a first step derisked the project by unstacking bets while also ensuring we didn’t disrupt the customer experience.
2. Don’t wait until the end to migrate the most complex use case. We decided early on to migrate our largest and most complex customer first. This allowed us to build confidence that we could later migrate other customers to Netflix Batch without issues, and resulted in the production migration lasting only 4 weeks.
3. We had to run Kueue with much higher QPS, Burst, and groupKindConcurrency than the default configuration to meet our throughput needs. This was derisked early on by running load tests in a development environment that mimics Titus.

Current State of Kueue at Netflix

Kueue is fully rolled out in production, with it managing millions of batch workloads. In the future, we’re looking at options to enroll more of Titus batch workloads into this more managed experience. We have also productionized more fair sharing and preemptions to address better utilization of reserved capacity. In addition, our learnings are being leveraged by other internal teams, including those building Kubernetes-native training infrastructure, to inform their job scheduling and queuing configurations.

Fair Sharing and Preemption

With Kueue, Preemption-based Fair Sharing allows Netflix Batch to maintain reservation semantics while lending resources to other tenants when those reservations are not in use. In addition, preemption allows Netflix Batch to preempt lower-priority workloads for higher-priority workloads. For our customers, this means that tenants can use more idle capacity from reservations, submit more jobs without the risk of starvation, and have quicker turnaround times for business-critical workloads.

An example preemption configuration on a ClusterQueue that we would be using is as follows:

apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
  name: "team-a-cq"
spec:
  preemption:
    reclaimWithinCohort: Any
    withinClusterQueue: LowerPriority

With these features deployed, Compute has seen a significant increase in average resource utilization.

Acknowledgement

This work would not have been possible without the great work of the entire Compute team at Netflix.

How Netflix Simplified Batch Compute with Kueue was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

By Zhuoning Yuan, Ta-Ying Cheng, Benjamin Klein, Bahareh Azarnoush

Introduction

At Netflix, we build technology to help storytellers bring their creative visions to life and to help members discover the stories they love.

To connect stories with diverse audiences around the world, we produce promotional assets, including trailers, teasers, and social short‑form videos, that build on and elevate the original footage. Through close collaboration with the teams crafting these assets, we identified a recurring gap in current tools. Transforming raw footage into a polished final asset often requires complex edits like seamlessly adding new visual elements, patching or replacing backgrounds, or removing unwanted objects without breaking the scene’s physical continuity. These tasks typically demand hours of specialized manual editing work. While recent generative video editing models show promise, they often struggle to preserve the integrity of the source footage. Many methods regenerate every pixel to make an edit, which can fail to isolate changes and inadvertently alter elements that should remain untouched. To execute these tasks effectively, artists need tools that empower them to dictate exactly what changes and how it changes.

Our research goal is to make this process easier for artists. We’re deliberate about where and how AI is applied, ensuring that the technology always serves the creative intent. That principle drives our recent work: exploring the benefits of generative AI in ways that protect and expand creative choice, and keeping artists in precise control of their final vision. Recent advancements in AI video editing have demonstrated impressive capabilities in streamlining complex manual editing workflows, but key challenges remain before they can reliably support professional use:

• Unintended edits: When editing a specific element in a video clip, many methods regenerate the entire video, which can inadvertently alter identity, performance, and other elements like objects, backgrounds, or critical scene details.

• Unnatural physics: When removing objects, many methods focus only on erasing the target while ignoring the scene’s physical continuity. This can lead to inconsistent motion and implausible interactions, making the results look unnatural.

Today, we’re sharing two research explorations that aim to address these challenges. We believe this work can help advance the field in a way that’s both meaningful and responsible:

• Vera: a layered video diffusion model. Vera generates only what needs to change as separate edit layers while leaving the rest of the video untouched, preserving the identities, performances, and other details from the source footage exactly as filmed.
• VOID: a video inpainting model for video object and interaction deletion. VOID performs physically plausible inpainting in complex scenes: it doesn’t just remove an object, but also reconstructs the scene as if the object was never there.

Along with this blog post, we’re also publicly releasing the research papers that detail the algorithmic innovations behind Vera and VOID. We hope these publications will enable other researchers to experiment with these ideas, build upon our findings, and further advance the field.

Vera: A Layered Video Diffusion Model

Existing video editing models regenerate the entire clip, coupling the intended edit with regions that should remain unchanged. This increases the risk of altering details of the original footage. To tackle this challenge, we introduce Vera, a novel layered video diffusion framework for content-preserving video editing.

Inference Pipeline

Given a source video and a text editing instruction, Vera jointly generates an edit layer and an alpha matte. These layers are then seamlessly composed with the original footage to produce the final edited result. By design, Vera supports complex tasks such as object addition and background change, while ensuring that the pixels outside the edited regions from the source video remain perfectly intact.

Training Data

One of the main challenges in developing Vera was the lack of suitable training data. Since no public dataset provides the high-quality layered data we need (clean input, alpha matte, edit layer, composite video), we built our own. Using a combination of existing open-source videos and human annotation, we constructed a layered video dataset with a total of 486k frames at 832×480 resolution. We organized it into three subsets of increasing complexity:

• Synthetic Composites: Clips with high-quality foreground alpha mattes are composited over diverse, automatically generated backgrounds. This subset provides strong and reliable supervision for alpha matting in object addition and background change tasks.
• Realistic Single-Object Videos: Real-world clips are processed through segmentation, matting, background inpainting/generation, and human quality filtering. This subset increases scene diversity and camera motion, improving composition quality across both tasks.
• Realistic Multi-Object Videos with Effects: This extends the previous subset by isolating individual objects with curated alpha mattes, including their associated effects such as shadows and reflections. This subset improves compositing and editing in more complex, dynamic scenes.

Model Architecture

Beyond data, model design is another key challenge. The three target outputs Vera generates — an edit layer (decoupled creative edits), an alpha matte layer (a grayscale mask that depends on the edit content and scene interactions such as occlusions), and a composite layer (natural footage) — have substantially different distributions. In practice, using a single shared architecture to reconcile these differences proved data-inefficient. To address this, Vera uses a MoT (Mixture-of-Transformers) design. Instead of a single DiT, we use three separate DiTs, one for each output:

• Each DiT maintains its own QKV projections and FFN weights, but we concatenate the output tokens from all three branches and then pass it to joint self-attention. This enables cross-layer interaction while allowing each branch to specialize.
• All three DiTs are initialized from the same pretrained T2V base model. We add two additional patch-embedding layers for the input video and an optional mask video. Source-video tokens are added to the composite tokens, while mask tokens are added to the noisy alpha tokens.
• All layers share the same RoPE (Rotary Positional Encoding). We also add zero-initialized learnable embeddings to the alpha and composite tokens to help the model distinguish between layers.

Evaluations and Results

To evaluate Vera, we curated a benchmark of test video-prompt pairs: 72 for object addition and 69 for background change, using open-source videos. The test set spans a range of difficulty, including slow and fast motions, various camera motions, single and multiple objects, and both simple and complex scenes. We evaluated the performance across three complementary dimensions:

• Content Preservation: Measures whether regions outside the targeted edit remain strictly unaltered, evaluated using pixel-level and perceptual similarity.
• Instruction Compliance: Measures how faithfully the edited video executes the text prompt.
• Video Quality: Assesses the temporal coherence and per-frame spatial quality of the final edited video.

In our results, both Vera-1.3B and Vera-14B significantly outperform existing baselines on content preservation, while maintaining similar video quality and instruction compliance performance compared to strongest baselines (please see the paper for full results).

To complement automated metrics, we ran a human preference study comparing Vera against five baselines. We collaborated with 19 creative reviewers who evaluated 512 video trials in total. In each trial, reviewers were shown randomized side-by-side comparisons between the Vera model and a baseline model. The human consensus strongly aligned with our quantitative findings: Vera-1.3B was preferred over all baselines for content preservation and instruction compliance. Furthermore, reviewers rated Vera’s video quality as comparable to baselines on background change tasks, and noted a clear advantage for Vera on object addition tasks.

VOID: Video Object and Interaction Deletion

Existing video object removal methods excel at inpainting content “behind” the object and correcting appearance-level artifacts such as shadows and reflections. However, when the removed object has more significant interactions — such as collisions with other objects — current models fail to correct them and produce implausible results. To address this, we present VOID, a video object removal framework designed to perform physically-plausible inpainting in these complex scenarios.

A Two-Pass Inference Pipeline

Given an input video, the user clicks on an object to remove. A VLM-based reasoning pipeline then analyzes the scene to identify other regions that will be causally affected, e.g., objects that will fall, collide, or change trajectory. This physical reasoning is encoded into a quadmask to guide the diffusion model:

• First Pass: VOID takes the video and the quadmasks as input and generates a physically plausible counterfactual video in which the object — and its interactions — are removed.
• Second Pass: Smaller video diffusion models occasionally suffer from “object morphing” when generating moving objects. If VOID detects this failure mode, it triggers a second pass that re-runs inference using flow-warped noise derived from the first pass, stabilizing the object’s shape along its newly synthesized trajectory.

Training Data

We built on top of the Kubric simulation engine and the HUMOTO human motion capture dataset to generate synthetic counterfactual video pairs along with their corresponding quadmasks. Specifically, the counterfactual videos are generated by re-simulating the exact scene from the original video, but with the target object(s) or human removed. This resimulation creates an alternate outcome based on strict laws of physics. For example, if a person holding a lamp is removed from the scene, the simulation ensures the lamp obeys gravity and falls to the ground. The quadmasks then capture the removed object (black), the affected regions (grey), their overlaps (dark grey), and the unchanged parts of the scene (white).

Model Training

During model training for VOID, we introduce two improvements over prior work: (i) quadmask conditioning, which explicitly identifies regions in each frame that may change after the object is removed, and (ii) a second-pass video appearance refiner that reduces artifacts such as unwanted object morphing. VOID is finally trained on the CogVideoX-Fun-V1.5–5b-InP backbone with Gen-Omnimatte’s checkpoint and fine-tuned for video inpainting with interaction-aware quadmask conditioning.

Evaluations and Results

Experiments across both synthetic and real data demonstrate that VOID preserves consistent scene dynamics far better than prior video object removal methods (please see the paper for full results). VOID successfully maintains object structure and produces plausible motion over time across a wide variety of real-world cases. By contrast, results from both open- and closed-source baselines consistently exhibit physically inaccurate artifacts. For instance, baselines generate water splashes without human impact (see top row of the figure below) or show spinning tops being disrupted without the presence of interacting hands.

To complement our quantitative evaluation, we conducted a user study with 25 creative reviewers to measure the perceptual realism and physical plausibility of our counterfactual edits. Each participant was randomly assigned 5 out of 75 real-world scenarios, resulting in 125 total comparisons. For each video, participants viewed the original input alongside the outputs of VOID and six baselines (seven models total) in a randomized order. Participants were asked to select the video that best reflected how the scene should realistically appear after the object was removed, factoring in visual quality, temporal consistency, blending, the realism of scene evolution, and the absence of artifacts. VOID was selected 64.8% of the time, substantially outperforming all baseline models.

Looking Ahead

Applying AI in ways that serve both member and creator needs is core to our research philosophy, and these projects reflect that approach. While Vera and VOID show promising early results, reaching production-ready quality will require addressing several limitations we encountered. For example, Vera struggles with some complex effects such as lightning or smoke due to the limited training data, and in some cases, it fails to keep background motion fully consistent with the input camera movement. Despite the various generalization capabilities VOID exhibits, we still observe domain gaps. For instance, it cannot handle videos with unusual camera angles or shots captured very close to the target object, and it currently has constraints on supported video length and resolution.

These limitations motivate continued investment in this line of research. Vera and VOID are important early efforts toward making complex video editing more controllable and accessible for artists. For this work, we used publicly available datasets with additional annotation efforts for experiments, and we hope that sharing our research will encourage the broader community to build on these ideas and advance them further.

Toward More Controllable AI Video Editing: An Early Research Exploration at Netflix was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Authors: Lequn Wang, Jiangwei Pan, and Linas Baltrunas

Introduction

The Netflix homepage is the first thing users see when they open the app and the primary way they discover content to enjoy. Almost every part of it is personalized, including which rows appear, which entities show up within those rows, and how everything is arranged on the page.

Constructing that homepage is a genuinely hard problem. It is not simply producing one ranked list. The homepage is a structured, two-dimensional layout, made up of recommendation rows and the entities within them. Here, an entity can be a movie, show, game, live event, or other recommendable item. Each choice can affect the value of the others. Traditionally, it is built through a complex, multi-stage pipeline, with separate components for candidate generation and ranking at both the row and entity levels.

We saw an opportunity to rethink this design. Large language models have shown that a single generative model can perform diverse tasks just by generating a response to a prompt. Inspired by this prompt-response paradigm, we trained a single generative model to build the homepage by directly answering one question:

Given everything we know about this user and this request, what homepage should we generate to maximize user satisfaction?

We call this approach GenPage. It treats the user history and request context as the prompt, and autoregressively generates the entire homepage as the response (Figure 1). Unlike most generative recommenders, such as TIGER, HSTU, and OneRec, which generate flat ranked lists, GenPage generates the rows, entities, and layout together.

This shift is motivated by several goals:

• End-to-end modeling. A single transformer model that constructs the page from raw input signals can replace a complex multi-stage recommender stack. This reduces the number of ML models to maintain, avoids misaligned objectives across stages, and eliminates much of the traditional feature engineering.
• Whole-page optimization via reinforcement learning (RL). Autoregressive page generation makes it possible to optimize for page-level rewards with RL. This can capture interactions across rows and entities, such as diversity or the balance between rows with different stopping power. For example, a Continue Watching row near the top of the page may strongly satisfy a user’s immediate intent, but also reduce how much of the page they browse. Modeling these interactions at the page level lets us align the system more directly with user satisfaction than entity-level objectives alone.
• Better scaling behavior. A generative transformer model gives us a clearer path to improving quality through more data, compute, and model capacity, without repeatedly redesigning the system.
• Flexibility and extensibility. The prompt-response paradigm is flexible by design. By simplifying feature engineering and enabling whole-page optimization, GenPage makes it easier to support new product experiences, such as additional content types like live events, games, and podcasts; layouts beyond the current two-dimensional structure; personalized UI components; and per-entity artwork personalization, all with fewer architectural changes.

Bringing GenPage into production at Netflix also required solving challenges specific to industry-scale recommender systems. Because the homepage is generated in real time, serving latency is a primary engineering constraint. We also need to handle entity cold start in a constantly evolving catalog, keep the model fresh as user interests and cultural trends shift, and enforce complex product and business rules on the generated output.

Despite these challenges, GenPage has already had substantial production impact. In an online A/B test against a mature, highly optimized multi-stage production recommender, GenPage delivered statistically significant gains on the core user engagement metric we use for launch decisions, while reducing end-to-end serving latency by 20%.

Offline, two findings stood out. First, enriching the prompt helped more than scaling model capacity in our current regime. Second, RL post-training increased homepage diversity even though diversity was not part of the objective.

We expect this approach to generalize to many personalization settings. In this post, we focus on Netflix homepage construction as a concrete case study, sharing our design, trade-offs, and lessons learned.

Data

Moving from a traditional recommender to a generative transformer requires us to rethink how the data is represented. Similar to how an LLM turns text into tokens, GenPage represents both the user context and the generated homepage as one sequence of discrete tokens (Figure 2). This sequence includes the full structured homepage layout, with multiple rows and the entities inside them, so the model can generate the page holistically rather than scoring each row or entity in isolation.

Each training example represents a homepage impression and consists of three components:

• Context: user engagement history, profile attributes, and request context.
• Page: the recommended rows and entities shown on the homepage, in layout order.
• Feedback: user interactions with that page, such as play, thumbs-up, or abandonment for entities on the page.

Only the context and page are tokenized as model inputs and outputs. Feedback is used to derive supervision signals via our internal reward system (see the Reward system section).

Instead of using an off-the-shelf text tokenizer, we build a domain-specific tokenizer for the homepage construction data. This is a proven approach in recommender systems and other specialized domains including computer vision, biology, and chemistry, where the raw data is not naturally represented as text. Compared with generic text tokenization, this gives us two key advantages:

• Computational efficiency. Custom tokenization significantly reduces sequence length, lowering inference cost and latency. For example, representing the event “User watched Orange Is the New Black for 50 minutes 30 days ago.” would require 16 tokens with the GPT-5 tokenizer, whereas our scheme compresses it to 4 tokens: [Entity_ID], [Action_Type], [Action_Time_Bucket], and [Action_Duration_Bucket].
• Product control. A direct mapping between tokens and product concepts, such as rows and entities, makes it easier to control what the model can generate. This is crucial for enforcing business rules on the final homepage.

Context tokens

Context tokens encode user engagement history, user profile, and request context.

We represent user history as a sequence of user actions. For each action, we extract key metadata, including the action type, entity ID, timestamp, and duration. These actions include both explicit signals, such as play, add to My List, and thumbs-up, and implicit signals, such as trailer views or visits to a details page.

User profile tokens capture attributes such as language and profile type. Request context tokens encode signals like time of day, day of week, and device.

Some data sources are too long to include directly as raw token sequences. A user’s full impression history, for example, would be prohibitively expensive to represent in full. In these cases, we use a summarized version. This is a pragmatic trade-off: while GenPage aims to operate on raw inputs as much as possible, handcrafted summaries still introduce a form of prompt engineering into the pipeline. Learning to compress these long data sources end to end is an important direction for future work.

To help the model distinguish between data sources, we insert special tokens that mark the start of each segment. Continuous signals, such as timestamps and durations, are bucketized into discrete ranges to keep the vocabulary finite.

Page tokens

Each entity, such as a show, movie, or game, and each row, such as Korean TV Shows, is represented as a single token. The homepage is serialized in layout order: left to right, then top to bottom. We update the entity and row vocabulary daily to incorporate newly added entities and rows. Entities that are still out of vocabulary at serving time are handled through semantic embedding fusion and fallback tokens, both described later.

In principle, the same paradigm can extend to any output that can be expressed as a linear token sequence. This includes layouts beyond the current two-dimensional structure, such as one-dimensional feeds or mixed layouts, as well as personalized UI components, such as the display size of each row, and per-entity outputs such as personalized artwork. We leave these extensions to future work.

Paginated recommendation

To make recommendations responsive to in-session user preferences, the homepage is often generated incrementally, a few rows at a time. Before each pagination request, we append the page tokens from previously generated rows to the prompt, along with the user’s latest engagements on those rows from Netflix’s real-time event-logging infrastructure. This allows the model to generate the next set of recommendations using both the user’s long-term preferences and their most recent in-session behavior.

Reward system

To quantify the long-term value of a recommendation, we rely on an internal reward system described in prior work. The reward system is tuned through online A/B testing to align with long-term user satisfaction and serves as the primary supervision signal for both supervised and reinforcement learning.

The reward system processes user feedback and assigns a scalar reward for every impressed entity on the homepage. For instance, a TV show binge-watched in one night reflects stronger user satisfaction and receives a higher reward than a movie watched for only 10 minutes. An impressed entity that the user abandons receives a negative reward.

We define the page-level reward as the sum of rewards across all impressed entities on the homepage.

Model architecture

GenPage uses a standard decoder-only transformer architecture, the same general architecture behind many modern LLMs. This choice keeps the model simple and flexible, while also letting us benefit from the broad ecosystem of tooling around transformer training and serving.

One architectural detail is that we untie the input embedding and output projection weights. This is useful because pretraining and post-training place different demands on the logits. Next-token prediction pretraining optimizes a softmax over the vocabulary, while weighted binary classification (WBC) post-training optimizes per-token sigmoid scores, as described below. Untying the weights gives the model more flexibility to adapt to both objectives.

Training recipe

Our training pipeline mirrors the LLM recipe: we first teach the model the “language” of the Netflix homepage through pretraining, then align its outputs with user satisfaction through post-training. For post-training, we explore two alternative approaches: weighted binary classification (WBC) and reinforcement learning (RL).

WBC is simpler to optimize and aligns directly with the entity-level objectives of our production ranking models. RL is harder to evaluate and optimize, but it is the key path to GenPage’s full vision of page-level optimization, with the flexibility to incorporate test-time reasoning and multi-token entity representations.

Pretraining via next-token prediction

We pretrain the model with a standard next-token prediction objective: given the context tokens and a prefix of page tokens, the model learns to predict the next page token. This stage focuses on representation learning, teaching the model the relationship between user contexts and successful homepages. Note that our context-page training examples resemble the prompt-response pairs used in LLM supervised fine-tuning (SFT) more than the raw text used in LLM pretraining. We nonetheless call this stage pretraining because we train the model from scratch rather than fine-tuning from an existing checkpoint.

Unlike LLMs, which often face a scarcity of high-quality labeled data, recommender systems have an abundance of user feedback. For pretraining, we use homepage impressions that received positive feedback when served in production, bootstrapping the model to generate pages similar to those produced by the existing production system.

However, pretraining mainly teaches GenPage to imitate the production system. It does not directly optimize the magnitude of the reward, and as GenPage becomes part of production, repeatedly training on pages generated by earlier versions of the model can risk model degeneration. To address these limitations, we explore two post-training approaches.

Post-training via weighted binary classification

One effective way to align the generative model with user satisfaction is weighted binary classification (WBC). At a high level, WBC turns generation into token-level value prediction: given the user context and the tokens generated so far, the model learns to estimate the value of generating each possible next row or entity token.

This objective is easier to optimize than page-level RL. By decomposing the homepage into per-token targets, WBC provides token-level credit assignment by construction, rather than requiring RL to infer how each generated decision contributed to the final page-level reward.

This training setup is enabled by our custom tokenization. Each page token corresponds directly to a specific entity or row, making it straightforward to assign a reward. For every impressed entity on the page, our reward system provides a scalar reward based on user feedback. For each impressed row, we derive a row-level reward by aggregating the rewards of the entities in that row.

From each reward, we derive a binary label from its sign, such as positive engagement versus abandonment, and a weight from its magnitude, such as binge-watching receiving a higher weight than a short play. We then optimize a weighted binary cross-entropy loss on the logit for the corresponding token. Under this setup, the logit for a token can be interpreted as the model’s value estimate for generating that token at that position.

Although the model is trained as a value predictor, it can still generate pages autoregressively. At each step, the model scores the candidate next tokens, greedily selects the token with the highest value, and appends it to the prefix. This process repeats token by token until the full homepage is generated.

Post-training via reinforcement learning

Our second post-training approach is reinforcement learning (RL). WBC is effective for optimizing entity-level metrics, but it does not directly optimize the homepage as a whole. RL treats page generation as a sequential decision-making problem, allowing the model to optimize a page-level reward while preserving the flexibility of autoregressive generation.

This opens the door to several important capabilities:

• Whole-page optimization. RL directly optimizes an aggregate page-level reward, allowing the model to account for interactions across rows and entities, such as diversity, stopping power, and page-level business constraints.
• Test-time reasoning. Analogous to its application in LLMs, RL can optimize reasoning capabilities for generative recommendation. Reasoning outputs can also be viewed as a form of automated feature engineering.
• Multi-token entity support. In our current tokenization, each entity and row is represented as a single token, so rewards map cleanly to individual tokens. In more complex settings, however, an entity may require multiple tokens, such as [Show_ID] plus [Episode_#] for an episode, or a sequence of semantic ID tokens. In that case, WBC’s per-token labeling becomes ambiguous because a single entity-level reward must be distributed across multiple tokens. RL avoids this issue by optimizing the sequence-level return, making it a more natural fit for variable-length, multi-token entities.

Inspired by the RLHF recipe used to align large language models, we adopt a two-step approach. First, we train a reward model that predicts the page-level reward for a generated page. This reward model is distinct from the reward system described earlier. The reward system converts observed user feedback into a scalar reward for a page that was actually shown, whereas the reward model predicts the page-level reward for a generated page without showing it to the user. This prediction is what lets RL optimize against arbitrary candidate pages during training.

Training against a reward model avoids the high variance of off-policy correction on logged or predicted propensities, but introduces the risk of reward hacking. Since the reward model is trained on data generated from the production policy, it is most reliable on pages similar to those the production policy generates. We therefore use a KL penalty to keep the policy close to the pretrained checkpoint, which itself was trained to mimic the production policy. This keeps the pages within the reward model’s region of coverage and limits opportunities for reward hacking.

For the RL algorithm, we adopt Dr. GRPO, a variant of GRPO that mitigates biases in the training objective. To train the model within this framework, we need the following components:

• Prompts: production user requests, represented by context tokens.
• Policy and reference models: both are initialized from the pretrained checkpoint; the reference model anchors the KL penalty discussed above.
• Reward model: a dedicated transformer-based reward model, also initialized from the pretrained checkpoint, predicts the page-level outcome reward, using the sum of entity-level rewards from our internal reward system as the supervision target. We also incorporate rule-based format rewards to guide the RL policy. For example, the page should resemble a list of rows, and business-critical rows or entities should not appear too low on the page.

Addressing production challenges

Cold start

New entities lack the rich interaction data needed to learn robust token embeddings. We address this through two complementary strategies:

• Context injection. We inject metadata about new or time-sensitive entities (e.g., Live Now events) directly into the context tokens, providing the model with semantic and time-sensitive information.
• Semantic embedding fusion. Rather than relying solely on entity ID embeddings learned from user interaction data, we represent each entity as a fusion of its ID embedding and a content-based embedding derived from semantic information such as synopses, cast, transcripts, genres, and video content. This fused embedding serves as the input embedding for the entity’s token in the transformer. During training, with small probability, we randomly replace an entity ID token with the generic fallback token (described below), so the model learns to make recommendations from the content-based embedding alone. This ensures that a new entity has a meaningful representation in the same latent space as established entities as soon as its content metadata is available — even before it has any interaction data.

Multi-cadence incremental training

At Netflix scale, daily retraining of a large transformer from scratch is prohibitively expensive, but recommendation models must remain fresh to capture shifting trends and new catalog additions. We address this with a multi-cadence incremental training strategy (Figure 3).

Our training pipeline operates on a cyclic schedule with two distinct rhythms. At a tunable cadence, we conduct a large-scale pretraining and post-training pass on data from a broad historical window. Between these passes, each day we perform an incremental update by continuing post-training from the previous day’s checkpoint, using a mix of the latest day’s data and a sampled subset of past data. This helps the model stay current with new trends and catalog changes while preventing overfitting and catastrophic forgetting.

To manage the daily influx of new tokens (e.g., new entities, rows), we employ fallback tokens. New tokens are initialized using fallback tokens of their type (e.g., [Row_Fallback_Token] for new rows, [Entity_Fallback_Token] for new entities). During training, we randomly replace a small percentage of known tokens with fallback tokens, teaching the model to handle unknown tokens gracefully.

Enforcing business rules

A Netflix homepage must satisfy structural constraints (e.g., organized as a list of rows) as well as product logic such as deduplication, row pinning, and category consistency (e.g., entities in a Comedy row must be comedies). While training signals can encourage rule adherence, they cannot guarantee strict compliance.

We enforce these rules at inference time through constrained decoding. At each autoregressive generation step, we compute a mask of eligible tokens based on the applicable business rules and apply it to the output logits, allowing only rule-compliant tokens to be generated. This is greatly simplified by our custom tokenization: because each entity and row is a single token, business rules map directly to token-level masks, avoiding the multi-token bookkeeping that constrained decoding requires over a text vocabulary. For example, to pin a specific row (e.g., popular games) at a fixed position (e.g., row position 2), we simply mask out all other tokens at that position.

Hybrid row decoding

Autoregressive generation ensures that each newly generated token is conditioned on the full preceding context, but generating every entity token one at a time can be expensive. We leverage the structure of the homepage to balance inference efficiency with the amount of contextual information available to each generated token.

Within each row, the first few entities are especially important: they receive the most user attention and strongly shape the row’s perceived quality and theme. To reduce inference latency, we use a hybrid row decoding strategy. The model autoregressively generates only the first few entities in each row. Conditioned on this generated prefix, we obtain logits for all eligible entities in a single forward pass and select the top-scoring remaining entities, subject to the same inference-time business-rule constraints described above.

This approach preserves autoregressive conditioning where it matters most while avoiding the latency and cost of decoding long rows token by token.

Offline experiments

We ran a series of ablations on Netflix internal data to understand how different components of GenPage affect model quality. Because the system was developed iteratively, individual ablations span different training configurations and data snapshots, so we report only relative comparisons within each study. Unless otherwise noted, experiments use ~200M-parameter models and report results on a held-out evaluation set.

Does pretraining help?

We compare WBC post-training with and without a preceding next-token-prediction pretraining stage. Figure 4 shows that pretraining yields substantial improvements across all metrics.

The gains may look small in absolute terms, but they are large in our production regime: setting aside the sample weighting, an Entity AUC lift from 0.91 to 0.92 means that for a randomly drawn positive-negative pair of impressed entities, the model’s misranking rate drops from 9% to 8% — a magnitude of improvement we rarely observe from a single change on a mature production system. Pretraining the model on the “language” of the Netflix homepage provides a strong initialization for post-training, mirroring the pretrain-then-post-train recipe behind modern LLMs.

How does performance scale with model size?

We sweep model size from ~120M to ~900M parameters (Figure 5) and report the next-token-prediction loss from pretraining and the WBC loss from post-training. Both losses decrease in a power-law-like fashion, mirroring the scaling trends seen in LLMs. This confirms that the generative approach scales favorably with model size, suggesting that recommendation quality can be further improved by scaling capacity.

How does performance scale with information in the user context?

Over the course of development, we progressively enriched the prompt, both by adding new data sources to the context and by refining how each source is tokenized. With model size held fixed, the WBC post-training loss decreases substantially as the context is enriched (Figure 6).

The model-size sweep and the context-enrichment sweep span different axes and are not strictly comparable: the model-size study covers roughly an order of magnitude in parameters, while the context study spans the full trajectory of our prompt design. Even so, the gap between the two is striking. Scaling the model from 120M to 900M parameters reduces WBC loss by roughly 1.3%, whereas the cumulative effect of enriching the context is around 6.9%. In several cases, a single well-designed context addition delivers a larger improvement than the entire ~7.5× model-capacity scaling.

This suggests that, in our regime, enriching the prompt — both what we put in the context and how we tokenize it — yields a substantially larger improvement than scaling model capacity. Personalization quality appears to be bottlenecked first by the information and representation available to the model, and only then by capacity. We expect context enrichment to dominate until the context is saturated, at which point model capacity becomes the primary driver.

Does RL post-training optimize at the page level?

In offline evaluations (Figure 7), RL post-training consistently improves the page-level reward over the pretrained checkpoint, but this is largely confirmatory: the reward is computed using the same model the policy is optimizing against. More interestingly, although diversity is not part of the RL objective, homepage diversity — measured via pairwise embedding distance among entities on the page — also increases over the course of training. This suggests that the RL-trained policy is optimizing the page as a whole rather than myopically optimizing each token in isolation.

Online evaluation

We conducted an online A/B test against the current production homepage recommender using GenPage. In this test, GenPage decoded over the existing production row and entity candidate sets, which help handle many business rules (such as eligibility).

Figure 8 shows the result: all variants delivered statistically significant improvements on the core user engagement metric we use for launch decisions (p < 0.001) against a mature, highly optimized multi-stage production baseline. The variants differed in their training-data configurations; that they all delivered comparable lifts suggests the gain is robust to these design choices rather than dependent on a particular configuration.

Alongside the engagement wins, we observed unintended shifts in the distribution of impressed entity categories (e.g., new vs. established titles, TV shows vs. movies). These shifts are not necessarily negative, but they are not something we explicitly optimized for, and they warrant deeper investigation. We suspect these shifts reflect GenPage personalizing more precisely than the production stack — consistent with an increase in homepage impression efficiency, i.e., users engaging with what they saw using fewer impressions. This sharper personalization appears to surface production-inherited components (such as the reward system) that aren’t yet aligned with the new generative paradigm. We plan to characterize the drivers of these shifts and, where appropriate, tune these components so the resulting distributions better align with desired product behavior.

We also observed strong responsiveness to in-session signals: the latest in-session actions quickly influenced subsequent recommendations and faded back to long-term preferences after a day or two, confirming that the model effectively attends to action timestamps. This responsiveness emerges naturally from the generative formulation, without the extensive manual feature engineering used in our production stack.

Contrary to the common assumption that generative models are slower, GenPage reduced end-to-end serving latency by 20% relative to the baseline. By replacing multiple ranking stages and heavy feature computation with a single transformer operating on raw tokenized inputs, we eliminated substantial serving complexity and computational overhead. Custom tokenization and hybrid row decoding further reduced the number of decoding steps, and thus latency. The 20% reduction was achieved without exhausting the available optimizations; further reductions are possible, and this headroom can be reinvested in capacity or richer prompts.

Conclusion

We presented GenPage, an early step toward end-to-end generative Netflix homepage construction: representing user context as a tokenized prompt and generating the entire homepage autoregressively in real time. This collapses the traditional multi-stage recommender stack into a single transformer that can be optimized end-to-end.

In online A/B tests against a mature, highly optimized multi-stage production system, GenPage delivered statistically significant gains on the core user engagement metric we use for launch decisions, while reducing end-to-end serving latency by 20%. Achieving this required adapting the LLM training recipe — pretraining followed by WBC or RL post-training — together with a set of domain-specific techniques: custom tokenization for serving efficiency and product control, context injection and semantic embedding fusion for entity cold start, multi-cadence incremental training for model freshness, constrained decoding for business-rule enforcement, and hybrid row decoding for inference efficiency.

Two offline findings stand out. First, in our current regime, enriching the prompt yields a substantially larger improvement than scaling model capacity — a takeaway we expect to generalize to other industry-scale personalization settings, at least until the available context is fully exploited. Second, RL post-training increases homepage diversity even though diversity is not part of the objective — an indication that page-level optimization captures interactions across rows and entities.

Several pieces of the full vision are still in progress: long context still relies on handcrafted summarization, and broader LLM-style capabilities — language, multimodality, and reasoning — have not yet been incorporated. One promising direction here is a hybrid tokenization combining our domain-specific tokens with generic text tokens, retaining structured control while inheriting the strengths of general-purpose LLMs; conceptually, this introduces an additional recommendation modality into an LLM.

More broadly, we expect many advances from the LLM ecosystem to transfer naturally to this setting, and the boundary between an LLM and a recommender system may increasingly blur. Our results suggest this is a viable path toward simpler, more flexible recommender systems that align more directly with user satisfaction and can more readily support new product experiences.

Acknowledgments

Contributors to this work (in alphabetical order): Abhishek Agrawal, Ashish Rastogi, Baolin Li, Casey Stella, Dan Zheng, Daneo Zhang, Ding Tong, Donnie DeBoer, Fengdi Che, Fernando Amat Gil, Grace Huang, Inbar Naor, Ishita Verma, Jason Uh, Jimmy Patel, Justin Basilico, Lanxi Huang, Lingyi Liu, Liping Peng, Louis Wang, Michelle Kislak, Nathan Kallus, Nicolas Hortiguera, Paran Jain, Qusai Al-Rabadi, Rein Houthooft, Ryan Lee, Santino Ramos, Scarlet Chen, Shaojing Li, Sheallika Singh, Si Cheng, Wei Wang, Yesu Feng, and ZQ Zhang.

GenPage: Towards End-to-End Generative Homepage Construction at Netflix was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

By Parth Jain, Rakesh Sukumar, Yingwu Zhao, Renzo Sanchez-Silva & Nathan Fisher
A deep dive into the engineering challenges of building a real-time service dependency map at Netflix scale: from streaming architectures and distributed aggregation pipelines to time-travel queries and the methodology that made it work.

Introduction

In our first post, we introduced the problem: engineers at Netflix needed a unified, real-time view of service dependencies to troubleshoot faster, understand blast radius, and navigate our distributed architecture. We described our multi-source approach, combining eBPF network flows, IPC metrics, and distributed tracing into physically separate graph layers that can be queried independently or merged into a comprehensive view.

That post explained what we built and why. This post is about how, the engineering reality of building this system at Netflix scale.

Here’s the truth: the first version worked perfectly… in our local environment. Production was a different story. Kafka consumers fell behind. Instances ran out of memory. Some nodes received 100x the traffic of others. Garbage collection pauses consumed more CPU than actual business logic.

What you’ll learn in this post isn’t a success story, it’s a learning journey. We’ll walk through the architecture decisions that enabled scale, the production challenges that tested those decisions, the optimization methodology that guided us through, and the lessons that apply to any distributed system. Along the way, we’ll share the innovations that made it possible to process millions of flow records per second, reconstruct topology at any point in time, and provide sub-second query responses, all while maintaining near real-time freshness.

Architecture Deep-Dive: Building for Streaming and Scale

Streaming-First: Why Real-Time Matters

Traditional service topology systems use batch processing, aggregating data hourly or daily, then storing complete snapshots. This approach works at a modest scale but has a fundamental problem: by the time you see the data, it’s already old. During a production incident at 3am, an hour-old dependency map is archaeology, not observability.

Our key architectural decision was to build streaming-first. Instead of batch jobs that process historical data, we continuously ingest flow records from multi-region Kafka streams and IPC metrics as Server-Sent Events, process them through reactive pipelines with backpressure handling, and provide near real-time topology updates, typically within tens of minutes, compared to the hours-old or day-old data that batch processing approaches provide.

This wasn’t just about freshness, it was essential for our use cases. Live events can’t wait for the next hourly batch. Incident response needs current data. Change validation requires seeing immediate impact. The architecture had to support continuous updates while handling massive scale without falling behind.

How Backpressure Enables Real-Time Processing
The streaming approach created new challenges, but also required solving a fundamental problem: how do you process millions of flow records per second in real-time without losing data when downstream systems slow down?

Traditional approaches fall short at our scale:

• Unbounded queues: Simple but dangerous. Keep buffering until you run out of memory, then the instance crashes.
• Drop-based flow control: Discard data when buffers fill. Fast, but now your topology is incomplete, you’ve lost connection information.
• Batch processing: Process everything, but hours later. By then, the incident is over (or worse, still happening with stale data).

We needed something different: the ability to slow down gracefully under load without losing data. This is where reactive streams with backpressure became essential.

Here’s how it works: when Stage 3 can’t write to the graph database fast enough, it signals Stage 2 to slow down. Stage 2 signals Stage 1. Stage 1 signals the Kafka consumer to pause. The data waits in Kafka until downstream capacity returns.

Backpressure propagates naturally through the entire system. When any stage becomes overwhelmed from traffic spikes, GC pauses, or external slowdowns, the pipeline automatically slows to a sustainable rate. No data is lost in most cases, no instances crash, the system degrades gracefully.

This is what enables “real-time” at our scale. During normal operation, we process with minimal latency. During load spikes or temporary slowdowns, we slow down rather than fall over. The data still gets processed, just a few seconds or minutes later instead of immediately. For topology updates, this trade-off is acceptable: slightly delayed real-time updates are vastly better than hour-old batch data or incomplete topology from dropped records.

The cost of this approach is complexity. Reactive streams are harder to reason about compared to traditional synchronous blocking models (we’ll discuss this more in the challenges section). But at Netflix scale, backpressure isn’t optional, it’s the mechanism that keeps the system running reliably under production load.

Multi-Layer Architecture: Physical Separation for Independent Optimization

As we covered in our first post, our multi-source approach uses three physically separate topology layers with different storage optimized for each:

• Network Layer: eBPF flow logs in graph database partition, comprehensive coverage but lacks application context
• IPC Layer: Application metrics in a different graph database isolated from the one for Network Layer, rich endpoint details but only instrumented services
• Tracing Layer: Distributed traces in columnar storage (Parquet), actual request paths but sampled.(We cover the tracing layer and its integration in our next post).

Physical storage isolation enables independent optimization, each layer has different throughput, query patterns, and evolution timelines. At query time, we execute parallel queries across relevant storage systems and merge results, providing unified views with sub-second latency while maintaining flexibility to evolve each layer independently.

The Three-Stage Distributed Aggregation Pipeline

The heart of the network layer ingestion is a three-stage distributed pipeline. This architecture solves a fundamental challenge with network flow logs: they only show individual network hops, not the true application-level connections we need to build a useful topology.

The Core Problem: Network Intermediaries

In cloud environments, traffic between applications rarely flows directly, it traverses intermediate network components like load balancers, NAT gateways, API gateways, and proxies. Network flow logs show individual hops: App A → Load Balancer and Load Balancer → App B appear as separate flows. But what engineers need is the logical dependency: App A → App B. Without resolving these intermediaries, our topology would be cluttered with infrastructure components rather than showing the service-to-service relationships that matter for troubleshooting.

The three-stage pipeline solves this:

Stage 1: Initial Aggregation (FlowLog Ingestion Service)

Multi-Region Kafka (4 regions)
  → Filter invalid flow logs
  → 5-minute time-window batching
  → Create initial aggregators per window
  → Distribute via consistent hashing
  → Stream to Stage 2 via SSE

Stage 1 consumes flow logs from multi-region Kafka, filters invalid records, batches them into 5-minute time windows, and creates initial aggregator objects. At this stage, we’re still working with raw network hops, identifying which flows involve intermediaries but not yet resolving them. Aggregators stream to Stage 2 for resolution.

Stage 2: Network Intermediary Resolution Layer (Intermediate GraphEntity Ingestion Service)

Stage 1 Aggregators (via SSE streams)
  → Group flows by intermediary (load balancer, NAT gateway, proxy, etc.)
  → Identify pairs: (Source → Intermediary) + (Intermediary → Destination)
  → Resolve to direct edges: Source → Destination
  → Track which intermediaries were traversed
  → Aggregate metrics across both hops
  → Re-distribute via consistent hashing
  → Stream to Stage 3 via SSE

This is the key step. Stage 2 performs graph resolution:

1. Collect flows by intermediary: Group aggregators where an intermediary is either source or destination, creating maps of flows going TO intermediaries (Source → Intermediary) and FROM intermediaries (Intermediary → Destination)
2. Resolve direct edges: For each intermediary, join its incoming and outgoing flows to create direct application edges (App A → App B), combining metrics from both hops
3. Result: Clean application-level topology showing App A → App B instead of App A → Load Balancer → App B

This resolution happens at aggregation time, not query time, with resolved edges flowing to Stage 3.

Why can’t we do this in a single stage? The fundamental issue is data locality. To resolve App A → Load Balancer → App B into App A → App B, we need both flows on the same instance to perform the join. But in Stage 1, flows are scattered across instances based on Kafka’s partitioning. Stage 2’s critical function is to redistribute aggregators by intermediary identifier, all flows involving “Load Balancer X” route to the same instance for resolution. This is the classic map-reduce pattern: Stage 1 maps, Stage 2 shuffles and reduces by intermediary, Stage 3 performs final aggregation.

Stage 3: Final Aggregation and Enrichment (GraphEntity Ingestion Service)

Stage 2 Aggregators (via SSE streams)Flow
  → Final aggregation across time windows
  → Enrich with external data (query key-value stores)
  → Convert to graph entities
  → Persist to graph database (throttled writes)

Stage 3 performs final aggregation of resolved edges, enriches graph nodes with external data sources (application health, ownership, metadata), converts aggregators to concrete graph entities (nodes and edges with all properties populated), and persists them to the distributed graph database with controlled throttling to respect storage system limits.

Why Three Stages, Not Two?

We initially used two stages: aggregate in Stage 1, resolve and persist in Stage 2. This worked in testing but failed at production scale, Stage 2 became overwhelmed by data concentration.

The problem: intermediary resolution requires collecting ALL flows involving an intermediary on the same instance. As a result, the instances handling flow logs for popular applications and their intermediaries became ‘hot nodes’ due to significant data concentration. Compounding this, data enrichment (querying external stores for health and metadata) meant the busiest instances were also doing the most I/O.

The solution: split responsibilities into three stages. Stage 2 focuses purely on resolution and redistributes. Stage 3 handles enrichment and persistence. Rather than routing all flows for a hot key to one owner, we redistribute in stages. Each flow is distributed, resolved, distributed again, and then persisted, which spreads the work across multiple instances and isolates compute-heavy resolution from I/O-heavy enrichment. Even when intermediaries see 100x typical traffic, no single instance becomes a bottleneck.

Why Server-Sent Events Instead of gRPC or Message Queues?

We initially used gRPC but it became a performance bottleneck, serialization overhead, connection pool management, and memory pressure for streaming responses consumed more CPU than business logic. Message queues added infrastructure complexity without benefit for our use case.

SSE proved ideal: lightweight HTTP-based protocol with minimal serialization, natural backpressure integration with reactive streams, and simpler connection model. The lesson: industry best practices like “use gRPC for service communication” don’t apply universally. For streaming large volumes of pre-aggregated data, lighter-weight alternatives may be more appropriate. Measure, don’t assume.

Why IPC Doesn’t Need Three Stages

The IPC layer uses single-stage aggregation because: (1) IPC metrics are already at application level, no intermediaries to resolve, and (2) data is partitioned correctly from the start — each node receives all IPC metrics for its assigned applications via consistent hashing, eliminating the need for redistribution. This highlights a key principle: data partitioning strategy determines processing architecture. When data arrives with the right partitioning, you can aggregate directly; when it doesn’t (like network flows requiring intermediary resolution), you need shuffle/redistribution stages.

Dynamic Load Distribution: How Hashing Works with Auto-Scaling

How do we decide which instance receives which aggregator when our Auto Scaling Groups dynamically add or remove instances? Traditional approaches assume static clusters requiring explicit rebalancing, coordination services, or manual data movement when cluster size changes.

Our Approach: Dynamic Consistent Hashing

We use consistent hashing with dynamic instance discovery from our service registry. Each instance queries the registry to get the current list of healthy ASG instances, maintains them in sorted order (ensuring all instances have the same view), and uses this list for the hash function findOwnerInstance(aggregator.primaryKey). When ASG scales up or down, the hash function naturally redistributes aggregators based on the updated instance list, no explicit coordination needed.

The key insight: leverage existing infrastructure. Our service registry already tracks ASG membership for health checking. Using it as our source of truth gives us dynamic cluster membership for free. Consistent hashing provides stable partitioning (most aggregators stay on the same instance during membership changes), while the sorted list ensures consistency.

The Result

Load follows infrastructure automatically. During traffic spikes or live events, new instances immediately receive their share. During deployments, aggregators seamlessly shift to healthy instances. This pattern proved crucial for production stability, no manual intervention, no coordination protocol, just automatic rebalancing.

The V1 Journey: Major Challenges at Production Scale

Getting the initial version (V1) to production taught us that scale changes everything. What works in development breaks in production. Every assumption gets tested. And fixing one bottleneck reveals the next.

Challenge 1: Kafka Consumer Lag

The Problem: Our multi-region Kafka consumers started falling behind. Consumer lag grew from seconds to minutes, then hours. Flow logs were arriving faster than we could process them. If this continued, we’d never catch up, and our “real-time” topology would become increasingly stale.

Investigation: We instrumented Kafka consumer metrics heavily. Key findings:

• Kafka had fewer partitions than optimal for our consumer group size
• Each fetch operation retrieved relatively few records
• Network socket buffers weren’t right-sized for our throughput
• Cross-region read latency added overhead

Solutions Applied:

1. Increased Kafka partitions: More partitions enabled more parallel consumers in our consumer group, distributing load across more instances.
2. Tuned fetch parameters: Increased records per fetch operation, reducing the number of network round-trips. This trades off per-message latency (we fetch larger batches) for throughput (more records processed per second).
3. Increased socket receive buffer size: Ensured network buffers never limited fetch operations. At our scale, default buffer sizes were too small.

Results: Throughput improved significantly, and lag reduced to acceptable levels, typically under a minute even during peak traffic.

Lesson: At scale, you can’t optimize in isolation. Fixing Kafka lag revealed the next bottleneck: our instances themselves couldn’t keep up with the higher ingest rate. The pipeline moved faster, which exposed downstream capacity problems.

Challenge 2: Hot Nodes and Data Amplification

The Problem: This was the most severe production issue we faced. Some instances in our Auto Scaling Group were receiving 100x more traffic than others. Memory usage spiked. Garbage collection pauses became frequent and long. More CPU time was spent in GC than in business logic. Eventually, hot instances would go DOWN, triggering cascading failures as their load redistributed to other instances.

Root Cause Investigation:
Flow logs for popular services dominate traffic volume. A service like our authentication layer or recommendation API is called by hundreds of other services, generating orders of magnitude more flow records than typical services.

Our initial architecture used consistent hashing to determine which instance owned aggregation for each destination service. All flow logs for a given destination are routed to the same instance, the “owner” for that destination. This design seemed reasonable: group related data for efficient aggregation.

But popular destinations created hot nodes. One instance might own authentication services, another might own a rarely-used backend service. The load distribution was wildly uneven, some instances handled 100x the flow records of others.

Worse, data amplification occurred during redistribution. Consider a service called by 100 upstream services across 10 instances. All 10 instances receive flow logs for that destination (because they all have local clients calling it). When they route aggregators to the owner instance, that instance receives 10 separate aggregators it must merge. The data volume multiplied during shuffling.

We profiled extensively using async-profiler and heap dump analysis. The results were clear: hot instances spent most of their CPU on garbage collection, trying to manage the rapid allocation and deallocation of aggregator objects as flow logs poured in faster than they could be processed. Memory pressure led to GC thrashing, which consumed CPU, which slowed processing, which increased memory pressure, a vicious cycle.

Solution: The Three-Stage Pipeline’s Dual Benefits
The three-stage pipeline we described earlier, designed primarily for proxy resolution, turned out to be exactly what we needed to solve the hot nodes problem as well. Here’s why:

Stage 1 performs initial aggregation locally before any distribution. Instead of sending every flow log to a remote instance immediately, each instance performs online aggregation of raw flow logs into time-windowed aggregators (over 5-minute periods) directly in memory; this allows the raw flow to be discarded and garbage collected quickly, significantly reducing memory pressure, and ensures only the aggregation results are transferred across the network to downstream stages.

Stage 2 focuses on proxy resolution but also provides intermediate redistribution. Aggregators from Stage 1 distribute via consistent hashing to Stage 2 instances. Now we’re moving compressed aggregators, not individual flow logs. After resolution, Stage 2 redistributes resolved edges again to Stage 3, providing a second hashing operation that further spreads load.

Stage 3 receives resolved aggregators that have been compressed twice and distributed twice. Even for extremely popular services, load has been spread across enough distribution points that no single instance becomes overwhelmed.

The key insight: architectural decisions driven by one requirement (proxy resolution) often solve other problems (load distribution) as beneficial side effects. The three-stage pipeline with graduated redistribution achieves both goals, it resolves proxies to show clean application-level topology AND prevents hot nodes by spreading load across multiple distribution points.

Switching from gRPC to SSE
As described earlier, this challenge also revealed that gRPC wasn’t the right protocol for inter-stage communication at our scale. We replaced gRPC with Server-Sent Events, dramatically reducing resource consumption on both sender and receiver sides.

Results:

• CPU usage became evenly distributed across instances, no more hot nodes with 10x the load of others
• Network bandwidth usage dropped significantly due to better aggregation and lighter-weight protocol
• Memory pressure decreased as we reduced the object allocation rate
• The system scaled gracefully with Auto Scaling Group changes

Lesson: Technology choices must match your specific use case. gRPC is excellent for request-response RPC patterns. For streaming large volumes of aggregated data in a pipeline, lighter-weight alternatives can be more appropriate. Let measurements guide the decision, not industry hype or existing team expertise.

Challenge 3: Memory and Garbage Collection

The Problem: Even after fixing hot nodes, we still saw high heap usage, frequent garbage collection pauses, and instances occasionally going DOWN. GC logs showed pauses consuming significant CPU time, in some cases, more than our business logic.

Root Cause: Multiple factors contributed: objects accumulating in heap while waiting for 5-minute aggregation windows to complete, unnecessary conversions between different object types as data flowed through stages, and immutability overhead, following Scala best practices, we used immutable data structures for aggregators, but every update created new objects, overwhelming the garbage collector at millions of records per second.

Investigation: Heap dumps and GC logs revealed flow log objects retained beyond their useful lifetime, unnecessary intermediate conversion objects, and constant creation/disposal of immutable aggregator versions. Minor GCs occurred every few seconds, major GCs took hundreds of milliseconds, the JVM spent more time on garbage collection than business logic.

Solutions Applied:

1. Faster processing: Process flow logs immediately, aggregate quickly, release references. Optimized Pekko stream stages to minimize object lifetime.
2. Eliminate unnecessary conversions: Route aggregators directly between stages instead of converting to intermediate types.
3. Mutable structures on hotpath: This was controversial, Scala best practices emphasize immutability. But at our scale, immutability created too many objects. We pragmatically chose mutable aggregators on the hotpath (immutability elsewhere), prioritizing performance over convention. Switching to mutable aggregators reduced heap allocation by over 50% and cut GC pause time significantly, though it required more careful code review.
4. Tuned time windows: Balanced data freshness against memory pressure.

Results:

• Heap usage decreased substantially
• GC pauses reduced to acceptable levels (tens of milliseconds instead of hundreds)
• CPU freed up for business logic instead of garbage collection
• Instance stability improved, no more instances going DOWN due to memory issues

Lesson: “Best practices” are starting points, not absolute rules. At unique scale, you may need to diverge from conventions. But do it deliberately, with measurement justifying the decision, and with awareness of the trade-offs. Don’t abandon immutability everywhere, just where performance data proves it’s necessary.

Challenge 4: Reactive Streams Complexity

The Problem: Our Pekko Streams pipelines would stall unexpectedly. Backpressure propagation didn’t work as expected. We struggled to debug why certain streams would stop processing without obvious errors. The reactive programming mental model, with its emphasis on async boundaries, backpressure, and demand-driven processing, proved harder to master than anticipated.

What We Learned:
Reactive streams with backpressure are powerful tools for building systems that handle load spikes gracefully. When downstream consumers slow down (due to temporary load, GC pauses, or external system slowdowns), backpressure allows upstream producers to slow down rather than overflow buffers or drop data.

But this power comes with complexity:

• Non-intuitive behavior: Traditional imperative code flows top-to-bottom. Reactive streams are demand-driven, downstream consumers pull from upstream producers. This inversion of control isn’t intuitive.
• Async boundaries: The .async operator in Pekko Streams creates a boundary where processing moves to a different thread. This can improve parallelism but also introduces complexity around buffer sizing, demand signaling, and error propagation. We initially misunderstood when to use .async and ended up with over-parallelized streams that created more overhead than benefit.
• Debugging difficulty: When a stream stalls, there’s no stack trace pointing to the problem. You must understand the internal mechanics, demand signals, buffer states, materializer state to diagnose issues.

Our Approach:

1. Deep learning investment: We invested significant time in understanding reactive streams concepts deeply. Reading documentation, experimenting with small examples, and building team expertise.
2. Simplified patterns: Where possible, we simplified our stream graphs. Complex branching and merging patterns are powerful but hard to debug. We preferred linear flows with clear stage boundaries.
3. Better monitoring: We added metrics at stream boundaries, tracking buffer sizes, element throughput, backpressure events. Visibility into stream internals helped diagnose issues.
4. Team education: We documented our learnings, shared patterns that worked, and built institutional knowledge about reactive streams.

Lesson: Powerful abstractions require investment. Don’t assume you understand a framework without validation. Build your mental model deliberately, test it with experiments, and be humble about your understanding. Reactive streams are worth mastering for systems that need to handle load gracefully, but expect a learning curve.

V2 Evolution: Continuous Refinement

V1 got us to production. The major architectural challenges like Kafka lag, hot nodes, memory pressure, were solved. But production at full scale revealed new optimization opportunities. V2 represents the continuous refinement that turns a working system into a production-ready system.

Challenge 5: Persistent Heap Pressure

The Problem: Despite V1 optimizations, we still observed higher-than-desired heap usage. GC metrics improved but weren’t optimal. Memory profiling showed room for improvement.

Root Cause: Deeper analysis revealed we were still doing unnecessary object conversions between stages. We’d convert aggregators to full graph entities (with all properties populated) before routing to the next stage, even though the next stage just needed the compressed aggregator state.

Solution: Architectural change to route aggregators directly through all stages, only converting to final graph entities at Stage 3 immediately before persistence. This eliminated two intermediate conversion steps and the associated object allocation.

Result: Heap usage dropped further, GC pauses became even less frequent, and memory headroom improved.

Challenge 6: Serialization Complexity

The Problem: Custom serialization logic for SSE messages caused occasional erratic errors that were hard to reproduce and debug. Different parts of the codebase used inconsistent serialization approaches.

Solution: Standardized on JSON encoding throughout the pipeline. While slightly less efficient than binary serialization, JSON’s human readability made debugging far easier, and the overhead was negligible compared to other operations. Consistency eliminated an entire class of bugs.

Result: Serialization-related errors disappeared. Debugging became easier because we could read SSE message contents directly.

Challenge 7: Stream Processing Inefficiencies

The Problem: Even after understanding reactive streams better, our Pekko configurations weren’t optimal. We had over-parallelized some stages and under-parallelized others. The .async boundaries weren’t placed optimally.

Solution: Through continued profiling and experimentation, we tuned parallelism parameters, adjusted buffer sizes, and refined async boundary placement. We added monitoring at stream boundaries to identify bottlenecks.

Result: Throughput improvements and more consistent processing latency.

Challenge 8: Uneven Graph Database Throughput

The Problem: Write distribution to our graph database wasn’t even. Some partitions received heavy write traffic while others sat idle. This caused throttling to kick in unevenly and limited overall write throughput.

Solution: Implemented batching of aggregators before writing to the graph database and improved distribution logic across partitions. Rather than writing each aggregator immediately, we batch them and write multiple entities in coordinated operations.

Result: More consistent write throughput and better utilization of database capacity.

Challenge 9: Data Enrichment at Aggregation Time

Beyond the core topology graph, we needed to enrich nodes with additional context. At Stage 3, before persisting graph entities, we integrate enrichment data from external sources, application health status, ownership information, and other metadata. Performing this enrichment at aggregation time rather than at query time avoids the performance overhead of post-query joins and ensures every topology node has full context when queried.

Pattern Recognition

Each V2 challenge followed the same pattern: production revealed an assumption, profiling identified the root cause, targeted fixes improved specific metrics. Measure, hypothesize, validate, iterate. This is how you build at scale, not by getting everything right upfront, but by continuous learning and improvement.

Time Travel: Continuous Topology Reconstruction

One of the most powerful capabilities we built enables querying historical topology: “What did the call graph look like when this incident happened?” This time-travel feature required solving an interesting architectural challenge, how to efficiently store and reconstruct topology across time.

The Problem

Engineers need to answer temporal questions: What did the topology look like during an incident? How have dependencies evolved? Traditional approaches, full snapshots or event sourcing — either have exponential storage costs or require slow log replay.

Our Approach: Time-Windowed Aggregators with Mutation Tracking

We combine three mechanisms:

1. Time-Windowed Aggregator Snapshots: Every aggregator stores startTs and endTs timestamps for its 5-minute window. These immutable aggregators persist in the graph database keyed by (entity_id, timestamp), providing checkpoint states every 5 minutes.

2. Property-Level Mutation Tracking: The graph database maintains mutation history at the property level, storing only changed properties with timestamps. This is much more efficient than full entity copies and provides sub-window precision beyond the 5-minute aggregation boundaries.

3. Query-Time Reconstruction: When querying historical topology, we query the mutation history API for the time range, retrieve all mutations, and reconstruct topology state by applying mutations in order.

This approach provides efficient storage (compressed aggregator states + sparse property mutations), fast retrieval (indexed mutation history, no log replay), and flexible analysis (arbitrary time ranges without pre-computing all possibilities).

Query-Time Re-Aggregation: We can further aggregate historical data at query time using the same aggregator classes from ingestion. This enables arbitrary groupby dimensions (availability tier, business domain, deployment cluster) that weren’t pre-computed, allowing exploratory analysis without exploding storage costs.

Lessons for Distributed Systems

While these challenges were specific to service topology, the lessons apply broadly to distributed systems at scale.

Scale Changes Everything

What works at 100 requests per second fails at 100,000 requests per second. The change isn’t linear, it’s qualitative. Approaches that are fine at modest scale hit fundamental walls at extreme scale.

Examples from our journey: immutable data structures create GC pressure at millions of allocations per second; single-stage aggregation fails catastrophically with power-law traffic distribution; standard gRPC becomes heavyweight for streaming aggregation at volume.

The lesson: be willing to break conventional wisdom when scale justifies it. But do it based on measurement, not speculation.

Optimize One Bottleneck at a Time

Distributed systems have cascading bottlenecks. Fix Kafka lag, and you discover hot node issues. Fix hot nodes, and you discover GC problems. Fix GC, and you discover serialization inefficiencies.

This isn’t failure, it’s the nature of complex systems. Each optimization raises throughput, which stresses the next weakest point. The approach: prioritize based on impact, fix the current bottleneck thoroughly with measurement confirming resolution, then move to the next one. Optimization at scale is continuous, not one-time.

Distribution Is Key to Scale

Single aggregation points are inevitable bottlenecks. Consistent hashing distributes load but doesn’t prevent concentration when data itself is unevenly distributed (power-law distributions like ours).

Our three-stage pipeline with graduated redistribution solved this. Load spreads across multiple distribution points at each stage. Even with highly skewed data, no single instance becomes overwhelmed. The general principle: use multi-stage processing with redistribution at each stage when dealing with skewed data at scale.

Current State and Impact

Service Topology operates in production today, processing flow logs, IPC metrics and traces from multiple regions and serving queries with sub-second latency. Teams across Netflix use it daily for incident investigation, blast radius analysis, dependency understanding, and production change management. The system has become essential infrastructure for maintaining reliability at scale.

Conclusion

Service Topology at Netflix represents a journey through building distributed systems at scale. We started with engineers struggling to understand dependencies across scattered tools. We built a multi-layer architecture using streaming aggregation, network intermediary resolution, and time-travel capabilities. And we learned that optimization at scale is continuous, measure, iterate, validate, repeat.

The challenges we faced, Kafka lag, hot nodes, memory pressure, required breaking conventional wisdom when data justified it. Each fix revealed the next bottleneck. But that iterative process, guided by constant measurement, is what makes systems work at extreme scale.

In our next post, we’ll explore the tracing layer integration, unified querying across heterogeneous storage, and how all three layers combine to provide comprehensive topology visibility.

Acknowledgements

Service Topology was built by Parth Jain, Rakesh Sukumar, Yingwu Zhao, Renzo Sanchez-Silva, and Nathan Fisher.

Special thanks to the many engineers across Netflix who made this possible — the Observability team who built the broader system, the graph database platform team who provided the storage foundation, and the Platform Modernization Engineering and Live teams who provided invaluable feedback and use cases throughout development.

Building Service Topology at Scale: Architecture, Challenges, and Lessons Learned was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

By AI Platform’s Model Runtime team and Inference team

Introduction

Most organizations consume LLMs through hosted APIs. Netflix went further — we run the full stack ourselves, from model deployment through inference, inside our existing production environment rather than a separate ML silo. Some of those decisions weren’t obvious, and a few revealed their trade-offs only under production load.

This post focuses on the choices where alternatives were seriously considered: engine selection, model packaging, API surface design, deployment strategy, and output constraints enforcement. The goal is to share not just what was built, but why — and what production revealed that the design phase didn’t anticipate.

Architecture Overview

Member-scale ML at Netflix is fronted by a unified JVM-based serving system that handles the end-to-end flow for downstream consumers: routing and A/B test logic, candidate generation, feature fetching, inference, post-processing, and logging at each stage. Both real-time and cached batch paths are supported. Figure 1 shows the two ways callers reach inference today: the gRPC path through this serving system and a direct HTTP path used by newer LLM-driven applications.

Where inference runs depends on the model. Small CPU models run in-process, avoiding remote-call overhead. Larger models need GPUs — the serving system handles pre- and post-processing locally but delegates inference to a remote service, Model Scoring Service (MSS). MSS is the shared inference backend, supporting XGBoost, TensorFlow, PyTorch, and LLMs behind a unified interface, with NVIDIA Triton Inference Server underneath managing model loading, batching, and GPU scheduling.

On top of Triton sits a Java control plane that handles deployment, versioning, health checking, autoscaling, and multi-region rollout. Model authors package their artifacts and configure the deployment; the control plane provisions GPU instances, configures Triton, and orchestrates zero-downtime upgrades.

Design Decisions and Implementation

Four decisions shape this platform — engine, packaging, API surface, and rollout — presented in dependency order, since each one constrains the next.

vLLM as the Paved-Path Engine

The platform was originally built on TensorRT-LLM, a performant inference engine at the time and already integrated with Triton — the compute backend in use within MSS.

By summer 2025, two things had shifted: open-source engines had largely closed the performance gap with specialized stacks, and our workload mix had broadened to include embedding generation, prefill-only inference for ranking and retrieval, autoregressive decoding, and custom models with non-trivial per-step constraint logic. We re-benchmarked against this mix and selected vLLM as our paved-path engine on operational fit:

• Loads custom model architectures without a multi-step compilation pipeline — faster iteration on non-standard models.
• Extensibility hooks for custom decoding logic — necessary for the constrained-decoding work described later.
• Debuggability — easier to inspect failures and intermediate state than with a compiled engine in earlier TensorRT-LLM.
• Familiarity — many ML practitioners were already using vLLM in research, which cut the research-to-production handoff cost.

Integrating vLLM into Triton

With vLLM picked, the next decision was how to package models for it. Triton supports two ways, and the choice has significant implications for maintainability — specifically, how tightly model artifacts are coupled to frontend upgrades.

• Python backend. The author defines explicit input/output tensor specs at packaging time. These specs are frozen in the artifact and must match what the third-party vendor’s frontend’s request builder expects, so every frontend upgrade that touches I/O specs requires a coordinated change to packaging code; otherwise, requests fail at runtime.
• vLLM backend. The artifact is just a JSON config pointing to the model weights and tokenizer. Triton’s vLLM backend reads this config and generates I/O tensor specs dynamically at deployment time — the author never defines them. Models and frontend evolve independently.

The vLLM backend is the architecturally correct default. Two things bit us in production:

• Triton/vLLM version mismatch. Triton’s vLLM backend is compiled against a specific vLLM API surface. When the two drift — for example, Triton 25.09 importing vllm.engine.metrics, a module removed in vLLM 0.11.2 — the backend fails to load entirely. The platform has to pin compatible versions when baking the service image, and prevent model authors from overriding the vLLM version at packaging time.
• Custom model logic. The vLLM backend expects a standard HuggingFace-compatible model and handles the full inference lifecycle. Models needing custom preprocessing, postprocessing, or non-standard execution — ensemble pipelines, custom tokenization — must use the Python backend, which gives full control over execute(). This escape hatch will likely remain necessary for a subset of models.

Ecosystem-Compatible HTTP Frontend

With engine and packaging settled, the next question is how callers reach the system. A key design goal of our system was that LLM models should NOT be special snowflakes. Every model — XGBoost ensemble or large-scale LLMs — is scored via the same gRPC call, so we reuse the same client libraries, health checking, and deployment pipelines. Given that the OpenAI-compatible API interface has become the de facto interface for the LLM ecosystem — inference engines, orchestration frameworks, evaluation tools, and client libraries all speak it — so we expose the OpenAI-compatible API as an additional frontend alongside gRPC.

The payoff shows up in the experimentation-to-production path: graduating from a hosted model to a fine-tuned self-hosted one — for quality, latency, cost, or data privacy — is nearly seamless. Same API, minimal code changes.

Behind the API, the implementation reuses NVIDIA’s Triton OpenAI-compatible frontend. It starts an embedded Triton server, wraps it in a TritonLLMEngine that converts request schemas into Triton inference requests, and serves responses through FastAPI. KServe HTTP/gRPC frontends are enabled alongside, so the same Triton instance remains accessible to the Java control plane over gRPC. Adopting Triton’s frontend directly exposed one gap: response_format — accepted by the schema — was silently dropped before reaching vLLM, so that a caller requesting JSON output proceeded without guided decoding constraints and could receive malformed JSON with no error surfaced by the platform. We git-subtreed and patched the frontend to translate response_format into vLLM’s guided decoding parameters at request time.

Deployment Strategies

With API surface and engine in place, the question that remains is how new versions roll out without dropping requests. GPU deployments take longer to bring up than CPU services, and the I/O schema may change between model versions — adding a coordination problem on top. The platform offers two strategies:

• Red-Black deploys a new version alongside the current one. Once the new instance passes health checks, traffic shifts in phases — the new version scales up while the old scales down at the same rate. If any step fails, the system triggers an atomic rollback. Red-Black is the right choice when the model interface is stable. Production revealed a coordination gap when a new version requires an I/O schema change (e.g., new tensor dimensions): the upstream consumer can’t update its config until the new model is fully live, so it inevitably sends “old” requests to a “new” deployment during the migration window, and those fail.
• Versioned solves that gap by maintaining an independent deployment for every (modelId, modelVersion) pair. Multiple versions serve simultaneously, decoupling model deployment from consumer updates: the consumer waits for the new version to be fully ready before switching its config, while the old version keeps serving legacy traffic. The platform cleans up older deployments after inactivity but always preserves the latest. The trade-off is a temporary increase in GPU cost during the transition overlap.

We recommend embedding variable configurations (e.g., tensor shapes) directly into the inference model to make it version-agnostic, so it can use the cheaper Red-Black path. Versioned is reserved for the rare cases where a breaking interface change is unavoidable.

Operational Notes

Beyond those four decisions, two operational details are worth flagging — both hit production gaps the design phase didn’t anticipate.

Boot sequence

Bringing a vLLM-on-Triton instance up involves several coordinated steps before the gRPC port opens. Two are non-routine.

• Model caching. Downloading large LLMs directly from S3 or Hugging Face at startup is slow enough to inflate cold-start latency past what schedulers tolerate. We materialize models on Amazon FSx at the time of model announcement, so warm starts hit a high-performance file system instead of object storage.
• Embedded vs standalone Triton. When consumers need the OpenAI-compatible API, Triton runs as an embedded server inside the OpenAI-compatible frontend process; otherwise, it runs standalone. This is configured per-deployment at packaging time.

The rest of the boot sequence is mechanical: extracting the model package, installing custom vLLM plugins via Python entry_points, cleaning the Prometheus multiprocess directory, and gating the gRPC port until the engine is ready.

Unified metrics endpoint

The Prometheus cleanup above hints at a wider observability gap. vLLM writes metrics to PROMETHEUS_MULTIPROC_DIR as .db files; Triton reports server-level metrics through its own Prometheus endpoint. Neither is aware of the other, and Triton’s built-in bridge surfaces only 9 of 40+ vLLM metrics — missing critical ones like token throughput, KV cache utilization, and prefix cache hit rates.

We added a lightweight HTTP proxy that merges both into a single /metrics endpoint: it fetches Triton metrics via HTTP, reads vLLM metrics from disk using Prometheus’s MultiProcessCollector, and returns the combined output. Existing dashboards and alerts work without modification.

Deep-Dive: Constrained Decoding at Scale

Some Netflix production workloads rely heavily on fine-grained control over token generation. Rather than applying business logic after inference — paying for invalid generations, then retrying or repairing — we push constraints inside the decode loop, so the model generates outputs that are compliant by construction. We implement this via vLLM’s custom logits processor interface, modeling each constraint as a state machine that evolves with the generated token history and emits token-eligibility masks at each step. Each request gets its own configured processor, since different requests apply different rules.

Getting this to scale ran across two engine versions: we initially deployed on vLLM V0 (V1 had feature gaps), then migrated to V1 in Q4 2025 once it matured. The two subsections that follow are the before-and-after.

Why the first implementation didn’t scale

Our initial pure-Python implementation worked functionally but hit a scaling bottleneck. In vLLM V0, custom logits processors run per-request: the GPU produces logits for the whole batch, the CPU copies them across and waits for the transfer, and then constraint logic runs sequentially for each request — sequentially because the GIL prevents Python from parallelizing the per-request work. CPU time in logit processing therefore grows linearly with batch size, hitting tail latencies. End-to-end latency becomes CPU-bound even though the model’s forward pass is batched efficiently on GPU. It’s a bottleneck invisible in single-request benchmarks that only surfaces under realistic concurrency. Figure 2 makes the serial pattern visible.

vLLM V1 enabled a batch-level design

The structural fix arrived in vLLM V1, which moved logits processing to batch level. We rewrote our custom processor to operate on batch-level data structures, computing masks across many requests together, and reimplemented the hot path in C++ with multi-threading to step around the GIL. The V1 API requires explicit tracking of batch membership changes via update_state(batch_update) — more complex than V0’s per-request interface, but necessary to maintain correct state in a dynamically evolving batch. Figure 3 shows logits processing time staying flat as batch size grows.

Operational hardening

Now, performance was no longer the bottleneck. But stateful constraint logic in the decode loop introduced two issues the design phase didn’t anticipate:

• Partial prefills. V1 performs chunked prefilling, so a request can be prefilled over multiple engine steps. BatchUpdate lacks the granularity to tell whether a request was fully or only partially prefilled, so we added internal tracking.
• Preemption. Under memory pressure, vLLM may evict a partially completed request’s KV cache and reschedule it later with a different prompt and output token list. This breaks the state machine’s assumption that the output token list grows monotonically. We detect when the token history shrinks between decode steps, reset the state machine, and reinitialize from the new prompt.

Wrap up

We set out to build an LLM serving platform for broad production ML requirements — low latency, deep customization, and integration with existing infrastructure. The result is a system on vLLM and Triton, unified behind a consistent API, designed to give ML practitioners a fast path from experimentation to production.

The lessons were often in the details — version pinning, silent API gaps, packaging trade-offs — but addressing them has made the platform meaningfully more robust and the developer experience smoother. Next investments reflect where we expect friction:

• System prompt compression to reduce prompt length without sacrificing quality.
• Asynchronous scheduling of vLLM V1.
• Vectorized logits processors that run as fused GPU kernels instead of CPU code.
• Lower-precision model variants to decrease memory footprint and increase throughput.

We’ll continue working closely with the open-source community as this space evolves.

Contributions

This system is the result of close collaboration and contributions from many teams within the AI Platform org at Netflix. In particular, Liping Peng designed and developed the model packaging workflow and drove the integration of Triton and vLLM with MSS to enable a unified pathway for serving LLMs. Hakan Baba, Nicolas Hortiguera, and ZQ Zhang led GPU capacity planning, system performance tuning, application integration and observability, as well as A/B test readiness and operational excellence efforts for all production models. Santino Ramos enabled vLLM for production models and optimized constrained decoding performance. Binh Tang developed the initial version of custom model serving and benchmarked different LLM serving frameworks. Lanxi Huang and Daneo Zhang built the serving development tools to enable user self-service. Lingyi Liu drove the overall system architecture and core technical decisions. Abhishek Agrawal and Shaojing Li provide management leadership to ensure alignment, prioritization and execution.

Acknowledgements

This work heavily leverages open-source ML libraries, such as Triton, vLLM and PyTorch, etc. We’re especially grateful to the teams and contributors from the community. We also thank our partner teams in Netflix AI for Member Systems for their close collaborations and innovation on the modeling side.

In-House LLM Serving at Netflix was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.