RSS Feeds

• Carrick expected to be in charge for Bournemouth game

• Compensation could be obstacle for Aston Villa’s Emery

Unai Emery and Julian Nagelsmann have emerged as leading targets to replace Michael Carrick as Manchester United’s head coach should the board decide on a change after the club’s poorest start to the season since 1986.

Carrick is expected to remain in place for Sunday’s visit of Bournemouth – he is due to hold his pre-game media conference on Friday – but the 45-year-old’s job is in the balance after two wins from 11 games this season.

Lawmakers in Albania's parliament clashed on Thursday as opposition MPs attempted to disrupt a session, throwing objects and ripping microphones from the desks of members of the ruling majority. Several opposition MPs held up images of students injured by police during protests in the capital, Tirana

Samartha Chandrashekar & Nick Parker

You click play on a Friday night, and in the second that follows, dozens of services answer in milliseconds. Most of them run on Titus, our internal container platform. Titus is, in essence, a Kubernetes-based container hosting platform: we keep both the control plane and the data plane (the container hosts that actually run containers) patched and up to date, so application teams get a consistent and reliable container experience. They hand us a container, and we run it, with no host, cluster, or kernel to own. Titus launches millions of containers every week across tens of thousands of EC2 instances in multiple AWS regions. It is multi-tenant by default: many containers from many teams share each host. That density is what makes container compute efficient, and it is why Titus has been the default home for so much of Netflix’s compute for years.

For most workloads, sharing a host works well. For a few, it does not, and those few include some of the most demanding real-time systems we run. This post is about them, and about single-tenant Titus, a mode that gives a task its own virtual machine while keeping everything else about Titus the same: the API, the deployment path, and the managed experience where the Compute/Platform team, not the application team, owns the host.

We will walk through why shared-kernel isolation runs out of road for certain workloads, how we measure isolation rather than asserting it, the options we weighed (microVMs, gVisor, EC2, and more shared-kernel tuning) and why a dedicated VM inside Titus won, how the mode is built on the Titus federation layer and autoscaling, and what we have seen along three axes: isolation, fleet efficiency, and the operational cost of keeping the fleet patched.

The cost of good neighbors

Multi-tenancy is a tradeoff. Packing many containers onto a large bare-metal instance keeps utilization high, but it also means those containers share a single Linux kernel and the underlying hardware. Titus isolates them with the tools the kernel provides: cgroups cap CPU, memory, and disk; namespaces give each container its own view of the system; seccomp and AppArmor restrict the system calls a container can make; the Completely Fair Scheduler divides CPU time; the iocost controller meters disk IO; and traffic shaping meters each container’s slice of the network. Each container also gets its own identity, IAM role, and security groups, so tenants are isolated for security and resource management.

These mechanisms provide isolation for what the kernel can partition. They cannot fully partition what is shared in hardware. A group of cores shares a last-level cache and a path to memory, so a container that suddenly pulls a lot of data from RAM evicts a neighbor’s cache lines and slows it down, even when both stay inside their CPU limits. There is no cgroup for memory bandwidth or cache residency, as there is for CPU shares. The interference is real, it is hardware, and it does not show up as either container exceeding its quota.

A noisy neighbor and a co-located app each stay within their cgroup CPU and memory limits, but both draw on a shared last-level cache and memory bandwidth that lack cgroup fences. The noisy neighbor takes most of it, so the co-located app stalls and its tail latency rises even though no one exceeded a quota.

We have written about this before. Predictive CPU isolation of containers at Netflix moved some placement decisions out of the kernel and into a service that forecasts each container’s CPU demand and re-pins containers to cores to reduce cache interference, formulated as a combinatorial optimization solved every minute. Noisy Neighbor Detection with eBPF instruments the kernel scheduler to surface per-container run-queue latency, so we can see a victim being starved in close to real time. And Mount Mayhem traced container-startup slowness to a global kernel lock amplified by modern CPU cache topology. Each of these raised the floor for everyone on the platform, and we are continuing that work.

A decade of shared-kernel isolation, at Netflix and across the industry, has not made co-tenancy invisible, because a noisy neighbor is not one problem but a family of them. Almost every physical resource a host shares is its own axis of interference, and a cgroup can fence very few of them:

• CPU performance indeterminacy. Co-located threads contend for the run queue, so scheduling latency rises even while a container remains within its CPU quota. Separately, a neighbor with a large working set evicts a tenant’s last-level cache lines and saturates memory bandwidth, neither of which a CPU share can partition.
• Network bandwidth and packets per second. Per-container shaping competes for link bandwidth, and receives processing runs in softirq context, charging one container’s packet work to whatever CPU a neighbor is running on. A flood of small packets exhausts a core’s softirq budget long before any bandwidth cap is reached.
• Storage bandwidth and IOPS. A neighbor saturating a device’s queue serializes everyone’s I/O, and on cloud block storage, the per-volume and per-instance throughput ceilings are a shared budget in their own right.
• Long tail. Page-cache reclaim, NUMA and cross-socket interconnect traffic, TLB-shootdown interrupts, PCIe and GPU contention, package-level power and thermal throttling, and contention on kernel global locks (the class of problem behind Mount Mayhem) each degrade neighbors in their own way.

What unites them is that they are physical, shared, and below the level a quota can reach. A platform can cap how much CPU time or memory a container is granted; it cannot hand each tenant its own slice of the memory bus, its own softirq budget, or its own portion of the last-level cache.

Of these, the network dimension has bitten hardest in our experience. Per-container bandwidth shaping uses a hierarchical token bucket (HTB) queueing discipline that serializes traffic with a global lock to account for each container’s bytes and packets. Under a flood of small packets from a single container, contention for that lock can burn kernel CPU and starve co-located containers of the time they need to process their own traffic, resulting in TCP retransmits and tail-latency spikes on services that did nothing wrong. The mechanism meant to prevent noisy neighbors becomes one.

The workloads most exposed to this are stateful, high-throughput streaming jobs: real-time data pipelines and stream processors that continuously move large volumes of data and maintain in-memory state. When one of these is perturbed, recovery is not cheap. A stream processor that falls behind or loses its place may have to restore from a checkpoint and pause processing for many minutes to catch up. For these jobs, intermittent interference from a neighbor is not a small slowdown that averages out; it is a stop-the-world event that pages an on-call engineer. The same teams report, accurately, that they never saw this on a plain EC2 instance.

Why we kept coming back to the VM

Full VM isolation, where a workload gets its own host and kernel, is the standard our customers are already happy with. The question was not whether VM isolation works; it was how to offer it without giving up what makes Titus worth using. We weighed four paths.

1. Keep improving shared-kernel isolation. This helps every workload on the platform, needs no opt-in, and we are still investing in it (more on the roadmap below). Its limitation is structural. Each round of tuning narrows the gap without closing it, because the underlying hardware remains shared. Cache, memory bandwidth, and the kernel’s own shaping and scheduling locks are not things a container boundary can fully contain. Shared-kernel isolation is the right default and the wrong promise for a workload that cannot tolerate a bad minute.
2. microVMs (Firecracker, Kata Containers). Lightweight VMs boot in about 150 milliseconds by emulating far less hardware than traditional VMs, which is why they power serverless platforms. They were the most appealing alternative on paper, and we carefully reviewed them. Three things made the cost hard to justify. First, the hardware virtualization they rely on is available only on bare-metal instances or through nested virtualization, and nested virtualization was not ready on the instance types we run at Netflix at the time, so adopting microVMs broadly would have constrained us to a narrow slice of the fleet. Second, there is no managed microVM service we could build on, which means we would have to run and maintain a virtualization layer ourselves: patching the hypervisor, live-migrating guests, and qualifying emulated devices. That is exactly the kind of operational surface we prefer not to own when we can avoid it. Third, even setting those aside, microVMs add virtualization overhead on CPU- and IO-bound work and do not support GPUs. Excellent isolation, but at a price in flexibility and operational load we were not willing to pay.
3. gVisor. A user-space kernel that intercepts system calls provides sandbox-grade isolation without a hypervisor, thereby avoiding the bare-metal constraint. The catch is performance and dependency. Intercepting syscalls in user space adds overhead on IO-heavy paths, which are precisely the workloads we most need to protect, and the open-source project lags behind its internal counterpart in fixes and is effectively maintained by a single vendor. The isolation we would gain came with a throughput tax on the exact jobs we were trying to help.
4. Move the workload to EC2. It is possible to run containers directly on EC2 instances. The problem is that the application team takes back the operational load that Titus exists to carry: choosing instance types, patching hosts, and sizing and scaling the fleet. For a team that came to Titus precisely to avoid those things, this is a regression in the experience, and the move between Titus and EC2 has historically been a one-way door that teams are reluctant to cross.

Spectrum of isolation options from lighter and more shared to stronger isolation: shared-kernel containers, then gVisor, then microVMs (Firecracker or Kata), then a full VM as used by single-tenant Titus, which is highlighted. Each option is annotated with its launch latency, overhead, and the entity that operates it; single-tenant Titus is a dedicated EC2 instance with no additional virtualization overhead.

There is also a question of what kind of isolation we actually need. microVMs and gVisor are built for security isolation between mutually distrusting tenants, the case where one tenant might actively try to attack or escape into another. Among Netflix teams, that threat model is more than the situation warrants. What these workloads need is resource and performance isolation: a guarantee that a neighbor cannot steal their CPU, cache, memory bandwidth, or network, not protection from a hostile co-tenant. A dedicated EC2 VM delivers exactly that, with hardware we do not have to emulate and a kernel we do not have to sandbox. We pay for the isolation we need rather than for a stronger security boundary.

microVMs and gVisor asked for a large, bespoke investment with real risk in order to approximate something we could get directly: a VM. Using EC2 to run containers provides the VM, but takes away the managed experience. So we took the direct route and brought single-tenant VM isolation into Titus itself, under the existing API and the existing shared-responsibility model.

How single-tenant Titus works

The idea is simple to state: a single-tenant task runs on its own EC2 instance, sized to fit it, with nothing else scheduled alongside. That instance is still a Titus container host that the Compute/Platform team owns and patches. The team running the workload sees the same Titus API and the same deployment path. Nothing about how they package or ship the job changes.

Side-by-side anatomy of a multi-tenant Titus, where multiple teams’ containers share one Linux kernel and the hardware beneath it and contend with one another, versus a single-tenant node where one task plus its platform sidecars owns the whole machine with no co-tenants.

Most of the work happens in the federation layer, the part of Titus that decides where a job runs (which cell, which capacity) before it is scheduled onto a node.

When a job is eligible for single tenancy, three things happen.

1. Federation decides eligibility and the envelope. Eligibility is governed by an allowlist maintained by the Compute/Platform team; a team opts in by targeting a single-tenant workload group rather than flipping a flag in the public API. For an eligible task, federation tags it as single-tenant and records the set of instance families it may run on, ensuring consistent performance across the workload. Consolidating this decision in federation follows the same principle we use for routing in general: placement intelligence belongs in one place, above the individual cells, where it can account for capacity and availability across the region.
2. Autoscaling provisions a dedicated node. Given a pending single-tenant task, our autoscaling implementation launches the smallest instance that satisfies the task’s requested CPU, memory, and GPU from the allowed families for the task just in time. Taking the tightest fit keeps the unused remainder of each dedicated node small. The next section covers this in detail.
3. The scheduler enforces one task per node. Single-tenant pods are configured with pod anti-affinity to prevent colocation with other pods, along with a priority class that exempts the task from preemption. Single-tenant nodes are also tainted to prevent unrelated workloads from being placed there, with only single-tenant workloads tolerating this taint. We also disable defragmentation/repacking for these nodes, so a single-tenant task is never relocated to repack the fleet. Together, these mean a single-tenant task has its host to itself for its entire life, undisturbed by a neighbor or by the platform’s own bin-packing.

Two design choices are worth dwelling on. First, single tenancy is allowlisted on purpose. If it were a free checkbox in the API, most teams would tick it, because who wouldn’t want a quieter host? And the efficiency that multi-tenancy exists to provide would erode across the fleet. Keeping the Compute/Platform team in control lets us direct dedicated isolation to workloads that need it and steer the rest toward shared capacity, while keeping the public API surface small as we learn. Second, we do not silently resize anyone’s job to fill its node. The job keeps the resources it requested; we recommend sizing it close to a node shape so the dedicated instance is well utilized, while keeping the workload portable and running identically whether in single-tenant mode or not, regardless of the host’s dimensions. Moving in and out of the mode is a configuration change, not a rewrite.

A few perpetrators, a lot of collateral damage

Noisy-neighbor pain is not spread evenly. In practice, a small number of workloads cause most of the disruption: a job that floods the host with small packets, churns the page cache, or saturates memory bandwidth degrades the performance of every other container unlucky enough to share its host. Some of these are also among the workloads most affected by contention, because a job can be both a source of interference and sensitive to it. The shape of the problem is a few perpetrators and many victims.

That shape is what makes single tenancy a precise lever rather than a blunt one. We do not need to give every workload its own machine. We need to pull the handful of perpetrators and the most disruption-sensitive victims onto dedicated nodes. Doing so protects their former neighbors directly and removes the reason the platform was spreading sensitive workloads apart with anti-affinity placement in the first place. The hosts left behind are quieter, and because they no longer have to be kept partly empty to honor spread rules, they pack more densely.

Before-and-after view of the Titus fleet. Before: a few red perpetrators are scattered across shared hosts, the amber workloads beside them are disrupted, and anti-affinity spreading leaves dashed stranded capacity. After: each perpetrator runs on its own dedicated single-tenant node, the remaining shared hosts are fully loaded with healthy workloads, and one host’s worth of capacity is reclaimed.

On the left is the fleet before: perpetrators sit on shared hosts, the workloads next to them suffer, and anti-affinity rules leave gaps that cannot be filled. On the right is the same fleet after: each perpetrator has its own right-sized node where it can be as noisy as it likes, with no one to disturb; the remaining hosts run full of undisturbed workloads; and the capacity that used to be stranded comes back. A few dedicated nodes buy isolation for the perpetrators and a calmer, denser fleet for everyone else.

Provisioning many small nodes

Single tenancy trades fewer large, shared instances for more small, dedicated ones, and that shift has consequences across the stack.

The first is the control-plane load. More nodes mean more for the Titus control plane to track and more VM launches against the EC2 control plane, the latter mattering most during a regional failover, when we scale up the fastest and can least afford a throttle. Titus is already a federated system: each region is divided into independent cells, which limit how large any single control plane can get, and let us add capacity by adding cells rather than growing a single stack without limit. We migrate workloads in stages rather than all at once, and we exercise the launch-rate path under controlled load on a schedule rather than discovering its limits during an incident.

The second consequence is capacity management, where the design changed the most. With one task per node, the fleet now spans many instance shapes whose demand shifts continuously as jobs come and go. Pre-staging static pools of every shape would be slow to manage and wasteful, so we instead provision single-tenant nodes on demand. It handles provisioning end-to-end, and several of its features line up with exactly what single tenancy needs:

• Just-in-time, right-sized provisioning. Single-tenant jobs are configured with a list of supported instance families based on their performance requirements. At launch, they are mapped to the best-fit node size by comparing requested resources against the selected instance families. New nodes of that instance type are brought up by the autoscaler to meet the pending demand. Single-tenant pods are configured with anti-affinity to all other pods, so the scheduler naturally lands one pod per autoscaled node. Newly vacated nodes may be reused by any pending deploys before they’re automatically torn down. The node pools themselves are shared across single-tenant workloads and describe a catalog of available instance families and sizes.
• Single tenant node pools. Single tenant may run in separate clusters or be shared with multitenant clusters. The pools are configured with a list of supported instance-type permutations and are pinned to a Titus container host AMI. This is where the managed experience lives: the host image and its configuration are ours, declared once and applied to every node created.
• Consolidation management. In multi-tenant environments, we default to a repack/defrag behavior that periodically relocates workloads. This doesn’t apply in single tenant, where nodes are explicitly each running at most one workload. We therefore reclaim only nodes vacated by workloads that have scaled in or terminated. Node termination is performed in two stages, allowing nodes to be reused for pending workloads before they are fully terminated.
• Drift-based patching. When we publish a new container host AMI, we support a “mutable” deploy to existing nodes without affecting any hosted containers, and an “immutable” deploy that updates the target AMI for newly created nodes. This means that many node upgrades don’t require disrupting workloads, but we reserve the right to roll the fleet with our users if needed.

The launch-latency tax, and why warm pools are harder here

Provisioning on demand does not make a node appear instantly, and launch latency is the tradeoff single tenancy asks us to manage most carefully. Bringing a single-tenant task to the point where it can serve traffic has two parts: standing the node up (booting a fresh EC2 instance, then pulling and extracting the container image, which is the largest part of host setup for most jobs) and the application’s own startup (a JVM loading and JIT-compiling its classes, a cache filling, and so on). A fresh instance takes tens of seconds before the image pull even begins.

In multi-tenant Titus, most of this is hidden. A new task usually lands on a Titus container host that is already running and already has the image layers cached, so starting it takes only a few seconds rather than minutes; the host boot was paid for once, long ago, and amortized across every task that has run there since. Single tenancy gives up that hiding place. Every single-tenant task waits for its own machine to boot and pull its image.

The usual remedy is a warm pool: keep some instances booted and idle so a pending task lands on one immediately. This works well for the multi-tenant fleet precisely because that fleet is close to homogeneous. A handful of large shapes cover almost everything, since any task can be packed onto a big shared host. A small amount of warm capacity in a few shapes gives a high hit rate, and its cost is spread across the many tasks that will use it.

Single tenancy inverts those economics, and this is the part worth being explicit about.

Press enter or click to view image in full size

Because every task gets a right-sized node, the fleet spans many instance shapes (several families across many sizes), and a warm instance only helps a pending task whose shape matches it. A warm r7i.2xlarge does nothing for a task that needs an m7a.8xlarge. To achieve a good hit rate, we would need to hold idle capacity in every family-and-size bucket, and idle capacity in a bucket with no task to fill is pure cost. The very property that makes single tenancy efficient at steady state, tight best-fit sizing, works against warm-pool reuse: more distinct shapes mean more buckets and a lower hit rate per dollar of standing capacity. And the pressure peaks exactly when it is least welcome. During a regional failover or a live event, when many tasks need fresh nodes at once, a thin warm pool spread across many shapes drains immediately, and the rest of the demand pays the full cold-boot cost.

So we treat warm capacity as a small, targeted lever rather than a default. We keep a little pre-warmed capacity for the most latency-sensitive single-tenant starts, and we deliberately narrow the menu of shapes to a curated set of sizes rather than the full continuum, so warm capacity concentrates into fewer buckets and trades a little best-fit efficiency for a better hit rate. Everything else provisions just-in-time and accepts the boot.

The more durable wins are in cutting the launch cost itself rather than pre-paying for it, and they help both single- and multi-tenant nodes, which is the better place to spend the effort:

• Faster image distribution. The image pull dominates the host setup, so newer OCI image formats and lazy-pulling snapshotters such as stargz and SOCI, which fetch only the bytes a container actually reads at startup rather than the whole image up front, reduce the largest component of the gap.
• Checkpoint and restore. For processes that are slow to initialize, snapshotting a warmed-up application with a tool like CRIU and restoring it, rather than cold-starting it, removes most of the second half of launch latency.
• Instance type selection. In multitenant configurations, we have historically used metal instances, which offer slightly better performance but come with a slower startup time. In a single-tenant configuration, node startup latency is more important, so we have moved away from these instances.
• Scheduling optimization. Through regular stress testing of fleet scaling, we found a scheduling bottleneck caused by Kubernetes preemption logic, leading to nodes being scaled up quickly but then left idle while waiting for workloads to be scheduled on them. This was ultimately unnecessary, as our single-tenant pools do not run preemptible workloads. Disabling preemption greatly improved scheduling performance when under strain.

None of this makes the tradeoff disappear, and it is worth stating plainly: a single-tenant node is slower to stand up than a slot on an already-warm shared host, and hiding that with warm pools costs more per node here than it does on the multi-tenant fleet, because the warm pool cannot be as homogeneous. But for the long-running services that single tenancy targets, paying that cost once at launch in exchange for a quiet life afterward is a trade these teams are glad to make.

Measuring the difference

Isolation claims are easy to assert and hard to trust, so we measure them with a repeatable benchmark and a single score.

The method is a differential one. We run a reference container, a healthy and representative application, and record its throughput and tail latency while it runs alone. Then we run it again next to a havoc container engineered to be a bad neighbor: a UDP flood through XDP to stress the network-shaping path, pointer-chasing and STREAM to pollute the last-level cache and saturate memory bandwidth, fsync storms against the disk, stress-ng to abuse the scheduler, and GPU-burn to monopolize an accelerator. We only trust a run once the host has reached a steady state, which we confirm with the kernel’s Pressure Stall Information: we wait until cpu_some or mem_some shows that tasks are actually stalling on a contended resource, not merely that the resource is busy. Each scenario is run many times with fixed seeds, and we report the median with a bootstrapped confidence interval, because noisy-neighbor distributions are heavy-tailed and a simple average can mask the spikes that matter.

From the reference container’s numbers, we derive a simple measure of how well isolation held. Throughput efficiency is the fraction of operations per second the app retains under attack; latency efficiency is the ratio of its undisturbed P99 to its P99 under pressure. A neighbor that doubles the reference app’s P99 drops the latency efficiency. As an illustration, a reference app whose P99 latency goes down next to a havoc container achieves a certain latency efficiency, and one whose throughput falls from a to b operations per second achieves a certain throughput efficiency. The composite score is a weighted blend of the two, with a latency-sensitive preset that gives greater weight to tail latency for interactive and streaming-path services. A perfectly isolated system scores 100%, and a single hard failure invalidates the run regardless of the average.

Noisy-neighbor isolation

cgroups container isolation isn’t perfect, and applications may experience worse performance due to sharing a machine with a determined neighbor. This could extend to disk and/or network IO where the same volumes or interfaces may be shared across workloads. For a distributed workload, this can lead to measurable inconsistencies in performance depending on where the replicas are placed. For more sensitive workloads, this variance can mean the difference between stability and getting paged.

The single-tenant option makes it possible to better isolate workloads that cause problems for their neighbors and/or are more sensitive to such problems. This can be used as a stopgap until the specific regression problem is resolved and consistency is restored, or kept permanent for load patterns that aren’t worth supporting in a shared environment.

Efficiency

The second effect is fleet efficiency, and it is the one people expect to go the wrong way. Dedicating a node to a single task consumes that node’s slack, so the naive expectation is that utilization falls. In aggregate, it can move the other way, for a reason related to how we dealt with poor isolation before single tenancy existed.

To protect sensitive workloads on shared hosts, the platform used anti-affinity placement, scattering them across the fleet to reduce colocation of the same workload on the same nodes. Spreading fights bin-packing: it strains capacity, because a host kept partly empty to honor an anti-affinity rule cannot be filled by a workload that would otherwise have fit there. When isolation-sensitive workloads move to dedicated nodes, the need for spreading and fragmentation goes away, and the remaining multi-tenant fleet can pack more densely.

Another source of fragmentation is services that are sensitive to disruption. These are typically stateful services where recovery is expensive. As these services become more common in a given population, they can lead to nodes being held up by pods with strict disruption requirements. This can lead to an inability to defrag and recompact the pool during off-peak times. The symptoms and resulting drop in fleet efficiency caused by disruption-sensitive workloads mirror what’s observed for workloads with anti-affinity rules.

Depending on the magnitude of the anti-affinity and/or disruption budget problem, moving such workloads to single-tenant pools can result in equivalent or even improved placement when all costs have been accounted for. It also becomes easier to quantify the cost to end-users. Rather than trying to attribute fleet-wide fragmentation to individual workloads, we can directly quantify waste as single-tenant node overhead.

The theory and the practice diverge here, and it was the practice that decided it. In principle, the efficiency ceiling is higher for multi-tenant bin-packing: pack enough tasks together, and a host fills almost completely. We do not reach that ceiling. AWS offers a discrete set of instance sizes, so a large shared host rarely matches the exact sum of the requests we place on it, and the gap between a host’s capacity and what we can actually pack onto it is a bin-packing hole we cannot fill. At the granularity of large instances, those holes are sizable. Right-sizing an instance to a single workload, picking the closest fit from many shapes instead of draping many workloads over one big host, captured more of the available efficiency in our experience than the higher theoretical ceiling ever delivered. The clean theory favored multi-tenancy; our measured fleet favored right-sizing.

Container-host maintenance

The third effect is operational, and it shows up in how we keep the fleet patched. A multi-tenant host is hard to take down for maintenance precisely because it is full: draining it means relocating many tenants at once, several of which carry disruption budgets that limit how fast they can move, so a routine patch becomes a careful, coordinated evacuation. A single-tenant host runs exactly one task, with consolidation and defragmentation turned off, so draining it means relocating only the one workload, and we can automatically handle that when a new AMI ships. The remaining multi-tenant hosts are easier to maintain, too, because the workloads that were most sensitive to disruption are no longer stranded on them.

What it changes, and what it doesn’t

Single-tenant mode is a configuration, not a migration. A team moves a workload onto a dedicated node, and back, with a labeling change, without rewriting anything and without leaving Titus. That turns the isolation decision into a reversible setting, which is the opposite of the one-way door between Titus and EC2 that teams used to face. A team can try single tenancy for a quarter, measure it, and step back if the trade-off isn’t worth it for them.

What does not change is the contract. The Compute/Platform team still selects the instance type, patches the host, and sizes and scales the fleet underneath. The application team still uses the Titus API, the same delivery pipeline, and the same observability. Single tenancy is a new point on the isolation axis offered within the existing managed experience, not a separate product that the team has to learn and operate.

When a dedicated node is the wrong call

A dedicated node is a real cost, and single tenancy is not free isolation. Two things are worth weighing before reaching for it.

First, a single-tenant task does not get the whole machine; it gets the machine minus overhead. Every Titus container host, dedicated or shared, runs the platform sidecars, system services, the kubelet, and the host environment itself, and that overhead is a roughly fixed slice of CPU and a few gigabytes of memory. On a shared host, that slice is paid once and amortized across the many tasks running there. On a dedicated node, a single task handles it. Because the cost is fixed rather than proportional, it is punishing as a fraction of a small instance and negligible on a large one: a small dedicated node can lose tens of percent of its resources to overhead, while on the largest shapes it rounds to a couple of percent.

Second, because a task is placed on the smallest instance that fits it, there is usually a gap between what the task asked for and the next available shape, and on a dedicated node, that gap is pure waste, with no neighbors to soak it up. The two effects compound: fixed overhead plus the best-fit remainder set a ceiling on how much of a dedicated node a workload can actually use, and that ceiling is lowest exactly where people are tempted to reach for isolation casually, on small instances. However, this sizing waste can be reduced through coordination with the service owner as needed.

The guidance follows from the arithmetic. Single tenancy pays off for large, long-running, performance-sensitive, disruption-sensitive workloads, sized close to a node shape and run as fewer, larger replicas rather than many small ones. It is the wrong call for small, bursty, or highly elastic workloads, where the overhead fraction is high, the dedicated node sits mostly idle, and multi-tenant packing is both cheaper and faster to scale. This is also why the mode is allowlisted rather than a self-serve checkbox: the platform can keep workloads on the side of that line where they belong, and leave the ones that would gain little from a node of their own on the shared fleet, where they are cheaper to run.

Who’s using it

The first adopters are the workloads that motivated the work: real-time data infrastructure. Managed Apache Flink stream processing, streaming data connectors, and telemetry and logging pipelines were both the most frequent sources of network contention and the most damaged by it, because their recovery is expensive and their disruption budgets are tight. These are stateful, high-throughput jobs where a few seconds of starvation can cascade into minutes of catch-up, and where the platform’s own repacking used to be a source of disruption in its own right.

Moving them onto dedicated nodes removed the class of problem rather than mitigating it. The teams running them report stable operation, having previously seen intermittent, hard-to-attribute interference, and their jobs are no longer relocated by fleet repacking, removing a second, self-inflicted source of disruption. Latency-sensitive serving paths and a handful of GraphQL and edge services have followed for the same reason: a predictable tail matters more to them than a few points of bin-packing efficiency.

This is an ongoing rollout, not a finished one. We are widening the set of eligible workloads as we learn, using the largest production jobs as the stress test for the platform’s scaling assumptions, and watching the control-plane and EC2 launch-rate metrics as the count of small nodes grows. We have also been running periodic stress tests to exercise scaling ahead of real user workload, and making improvements to autoscaling and scheduling logic along the way.

What’s next

We are not done on either side of the tradeoff. On single tenancy, we are reducing launch latency with smarter pre-warming of common shapes, broadening the allowlist as scaling data holds up, and extending the model beyond long-running services as cost attribution and capacity stories mature. On the multi-tenant side, which remains the right default for most workloads, we are addressing the dimensions that hurt the most. The largest is the network: replacing HTB’s global-lock shaping with an eBPF-based Expected Departure Time approach that rate-limits by timestamping packets rather than contending on a lock, and pairing it with a lower-overhead container network device. We are also evaluating hardware cache and memory-bandwidth partitioning (Intel RDT and AMD PQoS) to give the kernel a real lever on the last-level cache, hardware GPU partitioning (NVIDIA MIG) for accelerator isolation, and continued work on the cgroup v2 IO controller for disk. This isolation benchmark lets us rank these by return on effort rather than by intuition, and show customers, in the numbers, that the shared fleet is getting quieter over time.

Single-tenant Titus is also a step toward something larger. Netflix’s Platform teams offer two ways to get compute: virtual machines on EC2 & containers on Titus. We want to replace that with a single Compute API that takes intent rather than mechanism into account: how much isolation a workload needs, how quickly it must launch, how long it can run without disruption, and what business domain it belongs to. The platform then makes placement and runtime choices, and can make globally optimal ones because it sees the whole picture.

Conclusion

Multi-tenancy is the right default for a container platform, and we will keep making it better for everyone. But the right default is not right for everything. Single-tenant Titus gives the workloads that need it the isolation of a dedicated VM, provisioned and patched by our existing node management, under the same API and the same managed experience, without taking away the freedom to change their mind. We can now point to the difference rather than describe it: a tail that stops moving under a noisy neighbor, a fleet that packs more densely once spreading is retired, and hosts that are faster and safer to patch. For the streaming and data systems at the heart of Netflix’s real-time platform, a room of one’s own turned out to be the feature that mattered.

Single-tenant Titus: VM-level isolation on a multi-tenant container platform was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

GitLab is not affected by PHP2Shell (CVE-2026-109618 and CVE-2026-109619). The GitLab application is not written in PHP and does not run it to serve web requests, and our Security team has found no evidence of impact to GitLab-managed infrastructure. GitLab.com, GitLab Self-Managed, and GitLab Dedicated are not impacted.

PHP is a widely used programming language that runs on web servers and powers popular platforms like WordPress, Drupal, and Magento, so a flaw in PHP itself can affect many websites and applications. On October 21, 2026, the PHP project fixed two critical remote code execution flaws in PHP, together known as PHP2Shell (CVE-2026-109618 and CVE-2026-109619). PHP rates both 9.2 on CVSS v4.0. The flaws are in the PHP engine, not in any framework, library, or application code, so there is no application-level fix. Any web server that runs PHP is affected, including NGINX and Apache. The PHP project confirmed the vulnerability on Apache with mod_php, where an unauthenticated request can achieve remote code execution before any application code runs.

Upgrade every server and container that runs PHP to serve web requests to PHP 8.5.12, 8.4.27, 8.3.36, or 8.2.35, or to your distribution’s patched package now.

This guide is for security and platform teams with PHP projects on GitLab. It covers the fixed version for each PHP branch and distribution, how to use GitLab features to locate PHP projects across your groups, how to track remediation, and how to check for signs of compromise.

Upgrade PHP to a fixed version now

Upgrade the PHP runtime on every host and in every container image to a fixed version. Framework updates, Composer updates, input validation, and authentication don’t help, because PHP handles the request before application code runs.

If your PHP comes from a distribution package, upgrade that package. Fixed package versions as of October 22:

Sources: Ubuntu, Debian, Alpine, and Red Hat security trackers, as of October 22, 2026.

Ubuntu, and Red Hat for its 7.4 and 8.0 packages, keep the old upstream version number and backport the fix. Compare the package versions rather than php -v. Restart every process that runs PHP after upgrading.

Rebuild your container images. If your image uses a floating tag of the official php image, such as php:8, php:8.3, php:8.3-apache or php:latest, rebuild it with docker build --pull and redeploy it. Images that pin a patch version, such as php:8.3.35-apache, need the tag changed to the fixed version first. Images built on derivatives such as wordpress or drupal get the fix only after their maintainers publish an updated image, so check the upstream image before you rebuild.

Locate PHP projects across your GitLab groups

Any project that runs PHP to serve web requests needs a fixed version, including the ones that are easy to forget, such as an internal dashboard, a marketing site, or a CI job image. Finding them takes two views: a static search of what your repositories declare and your images contain, and a runtime inventory of what actually runs. GitLab covers the static view with three features. Use them in this order.

1. Container scanning

Container scanning reports vulnerabilities in operating system packages, so it finds PHP installed with apt, apk, or dnf, and it reports the two CVEs once they reach the scanner's vulnerability database. To turn it on for a project, add the template to .gitlab-ci.yml:

include:
  - template: Security/Container-Scanning.gitlab-ci.yml

container_scanning:
  variables:
    # Experimental, Ultimate only: also reports runtimes not installed by a package manager
    CS_BINARY_SCAN: "true"

You can also use a group-level scan execution policy to turn container scanning on for every project in a group at once. When a policy is enabled, the next time a pipeline is run, it will be run with container scanning. Container scanning results for the default branch can be reviewed in the Vulnerability Report after the scanner completes.

Additionally, you can set up Merge Request Approval policies with container scanning. With merge request approvals in play, if a code change contains PHP2Shell, the policy will block a merge without approval from the selected parties.

By default, container scanning detects only PHP installed as a package. The official images compile it from source, leaving no package record, so a project can pass container scanning and still run a vulnerable version.

Container scanning analyzer 8.7.0, released on October 29, adds an experimental binary scan that reports runtimes installed without a package manager, which closes that gap. It is available on Ultimate and off by default: set CS_BINARY_SCAN to "true" to turn it on. If you leave it off, Orbit and code search below cover it.

2. GitLab Orbit

GitLab Orbit indexes your projects, pipelines, security findings, and source code, including PHP, into a queryable graph, so you can ask an agent to review an entire group. With Orbit enabled on the top-level group, an agent can list every project that contains PHP code, including the projects that compile PHP or use the official php images, which container scanning misses. Orbit doesn't yet parse Dockerfiles (feature proposal issue #1416), so use that list to scope the code search in the next step, which confirms the base image and version in each project. You can also ask Orbit which projects have open findings for CVE-2026-109618 or CVE-2026-109619 once scans have run. Orbit indexes the default branch only and is in beta for Premium and Ultimate on GitLab.com and GitLab Self-Managed.

3. Code search

Exact code search can find the two types of files that indicate a PHP project. Run these at the top-level group with regular expression mode enabled (the .* toggle at the right of the search box):

• Container base images: This query searches for every file GitLab detects as a Dockerfile. The pattern matches any image whose name or tag contains php, so it also finds derivatives such as wordpress, frankenphp, or bitnami/php-fpm. Expect a few false positives that you can rule out by reading the result: lang:Dockerfile ^FROM.*php
• Composer projects: This query finds projects that use PHP even when they have no Dockerfile, or when the Dockerfile takes its image from a build argument such as FROM $BASE_IMAGE:

file:composer\.json$

Exact code search covers the default branch only. On GitLab.com it is on by default for paid plans. On GitLab Self-Managed instances an administrator must install Zoekt first. GitLab Dedicated doesn't support exact code search.

For each result, confirm the version. You can check the FROM php:<tag> line, or run php -v in the running container, and compare it with the table above. The group dependency list shows Composer packages and framework versions but not the PHP runtime version.

4. Compare with your runtime inventory

Container scanning and code search show what your projects declare and build, not what runs. PHP can run outside your GitLab projects, for example on web servers with mod_php or PHP-FPM enabled only in their configuration, on hosts configured by hand, or in containers that still run in clusters without an active pipeline. Compare the static results with an inventory of your running workloads from your configuration management (Ansible, Puppet, Chef, etc.), cloud, or VM inventory tools. For Kubernetes, operational container scanning scans the images that run in your clusters, with the same package-only detection as container scanning. Treat every PHP runtime that appears in only one of the two views as a gap to investigate.

Remediate at scale with GitLab Duo Agent Platform

GitLab Duo can run the discovery steps above for you, across all your projects, through Orbit and code search. From there, let agents do the repetitive work and keep people on review and merge.

Automate the fix. Use Agentic Chat to create an epic with one issue per affected project and assign the Developer Flow to each. The flow opens a merge request that updates the base image tag or package to a fixed version. A flow trigger on the Assign event starts it the moment the issue is assigned. Turn on automatic reviews with the Code Review Flow to get a first review that follows your custom review instructions. Review and merge each merge request, then close the issue once the fixed version is deployed.

Enforce with policies. The merge request approval policy from Step 1 blocks any change that reintroduces a vulnerable PHP version without approval, and the scan execution policy keeps container scanning on in every project so a regression appears in the vulnerability report.

Searches find candidates, but a security or platform engineer still has to review each match, resolve CI includes, and tell what the code declares apart from what actually runs. An example agent skill from the GitLab Developer Advocacy team handles this work. It runs on GitLab Duo Agent Platform and with agents that use GitLab MCP server or the glab CLI, such as Claude Code, Codex, and Cursor.

The skill checks container images, CI jobs, OS packages, configuration management variables, Composer pins, and environments. It returns a cited findings table with deployment evidence, and open gaps. The skill is read-only: It makes no changes, and remediation merge requests remain a separate, reviewed step. It cannot confirm what is running on hosts it cannot access, so verify those directly. Adapt the skill to your own environment before you use it.

Check for compromise

Reports of exploitation differ. Security researchers reported an exploit attempt on October 17, 2026, before public disclosure, and further reports of exploitation followed the release. The PHP project identified no functional exploits at the time of release. If an internet-facing server ran a vulnerable PHP version, review its logs from October 17, 2026, onward for:

• Multipart requests followed by memory-limit errors or repeated HTTP 500 responses
• Unusually large or malformed PHPSESSID response cookies
• Unexplained PHP worker restarts, or shell processes started by PHP workers
• Unexpected files on the PHP web server, outbound connections from PHP workers, or unexpected cron entries

A successful exploit can return HTTP 200 and leave the worker running, so an absence of errors doesn’t rule out compromise. If you find signs of exploitation, isolate the server and start an incident investigation. Where possible, capture memory, disk, and logs before patching and restarting. Restarting PHP workers may erase evidence. CVE-2026-109619 can also expose process memory, even when an attacker doesn’t achieve code execution. Rotate any credentials, tokens, or keys the PHP process could read.

Note: Scanners may flag the Gemnasium dependency scanning image (gemnasium:6), deprecated since GitLab 17.9, for its bundled PHP. It is used only as a command-line tool and never handles HTTP requests, so it is not exploitable, and a fixed version has already shipped under the same tag.

Timeline

Представьте: посетитель интернет‑магазина спрашивает в чате, сколько дней на возврат. Бот бодро отвечает: «30 дней, без вопросов». На сайте при этом написано 14. Через три недели покупатель приходит с товаром и скриншотом переписки. Спорить с ним бесполезно: «ваш же консультант сказал».

Мы делаем для сайтов ассистента. Это чат, который отвечает на вопросы о магазине или компании — доставка, оплата, условия, контакты, статус заказа — и зовёт живого оператора, когда сам не справляется. Когда собирали первую версию, быстро стало понятно: «не знаю» — далеко не худшее, что может сказать такой чат. Худшее — уверенная неправда. Языковая модель прекрасно умеет звучать убедительно. А вот отличать «знаю» от «похоже на правду» у неё получается гораздо хуже.

Ниже расскажу, как мы развели в ответах разговор и факты, где решение принимает код, а не модель, и где мы до сих пор верим модели на слово. Код на Python, но идея переносится на любой стек.

• 41-year-old set on ‘dream to be able to race in Africa’

• Rwanda to host despite criticism of human rights record

Lewis Hamilton has said he hopes to remain racing in Formula One until he can take part in the African Grand Prix which is set to take place in Rwanda in the 2030s. The seven-time world champion believes it is right that F1 is to hold a race there despite criticism of the country’s human rights record.

Last week F1 confirmed Rwanda was to host the African Grand Prix having concluded a 10-year agreement for the meeting on a purpose-built circuit, with the first race set for the early 2030s. It is lined up to be held on a circuit in Bugesera, 25 miles south of the capital, Kigali, which has yet to be constructed and marks a return to the African continent for the first time since it last held a race at Kyalami, South Africa in 1993.

City’s council passed Romanch’s Law, named after teen who died when he was ejected from carriage earlier this year

The New York city council has voted to ban horse-drawn carriage rides in the city by 2028 in a decision that has widespread implications for hundreds of jobs.

The council voted on Thursday on Romanch’s Law, legislation named in memory of 18-year-old Romanch Mahajan, who died this summer after he was ejected from a runaway horse-drawn carriage in Central Park.

Musk’s investment into the new media empire comes after Paramount’s $111bn merger with Warner Bros Discovery

Elon Musk has reportedly invested approximately $1bn in Skydance, the new media empire led by David Ellison.

The deal comes alongside fresh equity infusions from existing investors, including board member Laurene Powell Jobs, Steve Jobs’s widow. Semafor first reported the news.

Republican Mike DeWine said ad featuring estranged mother of Democrat Amy Acton was ‘wrong’

Donald Trump issued a swift rebuke against Mike DeWine, Ohio’s Republican governor, who on Thursday said gubernatorial candidate Vivek Ramaswamy was “wrong” to release an ad critical of his Democratic rival Amy Acton that featured her estranged mother.

DeWine had implored Ramaswamy to take the ad down, but hours later, Trump urged Ramaswamy to keep airing it and accused the Republican governor of disloyalty.

Изначально я планировал написать классический лонгрид про своё участие в чемпионате ROADEF 2026. Там я занял "почётное" 48-е место из 78 валидных участников, так что и кодом есть чем делиться, и историями с полей. Но тут на глаза мне попалось объявление на Кворке: какой-то суровый заказчик предлагал за 500 рублей разработать с нуля связку «Браузер + Поисковик».

Я сидел, смотрел на это и думал: кого же мне порадовать? Дорогого читателя или этого щедрого продюсера с биржи фриланса? Решение пришло компромиссное: мы сделаем Поисковик-Браузер для хабровчан и пробросим через сервер в народ. А заказчик с Кворка пусть оставит 500 рублей, купит мороженку.

(барабанная дробь) Встречайте SBrowserV2 — легковесный асинхронный поисковый движок.

OpsDeck за три недели: ИИ-агент, который сам ходит на хосты, и ещё восемь релизов

Три недели назад я писал здесь про OpsDeck — бесплатное открытое приложение, в котором DevOps-инженер держит в одном окне терминал, Kubernetes, SSH, базы, заметки и прочее, по чему обычно разложены пять вкладок и три консоли. С тех пор вышло девять релизов, в репозитории появились внешние контрибьюторы, а приложение заметно изменилось — настолько, что отдельный рассказ назрел.

Коротко о главном: локальный ИИ перестал быть подсказчиком команд и научился сам ходить на хосты — по правилам, которые вы ему задаёте, и с вашим разрешением на любое изменение. Об этом — первая половина поста. Дальше — что ещё появилось и чему меня научили ваши баг-репорты.

Datacenter and chip costs fueled sharp decrease in AI sector although non-AI stocks still had a strong showing

The latest shudder for stocks in the AI industry pulled markets lower worldwide on Thursday, while oil prices hit their highest levels since May on worries about the war with Iran.

The S&P 500 fell 1.2% for its worst day since July, while the Nasdaq composite tumbled 2.6%. The weakness for tech stocks was so overwhelming that it masked strength for the rest of the US market, in which the majority of stocks actually rose. The Dow Jones Industrial Average added 11 points, or less than 0.1%.

• 41-year-old set on ‘dream to be able to race in Africa’

• Rwanda to host despite criticism of human rights record

Lewis Hamilton has said he hopes to remain racing in Formula One until he can take part in the African Grand Prix which is set to take place in Rwanda in the 2030s. The seven-time world champion believes it is right that F1 is to hold a race there despite criticism of the country’s human rights record.

Last week F1 confirmed Rwanda was to host the African Grand Prix having concluded a 10-year agreement for the meeting on a purpose-built circuit, with the first race set for the early 2030s. It is lined up to be held on a circuit in Bugesera, 25 miles south of the capital, Kigali, which has yet to be constructed and marks a return to the African continent for the first time since it last held a race at Kyalami, South Africa in 1993.