• Training and serving frontier AI models depends on fast, reliable networks that move data between GPUs without wasting compute cycles. 
• To meet this challenge at scale, Meta designed MetaRoCE – a clean-sheet RDMA transport protocol purpose-built for AI workloads on commodity Ethernet. 
• We’re releasing the MetaRoCE specification, a reference software implementation and a compliance test suite through the Open Compute Project (OCP) to enable the broader industry to adopt, implement, and build on it.

At Meta, we’ve been a strong driver behind the industry’s growing consensus that Ethernet should be the fabric of choice for AI infrastructure. We’ve already shown that RoCE can power distributed AI training at scale. Now, we’re building on that work with MetaRoCE, protocol designed from the ground up for Ethernet at million-GPU scale.

We’ve scaled up clusters of hundreds of thousands of GPUs, spread over multiple data centers and regions. Whether these clusters are training the next frontier model or serving inference at global scale, the network is in the critical path.

 Collective operations like all-reduce and all-to-all synchronize thousands of accelerators during training, and the slowest transfer sets the pace for the entire job. In inference, low-latency communication between distributed model shards directly impacts response times for hundreds of millions of users. Even small amounts of network friction directly strand significant compute capacity.

Standard RoCE expects the network to deliver every frame in order, leveraging PFC and discouraging the packet spraying that provides performance in multiplane and large scale networks.  MetaRoCE is built to provide high throughput, low tail latency, and operational simplicity as the network grows in the number of accelerators and the distances between them.

How MetaRoCE Works

MetaRoCE’s core insight is simple: The fabric sees packets, but the NIC sees intent. Traditional architectures centralize intelligence in the fabric, relying on switches to enforce losslessness and maintain order.  

By moving intelligence to the endpoint MetaRoCE decomposes the network into many fine-grained logical paths, each with its own real-time telemetry – per-path RTT, ECN state, and utilization. This visibility unlocks capabilities that are difficult to achieve with traditional RDMA.

Native Out-of-Order Delivery

MetaRoce sprays packets across many paths, so they arrive out of order by design. The transport treats out-of-order arrival as the normal case. Every packet carries its own destination, so data is written straight to its final memory location as it lands, with no reorder buffer and no head-of-line blocking. 

Writes carry their destination in every packet. Sends carry the match to a posted receive buffer, so a Send lands correctly even when the messages ahead of it have not arrived, and without a round trip to learn where the data goes. Collective libraries can use two-sided messaging where it suits them rather than reducing everything to Write.

Native Multipathing

MetaRoCE gives each connection first class paths and sprays across them packet by packet. Each path carries a distinct UDP source port as its ECMP entropy, which the NIC can change at any time to move traffic off a bad route. On multiplane fabrics, plane selection falls entirely to the NIC, and the fabric is used only as well as the NIC sprays. Because each path keeps its own window and round trip estimate, the transport can tell congestion from failure and rebalance explicitly, so a hot or broken link slows one path instead of stalling the connection.

Loss Tolerance by Design

MetaRoCE treats the Ethernet fabric as lossy and does not ask it to be otherwise – no PFC, no pause frames. Because each path carries its own ordered sequence, a gap in its 256-bit selective acknowledgment bitvector is evidence of loss rather than of reordering. In other protocols a SACK mostly avoids resending data that already arrived; here it triggers retransmission of exactly the missing packet, on the path that lost it, the moment the gap appears.

Congestion Control From Both Sides

MetaRoCE combines a conventional ECN-based, sender-driven AIMD congestion control with receiver-driven fair-share rate hints. Windows are kept per path as well as per connection, so a congestion mark trims the path that saw it and steers the next packets toward paths that are clear. In every acknowledgment, the receiver returns the share of its inbound bandwidth it has allocated to that sender, so senders approach the right speed directly rather than searching for it. Incast resolves in one or two round trips, with better fairness and lower tail latency.

Topology Independence

MetaRoCE asks the fabric for two things every switch already has, ECN marking and ECMP. It does not require packet trimming, in-network telemetry, credit-based flow control, or switch-side spraying, and it does not break when a fabric offers them. The same transport runs over fat-tree, multiplane, deep-buffer, and shallow-buffer fabrics, and over vendor clouds whose configuration you don’t control. Nothing proprietary is involved, so the fabric stays free to optimize for cost and cabling.

Unified Connections at Scale

A queue pair (QP) carries both an ordered stream of messages and bandwidth. Traditional RDMA gets more of either by opening more QPs (dozens per node pair), each with a congestion window blind to the rest and its own state on the NIC. 

MetaRoCE separates the two. A single connection carries many independent ordered streams above, one per communicator or collective, and many paths below, under one congestion controller. The connection state stops growing with the parallelism of the workload.

The application layer remains mostly untouched – existing RDMA Verbs APIs and software stacks work without modification. Enhanced features like multiplane support are supported through extension APIs.

Meta-RoCE in Practice

To accelerate hardware validation, we worked with AMD to implement MetaRoCE on their Pensando programmable NICs.  

On a 64-node AMD GPU cluster running RCCL collectives, we directly compared MetaRoCE against RoCEv2 across all-reduce and all-to-all operations. The results were consistent with the design goals:

MetaRoCE consistently delivers higher throughput and lower flow completion times than RoCEv2.

Under packet loss conditions that would degrade RoCEv2, MetaRoCE maintains ~86% throughput at 1% packet loss and continues delivering useful bandwidth even at extreme 10% loss rates – converging gracefully rather than collapsing.

Multiplane validation across 4-plane and 8-plane topologies with up to 4,000 concurrent connections confirmed that throughput scales linearly with plane count.

During simulated plane failures, the protocol demonstrates graceful autonomous recovery – traffic redistributes without application involvement or operator intervention.

These results support MetaRoce’s core design choice. By designing for loss from day one and pushing intelligence to the edge, you get a transport that performs better in ideal conditions and degrades gracefully when things go wrong.

Open By Design

AI infrastructure benefits from shared standards that accelerate innovation across the ecosystem. MetaRoCE extends the same open, multi-vendor philosophy that the Open Compute Project (OCP’s) Ethernet Scalable Unified Network (ESUN) initiative established for the fabric into the transport layer. 

That’s why we’re opening MetaRoCE:

Open specification via OCP: The full protocol spec is being contributed to OCP, available for any vendor to implement and build interoperable hardware.

Multiple NIC implementations: MetaRoCE is designed to run across diverse NIC architectures – programmable and fixed-function alike. We’ve proven it on AMD Pensando hardware, with additional implementations underway from other vendors.

Production compliance suite: We have developed a compliance suite that gives hardware vendors the tools to prove their implementations match the protocol spec. 

Software reference implementation: Our libsoftmetaroce library provides a complete, functional transport stack that runs on commodity Linux over standard UDP sockets without specialized hardware. It serves as the authoritative behavioral model for silicon development and the foundation of our unified compliance framework.

The Road Ahead

With MetaRoCE, we’ve made strong progress on scale-out networking – high-performance, resilient transport within the data center on commodity Ethernet. But AI infrastructure spans multiple distance and latency regimes, and each brings distinct challenges we’re actively working on:

Scale-up: Within a rack, accelerators trade small messages where every nanosecond matters. MetaRoCE removes two main sources of latency, the reorder buffer and PFC. We are now optimizing the fast signaling path for short memory operations issued directly from one processing element to another.

Scale-across: Scale-across enables a single job to span buildings thousands of kilometers apart. Round trips stretch into milliseconds, and small differences between paths add up. Treating paths as first class entities is what lets MetaRoCE adapt, preferring the uncongested ones and seeking fairness at every level. The work ahead is in fairly sharing contended long-haul links.

Storage/Kv-cache use cases: Distributed storage invites incast,where a single read fans out to many servers and they all reply at once. Receiver-driven rate hints let whichever side is receiving(a storage server taking writes or a client taking reads) moderate the inbound rate, whether the request went to ten servers or a thousand. The new dimension is keeping that rate accurate with networks of varying speed and requests of varying size.

Help Us Build the Future of Ethernet for AI Infrastructure

In October, we’ll release the MetaRoCE specification, a DPDK-optimized software reference implementation, and our production compliance framework at the 2026 OCP Global Summit. 

We’re building this in the open because the challenges ahead benefit from broad industry collaboration. If you’re building NICs, switches, or AI infrastructure, we invite you to join us.

The post MetaRoCE: A New RDMA Transport Built for AI-Scale Ethernet appeared first on Engineering at Meta.

• MTIA 300 is the first of Meta’s family of in-house training and inference accelerators optimized for training ranking and recommendation models.
• We’re sharing how MTIA 300’s built-in NIC chiplets allow it to meet the communication needs associated with training recommendation models with superior performance over general-purpose GPUs.
• By co-designing MTIA’s communication library, HCCL, alongside the chip we’ve taken a fundamentally different approach to chip design and made communication a first-class citizen.

Deep learning models may deliver personalized content—from short videos to friend posts—to people on apps. As these models have grown in complexity, so has the importance of the compute that trains them, and the network that connects those accelerators.

Training recommendation models is a unique infrastructure challenge. Unlike large language models, which need enormous floating-point throughput, recommendation models are bottlenecked by a need for fast and efficient communication between the accelerators that train them. Their embedding tables can contain over 99% of the model’s parameters, requiring hybrid parallelism that generates frequent AllReduce, AllToAll, and AllGather collectives across hundreds of accelerators. On chips like GPUs these communication operations compete with training computation for the same resources, often leaving expensive hardware underutilized.

We’ve addressed this challenge starting on the Meta Training and Inference Accelerator (MTIA), our family of homegrown AI chips, with MTIA 300, the first of the MTIA family optimized for training recommendation and ranking models. By co-designing MTIA 300 with HCCL, a communication library co-designed with the hardware from scratch, we’ve made communication a first-class citizen in the chip’s design, not an afterthought handled by general-purpose compute cores.

Integrating the Network Directly on the Chip

With MTIA 300, the network interface lives inside the chip package itself (see Figure 1). Two network chiplets, each containing six custom 800 Gbps RDMA NICs, provide 1.2 TB/s of total I/O bandwidth without ever crossing a PCIe bus. This eliminates the host-device-NIC bottleneck present in traditional GPU architectures, where the CPU must mediate between the accelerator and the network. (More details about the silicon design are available in our recent paper from the ISCA 26 conference.)

Because we use the same 12 Ethernet-based NICs for scale-up communication (within a rack of 16 nodes, at up to 1 TB/s) and scale-out communication (across racks, at 200 GB/s), we can flexibly partition the NICs to adjust to changing needs. 

As model requirements shift, we can reconfigure this split by reconfiguring the network rather than changing the hardware. To minimize per-transaction latency, we introduced express doorbells. The work request write itself serves as the doorbell, eliminating an additional memory read and saving ~800 ns per operation.

Offloading Communication From the Compute Grid

On GPUs, libraries such as NCCL execute collective communication as GPU kernels that consume streaming multiprocessors—the same hardware needed for training computation. When collectives and training kernels run simultaneously, both slow down.

MTIA 300 takes a different approach. Alongside its 12×6 grid of processing elements (PEs) for computation, the chip includes 16 dedicated message engines (MEs) that handle all communication independently.

Each ME contains: 

• an RISC-V core for orchestrating w
• an NIC interface that routes requests to the correct NIC
• a near-memory compute (NMC) block that performs reductions at 128 bytes/cycle 

Positioned at the chip edges next to HBM and cache, the NMCs collectively deliver more than 2.8 TBs of reduction throughput—more than double the I/O bandwidth—enabling line-rate execution of AllReduce and ReduceScatter collectives without touching the compute grid.

The result is near-perfect isolation. Running large GEMMs concurrently with collective operations introduces less than 0.5% degradation to compute throughput, as opposed to traditional GPUs that can see over 20% degradation because communication is handled by the same GPU resources.

A Compiled-Communication Model

Our communication library, HCCL, was co-designed with MTIA 300. Rather than driving communication from the host during execution, HCCL compiles each collective into a complete set of subgraphs—arrays of work-queue entries with explicit dependencies—dispatched to the MEs for fully autonomous execution. Once work reaches the device, the host is uninvolved. Figure 2 shows how the CPU is no longer involved after copying the instructions into HBM.

This compiled model integrates naturally with PyTorch’s c10d and torchcomms interfaces. Collectives traced through torch.compile are compiled into a single graph alongside compute operators. HCCL selects topology-aware algorithms that exploit the asymmetric bandwidth between scale-up and scale-out, minimizing cross-rack traffic where bandwidth is constrained. For inference workloads, we developed additional paths: one-sided communication where PEs submit work directly through express doorbells, and device-triggered collectives where compute kernels signal hardware-offloaded communication on a parallel stream without breaking graph execution.

Performance in Production

HCCL achieves up to 940 GB/s of communication bandwidth within a single rack. On a 150-billion-parameter production-recommendation model running across 40 accelerators, MTIA 300’s total communication time is 3.9 times faster than the equivalent GPU cluster.

MTIA 300’s design enables further co-design strategies: Its 216 GB of HBM3E allows larger local batch sizes (reducing trainer count and communication overhead); its 1:1 CPU-to-accelerator ratio enables CPU offloading of numerically intensive optimizer operations; and its high network bandwidth lets us use higher-precision datatypes to maintain precision. 

Looking Ahead

While MTIA 300 was designed for training recommendation models, the architectural principles—integrated networking, offloaded collective execution, and system-level co-design of compute and communication—position it for a broader set of workloads. As AI inference evolves toward reasoning, agentic, and long-context use cases, the communication demands a shift: Messages become smaller, more frequent, and latency-sensitive, with tighter per-collective budgets. 

An architecture that treats the network as a first-class system constraint, optimizing not just bandwidth but also latency and message rate, is well suited to meet these emerging demands. The patterns established in MTIA 300 and HCCL are the foundation for Meta’s next-generation AI silicon.

Learn More About MTIA 300 

To learn more about MTIA 300’s silicon design and the work detailed here, read our papers:

• “MTIA 300: Meta’s First Training Chip Featuring Built-in NICs and Collective Offloading Engines “ (From ISCA ‘26)
• “HCCL: Collective Communication for Meta Training and Inference Accelerators” (to be published at SC26).

The post MTIA 300: Meta’s First Training Chip with Built-in NICs and Communication-Offloading Engines appeared first on Engineering at Meta.

WhatsApp is committed to helping people stay safe while protecting the privacy of their messages. As scam tactics evolve — from impersonation to social engineering to AI-generated lures — we’re always evolving as well, so that our protections stay ahead of scammers while protecting people’s personal messages with end-to-end encryption.

Today, we’re sharing an early look at Scam Alert, a new, optional feature that runs an on-device machine learning model to alert a user about potential scam messages. No message content leaves the device for classification or is auto-reported to WhatsApp, Meta, or anyone else. The feature complements end-to-end encryption while enabling a user-controlled, optional scam alert when the model believes there’s a likely scam.

Before we make this feature available to all WhatsApp users, we are publishing this early technical overview alongside the feature’s limited rollout in Beta, and will continue working with our Bug Bounty community to stress-test this system. To help validate our implementation, we welcome feedback from the broader security research community.

Design Principles

Recent advances in on-device machine learning models make it possible to run accurate text classification entirely on mobile hardware without the performance, battery, or model-size tradeoffs that previously made on-device classification less practical. Scam Alert is well-suited to this approach: The model is small enough to run on-device, simple enough to publish for independent review, and effective without server-side components. The architecture we chose reflects a set of deliberate choices about what this system can and cannot do.

To that end, we designed Scam Alert to meet the following principles that adhere to the core guarantees of end-to-end encryption:

• On-device only: The model and the message data it processes all remain on the device.
• No automatic reporting: WhatsApp is unable to initiate sharing of any user data without the user’s action. The only way message content, or even the fact a scam was detected, reaches our servers is if the user explicitly chooses to report it, which is consistent with how user reporting works on WhatsApp.
• User control: Scam Alert is a user-controlled tool that provides additional information to the user, and the user can turn it off or on at any time. 

How Scam Alert Works

Scam Alert is optional. Once the user turns it on, Scam Alert downloads a machine learning model to the device, where it runs inferences to classify whether incoming messages from non-contacts match known scam patterns. The model is trained on patterns observed in scam conversations from reports that users have sent to us. It performs probabilistic classification based on conversational structure and linguistic signals. No content is automatically reported to WhatsApp, Meta, or any third party. 

If the model identifies a message as a likely scam attempt, the user sees a warning in the chat, which is not visible to the other person. From there, the user can decide what to do: block, report, or continue the conversation. If they decide that a warning is incorrectly flagged, the user can mark the chat as trusted, in which case the warning is removed and Scam Alert will not flag that chat again. If a user marks that they trust a chat, they can also opt in to share the last 5 messages received with WhatsApp to help improve the feature’s accuracy. 

Foundational Safeguards

To uphold the principles above, we designed Scam Alert with the following foundational requirements and safeguards, with each architecturally enforced and independently verifiable by security researchers through an expanded bug bounty program and by users themselves through in-app logs. 

1. On-Device Processing and Privacy-Preserving Analytics: All inference happens on-device and no message content leaves the user device for classification. The minimal telemetry needed to measure whether the feature is working (i.e., aggregate and anonymous warning counts and user action counts) is processed within a confidential computing environment and sent to WhatsApp as differentially private aggregates. The confidential federated analytics pipeline is built on top of confidential virtual machines (CVMs), a type of Trusted Execution Environment (TEE). We chose this approach so its behavior can be independently verified.
2. No Targeted Model Delivery: Neither Meta nor WhatsApp can deliver a specific model to a specific user. Every model version, including experimental variants, is published on a public transparency ledger before it is deployed.
3. Verifiable Model Behavior: We publish model weights so that independent security researchers can verify that we’ve purpose built this for scams only.

The rest of this post details the technical implementation of each requirement.

On-Device Processing and Privacy-Preserving Analytics

As referenced above, all inference happens on-device. But we need to know that the feature itself is working – i.e., it is indeed catching real scams – and know if we need to update it to stay ahead of constantly evolving scams and improve the model over time.

To that end, our approach follows a set of data minimization principles. Message content does not leave the device, and logging is limited by design to only the signals that are needed to measure whether the feature is working as intended. Even those signals are processed within a confidential computing environment built on TEEs, which ensures that processing occurs in a secure environment that no one, including Meta and WhatsApp, can access. Our experience building and securing systems like Private Processing has informed the design of this system. Only anonymous, differentially private aggregates are made available to Meta and WhatsApp. Differential privacy works by adding carefully calibrated noise to provide a mathematical guarantee that adding or removing any single person’s data has a negligible effect on the anonymous, aggregated numbers. Hence these aggregates show how the feature performs across the population while telling us nothing about any individual.

For Scam Alert, that data is limited to two categories of approximate, aggregate counts:

• Warning Counts – how many times the on-device model surfaced a scam warning. This tells us whether the model is triggering at the right rate, which is essential for measuring precision and catching regressions across model versions.
• User Action Counts – when a user sees a warning, they can trust the sender or block and report. We log which action category was taken as an aggregate count. This tells us whether users find the warnings accurate, which is essential for measuring false positive rate.

Confidential Federated Analytics

To anonymize these warning and user action counts, we built a confidential federated analytics pipeline designed around the following privacy and security guarantees, each architecturally enforced and externally verifiable:

• On-Device Data Minimization: For Scam Alert, raw signals do not leave the device. The client aggregates them locally into counts and sends only those aggregated counts. These metrics are sent at randomized times, contain no device identifiers and limit any timestamp information to coarse time intervals. This ensures that neither the act of transmitting these metrics nor the metrics themselves can be used to identify a user.
• Confidential Processing: These metrics are processed within TEEs, secure hardware environments built on CPU-based confidential virtualization technologies, which allow attestation of software based in a hardware root of trust. Before any data is transmitted, the client checks these attestations and confirms them against a third-party log of acceptable binaries. Data is encrypted between the client and the TEE, so that no one in between, including Meta, WhatsApp, or any third-party relay, can access it.
• Secure Aggregation: Individual device metrics are not readable by Meta, WhatsApp, or anyone outside the TEE. They are merged into running aggregates, and only aggregated statistics, above a minimum cohort size and with differential privacy noise applied, are made available to Meta or WhatsApp. For Scam Alert, the device sends pre-aggregated counts directly to TEE for secure aggregation.
• Enforceable Guarantees: Before any data is transmitted, the client verifies that the code running in the TEE matches what was published on the third-party ledger and that the privacy parameters (such as differential privacy ε and δ, and k-anonymity thresholds) meet locally enforced guardrails. If verification fails or the privacy parameters are insufficient, the client refuses to transmit data. Any attempt to modify the processing guarantees either causes the system to fail closed or is publicly discoverable.
• Encrypted Recovery Checkpoints: Because the pipeline aggregates data over long periods, the system periodically saves encrypted checkpoints of its in-progress aggregates, so that a crash does not force a measurement to restart from scratch. These checkpoints contain only the partial aggregate counts already being computed, and they are encrypted: the keys never leave the TEEs, so only confidential federated analytics TEEs running the same attested binary can decrypt a checkpoint, and neither Meta nor WhatsApp can read it. Checkpoints are retained only for the bounded period needed to recover.
• Non-targetability: An attacker cannot target a particular user without attempting to compromise the entire system. All metrics are routed through an OHTTP relay that strips the requester’s IP address, and authenticated using anonymous credentials so that the system can verify that metrics come from a legitimate WhatsApp client without knowing which one. This limits the impact of small-scale attacks by ensuring that they cannot be used to target the data of a specific user.
• Verifiable Transparency: we will provide in-app capabilities for users to review what data was shared with the confidential federated analytics pipeline, the privacy parameters applied (such as differential privacy ε and δ, and k-anonymity thresholds), and details of how each secure session was established. We will be publishing the CVM image binary powering the pipeline, along with the source code of its privacy-relevant components, so that security researchers can independently verify that the published code is exactly what runs in the TEE. We will be expanding our Bug Bounty program to include the confidential federated analytics pipeline and will publish a detailed engineering white paper on its design.

The foundations of this pipeline were established in Meta’s peer-reviewed federated analytics work, publicly outlined in “PAPAYA Federated Analytics Stack: Engineering Privacy, Scalability and Practicality” (USENIX NSDI 2025). Here, we describe how Scam Alert applies and extends that foundation.

How Confidential Federated Analytics Works

The pipeline works as follows:

1. On-Device Data Collection: Data is collected and stored in a dedicated local store, isolated from other application data. The confidential federated analytics system can only access data that the application has explicitly made available to it. Hardcoded privacy guardrails enforce data lifetime, scope, and access. For Scam Alert, this is only counts of warning events and user actions.
2. Job Selection: At randomized intervals, when the device is idle and subject to a self-enforced daily resource limit, the client connects to the application server via OHTTP. The client authenticates using anonymous credentials that prove it is a legitimate WhatsApp client without revealing which one, and fetches the list of active jobs for the pipeline. For each job, the client checks whether the privacy parameters (ε, δ, and k-anonymity thresholds) meet locally enforced guardrails, whether the device has new metrics to send, and whether participating would exceed its daily limits. The client can reject any job that does not meet these criteria.
3. Local Transformation: For jobs the client accepts, it retrieves the relevant data from local storage and performs the requested on-device aggregation by converting raw signals into anonymous counts over the specified time period (e.g., daily warning counts, daily user action counts). Only the aggregated counts are transmitted; the raw signals remain on the device and are automatically deleted after a retention period.
4. Attestation, Authentication, and Session Establishment: The client establishes a Remote Attestation + Transport Layer Security (RA-TLS) session with the orchestrator TEE. The attestation quote contains measurements of the orchestrator, which the client cross-checks against a third-party transparency ledger to ensure it is connecting only to code that satisfies our verifiable transparency guarantee. The client authenticates using anonymous credentials and the connection is routed via a third-party OHTTP relay that strips the requester’s IP address. The relay cannot read the metrics, as they are encrypted end-to-end between the device and the TEE.
5. Orchestrator TEE — Ephemeral Processing: The encrypted metrics arrive at a stateless orchestrator hosted on a TEE. The orchestrator validates that the privacy configuration from the client matches the job’s configuration, batches metrics from multiple devices, and forwards the batched metrics to the appropriate aggregator TEE. For Scam Alert, these metrics are pre-aggregated counts.
6. Aggregator TEE — Secure Aggregation: The aggregator TEE merges incoming metrics into running histograms. Individual metrics are discarded after aggregation. Periodically, the aggregator enforces k-anonymity thresholds to suppress results with too few contributors and applies differential privacy noise. The number and timing of releases are limited to ensure the overall privacy budget (ε, δ) is not exceeded across all releases. Only the noisy, thresholded aggregates are sent to WhatsApp.
7. Datastore: Only differentially private, anonymous aggregates, such as total warning counts and action rates across all users, leave the TEE boundary. These aggregates contain no message content, no per-user data, and no conversation-level signals. They are used solely to measure whether the feature is working as intended.

The confidential federated analytics pipeline is built so that, by the time any totals reach WhatsApp, the counts have been aggregated across many users and had differential privacy noise added – so we only ever see approximate counts of how many warnings were shown and how many were acted on. We will not know what the messages was, who sent or received them, or which conversation triggered a warning.

Threat Model and Defense-in-Depth

This confidential federated analytics pipeline operates in a highly adversarial environment. Our threat model accounts for three categories of attacker: third-party or supply chain vendors with access to system components, malicious or compromised insiders with access to infrastructure, and external actors attempting to exploit the pipeline’s attack surface.

External actors attempt to intercept or extract unaggregated data in transit or during processing.

Data in transit is encrypted between the device and the TEE and routed through a third-party OHTTP relay. The relay’s role is limited to stripping the client’s IP address – it cannot decrypt, inspect, or modify the data itself. Because this data is already not visible to the third-party, a compromised relay cannot access the data or associate datasets with a specific user. During processing, data is protected by TEE code isolation, with entry points limited to a small set of reviewed components.

Insiders with infrastructure access attempt to access unaggregated data within the TEE.

The TEE prohibits remote shell access, including from the host machine. Neither Meta engineers nor networked systems can gain access to the CVM shell at runtime. Software is built exclusively from checked-in source code and artifacts, where any change requires multiple engineers to modify the build artifacts or build pipeline. All code changes are auditable, enabling both continuous internal audits and external security researchers to inspect our binaries. Unaggregated data is never readable outside the TEE; when stored, it is encrypted under keys released only to a TEE running the same attested binary, and retained only for a bounded period before being merged into running aggregates and discarded.

Attackers with physical or remote access interfere with the TEE to bypass confidential processing guarantees.

Because TEE guarantees are not absolute, we apply defense-in-depth: encrypted DRAM, CVM hardening, enhanced host monitoring, and OHTTP relay routing that prevents directing a specific user’s data to a specific machine. A targeted attack would require compromising the entire system in a way that is publicly discoverable through verifiable transparency.

No Targeted Model Delivery

The model is downloaded from a CDN, not hardcoded into the app, so that improvements in accuracy and coverage of emerging scam tactics can reach people without requiring a forced app upgrade. And to that end, we are also ensuring that there is no path to deliver a different model to a specific user.

Every model version — including its SHA-256 hash — is published on a third-party append-only transparency ledger before it is served to anyone. An append-only ledger is a public log where entries can be added but never modified or deleted, ensuring a tamper-evident history that researchers can inspect.

We designed the model download system around three guarantees:

1. Auditability: Every model version is publicly recorded on a third-party append-only transparency ledger before it is served to any user. The ledger is tamper-evident, so entries can be added but never modified or deleted. Delivering a targeted model would require publishing it on a ledger that anyone can inspect, making the attempt publicly discoverable. To check the ledger, users can download their in-app transparency log and use the namespace and epoch in the report to confirm, using the following format: https://akd-auditor.cloudflare.com/namespaces/<namespace>/audits/<epoch>.
2. Anonymous Download Requests: The model download endpoint has no way to determine which user is requesting a model. All download requests are authenticated using anonymous credentials and routed through an OHTTP relay that strips the requester’s IP address, and the request payload contains no identity selectors. The model assets themselves are served by a CDN, which delivers only publicly published files and has no role in choosing which model a device uses, so it cannot be used to target a specific user.
3. Non-targetability: No one can deliver a specific model to a specific user, even through experimentation. The client randomizes the timing of download requests, and experiment group assignment happens entirely on the device: the client assigns itself to a group using locally generated randomness, so the server cannot steer a specific user to a specific model variant.

How Model Download and Verification Works

1. Model Publication: When a new model version is ready for deployment, the server computes SHA-256 hashes of model weights, tokenizers, and other assets, and constructs a manifest — a JSON document containing these hashes, the model version, and a timestamp. The manifest digest (SHA-256 of the manifest) is submitted to a third-party signer (Cloudflare) for signing using Ed25519 keys. Meta does not hold the signing key. The signed manifest digest is then published to the third-party append-only transparency ledger, and the model assets are uploaded to the CDN.
2. Anonymous Download Request: The client connects to the model download endpoint via OHTTP, authenticating with anonymous credentials. The OHTTP relay strips the client’s IP address — the relay can see the IP but cannot decrypt the request, and the server can see the request but only the relay’s IP. The server returns the manifest, the digital signature, and CDN URLs for the model assets.
3. Client-Side Verification: Before using any downloaded model, the client performs a multi-step verification. First, it computes the manifest digest and verifies the digital signature against hardcoded Cloudflare Ed25519 public keys — confirming the manifest was signed by Cloudflare, not forged by Meta or any other party. Next, it cross-references the manifest digest with the transparency ledger and enforces freshness checks to prevent replay attacks with stale entries. Finally, it downloads the model assets from the CDN and verifies that the SHA-256 hash of each asset matches the manifest. If any step fails, the client refuses to load the model.
4. Model Loaded on Device: Only after all verification steps pass does the client install and use the model.

Private Experimentation

Before rolling out a new model version globally, we must evaluate its accuracy and effectiveness by testing model variants with subsets of users. This experimentation must not create a path for targeting (ie delivering a specific model to a specific user). The model download flow is designed to prevent this:

• Client-Side Group Assignment: the download response includes the available model configurations, fetched at randomized intervals via the same anonymous OHTTP and ACS flow. Using a locally generated random seed, the client assigns itself to a group and selects which model to use from those configurations. The server cannot influence which model variant a specific user receives.
• Experiment Configuration Tamper Checks: The client enforces integrity checks on experiment configurations. Group properties cannot be modified after publication, experiment sizes can only be expanded (never reduced), and groups must meet a minimum size threshold, preventing an attacker from narrowing a group to target individual users.
• Experiment Models on the Ledger: Every experiment model variant is published to the transparency ledger before being served, with the same signing and verification flow as production models. No model — whether production or experimental — reaches a device without being publicly recorded.
• Anonymized Experiment Metrics: performance metrics from different experiment groups are measured through the same confidential federated analytics pipeline described earlier in this blog. If an experiment group is too small to meet k-anonymity thresholds after aggregation, the TEE suppresses the data entirely.

Because the entire verification and experimentation flow runs on the client, security researchers can examine the app binary to confirm these checks are performed.

Verifiable Model Behavior

We outlined above how neither Meta nor WhatsApp can deliver a specific model to a specific user, and how the confidential federated analytics pipeline preserves privacy. But neither answers a more fundamental question: how can anyone verify that the model is built only to identify potential scam messages, unless its behavior can be independently examined?

We designed the model verification system around two guarantees:

1. User Visibility: Users have direct, on-device access to what the model did — which messages were scanned, what the outcome was, and which model version was used.
2. Independent Verifiability: As referenced previously, external researchers can obtain the exact model that runs on user devices, verify its integrity against the transparency ledger, and analyze its behavior independently.

How Model Transparency Works

• Published Model Artifacts: Every model version’s hashes are recorded in the signed manifest on the same third-party transparency ledger used to verify model delivery. The transparency ledger is public, so anyone can inspect it to verify which model versions are being served and confirm that every version, including experimental variants, is publicly recorded.
• Client-Side Transparency Logs: On the device itself, users can enable transparency logs that record which messages were flagged by the on-device machine learning model. Users can see this by going to Account > Request Info > Scam Alert Activity. Included in the transparency logs are the outcome of each analysis (whether the model flagged the message and whether the warning was shown), and which model version was used. These logs give users direct, granular visibility into how the system operates on their device.
• Bug Bounty Program: Ahead of Beta rollout, we engaged external security researchers through our Bug Bounty program to help us stress-test our system:
  • Privacy Architecture Review: Researchers were provided with an early access APK build with the feature to confirm no message content leaves the device and there is no auto-reporting, and;
  • Model Integrity Review: Experienced AI/ML researchers receive the model weights to confirm the model is purpose built for scams and nothing else. 

We will be expanding our Bug Bounty scope to include the models to test them against their own inputs, analyze their behavior across a range of scenarios, and report any findings where the model deviates from its declared purpose or where its capabilities can be systematically evaded.

Building Verifiable Trust and Next Steps

The confidential federated analytics pipeline ensures that even the act of measuring model performance protects user privacy. The transparency ledger and third-party signing ensure that every model we ship is publicly recorded and tamper-evident. And published model artifacts, client-side transparency logs, and a dedicated Bug Bounty program ensure that what the model does can be independently verified.

As mentioned above, this feature is only beginning to roll out in a limited Beta capacity. We will continue iterating and improving on it during the Beta phase before production, but we wanted to take this opportunity to outline our principles.

Scammers will continue to evolve their tactics. To keep staying ahead of them, so will we.

We welcome feedback from users, security researchers, and the broader security community through our security research program: Contact us.

Acknowledgements

Thank you to Ronald Anthony, Shafin Anwarsha, Samyukta Mogily, Lenny Grokop, Riccardo Tortul, Harish Srinivas, Kiran Teja Tummuri, Roman Dashchakivskyi, Chao Zhang, Jitendra Mohanty, and the many others across the company who helped make Scam Alert possible.

The post How We’re Building Scam Alert on WhatsApp With End-to-End Encryption and Verifiability Guarantees appeared first on Engineering at Meta.

Every day, Meta’s recommendation platforms handle billions of user interactions, generating rich temporal signals that capture individual preferences and intent across products, ads, and content. In our 2024 post on sequence learning for ads recommendations, we showed how modeling the order and timing of user actions (rather than relying on static, manually engineered sparse features) produces richer, sequence-aware representations of user interests and ad preferences. 

This post goes a step further, introducing two architectural breakthroughs that let us scale sequence learning advancements from foundational innovations into a production platform with predictable, LLM-style scaling laws: (1) a multi-stage sequence model that decouples heavy offline user modeling from lightweight online ranking tasks and (2) a learning technique based on dense tokenization and target-aware attention that efficiently learns feature interactions directly from data. 

Together with our broader model innovations, these advancements have contributed to a cumulative lift of 6% in conversions on Instagram, 3% in conversions on Facebook and 3.5% in ad clicks on Facebook. This unified platform for sequence modeling is a core component of Meta’s Generative Ads Recommendation Model (GEM), helps to harness the comprehensive user behavioral understanding of this learning paradigm to maximize the benefit to advertisers. 

The Historical Challenges of Sequence Modeling 

Ads recommendation systems must retrieve and rank thousands of ads within milliseconds, processing millions of candidates per second. To manage this scale, some approaches to sequence models rely on hybrid model configurations where a specific model processes user event sequences and another model handles sparse feature interactions. 

While effective at meeting production demands, this hybrid approach has potential tradeoffs: 

• Lossy knowledge transfer between components
• Continued reliance on manual feature engineering
• Scaling ceilings from interference between ranking and sequence model components

Scaling both temporal sequence lengths and the transformer models that process them can turn the tradeoffs of the hybrid approach into a bottleneck, limiting the ability to improve the ads experience of users and the performance of advertisers’ campaigns. 

We’ve made two fundamental architectural breakthroughs in sequence learning that resolve the core tension between model complexity and serving efficiency: (1) a multi-stage sequence model that decouples offline user modeling from online ranking and (2) a dense tokenization with target-aware attention learning paradigm. Together, they provide a flexible production strategy that helps generalize sequence learning models and establish an LLM-style scaling law that predictably balances model performance with compute. 

Introducing the Multi-Stage Sequence Model

To address scaling efficiency, a multi-stage model has been developed that enables scaling of a transformer-based sequence model in a compute efficient manner. Separating the sequence model into two complementary stages (upstream/offline user modeling and downstream/online ranking), enables model capacity to scale so that performance keeps improving without proportional increases in serving resources. 

In Figure 1, the left panel shows the offline user model. It processes long user histories asynchronously and produces cached embeddings that capture deep behavioral patterns. The right panel shows the online ranking model that combines these cached representations with real time ad candidate signals to produce the final ranking. The arrow between the two stages carries the user feature embeddings from offline → online ranking models.

Two Key Stages of the Model

First Stage: Offline User Model

User-side features are processed asynchronously using deep transformer upstream models. These models scale to several transformer layers with sequence lengths in the thousands and generate embeddings that are precomputed and cached at the user level. The upstream model strictly separates user features from ad and context features to ensure user embeddings remain independent of any particular ad candidate.

Second Stage: Online Ranking Model

The offline user model representations are complemented with online ranking models that use fresh user signals and ad candidate information for real time ranking. This stage is optimized for speed, meeting strict latency budgets while leveraging the deep representations computed offline. 

Separating the sequence modeling system into two distinct, yet complementary, stages enables an increase in model complexity along a scaling curve for the Offline User Model without causing a spike in serving costs for the Online Ranking Models. 

Sequence Model Architecture Innovations

Dense Tokenization

This tokenization approach integrates sparse features with sequential behavioral data into a single dense vocabulary, enabling attention mechanisms to discover interactions independently. Unlike traditional recommendation systems, which rely on manually engineered representations to capture sparse cross-feature interactions, this approach lets the model learn those interactions directly from the data. 

Target-Aware Multi-Head Attention

Tokenized sparse features and ad candidate information are fused with user behavior sequences, then processed by a memory-efficient form of multi-head attention that lets each layer weigh a user’s past behaviors against the specific ad being scored. Stacking multiple aligned attention blocks with stable attention distributions allows each layer to capture higher-order interactions between the target ad and the user’s historical behavior, progressively distilling long sequences into compact representations.

A Predictable Scaling Curve 

LLM-Style Scaling Law

When running on real-world ads traffic, the multi-stage sequence model demonstrates the emergence of predictable scaling laws for ads recommendations that are analogous to those observed in large language models. Performance improvements follow a log-linear relationship with respect to compute, with a marked improvement in scaling efficiency over other transformer-based sequence models. Figure 2 conveys these scaling properties by showing the relationship between compute (FLOPs) and model performance (measured by normalized entropy, NE) across several dimensions: model depth, content/semantic enrichment, model width, and sequence length.

Levers for Scaling

Unlike LLMs, which process dense and continuous text, ads recommendation systems must integrate sparse ID features with temporal user sequences. The fact that LLM-style scaling emerged despite the structural differences provides a strong indicator of model architectural fit for further sequence learning applications. 

We have identified four levers that we anticipate will help unlock the frontier of the scaling law:

1. Balanced Model Shape

Optimal performance requires balanced growth across model depth, width and sequence lengths. If scaling only occurs on a single axis, the other axes will likely bottleneck the performance improvements, potentially leading to diminishing returns. This mirrors findings from LLM scaling law research, a principle we call the scaling synergy principle. 

2. Multi-Stage Tunability

The multi-stage architecture provides a tunable lever to scale either the offline or online model up/down. Scaling the online ranking model drives steeper improvements per unit of compute that is bounded by serving/request time requirements. Scaling the offline model (shown in Figure 2) follows a more gradual curve, but its async inference avoids latency constraints, allowing scale in at an unhindered rate.  

3. Sequence Composition

Performance continues to improve as sequences get longer, but an impactful finding is that sequence diversity beats sequence homogeneity.  A balanced mix of action types (e.g., views, clicks, conversions) yields better results than sequences composed of a single action type. This finding suggests that a diverse mix of engagement types and broad temporal coverage produce richer behavioral representations of users than homogeneous sequences of high signal actions in isolation.

4. Semantic Feature Representation

Semantic content features from foundation models complement traditional collaborative filtering (i.e. which users interacted with which items) signals. They are especially helpful in cold-start scenarios (e.g., new ads or advertisers with limited historical engagement data). By addressing this persistent challenge of recommendation systems, we improve overall signal coverage to a fundamental sparse problem in recommendation systems.

The Impact of Multi-Stage Sequence Modeling

The multi-stage sequence modeling architecture is delivering impact across three dimensions: 

Deeper User Representation

By modeling thousands of user event sequences (e.g., clicks, views, and purchases) the offline model generates highly nuanced user representations. This depth of behavioral understanding improves ad relevance and conversion rates across Meta’s Family of Apps. Together with our broader modeling innovations, these sequence-derived representations drove a cumulative lift of 6% in conversions on Instagram, 3% on Facebook and 3.5% in ad clicks on Facebook.

Scaling Efficiency

The two-stage design delivers performance improvements with greater compute efficiency compared to hybrid approaches. Initial evaluations improved ads ranking quality with minimal impact to serving resources, confirming that model complexity and production efficiency can scale together. 

Platform Integration

As a core part of GEM, this model architecture for sequence learning has been designed for generalization, where the same multi-stage backbone and scaling properties can extend to any ads ranking task with minimal adaptation and overhead. 

Current Work: Continued Scaling

The sequence model scaling law shows no signs of saturation. With architectural parity achieved, scaling model complexity can draw on techniques proven in the LLM domain (e.g., mixture-of-experts, cross-user compute sharing, advanced attention mechanisms) with potential to continually scale at the optimal performance/efficiency tradeoff. 

Read the Paper

A detailed technical publication of this model architecture and its scaling properties is available in our paper, “LLaTTE: Scaling Laws for Multi-Stage Sequence Modeling in Large-Scale Ads Recommendation.” 

The post From User Sequences to Scaling Laws: A Multi-Stage Architecture for Meta’s Ads Ranking appeared first on Engineering at Meta.

• Meta’s Generative Ads Recommendation Model (GEM), the foundation model behind ads recommendations across Instagram and Facebook, now trains at LLM scale on several thousand of the latest-generation GPUs. This post goes into the details on how we achieved: doubling end-to-end (E2E) training efficiency to 20–25% Model FLOPs Utilization (MFU) while scaling training FLOPs 4x in 12 months, by co-designing kernels, precision, parallelism, networking, and memory together.
• Training GEM presents unique engineering challenges at the intersection of recommendation systems and LLMs as the model combines a hybrid architecture plus recommendations-domain data properties that are unlike typical LLM workloads. 
• AI infrastructure optimized for LLM training (kernels, parallelism, low precision recipes etc.) does not directly transfer, requiring significant innovation and hardware/software co-design to reach LLM-scale training for recommendation models efficiently.
• We tackled these challenges through complementary compute efficiency and scaling efficiency innovations:
  • Compute efficiency: Achieved through a customized recommendation kernel library — Jagged Flash Attention (JFA), Generalized Dot-Product Attention (GDPA), BlockAttention, etc. — and mixed ultra-low precision training (including MXFP8 attention and MLP) optimized for recommendation workloads, purpose-built to exploit latest generation GPU’s architecture.
  • Scaling efficiency: Topology-aware 5D parallelism with Streaming Multiprocessor (SM)-free collectives — 2D FSDP + Expert Parallelism for dense parameters, combined with Fully Sharded 2D Model Parallelism for sparse parameters — co-designed with Meta’s multi-tiered network hierarchy to reduce communication overhead.
• The results: we doubled GEM’s E2E training efficiency to 20-25% MFU while scaling total training FLOPs 4x over the past 12 months.

GEM’s Architecture And Its Unique Training Challenges

GEM is the central recommendations foundation model behind Meta’s ads system. It has a hybrid architecture with trillions of sparse embedding parameters and billions of dense parameters. GEM is trained on ad content and user engagement data with two categories of features: sequence features (e.g., user activity history) and non-sequence features (e.g., user location, ad creative representation). Customized attention mechanisms are applied to each group independently, while also enabling cross-feature learning.

The interplay between this hybrid architecture and rec-domain data properties is what makes GEM’s training uniquely challenging.

Challenge 1: Achieving High Per-GPU Utilization 

Today’s data center GPUs and their software stacks are mostly optimized for LLM workloads, whereas recommendation workloads have a fundamentally different profile due to unique data characteristics and rich user & ads signal interaction patterns that make it extremely difficult to achieve high GPU compute utilization for training a foundational recommendation model of GEM’s size.  

• Jagged Inputs: Training samples have highly variable sequence length as user activity history can vary wildly. Padding to max length would waste up to 50% compute.  
• Diverse interaction patterns and asymmetric sequences: Self-attention operates on extremely long sequences (activity history) but short attention window; cross-attention learns user x ads interaction with long queries but short key/value; pooled multi-head attention (PMA) compress user activity history, resulting in short queries but long key/value. These asymmetric shapes make intra kernel pipelining less effective to saturate compute units.     
• Memory-bound operations: e.g., small embedding dimension for MLP and various normalizations for model quality and training stability leave compute units underutilized.   
• Numerical sensitivity: Ads optimization tasks (CTR/CVR prediction) are highly sensitive to numerical change (e.g., precision), making naïve low-precision training prone to quality regression.

Challenge 2: Scaling Efficiently Across Thousands of GPUs 

Training GEM across thousands of GPUs with trillions of sparse embedding parameters and billions of dense parameters requires scaling efficiently, not just scaling up. Simply adding more GPUs does not translate to proportional speedup. In distributed training, E2E latency per training step is determined by: 

E2E Latency = Max across GPU Rank (Max(Local Compute Time, Communication Time))

Near-linear scaling requires four conditions: 

• Total compute time >> total communication time. 
• Communication hidden behind compute without contention.  
• Minimal recomputation from memory pressure.  
• Good load balancing across ranks.
  

 GEM’s workload threatens every one of these: 

• O(Trillion) sparse parameters and O(Billion) dense parameters drive heavy communication with mixed compute patterns.
• Architecture diversity across layers makes overlap windows uneven; resource contention between communication and computation makes hiding communication non-trivial.
• Long sequences with large activations push memory usage toward its limit, forcing activation recomputation that erodes efficiency.
• Jagged sequences across samples create data-driven load skew that varies across ranks.

Our Approach and Efficiency Framework

Given the challenges outlined above, we needed a framework that turned a sprawling co-design effort into a small number of technical levers. We measure training efficiency through E2E MFU, which decomposes into two factors:

E2E MFU = Local MFU (compute efficiency) × Scaling Ratio (scaling efficiency)

These factors describe two related but distinct optimization problems.  

Local MFU (compute efficiency)  measures how well a single GPU’s compute units are utilized — how close the workload runs to the hardware roofline. It is determined by kernel design, numerical precision, and how well the workload’s compute patterns (data dimensions, sequence lengths) map onto GPU architecture (Tensor cores, memory hierarchy, streaming multiprocessor scheduling).

Scaling Ratio (scaling efficiency) measures how much single-GPU performance is retained when distributing across thousands of GPUs. A scaling ratio of 1.0 means perfect linear scaling; in practice, communication overhead, load imbalance, straggler effects, and activation recomputation from memory pressure all erode it.

To isolate local MFU, we run model layers individually on a single GPU and compute a weighted average MFU without activation recomputation or communication exposure. The scaling ratio is derived as the ratio between local and E2E MFU.

This decomposition matters because it lets us treat compute efficiency and scaling efficiency as related but distinct optimization problems, each with its own dedicated set of techniques:

• Compute efficiency is a kernel-level and numerical-precision problem. The levers are kernel design and ultra-low-precision training — both targeting the per-GPU roofline.
• Scaling efficiency is a distributed-systems problem. The levers are parallelism strategy, network topology mapping, networking efficiency, memory management, and load balancing — all targeting the gap between single-GPU and multi-GPU throughput.

Both must be addressed to maximize end-to-end MFU.

Optimizing Compute Efficiency With Recommendation Kernels and Ultra-Low-Precision Training

To address the recommendations-system-specific challenges mentioned above and push up GPU FLOPS utilization, we built a custom kernel library and an ultra-low-precision training recipe custom-built and optimized for recommendation workloads on the latest GPU hardware. 

• JFA — eliminates the up-to-50% compute waste from padding jagged inputs.
• BlockAttention — reduces long user-history self-attention cost from O(L²) to O(L) while preserving model quality and efficiency 
• GDPA — unifies and accelerates GEM’s diverse, asymmetric attention modules where FlashAttention’s dense long-sequence assumptions break down
• MXFP8 attention + MLP — turns lower-precision Tensor Core throughput into real end-to-end speedups without regressing precision-sensitive CTR/CVR objectives 

Inside the Customized Kernel Library for Recommendation   

Jagged Sequence Flash Attention 

FlashAttention is designed for dense, fixed-length sequences common in LLMs. In recommendation models, user sequences are inherently jagged — varying from hundreds to tens of thousands of tokens per sample — and padding to max length could waste up to 50% of compute. 

Standard FlashAttention implementations assume uniform sequence lengths for efficient tiling and parallelization; with jagged inputs, naive approaches either pad (wasting compute) or leave SMs idle when short sequences finish early. We developed JFA, a custom FlashAttention implementation that operates directly on variable-length jagged tensors, eliminating padding overhead while supporting rec-specific features such as custom attention biases, asymmetric query/key-value lengths, and efficient backward passes.

We evolved JFA through four generations, progressively closing the gap from being slower than padded SDPA (scaled dot-product attention) to matching SOTA CUDA/Cutlass performance on latest-generation GPUs:

• Jagged masking via subtraction scheme: Traditional 2D masking for jagged boundaries (marking invalid positions with -inf) consumes significant non-tensor-core instructions (~28% of executed instructions). We replaced this with a novel subtraction scheme — masking Query/Key with zeros (which the Tensor Memory Accelerator (TMA) does for free) and subtracting the extra exponents — producing numerically equivalent results without the masking overhead.

• Backward parallelization: FlashAttention’s backward pass requires accumulating dQ across sequence tiles, typically via costly atomic adds. We explored multiple schemes (seq-parallel with atomics, no seq-parallel, seq-parallel with recompute, split dQ/dKdV) and found that for rec workloads with high batch x heads, a non-seq-parallel scheme with split dQ computation delivers 21-40% backward speedup by eliminating both atomic writes and redundant recomputation.
• Warp specialization and persistent kernels: Upgrading to Triton Low-Level Extensions (TLX) enabled explicit warp specialization, along with use of TMA, and persistent kernel scheduling — unlocking 30-100% TFLOPS improvement by leveraging the latest hardware feature. 

JFA v4 (TLX) achieves 40-140% TFLOPS improvement over JFA v2, which delivers consistent gains under production jagged distributions (sparsity 0.5), contributing to 18.5% relative local MFU gain and 12% QPS gain.

Generalized Dot-Product Attention (GDPA) 

GEM uses diverse attention-like interaction patterns — self-attention, PMA, and cross-attention — that share a common structure: two matrix multiplications with an element-wise activation in between, but replace softmax with activations like GELU or SiLU. We unify these modules under a single GDPA kernel optimized for production RecSys training workloads on latest generation GPUs.

Existing FlashAttention kernels are designed for LLM-style dense, long-sequence inputs and perform poorly under real production traffic. We observed a 2.6x forward performance gap and up to 4x worst-case gap between real-world workloads and synthetic benchmarks driven by short/asymmetric K/V sequences, jagged inputs, and large batch sizes that break pipeline occupancy assumptions.

We redesigned the kernel pipeline, scheduling, and math to close the performance gap between real-world traffic and hardware roofline.

• Pipeline redesign for non-softmax activations: Eliminating the softmax correction stage frees four warps and their registers. For short K/V sequences, outer-loop software pipelining recovers ~10% performance lost by inner-loop pipelining when the inner loop runs only 1–2 iterations.
• Software-level tile scheduling for jagged tensors: precompute valid tiles on CPU, skip empty tiles entirely, and apply zigzag assignment across SMs — reducing workload skew from 6x to near-balanced.
• ALU-only activation approximation: Replace GELU’s SFU-bound tanh with a 6th-order Taylor expansion (ALU-only), accurate within the bounded input range enforced by QK-norm (query/key normalization). Eliminates SFU contention in both forward and backward passes.

With these optimizations, the optimized GDPA kernel achieves 2x forward speedup (1,145 BF16 TFLOPs, ~97% Tensor Core utilization) and 1.6x backward speedup over baseline. Under short K/V production settings, it achieves up to 3.5x forward speedup over Flash Attention 4 (FA4). Applied across the full model, these kernels deliver over 30% end-to-end training throughput improvement.

BlockAttention  

For GEM self-attention, the core efficiency challenge was scaling long user sequences without paying the quadratic cost of full attention. We first moved the layer from full self-attention to sliding-window attention, limiting each token to nearby events and reducing complexity from O(L2) to O(L * window). This made longer sequences practical. The Sliding Window Attention (SWA) kernel skipped off-window tiles in JFA and reduced long-sequence self-attention latency by up to 68% with neutral NE (normalized entropy, a model-quality metric).

We then pushed the structure further with block-aligned attention. Since GEM could safely use fixed 64-token blocks, each Q block only attends to its corresponding K/V block, turning attention into independent 64×64 problems. This removes the partial-window masking and multi-tile iteration still present in SWA, and lets a dedicated TLX kernel eliminate FlashAttention overheads such as online softmax correction, logsumexp HBM traffic, and separate Di preprocessing. 

Fusing RoPE backward into the attention epilogue removes another memory-bound kernel and keeps gradients in FP32 registers. Together, TLX block attention + fused rotary improves self-attention layer MFU by +30.6% over Triton block attention, or roughly +44% over the SWA baseline.

Mixed Ultra-Low-Precision Training 

On a GPU, lower precision directly translates to higher Tensor core throughput. For the latest generation GPU, FP8 delivers 2x peak FLOPS over FP16, and FP4 delivers 4x. We expect the peak FLOPS of low precision to increase faster in next-generation GPUs. This makes low-precision training increasingly attractive as hardware vendors scale low-precision FLOPS faster than FP16. 

However, making low-precision training work without quality regression — addressing both numerical stability and quantization overhead — remains an industry-wide challenge. We developed MXFP8 Attention and MLP with numerical stability enhancement, which addressed both training stability and quantization overhead.     

Low Precision Flash Attention  

We extended the FA4 kernel with end-to-end MXFP8 blockscaled MMA for both forward and backward passes leveraging latest generation GPUs’ native support for low precision. The main challenge is that low precision attention is not just a datatype swap. Scale factors must be generated along each GEMM’s (General Matrix Multiplications) K dimension, staged through shared memory (SMEM) / tensor memory (TMEM) despite FA4’s already full TMEM footprint, and computed online for intermediates such as softmax P and backward dS. 

To make the Tensor core speedup survive at module level, quantization was fused into upstream normalization and projection kernels, emitting FP8 activations and tensor-core-friendly scale layouts directly while avoiding extra BF16 global-memory traffic. For GEM’s jagged recommendation workloads, FP8 data stays at unpadded positions and only compact scale factors are scattered/padded for TMA. This turns MXFP8 block-scaled MMA support into practical E2E attention speedups without introducing model quality regressions.

To meet our unique requirements we had to develop three new innovations at the kernel level:

• TMEM scale factor placement: The original FA4 fully utilized 512-column TMEM for accumulators, leaving no room for block-scale factors. We solve this by overlapping scale factors with temporarily-unused TMEM regions (e.g., placing S(i) scale factors in the S(1-i) accumulator region), requiring only one additional lightweight barrier that is hidden behind existing GEMM latency.   
• Online P-to-MXFP8 conversion: Softmax output (P) is quantized to MXFP8 in-place within the softmax warp, reusing the row-max already computed for softmax normalization to avoid redundant reductions. Scale factors are derived via optimized PTX bit-manipulation sequences instead of expensive log2/round/clamp operations.
• Block-wise Quantization: We use [32, 32] square quantization computing one scale factor per 32×32 block via redux.sync.max.abs.f32 warp-wide reduction — making quantization transpose-invariant so each tensor is quantized only once. This is useful for the backward pass, where transposed Q,K values are needed.
  

On GEM representative shapes, measured on Meta internal power capped latest generation GPU, we achieved >1.3x speedup for the forward kernel with MXFP8. For the backward kernel, we achieved >1.5x speedup with MXFP8.

Handling Quantization Overhead 

Quantization overhead mainly comes from two sources, model parameters (weights) and intermediate tensors (activations). If handled naively, the extra casting, scaling, and data movement can offset the compute speedup from low-precision Tensor cores.

• Weight – quantization on Fully Sharded Data Parallel (FSDP) shard

• • Pre-all-gather shard quantization: quantize each rank’s local shard before FSDP all-gather to amortize the quantization cost across ranks, this avoids re-quantizing the fully gathered weight on every rank.
  • Quantized FSDP communication: communicate low-precision payloads (vs. BF16) to reduce all-gather volume and cut all-gather latency which further neutralizes the quantization overhead. 

• Activation – kernel fusion

• • Linear modules: Instead of doing a separate quantization step with extra kernel launch + HBM traffic, we fused activation quantization into the preceding normalization (PreNorm fusion) to avoid the overhead. 
  • Attention modules: In addition to PreNorm fusion, we also fused quantization into the preceding projection so the attention kernel consumes low-precision activations directly with no extra quantization step.

Addressing Numerical Stability  

Quantization errors, outliers, and rounding bias can make low-precision training  numerically fragile, especially for gradient computation. We addressed these challenges with:

• Outlier mitigation:
  • We applied Random Hadamard Transforms spread outliers and smooth distributions prior to low precision quantization.
• Recipe tuning (fine-grained controls):
  • We used stochastic rounding to eliminate deterministic rounding bias.
  • Skipping / higher-precision weight-gradient (WGrad): We observed activations and gradients can exhibit more severe outlier behavior; selectively skipping WGrad or using higher precision can materially improve model quality.
• Mixed precision: 
  • We use ultra low precision  where it will have the most benefit  (e.g., large GEMMs) and fall back to BF16 (e.g., later layers in the model are more sensitive to quantization errors) where ultra low  precision is insufficient to meet model quality targets.

Scaling Efficiency: 5D Parallelism, Networking, Memory, And Load Balancing 

As mentioned above, for large scale distributed training:  

E2E Latency = Max across GPU Rank (Max(Local Compute Time, Communication Time)) 

Near-linear scaling requires four conditions: total compute time > communication time, compute / communication overlapping without contention, minimal recomputation, and good load balancing.  Our optimizations address each condition to push up GEM’s scaling efficiency. 

5D Parallelism, Optimized with Meta’s Network Topology

GEM’s hybrid architecture requires distinct parallelism strategies for each component as dense and sparse parameters have different compute and communication patterns. We use 5D parallelism to scale GEM’s training efficiently across thousands of GPUs: 2D FSDP with Expert Parallelism (EP) for dense parameters, and Fully Sharded 2D Model Parallelism for sparse parameters. 

The design principle is to match communication volume to available bandwidth across the topology hierarchy. When a collective becomes a bottleneck on a given tier, we introduce a new parallelism dimension that reduces message volume or group size on that tier.

Meta’s training cluster used by GEM has a three-tier network hierarchy: Eight GPUs per host connected via NVLink , hosts within an AI zone connected via RoCE, and AI zones connected via oversubscribed RoCE with bandwidth reduction. 

Dense Parallelism Evolution: From 1D to 3D Parallelism  

GEM’s O(Billion) dense parameters are sharded using FSDP. Parameters are distributed across GPUs and reconstructed via all-gather before computation, with gradients synchronized via reduce-scatter. We add two dimensions on top of FSDP — a replica (DDP) dimension (making it 2D FSDP) and EP — for a total of three dense parallelism dimensions (3D dense parallelism).

This topology-aware distributed training is what makes 3D dense parallelism efficient — each dimension’s communication cost is matched to the bandwidth available at its topology level.

Why 2D FSDP: Reducing Group Size for Better Bandwidth

At several thousands GPU scale, standard FSDP requires collectives across the full rank count, where effective bandwidth degrades with group size —  particularly when spanning multiple AI zones. 2D FSDP solves this by splitting the communication into two topology-aware tiers:

• FSDP shard group : Parameters are sharded and reconstructed via all-gather / reduce-scatter across a much smaller group (e.g., 128-256 GPUs). The reduced group size achieves higher effective bandwidth. 
• DDP replica group : Gradients are synchronized via all-reduce across replica groups. Because parameters are already sharded by FSDP, each rank sends only a fraction — the message size is small enough to even tolerate the lower cross-zone bandwidth.

We aggressively pre-fetch parameter all-gathers, pipelining each module’s communication with the previous module’s compute to maximize overlap. This works well for most modules — however, large modules like DHEN (Deep Hierarchical Ensemble Network) experts have parameter sizes where communication time still outweighs neighboring compute time, becoming exposed and slowing down E2E efficiency.

Adding Expert Parallelism: Pushing Heavy Communication to the Fastest Links

To address communication exposure from large dense expert modules, we layer EP on top of 2D FSDP. With EP, each rank holds only one expert, shrinking the FSDP all-gather to a single expert’s parameters — reducing both group size and message size.

The extra EP communication is placed on intra-node NVLink with high bandwidth  making it easily hidden. The forward and backward passes coordinate FSDP and EP collectives:

• Forward: FSDP all-gather expert params (16-way, inter-node) → EP all-gather activations (2-way, intra-node NVLink) → compute local experts on full batch → EP reduce-scatter outputs (2-way, intra-node NVLink). 
• Backward: FSDP all-gather expert params (16-way, inter-node) → EP all-gather output gradients (2-way, intra-node NVLink) → compute expert gradients → EP reduce-scatter input gradients (2-way, intra-node NVLink) → FSDP reduce-scatter param gradients (16-way, inter-node).

Sparse Parallelism Evolution: From 1D to 2D memory overhead free parallelism 

GEM’s sparse parameters (O(Trillion) embedding tables) present unique scaling challenges distinct from dense parameters. Embedding tables require model-parallel sharding with all-to-all communication for feature distribution, and their sheer size makes memory overhead a primary constraint. We evolved through three generations of sparse parallelism to address these challenges.

V1 → V2: Solving Imbalance and Communication Bottlenecks

At several thousands GPU scale, 1D model parallelism hits two fundamental bottlenecks for good efficiency: 

• Load imbalance: Distributing embedding table shards across thousands of ranks results in severe workload skew — each rank holds too few shards for balanced partitioning.
• Communication latency: All-to-all collective group size scales with total rank count. Cross-node bandwidth degrades rapidly with group size, particularly when jobs span multiple AI zones where bandwidth is oversubscribed 

2D model parallelism addresses both by partitioning ranks into smaller model-parallel groups (e.g., 256 GPUs), with multiple replica groups performing data parallelism. Each replica group independently shards and communicates within a much smaller scope, reducing all-to-all latency and improving load balance — delivering significant QPS gains over 1D at large scale.

V2 → V3: Eliminating Memory Overhead

The tradeoff of V2 is memory: each replica group must hold a full copy of its assigned shard’s parameters. For GEM’s trillion-parameter sparse tables, this O(T) overhead can consume significant HBM — blocking further model scaling.  

Fully Sharded 2D removes this overhead by further sharding each replica’s parameter copy across its groups. Each rank stores only a fraction of the shard, and parameters are reconstructed on-demand:

• Forward: All-gather table shards → all-to-all feature distribution → embedding lookup → all-to-all embedding return
• Backward: All-gather table shards → all-to-all gradient exchange → local update → reduce-scatter parameters

The extra all-gather and reduce-scatter from V3 are mapped to intra-node NVLink. We overlap these collectives with concurrent dense compute through pipelining, and schedule the all-gather to release reconstructed copies before peak memory usage.

With these optimizations, we’re able to make sparse scaling nearly overhead-free at GEM’s training scale with very minimal communication exposure.

Networking Efficiency : Getting Communication Off the SMs

With 5D parallelism, GEM hides most communication behind compute kernels through pipelining. However, communication collectives could also occupy SMs, which creates SM contention. Communication kernels occupy SMs (e.g. ~24 SMs for all-gather, reduce-scatter) that would otherwise be utilized by compute kernels running in parallel, costing up to 15% efficiency. What makes it worse is that compute kernel performance could drop more than SM occupancy loss, since wave scheduling could end up with more waste. 

Hence, our primary networking efficiency push is SM-free communication — offloading data movement from SMs to dedicated hardware engines.

For pure data-movement collectives (e.g., all-gather), we use NCCLX — Meta’s extension to the NCCL library — for copy-free, SM-free communication. NCCLX leverages hardware features to move data without SM involvement: the Copy Engine (CE) handles intra-node NVLink transfers and RDMA handles inter-node transfers, reducing SM usage from 24 to 1 for all-gather. This reclaims ~23 SMs for compute, yielding ~5% E2E QPS gain at full training scale.

For collectives that require reduction (e.g., All-Reduce), we found NVLink SHARP with in-network reduction a viable option to reduce SM usage by offloading the reduction computation from SMs to the network switch hardware.

Memory Efficiency: Large Local Batches Without Paying the Full Memory Bill 

Per-GPU memory breaks down into three categories: activations, embedding tables, and dense parameters (including optimizer states). After parallelism shards embedding tables and dense parameters across GPUs, activations dominate per-GPU memory and scale with model and batch size. 

We used two techniques to address this:

Compiler-based Automatic Activation Checkpointing (AutoAC) 

PyTorch’s compiler-based activation checkpointing already beats traditional all-or-nothing recompute by reasoning over individual nodes in the joint forward–backward graph — saving expensive ops, recomputing cheap pointwise ops. But it still applies a single memory budget across the whole model, which leaves performance on the table when regions (compiled subgraphs between graph breaks) differ in recompute ROI (latency saved per GB of activation). We replaced the global budget with a customized per-region budget schedule, so memory flows to the regions with the highest payoff. This pushes the memory–latency tradeoff past what any uniform budget can achieve.

Activation Quantization

On top of AutoAC, we further squeeze the memory usage via activation quantization. It operates on the checkpointed tensors — the set of intermediate activation tensors that AutoAC has already determined need to be stowed for the backward pass. When enabled, it quantizes these saved activation nodes (e.g., from BF16 to FP8/MX4) at the boundary between the forward and backward graphs. 

With these optimizations, we’re able to use large local batch sizes (up to 1K+ samples) with modest activation recompute cost to train the GEM model efficiently. This is important for scaling since small batch size and heavy activation recompute both hurt MFU.

Load Balancing: A Recommendation-Specific Straggler Problem

LLM training could avoid load balancing by padding all sequences to fixed length. For GEM, user sequences are inherently jagged, and padding wastes 50%+ of compute. Jagged kernels avoid per-rank waste but create a new problem – data-driven compute skew that varies every iteration.

The heaviest rank consistently exceeds the average by ~15% each iteration.

Choosing the Right Rebalancing Strategy 

We considered local and global rebalancing strategies to address workload imbalance:

The overhead associated with the global approach — a collective on every training step — negates the very efficiency gains it aims to deliver. We developed a new technique that we call Base Batch Shuffling (BBS), where distributed readers generate small sub-batches (128 samples), which are sorted by total sequence length and interleaved (heaviest paired with lightest) when merged into full training batches (1k+ samples per rank) — capturing most of the theoretical optimal balance with zero cross-rank communication.

BBS delivered 4% efficiency gain on GEM training, comprising 4% QPS improvement and 4% peak memory reduction. Upon activation, the maximum-over-average workload gap immediately dropped.

On to the Next Level of Scale and Efficiency      

Training a foundation model at the intersection of LLMs and recommendation systems is a co-design problem, not a software problem or a hardware problem alone. The 2x efficiency gain we describe here came from carefully considering every layer of the stack for optimization— kernels, precision, parallelism, networking, and memory all had to move together. We expect the next 2x to come in a similar way and with even faster iteration speed as we embrace agents to automate some of the optimization cycles. As we continue to scale the GEM model, we expect to keep pushing system boundaries and extreme co-design across different layers of the AI infra stack to further advance compute and scaling efficiency. We’re sharing this work in the hope that the broader community sees similar opportunities in the workloads they run.

Acknowledgements

The post GEM Training: How Meta Doubled the Efficiency of Its LLM-Scale Ads Foundation Model appeared first on Engineering at Meta.

• We’re introducing SilverTorch, a reimagining of recommendation systems that unifies all retrieval components for user generated content under a unified architecture. 
• SilverTorch shows up to 23.7x higher throughput compared to the state-of-the-art approaches. It’s also showing 20.9x more compute cost efficiency compared to a CPU-based solution while also improving accuracy. 
• Our research paper, “SilverTorch: A Unified Model-based System to Democratize Large-Scale Recommendation on GPUs,” accepted to the full paper track at SIGIR 2026, contains full technical details.

The retrieval system within industry recommendation systems have consisted of microservices stitched together, with neural networks inconsistently integrated. Our recommendation can scale to serve people across multiple platforms. Retrieval is responsible for narrowing from millions of pieces of content (e.g., reels and photos) down to thousands before passing them to ranking systems, all in less than 100 milliseconds.

However, the microservice based design had hard constraints on model complexity and the number of candidates evaluated, ultimately creating a ceiling on the quality of recommendations that people on our platforms see.

To break through this ceiling, we’ve fully reimagined our retrieval ecosystem into a unified model-based system – SilverTorch.

SilverTorch operates under a new paradigm we call Index as Model. We’ve built our retrieval system as a single neural network and now express different microservices as model modules within this integrated neural network. Under Index as Model previous microservice-based item indices used for retrieval become a tensor inside the model.  As a user opens up their app, one request flows through a SilverTorch model, completes all critical retrieval functions (searching for items similar to the user’s interests, filtering for eligibility, reranking and scoring engagement likelihood against multiple user engagement actions), and returns a list of high-quality content candidates to ranking. This new design effectively allows us to increase modeling complexity and the number of candidates evaluated without breaking the sub-100 milliseconds bar.

SilverTorch  makes retrieval significantly more efficient, runs at scale, and enables better recommendations.

• Higher throughput, lower total cost of ownership (TCO). In an 80M-item end-to-end evaluation, SilverTorch served 23.7× more requests per second than a strong traditional multi-service baseline built on the same model architecture, while improving estimated TCO efficiency by 20.9×.
• Proven at scale. Results show SilverTorch can scale across a family of apps as the major retrieval system behind the feed and video content people see.
• Better recommendations. By making neural reranking and multi-task scoring practical within tight latency budgets, SilverTorch has consistently enabled retrieval quality improvements that would have been impractical under a microservices architecture. 

Moving From Microservice Mesh to One Integrated Neural Network

The Microservice Paradigm We Replaced

Traditional recommendation retrieval is built as a mesh of microservices. When a user opens a social media platform, the request hits an orchestrator, which fans out to a user-tower model service (which computes a vector representation of the user’s interests, called a “user embedding”), a combined retrieval service (which finds and filters candidate items based on similarity to the user vector and eligibility rules like language and geography), and a scoring service (which ranks the survivors). The orchestrator merges results and hands them downstream. Each service has its own codebase, often in a different programming language, with its own deployment lifecycle.

This worked well in the CPU era. But as retrieval systems grew in scale and sophistication, three problems compounded into structural limits that no component-level optimization can fix:

• Latency lost to data movement. Every hop between services costs network round-trip time and serialization overhead, eating into our sub-100-millisecond retrieval budget that should fund actual computation. And because filtering, search, and scoring are designed independently, they cannot be jointly optimized.
• Version inconsistency. The user-tower model, the item index, and the filtering rules each update on their own cadence. When the user model ships v2 but the item index is still on v1, the system queries v1 embeddings with v2 user representations — creating quality gaps no downstream ranking can recover.
• Siloed development environments. Machine learning (ML) engineers write PyTorch. Infrastructure engineers write C++. Different release cycles, different testing setups, different mental models. Every retrieval improvement requires translating an idea between two environments — weeks or months per cycle.

Component-level optimizations like Faiss-GPU help by making the specific microservice faster, but they don’t resolve the underlying structural limits. The architecture is still a system of services with artifacts handed between them.

The Shift: All Components Are Model Modules

SilverTorch rethinks the paradigm from the ground up. Instead of designing a microservices system and inserting neural networks into it, we start with the neural network and design outward. We call this Index as Model: Every retrieval component — the item index, eligibility filter, scoring layer and user tower — becomes a tensor or operator inside a single PyTorch model. That means one artifact to deploy, one forward pass to run and one source of truth for what’s in the system. 

Inside the Model

Inside this single neural network, different regions of the network handle different jobs. Approximate nearest neighbor (ANN) search regions find items most similar to the user’s interests without checking every item in the catalog (a librarian who has organized the books well doesn’t walk every shelf). Eligibility filtering regions check that each candidate is allowed to be shown: right language, right country, right content policy. Multi-task reranking regions predict the likelihood of multiple engagement actions (like, share, comment) at once, then combine them into a composite score. Some regions are hand-written by engineers; others are trained end-to-end via backpropagation. From the runtime’s perspective, all of them are nn.Module — the standard building block of PyTorch — and indistinguishable from each other.

The Redesign: Pure PyTorch Modules for Every Stage

How Each Component Worked Before

Before SilverTorch, every module in the production retrieval pipeline — ANN search, eligibility filtering, neural reranking, composite scoring — had a well-known classic implementation, mostly built as standalone services in C++. 

These implementations are mature and battle-tested, but each is a standalone service with its own data structures, memory, and execution model. We can chain them — run ANN, then hand its output to filtering — but we cannot easily implement cross-module optimizations like “pick the most promising clusters first, filter only inside those clusters, then score only the survivors.” This level of co-design requires modules to share memory, an execution graph, and a compilation step.

The Pure PyTorch Decision

To enable that co-design, we made a decision that every module would be reimplemented in pure PyTorch. Under this paradigm:

• All data is expressed as tensors.
• All logic is tensor-in, tensor-out.
• Every module is an nn.Module that conforms to PyTorch’s standard interface.
• At execution time, the ANN and Bloom index filter modules are indistinguishable from a trained ML reranker — both are nn.Module, both take tensors in and produce tensors out.

With every module as an nn.Module, the boundary between ML engineering and infrastructure engineering dissolves — they live on the same layer, freely composed and jointly optimized in a single PyTorch training script. And because the whole system reduces to a single PyTorch model, we get to benefit from the broader AI industry’s work on making PyTorch models faster,  like PyTorch’s own torch.compile that automatically rewrites a PyTorch model into more efficient GPU kernel code. Every advance in that ecosystem improves SilverTorch’s serving performance.

The pure PyTorch decision did not mean taking CPU-era retrieval components and wrapping them in nn.Module. It forced us to rethink retrieval primitives in forms native to GPU execution and to the model graph itself. Bloom index filter and fused Int8 ANN search are two examples. In both cases, the gain comes not from porting an old service into PyTorch, but from redesigning the underlying algorithm around GPU memory behavior, tensor layout, and execution inside the same forward pass. That is the fundamental playbook of SilverTorch: once retrieval components live inside one PyTorch model, co-design becomes possible, and that co-design is what unlocks the gains.

Bloom index filter is one example of how SilverTorch redesigns retrieval for GPUs. In traditional systems, filtering is usually handled by an inverted index, which is efficient on CPUs but harder to run well on GPUs. The problem is that recommendation filtering often has to check many item attributes at once, such as language, location, or eligibility rules, and posting lists can also vary dramatically in length across attributes and queries, creating intra-warp load imbalance and warp divergence on GPUs. Threads assigned short lists become inactive early, while the warp remains occupied until the lanes processing the longest lists complete. 

SilverTorch replaces that with a Bloom index stored directly inside the model. Each item gets a compact signature when it is published, and at serving time the model can quickly check whether an item matches the request using simple bit operations. This turns filtering into the kind of dense, parallel work GPUs are good at, and because the filter result is already inside the model, it can flow directly into ANN search without a separate service call.

Fused Int8 ANN search follows the same idea. General-purpose ANN libraries are built to find nearby items, but recommendation systems need more than a small nearest-neighbor lookup. They often need to pull back a much larger pool of candidates so later stages can make better relevance decisions. 

SilverTorch reimplements ANN search as part of the model itself. It stores item embeddings in a compact Int8 format, which cuts memory use roughly in half compared to typical 16 bits, and runs search with a fused GPU kernel. That reduces data movement and makes the retrieval stage cheap enough to return many more candidates, giving downstream models more room to find the best recommendations. Our Int8 quantized ANN search shows limited quality loss compared to brute force while significantly improving serving performance. It frees headroom for ranking more items with more sophisticated layers and improves the end-to-end retrieval accuracy, and the algorithm supports large top-k and probe counts; in practice, we observe no retrieval recall loss with 64 probes and top-2048.

Benefits — What Shows Up Outside the System

SilverTorch delivers concrete impact along three dimensions: compute cost efficiency, recommendation quality, and engineering velocity.

Compute Cost Efficiency

By moving ANN search, eligibility filtering, and composite scoring onto the GPU and combining them through SilverTorch’s co-design, we serve far more requests per second on the same machine. More requests per second means fewer machines needed for the same workload, and fewer machines means lower compute cost per request.

Below is a comparison on a production retrieval workload of 80 million items, with real production traffic replayed against each system under the same latency budget:

SilverTorch’s 13.35× cost-per-request advantage compounds from several sources: The fused Int8 ANN kernel is 2.2-14.7× faster than Faiss-GPU; the Bloom index is 291-523× faster than the CPU inverted index; the probe-then-filter co-design cuts filter compute by another 30×. Int8 quantization in the model graph cuts memory in half compared to full-precision baselines, leveraging the GPU’s dp4a instructions, with no measurable recall loss.

Recommendation Quality

SilverTorch improves recommendation quality by turning retrieval into a much broader and more expressive pre-ranking stage. In traditional service-based systems, retrieval is usually constrained to a relatively narrow ANN result set, scored mostly by simple embedding similarity, with richer relevance modeling deferred to late-stage ranking. 

SilverTorch unlocked headroom. By keeping ANN search, filtering, and scoring inside one model, it can widen the funnel substantially. Instead of handing only a small set of candidates downstream, it can bring one to two orders of magnitude more candidates through additional learned relevance layers before final ranking. That makes retrieval contribute meaningfully to recommendation quality, not just a fast pruning step.

Neural reranking. SilverTorch introduces a neural network based reranking layer that goes beyond dot-product similarity and applies richer user-item interaction modeling to a much larger candidate set. These layers can take the form of multi-layer perceptrons, stacked self-attention, or more structured interaction models such as mixture of logits. Because the item representations and cross-features remain in GPU memory and are executed within the same model, SilverTorch can afford to apply these more sophisticated ranking layers earlier in the pipeline, over far more candidates than conventional retrieval systems typically can.

Multi-task scoring. SilverTorch also makes retrieval natively multi-objective. A scoring layer combines predictions for different user actions into a single composite score, so retrieval is no longer optimizing around one coarse similarity signal. Instead, it can evaluate a broad candidate pool against a richer notion of user engagement before late-stage ranking begins. The result is a wider funnel with more intelligence inside it – more candidates survive early retrieval, and they are screened by more sophisticated, multi-objective scoring before being passed to the final ranking.

Engineering Velocity

Lastly, SilverTorch accelerates how quickly the team can build and ship retrieval improvements. Because the entire pipeline lives in one PyTorch codebase, an engineer working on a new retrieval idea writes PyTorch and only PyTorch. There is no longer a need to translate an algorithm from a research notebook into a C++ service, coordinate with a separate infrastructure team, and run a multi-week integration cycle. The time required to build and publish a new innovation dropped from weeks to days.

Engineering for Scale and Freshness

SilverTorch is designed with scalability and index freshness in mind to ensure that it can support a massive scale recommendation system and distribute newly created content in near real time.

Scale Up and Scale Out

Our strategy is to scale up first. We make the most of the single high-performance GPU by carefully orchestrating its memory hierarchy (on-chip SRAM, GPU-resident HBM, host DRAM, remote DRAM) so data lives close to where it’s computed. Once we’ve maximized a single GPU, we scale out within a host, taking advantage of high-bandwidth interconnects between GPU cards on the same machine. 

When the neural network exceeds a single host’s capacity, we use document sharding: split the item inventory (videos, posts, photos) across hosts, like splitting a large library’s catalog across branches. 

For the very large sparse networks inside the model — embedding tables that map every item and every user feature to a learned vector — we use TorchRec, PyTorch’s library for sparse-table sharding. TorchRec spreads these tables across HBM, GPU host DRAM, and even remote CPU-host DRAM, decoupling sparse data movement from computation. 

Index Freshness

With index as a model module, maintaining index freshness equates to updating the model weights of a neural network in production, at scale, without taking the model offline. 

SilverTorch decouples freshness from the full model publish cycle through streaming updates. As model parameters get updated based on the latest training, we periodically publish the full model as a complete snapshot. Between publishes, a continuous streaming service reads real-time signals — new items, updated engagement features, changed eligibility — and applies targeted updates in-place to the specific tensors in the in-memory model. Updates land without interrupting serving and without redeploying the model.

The result shows up in the recency of recommended content. Same-day posts now represent a significant portion of recommendations on social media platforms compared to previous systems.

The Evolution of SilverTorch and What’s Next

SilverTorch is a journey from a system of microservices with neural networks bolted in to a full model-based recommendation retrieval. Two things stand out in retrospect: Full model-based retrieval is viable and efficient at production scale — the architecture breaks down the wall between infrastructure and modeling, and they become one unified practice. It also unlocks better user experience — capabilities like multi-task scoring and neural reranking that prior systems couldn’t run inside the latency budget.

The technical work went through three stages: We first reproduced every baseline retrieval module — ANN, filtering, scoring — in PyTorch. This step alone yielded benefits from high-speed GPU memory and reducing data movements. We then rethought each module in a PyTorch-native, GPU-native way. This is where SilverTorch’s fused Int8 ANN and Bloom index filter came from, designed to compose rather than to stand alone. Finally, we enabled backward propagation for select hand-written modules so they can be trained jointly with the rest of the model.

Looking Ahead

Index-as-Model is the right paradigm for the next generation of recommendation systems, and it’s widely adopted within Meta across different apps. As recommendation systems increasingly incorporate large language models (LLMs) for understanding user intent and content semantics, SilverTorch’s architecture provides a natural integration point:

• An LLM can be plugged into SilverTorch as just another module — the system treats it identically to any other component.
• LLM-based item generation and SilverTorch’s filtering use the same GPU-parallel patterns.
• Item knowledge can be updated in real time through the same streaming infrastructure.
• The LLM and traditional scoring share the same GPU memory — no data movement between services.

In short, SilverTorch lets us integrate LLM capabilities directly inside the retrieval model, rather than orchestrating them as a separate service that sits alongside it. That tighter coupling is what raises the system ceiling for what LLM-powered recommendation can do at production scale.

Read the Paper

For more technical details, see our paper accepted as a full research paper at SIGIR 2026: “SilverTorch: A Unified Model-based System to Democratize Large-Scale Recommendation on GPUs.”

Acknowledgments

We would like to thank the following individuals and our partner teams across Meta for their collaboration in bringing this system to life.

Ryan Chang, Yijie Deng, Fei Ding, Eric Dong, Fan Duo, Zheng Fang, Pawel Garbacki, Hui Geng, Kevin Greer, Max Gu, Ke Huang, Chirag Jain, Anna Jung, Eric Kim, Da Kuang, Xialu Li, Sam Lin, Ziqi Liu, Yiming Ma, Lei Mao, Xiaoheng Mao, Peter Park, Lanbo She, Fangcheng Sun, Jin Sun, Shuo Tang, Harry Tran, Alex Wang, Byron Wang, Jiazhou Wang, Liang Wang, Wenting Wang, Zhen Wang, Zheng Wei, Hong Wu, Peng Xia, Judy Xiang, Bi Xue, Lan Xue, Chao Yang, Shuguang Ye, Hongzhang Yin, Min Yu, Keke Zhai, Qianqian Zhang, Rui Zhang, and Yingjiao Zhao.

Rui Li, Qifan Wang, Shengzhi Wang, Yubo Wang, Yueming Wang, Jiaqi Zhai, Erheng Zhong, and the RecSys Modeling team.

Xinyao Hu, Yanzun Huang, Rui Jian, Min Ni, Qunshu Zhang, Yuting Zhang, Yanli Zhao, and the RecSys Foundation team.

Bruce Deng, Congle Zhang, Luyi Guo, Min Li, Yang Liu, Kai Ren, Guoqiang Jerry Chen, Yimin Tan, Honghao Wei, Li Yu, Lu Zheng, and the Facebook team.

Lihan Bin, Xianjie Chen, Mingze Gao, Abhishek Kumar, Zhengyu Su, Haotian Wu, and the Instagram team

Shujian Bu, Chenglin Lu, Rui Wang, and the Threads team.

Shiyan Deng, Lu Fang, Hongyi Jia, Xudong Ma, Lujia Zhang, and the AI Infrastructure team

Rongrong Hu, Shuyi Zheng, and the Meta AI team.

The post SilverTorch: Index as Model — A New Retrieval Paradigm for Recommendation Systems appeared first on Engineering at Meta.

• We’re introducing Instantaneous PowerLoss Storm, a new testing paradigm within Meta’s infrastructure for handling and mitigating instant or zero-notice power loss in our data centers. 
• We’re sharing: how we built readiness to tolerate instant failures into our existing systems with defense-in-depth strategies; tradeoffs made in implementing it, and how we validated our readiness.

Disaster preparedness is not optional. Hurricanes, wildfires, power supply and network disruptions, and countless more disaster scenarios all pose risks to our data center (DC) infrastructure.

Early warning systems and tried-and-tested mitigation strategies already serve us well in situations where we have a few hours or more advanced warning. While these strategies have matured over time as we have expanded our DC presence, the ever-increasing size and variety of our infrastructure has demanded an increased level of preparedness for zero-notice disasters (ones that occur without any warning), such as instantaneous power loss, with minimal impact to overall fleet availability.

Instantaneous PowerLoss Storm is a new testing paradigm within Meta’s long-established Disaster Readiness (DR) “Storm” program that forms the last line of defense, and the ultimate safety net, to handle and mitigate instant or zero-notice power loss from known, emerging, and unknown risks.

How We Built Readiness To Tolerate Instant Failures Into Our Existing Systems With Defense-in-Depth Strategies.

The capability to handle instant power loss had to be built from the ground up into our DC stack, from mechanical and electrical facilities to server racks, from storage to compute and the core Twine container orchestrator.  Fortunately, each of these architectures was already developed with power loss tolerance as an integral component.

Providing the ability to persist in-memory data when racks have lost power using batteries and Power Loss Siren (PLS) is one such capability. Having a robust DC region-wide asynchronous signaling mechanism for Twine services in the form of unavailability events (UE) is another. (A DC region — referred to as a “region” below — is one where multiple DC buildings are co-located and share common network and power connectivity).

While these abilities were battle-tested and hardened on singular fault domains within single DCs, we identified outstanding vulnerabilities in scenarios encompassing an entire region. Also, testing a region required us to confront problems of not only scale (a typical region is normally 50-60x the size of the typical fault domains) and replica placement, but also of autonomous bootstrapping.

Bootstrapping refers to kickstarting a powered-off region and requiring millions of services to start all at once and discover each other autonomously. We describe two of the problems we encountered with bootstrapping below that required us to adopt a belt-and-braces approach to cover all possible eventualities and contingencies.  

A prominent one to call out — one that haunted us from our earliest days — is that of dependencies, and in particular the dreaded circular dependency, “ouroboros,” risk! Our Twine orchestrator has a set of control plane services —  Scheduler, Allocator, Broker, Zelos (co-ordinator), and so on — without which we cannot run or start any other services in the region.  While the risk from circular dependencies during regular operations is low, the risk and impact are far higher when bootstrapping an entire region. It’s a true chicken and egg problem.

We solved this by identifying critical startup dependencies among the control plane services, and we continuously detect those early and often with Belljar tests in our CI / CD pipelines. These helped uncover and eliminate most, if not all, dependency risks before they are deployed to production. Given the rapid evolution of our Infra, and as a belt-and-braces solution, we also required the capability to break any circular dependencies that may have unexpectedly occurred. A purpose-built Twine recovery kit provides this “jumpstart” capability to recover those Twine services that power Twine itself. Together with Belljar and Twrko, we have been able to successfully put the specter of circular dependencies to rest.  

We also encountered a “boomerang” problem in the same vicinity —  the generator of a critical signal being impacted by the same signal. The UEs used to orchestrate shutdown and recovery of services ended up shutting down the orchestrator control plane services themselves, resulting in orphaned services that could not be “reaped” (because they never received a UE). While this problem could have been solved with intricate solutions such as excluding a preset set of services from the UE dispatch list, we decided to adopt a simpler and more sustainable approach by allowing control plane services to simply “ignore” shutdown signals associated with power-related UEs.     

Tradeoffs Made When Striking the Right Balance Between Reliability and Velocity of Growth.

While it is feasible to build watertight tolerance to instant loss, this can come at opportunity costs for infra or risk overengineering our systems. The latter even has the potential to introduce risks of false positives impacting regular operations. Hence, we needed to make certain tradeoffs to strike the right balance between reliability and engineering.

We began by drawing the line on which impacts must be avoided. Data loss of storage and database systems, permanent damage to DC facilities (mechanical/electrical), or sustained impact beyond a single region are some that we prominently noted as table-stake requirements. Transient service errors, rack failures (within a predefined threshold), and bounded staleness in service routing tables or in region unavailability detection (this is a hard problem for asynchronous systems) were deemed as tolerable risks. In general, only issues which cannot be mitigated through post-incident remediations, and within a reasonable mean time to respond (MTTR), fell outside the boundary of tolerable impact. 

How we validated our readiness through the exercise of Instantaneous PowerLoss Storm, and how this is enabling us to push the envelope further.

Validation of the above expectations and preparation, by de-energizing a large production region, carried significant risks with several known and unknown unknowns. To solve this chicken-and-egg problem of needing to take risk to address risk, we established an incremental approach where we validated self-contained problems such as dependencies when turning up new/pre-production regions, as well as by running tests in “shadow” regions which replicate production regions. Subsequently, we were able to successfully test in our newest (and thus smallest) production regions with limited blast-radius. Finally, we powered off large production regions housing critical storage, AI, and data warehouse workloads. At this stage, we named these Storm exercises Instantaneous PowerLoss Storms.

From 10,000 feet, the Storm consists of a power supply fault being injected to cause immediate de-energization of the entire region, and after a short MTTR remedial “drain” actions undertaken to cordon off the impacted region from global controllers/schedulers. We also aimed to avoid undertaking any preemptive actions prior to the test to truly represent an unexpected loss of power. MTTR chosen for the test mirrored typical MTTR seen during real incident scenarios. 

Each of these exercises helped to train our infrastructure and engineers iteratively towards the long term goal of handling loss of a region as seamlessly as loss of a sub-regional fault domain. 

Stepping Stones Into the Future: Slow is Smooth. Smooth is Fast

Even with all precautions, this has not been an entirely smooth path but one with multiple opportunities for learning and improvement that not only improved our testing capability but also pervaded throughout our Infra with several architectural improvements to our existing systems. 

In tandem, our infra has been evolving rapidly to meet myriad use cases of capacity and AI. Moving fast is possible only when we have strong foundations. Reliability and velocity are two facets of the same coin. You cannot have one without the other. The ability to recover a region from instantaneous failure has laid a strong foundation that has helped enable us to innovate in DC designs and validate them, build reliability in lockstep with rapid capacity deployments, and push the envelope further in what risks we can tolerate. 

 While previous Storms mostly validated storage and database backends, we are adopting the same incremental strategy towards validating regions with live client traffic against instantaneous failures. (More on this in an upcoming post!) We are also continually revisiting and revising tradeoffs in light of new challenges emerging during this growth phase.

The post Lights Out, Systems On: Validating Instant Power Loss Readiness appeared first on Engineering at Meta.

• Adopting AV1 for real-time communication at Meta has been a multi-year effort spanning codec selection, device eligibility, rate control, and error resilience.
• We’re sharing the technical and operational challenges while deploying AV1 and expanding coverage, and how we addressed them for real-time communication.
• We’re presenting several technologies for improving AV1 call quality, including rate control and error resilience.

The AV1 video codec, first standardized by AOMedia in 2018, has rapidly evolved and gained widespread industry support. Today, leading companies like YouTube, Netflix, and Meta stream video using AV1 at scale. Meta introduced AV1 for real-time video calls on high-end devices in 2023, aiming to deliver superior call quality. Since then, we have made notable progress in expanding AV1’s reach and improving the experience for AV1-powered calls. Today, AV1 is enabled on the majority of mobile devices in Meta Real-Time Communication (RTC) applications such as Messenger and WhatsApp.

Why Is Meta Interested in Adopting AV1 for RTC? 

The motivation for switching to a more advanced video codec is straightforward — it delivers the same visual quality while using much less bandwidth. In offline tests, we observed at least a 20% bitrate reduction with AV1 compared with H.264/AVC under our product settings on low-end and mid-range devices. If devices can accommodate higher encoding complexity, the bitrate reductions are even greater. For real-time video calls, this means people on slower or limited networks can enjoy significantly better video quality. This is important to our users because, to meet low-latency requirements, the RTC product must handle bitrate fluctuations. In real-world networks — especially in emerging markets — video bitrates for RTC products typically range from 10 kbps to 400 kbps. Maintaining good video quality below 100 kbps remains challenging.

To evaluate the user experience across codecs, we enabled AV1 in the Messenger app and conducted a side-by-side comparison using two Android phones. In the examples below, AV1 is displayed on the right and H.264/AVC on the left, both limited to 100 kbps. The H.264/AVC video appears noticeably blurry, while the AV1 video remains much clearer — highlighting the significant advantage of AV1 for video calls under bandwidth constraints.

An increased focus on screen content, needs support from high-quality computer generated content encoding. Traditionally, video encoders aren’t that well suited to complex content such as text with a lot of high-frequency content, and people are very sensitive to reading blurry text. AV1 has a set of coding tools — palette mode and intra-block copy — that drastically improve performance for screen content. 

Palette mode is designed according to the observation that the pixel values in a screen-content frame usually concentrate on the limited number of color values. It can represent the screen content efficiently by signaling the color clusters instead of the quantized transform-domain coefficients. In addition, for typical screen content, repetitive patterns can usually be found within the same picture. Intra-block copy facilitates block prediction within the same frame, so that the compression efficiency can be improved significantly. AV1 has the benefit of providing these two tools at the main profile.

The Challenges in Adopting AV1

While the comparison clearly illustrates AV1’s advantages, there are significant challenges to its adoption in RTC. Unlike video on demand (VOD), RTC systems must manage end-to-end video latency, which ideally should remain below 300 milliseconds. If latency exceeds this threshold, people begin to notice delays in the conversation.

Maintaining both high video quality and low latency is challenging. For example, multi-pass encoding techniques — which can improve quality — introduce additional delay. On the decoder side, extensive buffering further increases latency. Additionally, any sudden spikes in bitrate can cause video freezes during calls, degrading the user experience.

RTC products must also dynamically adapt to network conditions during a call. Two challenges are fluctuations in network bandwidth and packet loss.To cope with bandwidth changes, the video encoder adjusts parameters such as resolution and frame rate. However, switching resolutions typically requires a new key frame, which can cause a sudden bitrate spike and temporary video freezing. Similarly, packet loss can trigger retransmissions or force the encoder to send another key frame, both of which may lead to video freezes. Effectively managing these issues helps enable delivery of high-quality, uninterrupted video calls.

Additionally, the RTC client must perform both real-time encoding and decoding, both of which consume significant power — making power efficiency important, especially on mobile devices.

Encoder and Decoder Selection

Choosing the right encoder and decoder is the most critical step in adopting a new codec. The computational complexity of video codecs is a significant consideration for mobile devices. While AV1 offers improved compression efficiency through advanced coding tools, these benefits come at the burden of increased computational demands, particularly during encoding.

To assess this increased complexity, in an offline experiment we integrated an open-source AV1 encoder and measured power consumption on a Pixel 8 device during a video call. The results showed a 14% increase in power usage compared to H.264/AVC — a significant challenge for mobile deployment. To address this, we adopted an internal low-complexity encoder that has similar power consumption as H.264 baseline, as detailed in the next section.

Beyond power, AV1 encoding also increases memory usage compared to H.264/AVC, leading to app crash regressions that further complicate mobile adoption.

Low-Complexity Encoder

A strong encoder should balance visual quality against computational complexity. Low complexity encoding helps enable AV1 encoding on mid-range and low-end devices.

Compared to older codecs like H.264/AVC, newer codecs such as AV1 deliver better compression efficiency. However, these benefits are thought of to come only with higher computational complexity — this represents an obstacle to extending AV1 coverage to low-end devices.

However, a newer codec should not necessarily require a higher-complexity encoder. Because modern codecs support a larger set of coding tools, a well-designed encoder has more opportunities to find better trade-offs between quality and complexity. These trade-offs are also referred to as presets. Ideally, the encoder offers multiple presets, spanning a range from high to low complexity while still maintaining a consistent compression efficiency gain. An ultra-low-complexity preset comparable to H.264/AVC could enable shipping AV1 on low-end phones. 

To address this, we adopted a low-complexity encoder implementation of AV1 for the RTC use cases. In addition to optimizing the quality of the high-complexity preset, we developed an ultra-low-complexity preset. This new preset delivers encoding complexity comparable to H.264/AVC. With it in place, we designed a mechanism that adjusts the encoder preset based on device capabilities, enabling us to ship AV1 to a much broader range of devices.

Decoder Selection

After selecting the encoder, the next step is choosing the decoder. Although video decoders are generally less complex than encoders, we found that decoding complexity remains significant on mobile devices and video calling usecases, especially low-end models. In our initial A/B tests, some low-end devices could not perform real-time decoding, resulting in video freezes and audio/video synchronization issues.

We compared several open-source decoders and, after A/B testing, we selected dav1d for its superior power efficiency and reliability. Our experiments also showed an increase in talk time with the dav1d decoder.

Binary Size

Integrating the AV1 encoder and decoder into the mobile app introduces another challenge: binary size. Using libAOM as an example, AV1 support adds 1.7 MB to the application (600 kB compressed). While this may sound negligible, it’s a major challenge for a company that serves billions of users. Binary size affects update success rates, application startup time, and software health metrics like memory usage and crash rates which can negatively impact user experience. A larger binary leaves more people on older app versions and delays incoming call setup. For example a 600 kB increase could consume an entire year’s binary size budget for a large organization.

We explored several approaches to reduce the binary size. 

• Our initial approach was to use a dynamic-download framework to deliver AV1 as a separate component. However, download failures — whether from poor network conditions, device issues, or random occurrences — degraded the user experience, making this approach insufficient.
• We then focused on direct binary size optimizations. For example, the quantization matrix (QM) tool accounts for about 10% of the encoder’s library size; optimization could halve it. We also contributed size reductions optimizations to the dav1d project.

This strategy extends to end-to-end pipeline optimization, removing unused tools from the library entirely. For instance, removing QM frees 60 kB of binary space. At the application level, we can share codec libraries across features — such as video message transcoding — and leverage built-in platform codec support to avoid bundling additional libraries.

Expanding AV1 Coverage

After selecting the encoder and decoder, the next challenge was identifying which devices are eligible to use AV1. Compiling eligible iOS models was straightforward given the limited number of variants, but Android posed a far greater challenge due to the vast number of device models.

We initially tried selecting devices based on memory, release year, and Android OS version, but none of these strategies proved sufficiently reliable. Ultimately, we leveraged Meta’s in-house ML-based device eligibility framework to generate a reliable list of eligible Android devices.

AV1 Device Eligibility

We created a machine learning (ML)-based device eligibility  framework to support advanced video and audio features based on device capability:

The idea is to use large-scale real-world statistical data to categorize device capabilities, rather than relying on lab data. This helps us scale our device eligibility system and make more accurate decisions. We propose an ML-based device eligibility approach that uses low-level performance statistical metrics collected through our logging pipeline to assess a device’s AV1 capability. The model takes these measurements as input features and outputs an rtc_score, which quantifies the device’s overall AV1 performance. This score then informs decisions such as optimizing call settings and determining whether a device can run the AV1 codec efficiently.

In 2025, we iteratively refined our model using AV1-specific data and significantly expanded device support. Our first milestone, Model V1.1, rolled out in August 2025 and broadened AV1 traffic across an increasing set of devices. That additional traffic contributed to a dedicated AV1-only dataset that became both larger and more representative over time. With this richer data, we built Model V2, introducing a two-tier approach that differentiates between higher-end and lower-end devices—reflecting the reality that entry-level phones and flagship devices can have very different AV1 encoding capabilities. Across these iterations, we substantially increased AV1 enablement across the device landscape, with an approach designed to keep improving as traffic grows and more data becomes available. 

As AV1 traffic continues to grow, we expect iterative optimization will further improve both call duration and quality.

Codec Complexity Adaptation

Device eligibility  lets us identify capable devices, but we discovered an additional challenge: During A/B tests, we observed calls with significant audio/video sync regressions, primarily caused by devices unable to encode or decode video in real time. Surprisingly, even a 2023 smartphone with an octa-core processor could not handle encoding at 320×180@15fps. This issue affected both H.264 and AV1, though it was more prevalent with AV1. We suspect these devices throttle CPU frequency during calls, reducing their effective capability.

As a result, enabling AV1 purely based on device name is not sufficient. We needed a more robust mechanism to adjust codec complexity based on both local and peer device status. We developed three mechanisms: adaptive encoder preset adjustment, encoding latency-aware codec switching, and decoding latency-aware codec switching.

Adaptive Encoder Preset Adjustment

We designed multiple encoder presets ranging from low to high complexity. A monitoring mechanism continuously tracks encoding latency during calls to select the appropriate preset. If encoding latency becomes too high — meaning the device is close to being unable to encode in real time — we reduce encoder complexity. Conversely, if the device can sustain higher complexity, we increase the preset to achieve better quality.

Local Device Encoding Latency-Aware Codec Switch

If lowering the encoder preset still does not reduce encoding latency to an appropriate level, we apply codec switching. In this case, the device switches to H.264/AVC, which may be  less computationally intensive than AV1 for that specific content. To enable this, we negotiate support for both codecs at call setup, and the client continuously monitors device conditions to determine the most appropriate codec. Encoder preset and codec selection are decided jointly to optimize call quality and prevent codec-selection oscillation.

Peer Device Decoding Latency-Aware Codec Switch

Because AV1 also has higher decoding complexity, we want to ensure the peer device can decode AV1 frames in real time. This is especially important when a high-end phone calls a low-end phone: the sender may be able to encode AV1, while the receiver may not be able to decode it in real time.

To address this, each device continuously feeds back its video decoding latency during the call. If the sender detects that the peer cannot decode AV1 in real time, it switches back to H.264/AVC.

Together, these mechanisms adaptively adjust both the encoder preset and the codec based on encoding and decoding latency. Beyond latency, we also consider other device health signals, such as battery level. For example, when the battery is low, we switch to H.264/AVC. This helps maintain call quality and extends call duration.

Asymmetric Codec Design

With the improved codec-selection strategy, we rolled out AV1 support to mid-range and low-end Android devices. While some mid-range devices cannot perform real-time AV1 encoding, many can decode AV1 in real time. This enables an asymmetric codec design: mid-range devices continue to encode and send H.264/AVC, but can receive AV1 from high-end peers. As a result, we significantly increased AV1 coverage across Android devices.

Improving AV1 Call Quality

The preceding sections described our framework for enabling AV1 on a wide range of devices. With this system in place, AV1 now powers the majority of mobile devices in Meta RTC (Real-Time Communication) applications. . The next challenge is further improving AV1 call quality.

As discussed earlier, RTC products must dynamically adapt to network conditions during a call. Two notable challenges are fluctuations in network bandwidth and packet loss. Accurate rate control helps address bandwidth changes. Error-resilient strategies play an important role in ensuring reliable quality in the presence of packet loss.

Accurate Rate Control

In RTC, maintaining a constant bitrate (CBR) is important. Any instantaneous bitrate overshoot can lead to congestion and video freeze on the peer’s side. RTC applications are sensitive to instant bitrate overshoots, so simply checking average bitrate is insufficient. We use Video Buffering Verifier (VBV) delay as a metric to evaluate CBR accuracy.

VBV Delay

The Video Buffering Verifier (VBV) is a leaky-bucket-based measurement used to ensure that an encoded video stream can be correctly buffered and played back at the decoder.

We use a similar method to measure CBR rate control accuracy. The figure below shows an example:

Assume the current network bandwidth allocated to video is 100 kbps and we ask the encoder to encode frames at 100 kbps. The encoder encodes Frame (Frm) N at 20 kbits. At the same time, Frame (Frm) N-1 has not been fully transmitted, and 5 kbits remain in the buffer (likely from an overshoot on Frame N-1).

Sending Frame N would therefore take at least (20 kbits + 5 kbits) / 100 kbps = 0.25 s = 250 ms. Consider a system in which the desired VBV delay for RTC is below 200 ms. In this example, encoder overshoot and a large VBV delay are likely to lead to a poor user experience—for example, higher latency, network congestion, or video freezes. This highlights the importance of accurate rate control for RTC use cases.

Rate Control Optimization

We made several rate-control improvements to ensure the encoder does not overshoot. During encoding, the encoder tracks VBV buffer status and uses it to guide bitrate allocation. When an overshoot occurs, it reduces the rate of subsequent frames to keep VBV delay under control. In our experience, many video encoders do not handle this well, allowing VBV delay to grow and potentially cause network congestion.

Similarly, encoders often allocate a high bitrate to intra-only (key) frames to maintain quality consistency between key frames and inter frames. Some encoders even “boost” key-frame quality to improve reference-frame quality. In RTC, however, we want to avoid bitrate spikes. The encoder therefore strictly controls key-frame bitrate and reduces the rate of subsequent frames to compensate for any overshoot.

Rate control in RTC also presents challenges:

• Frequent target bitrate changes. The client may update the encoder target bitrate frequently. A robust encoder must keep VBV delay under control — especially when the target bitrate drops sharply.
• Frequent resolution changes. The client may also change resolution often during a call. A rate-control algorithm should therefore remain stable and effective under frequent resolution changes. In addition, AV1 supports a useful feature to address this issue, called Reference Picture Resampling (RPR), which allows resolution changes without generating a key frame. This can reduce bitrate spike significantly and improve the video freeze.

Because the video encoder interacts closely with the network congestion-control module, we found that preventing undershoot is as important as preventing overshoot. In our early versions of the rate-control algorithm, we used conservative rate allocation to avoid overshoot, but this increased the tendency to undershoot. Undershooting can mislead bandwidth estimation, slow bitrate ramp-up, and ultimately degrade video quality. We therefore revised the algorithm to address undershoot and improve bitrate accuracy.

Overall, an accurate rate-control algorithm that produces a stable bitrate — without significant overshoot or undershoot — can substantially improve video-call quality.

Error Resilience

RTC imposes strict latency constraints, while modern video codecs rely on long, tight chains of inter-frame dependencies. When a packet is lost, the receiver must send a NACK and wait a round trip for retransmission. If that fails, the dependency chain breaks and the video freezes. The receiver then requests a keyframe, which costs another round trip, but because keyframes are roughly 10x larger than typical P-frames, they can congest the network and increase packet loss, creating a problematic cycle. To mitigate this, we tuned AV1 for fast recovery and drift containment under packet loss by leveraging temporal layers (TL) and Long-Term Reference (LTR) frames.

Temporal Layer (TL)

Temporal layers are a form of temporal scalability used in modern video codecs (including AV1) where the encoder organizes frames into a time-based hierarchy. The base layer (temporal layer 0) provides a lower frame rate on its own, while enhancement layers (temporal layer N) add intermediate frames to reach higher frame rates when conditions allow. Figure 4 shows the two-layer structure we use for AV1.

A notable property of this structure is that the base layer maintains continuity, without relying on enhancement-layer frames.If enhancement-layer packets are lost or arrive too late, decoding can still proceed using the base layer without stalling. We take advantage of this by prioritizing robustness by layer: We apply FEC to protect base-layer data rather than spending redundancy on enhancement data. We also treat enhancement-layer retransmissions more conservatively — when round trip time (RTT) is low, retransmitting a missing enhancement packet can help; when RTT is high, we may skip retransmissions without breaking the decode flow.

There is a trade-off: Compared to a tightly dependent prediction chain (where each frame references the immediately preceding frame), a temporal-layer structure is typically less compression-efficient, so leaving TL enabled all the time can degrade quality at a given bitrate. But TL’s benefits show up mainly under lossy or unstable networks, which are only a subset of real-world calls. For that reason, we enable TL adaptively. The sender monitors network feedback, turns TL on when loss rises, and turns it back off once conditions recover. This gives us resilience when we need it without sacrificing efficiency when we don’t.

Long-Term Reference (LTR)

LTR is an error-resilience feature that allows a video encoder to store reference frames in the buffer longer than regular reference frames and send LTR-predicted (LTRP) frames as requested. When the decoding chain is broken due to frame loss, an incoming LTRP frame—predicted from a previously decoded LTR frame—instantly resynchronizes sender and receiver, recovering from the loss. Figure 5 illustrates how LTR and LTRP frames work in lossless and lossy scenarios.

Implementing LTR requires close coordination with the network layer. Figure 6 shows how the AV1 encoder interacts with the network layer. The encoder periodically emits LTR frames and pins them in its bounded reference buffer of size 4, evicting the oldest pinned LTR when a new one is added. From the network layer’s perspective, however, an encoded LTR frame looks the same as any other frame, so the network cannot tell when to send an ACK back to the encoder. To make this reliable, the encoder sends an explicit LTR indicator when handing the frame to the network layer. This differs from H.264, where LTR and non-LTR reference frames are distinguished by bitstream syntax — the network layer can parse the H.264 slice header to recognize an LTR frame and ACK the sender upon receipt.

The explicit LTR indicator is a binary flag carried in our proprietary RTP header extension, which we use to transport per-frame metadata on the primary channel. We also expose the frame_id to the network layer through LTR bitstream syntax. ACK feedback is sent via a separate proprietary RTP header extension. Each ACK includes the corresponding frame_id, allowing the sender to unambiguously identify which LTR was received. When servicing an LTRP request, the encoder always uses the most recently ACKed LTR as the prediction reference.

The network layer requests an LTRP frame from the encoder in two cases. The first is reactive recovery, when the receiver experiences a freeze and sends an RPSI to request an LTRP. The second is proactive protection, when the sender detects elevated packet loss via a feedback channel and asks the encoder to send LTRPs periodically. While the proactive path can be somewhat redundant, it significantly improves reliability and reduces freezes. From the encoder’s perspective, the reason does not matter — it simply receives an LTRP request and responds based on whether it has an ACKed LTR reference in the buffer. If an LTR is available, the encoder produces an LTRP frame. If not, it assumes resynchronization is needed and sends a key frame instead.

While LTR is more efficient for loss recovery than forcing a key frame or relying on retransmissions, it can reduce overall coding efficiency because an LTRP frame may reference an older LTR with weaker temporal correlation, making motion prediction less accurate. We mitigate this by leveraging an existing encoder design choice — the encoder already emits a periodic, slightly higher-quality frame to improve overall quality. We simply mark that frame as LTR, so the LTR remains high quality even as it ages.

Meta’s Ongoing Journey With AV1

Adopting AV1 for real-time communication at Meta has been a multi-year effort spanning codec selection, device eligibility, rate control, and error resilience. By combining a low-complexity encoder with ML-based device eligibility, adaptive codec switching, and robust error-resilience mechanisms, we have enabled AV1 on the majority of mobile devices — delivering meaningful quality improvements, especially for users on bandwidth-constrained networks. This initiative complements our ongoing efforts to expand AV1 for VOD applications. As device capabilities continue to improve and ML models leverage more data, we expect AV1 coverage and call quality to keep advancing.

Meanwhile, we are working on extending AV1 to group calls. Unlike 1:1 calls, participants in group calls must decode multiple video streams, which makes increasing AV1 coverage in group calls more challenging.  While software AV1 implementations aid the steady expansion of AV1 coverage, higher quality and improved features will likely require AV1 hardware support.

The benefits of AV1 are clear, and most content and RTC service providers are moving to AV1 as their flagship codec. We encourage SoC vendors to invest in HW AV1 across all device tiers to meet the AV1 requirements to deliver an improved viewer experience, device battery savings and enhanced network operator infrastructure efficiency.

The post Adopting AV1 for Real-Time Communication (RTC) at Scale appeared first on Engineering at Meta.

Smart glasses like the Ray-Ban Meta and Oakley Meta Vanguards need to pack enough energy to power features like cameras, speakers, AI workloads, and even a display. But it all has to fit into the glasses’ temple arms.

So how do you place a battery with enough power to run a pair of smart glasses all day into a form factor narrower than an adult’s pinky finger? You have to rethink how batteries are made. 

In episode 86 of the Meta Tech Podcast, host Pascal Hartig sat down with Karthik and Myuran, the engineers behind Meta’s steel can battery technology, for a conversation on powering the newest and next generation of wearables. 

Why Traditional Batteries Fall Short for Smart Glasses

Traditional pouch cells — the batteries in most phones and laptops– can’t cut it for devices like smart glasses because they’re difficult to reshape and shrink down. Their folds waste volume, their tolerances eat into precious millimeters of space, and at smaller sizes they can have difficulty providing peak power for multitasking (for example, if someone is using the camera and asking the AI model to perform a task at the same time). 

Smart glasses need a battery that can claim every micron of space – something rigid, precise, and shaped to the product rather than the other way around.

Enter Steel-Can Cells (at Never-Before-Seen Widths) 

Steel-can batteries aren’t new. Power tools and watches use them. But Meta’s AI glasses needed batteries with widths as narrow as 7mm, narrower than anything that existed before. Getting there meant rethinking nearly every internal component of the battery. 

The Electrode Architecture

Traditional steel-can cells use a wound “jelly roll” of electrode material. Meta’s engineers replaced that with die-cut stacked layers, similar to wiring small resistors in parallel. The result is dramatically lower impedance, which matters when peak power is required so that the device can avoid brownouts if a lot of power is being demanded at the same time (because someone may be making a recording while asking the AI a question at the same time). 

Tolerances

A steel-can cell holds its shape to roughly 100 microns. On a 10mm-wide battery, that gives back real usable volume that translates directly into additional energy density and runtime.

New Challenges With Each Generation

From Gen 1 to Gen 2 the Meta Ray-Ban’s, cell capacity grew from 160 mAh to 210 mAh — roughly a 30 percent bump. Yet the product shipped with claims of double the runtime. The chemistry didn’t change. The extra gains came from system-level efficiency improvements across hardware and software such as better power management, tighter firmware control, and a form factor that allowed for a larger cell

The Oakley Meta Vanguards actually feature a battery in each temple arm, which introduced a real systems puzzle at the intersection of electrical, firmware, and mechanical engineering. The cells in each temple arm are symmetric, but the electronic loads aren’t split evenly between the two sides. That creates cross-charging risks and sequencing complexity at boot and shutdown. 

Then the Meta Ray-Ban Display glasses introduced the most demanding power profile yet. Its screen draws sustained power rather than short bursts, which required designing a 248 mAh steel-can cell, the largest in Meta’s lineup.

More Power to the Wearables

The ultra-narrow steel-can approach we developed for our smart glasses is proving adaptable to other form factors across Meta’s hardware portfolio.

Meta is now focused on scaling and democratizing this technology across multiple vendors, ensuring we have resilient supply and can bring these batteries to the next generation of wearables.

Listen to the full episode to hear the complete story — from first sketch to global shelf — including details on cross-charging two-battery systems, software versus hardware iteration cycles, and what it’s really like to collaborate across time zones to build something the world has never seen.

Listen now

You can also find the episode wherever you get your podcasts, including:

• Spotify
• Apple Podcasts
• PocketCasts

Timestamps

• 0:06 — Intro and News
• 1:49 — Guest intros
• 4:16 — The problem with existing batteries
• 6:40 — Pouch vs. steel-can batteries
• 10:27 — What lower impedance means
• 12:25 — Power requirements
• 16:02 — Synchronizing two batteries
• 23:11 — Manufacturing never-done-before batteries
• 28:12 — Software vs. hardware iteration cycles
• 30:51 — Collaborations across the globe
• 37:00 — Market compliance
• 42:24 — Outro

The Meta Tech Podcast is a podcast, brought to you by Meta, where we highlight the work Meta’s engineers are doing at every level – from low-level frameworks to end-user features.

Send us feedback on Instagram, Threads, or X.

And if you’re interested in learning more about career opportunities at Meta visit the Meta Careers page.

The post How Meta Engineered Ultra-Narrow Batteries for AI Glasses appeared first on Engineering at Meta.

Privacy controls — systems that enforce retention, access, allowed-purpose, downstream-sharing, or anonymization policies — require a reliable understanding of data to function. Before such a control can operate effectively, it must know exactly what it is looking at. This can be complex, as demonstrated by a field simply named “age“: In one context, it might describe a person and require strict protections, while in another, it could be a cache time-to-live (TTL) numerical value in an infrastructure pipeline.

This is the everyday problem behind privacy-aware infrastructure (PAI): The inputs are noisy and probabilistic, but the outputs need to be precise enough to drive enforcement. 

AI-native products make that problem harder. They introduce new data modalities, faster iteration cycles, derived features, embeddings, multimodal inputs, and changing policy interpretations. Manual review remains important for judgment and accountability, but it cannot keep up with the volume and pace of change.

At Meta, we apply a hybrid pattern for asset classification at scale:

• Build a rich context before asking a model to reason.
• Use LLMs to handle ambiguity, cold start, and novelty.
• Keep human-reviewed labels separate from model-generated recommendations.
• Distill stable behavior into deterministic, versioned rules for routine enforcement.

The end goal is not “LLMs everywhere.” Instead, it is a system that can learn from ambiguous signals while moving production enforcement toward logic that is low latency, replayable, and easier to audit.

The LLM does not make the production decision in the common case, deterministic rules do. We use LLMs deliberately and narrowly, to interpret novel or ambiguous assets, and then to distill what they learn into versioned human-reviewed deterministic rules, which steadily shrinks the LLM’s role in production over time. Humans stay in the loop where it matters most. People adjudicate the reviewed reference labels, and they review and approve rule promotions that could change how protection is enforced.

PAI addresses four operational concerns: 

1. Understand what data exists and how it is governed. 
2. Discover which data flows are relevant to a policy question. 
3. Enforce retention/access/purpose/sharing constraints.
4. Demonstrate compliance through verifiable evidence.

Asset classification sits at the understand layer. It provides the foundation that every downstream concern depends on.

Why Asset Classification Matters

Asset classification is the foundation for many privacy controls. Before a system can enforce retention, access, allowed-purpose, downstream-sharing, or anonymization policies, it needs a reliable view of what the asset is and how it should be governed.

An asset can be more than a table or column. It can be a nested field inside a payload, a log key, an event parameter, an API field, a machine learning (ML) feature, an embedding, or a derived dataset produced by an intermediate pipeline. That breadth matters because AI-native systems often transform data across many representations. A single source signal can move through pipelines, become a feature, appear in a model-training workflow, or be joined with other derived signals. Classification has to follow the meaning of the data, not just its shape.

There are four recurring challenges:

First, noisy and weak signals: Dozens of context fields are fetched per asset, which forces the model to rediscover what matters each time. High token usage dilutes attention, and decision boundaries get buried in irrelevant or misleading fields. A field called age in a caching pipeline is a concrete example: Without code resolution and lineage analysis, a classifier will trigger false restrictions on the entire pipeline.

Second, the relevant context is distributed. Code, lineage, ownership, semantic annotations, documentation, and usage patterns often live in different systems. A good classifier needs to assemble that context before making a decision.

Third, requirements evolve. Product teams move quickly, and policy interpretation can change as new product capabilities appear. A static rule set or periodic manual review process can leave gaps between reviews.

Fourth, classification is only useful if it feeds enforcement. A false positive can trigger unnecessary restrictions downstream. A false negative can leave a protection gap. The classifier sits near the front of the enforcement pipeline, so its error profile affects every system that depends on it.

This creates the central tension: Classification needs to reason under ambiguity, but enforcement needs decisions that can be explained and reproduced later.

The Pattern

Our approach is built around three principles that emerged from building and operating the system:

First, context beats prompts. Most classification failures were not caused by weak instructions; they were caused by weak or missing evidence. Hours of prompt optimization produced marginal improvement when the model was reasoning over raw, noisy fields. Structuring context into evidence briefs, with supporting signals, contradicting signals, provenance, and masked circular fields, produced much larger accuracy improvements. The practical lesson is simple: Focus on what goes into the model before optimizing how you ask.

Second, decouple evaluation from optimization. LLM outputs are useful recommendations, but they cannot become their own ground truth. The evaluation loop needs to stay independent from the classifier: different models, different prompt strategies, frozen reference sets, human-reviewed labels, and regression gates. If evaluation and optimization share the same loop, the system can end up measuring drift instead of progress.

Third, distill stable behavior into deterministic rules. LLMs are useful for ambiguity, cold start, and new patterns. They are not the right default enforcement mechanism at scale. When the system finds stable, validated patterns, those patterns should become versioned, auditable rules that run without the LLM. Over time, the classifier should progressively shrink its own LLM surface area, leaving model inference for novel or ambiguous assets while routine enforcement becomes deterministic, low-latency, and replayable.

These principles translate into a concrete operating pattern: Define a stable classification contract, build a context mesh, route decisions through a deterministic-first funnel, and keep the learning loop safe with independent evaluation and reviewed labels.

To execute on this pattern, we break the work down into seven practical stages. These stages transform the high-level architecture into a concrete, repeatable process.

The rest of this post walks through those pieces using asset classification as the case study.

1.) Start With the Contract

A classifier should behave like a platform service. That means its contract should be small, explicit, and stable. For each asset, the classifier receives an identifier and a bundle of context. It returns a structured result with:

• A category from the classifier’s taxonomy.
• A confidence score – a raw model self-assessment whose calibration we evaluate against reviewed labels (see below).
• A decision trace showing which evidence influenced the result.
• The rule that matched, if the decision came from deterministic logic.
• Version information for the context, rules, and prompt used to make the decision.

The taxonomy is domain-specific. One classifier might distinguish user data from operational data. Another might classify whether an asset is eligible for a particular AI-training use case. We avoid forcing every classifier into one universal taxonomy. Instead, each classifier owns one scoped question, and downstream systems compose the answers when they need multiple facets.

That scoping is important. A narrow classifier is easier to evaluate, easier to debug, and easier to govern. It also makes the decision trace more meaningful because the classifier is explaining one decision, not trying to solve every policy question at once.

2.) Build Context Before Prompting

Most classification failures are not prompt failures. They are context failures. If the only signal is a field name, the model has to guess. If the system can also provide code references, lineage, ownership, semantic annotations, and nearby usage, the model can reason from better evidence.

In practice, the context mesh may include:

• Source-code resolution, including where a field is defined or used.
• Ownership and organizational metadata.
• Semantic annotations, such as data type or origin.
• Lineage signals that show where data came from and where it flows.
• ML heuristic outputs from scanners or embedding-based classifiers.
• Code search results that show references, logging declarations, or call sites.

The point is not to pass everything to the LLM. More context is not automatically better. Some fields are redundant. Some are noisy. Some can create circular reasoning if they already encode the label we are trying to predict.

So the system creates an evidence brief – a compact summary of the strongest supporting signals, contradicting signals, and provenance chains. Instead of asking the model to sift through raw context, we ask it to reason over the evidence that is most relevant to the classification decision.

Without this structuring, the model receives dozens of raw fields per asset and must rediscover what matters leading to high token consumption, diluted attention, and decision boundaries buried in noise. The evidence brief solves this by pre-ranking signals. For a field like user_payload.email_address, an evidence brief might say:

• Supporting signal: Lineage connects the asset to a user-facing logging pipeline (weight 0.8).
• Supporting signal: Semantic annotation indicates EMAIL-like data (weight 0.9).
• Contradicting signal: Ownership metadata points to an infrastructure team, not a user-facing product (weight 0.3).
• Suppressed signal: An existing privacy label was removed to avoid circular reasoning.

That last point matters. A model should not be allowed to “discover” the correct answer by reading a field that already contains the answer. Masking is not just prompt hygiene, it is a system invariant. Fields masked from the LLM are also blocked from learned rule distillation so the model cannot smuggle the answer into a rule by way of a circular field. Deterministic rules that use high-risk fields require explicit review.

Over time, the system can also learn which context fields are useful. Fields that consistently improve classification can be prioritized. Fields that are unstable, redundant, or harmful can be suppressed. This turns signal quality from a matter of intuition into something measurable.

3.) Use a Decision Funnel

Once the context is assembled, the classifier routes the asset through a decision funnel.

The first path is deterministic. If a known, versioned rule matches the asset, the classifier can return a decision quickly and with a clear explanation. Deterministic rules work well for stable patterns – a well-understood namespace, a semantic annotation with high precision, or a combination of signals that has been validated over time.

The second path is LLM-based. If the asset is novel, ambiguous, or outside current rule coverage, the classifier asks the model to reason over the evidence brief. The model returns a candidate label, confidence indicators, a decision path, and cited evidence.

In our production deployment, Figure 7 shows how cheap deterministic rules resolve the large majority of traffic, roughly 85%, in single-digit milliseconds. The LLM is reserved as a fallback for the roughly 15% that is novel or ambiguous. That path is slower — on the order of seconds — and roughly 400 times the compute cost, so it is budgeted separately. Both paths emit the identical result schema. The masking invariant is enforced on each.

That confidence deserves a careful read. The raw score is a model self-assessment, a number the model produces from its own judgment, not an inherent probability of being correct. So we evaluate its calibration against reviewed labels. Raw scores are compared to the correctness rate actually observed on the human-reviewed reference set, which tells us how well a given score tracks a real probability of being right. Confidence-based routing in the funnel, for example, accept automatically versus route to human review, should use calibrated scores where that calibrated path is enabled, rather than the raw number

Both paths emit the same result format. Downstream enforcement systems do not need to know whether a decision came from a rule or from model-based reasoning. They receive a category, confidence, trace, and versioned decision metadata.

This split is what makes the pattern practical. LLMs are useful for ambiguity and cold start. Rules are better for routine enforcement. The more stable behavior we can distill into rules, the less often the serving path needs model inference.

Rule coverage becomes an important operational metric. If coverage rises while quality holds steady, the classifier is moving toward a healthier steady state: fewer routine calls to the model, lower resource use, lower latency, and decisions that are easier to replay.

A critical system invariant: Fields masked from the LLM are also blocked from learned rule distillation, so a masked signal cannot re-enter the decision through an automatically distilled rule. In one production deployment, a subtle bug in how masked context was handled during rule evaluation caused rules to silently fall through to LLM fallback, so rule coverage appeared to plateau even as the rule set grew. Fixing that handling immediately increased rule coverage and cut LLM inference calls significantly. 

The lesson: Masking is not a prompt-engineering concern, it is a system invariant. And deterministic rules that rely on high-risk fields require explicit review rather than inheriting masking implicitly.

4.) Solve Cold Start Deliberately

On day zero, a classifier has a hard problem: There may be millions of assets and very few reviewed labels. Random sampling is not enough. The categories that matter most for privacy can be rare, and rare categories are easy to miss if you wait for examples to appear naturally.

Instead, we seed the process with policy-guided examples:

• Rare sensitive categories.
• Borderline cases where policy interpretation is difficult.
• Negative examples that look sensitive but are not.
• Assets where context signals disagree.

The goal is not to eliminate human review. It is to focus human attention on the cases where judgment matters most.

5.) Keep the Learning Loop Safe

Once the classifier is live, it needs to improve without grading its own homework.

We separate two loops:

The reference loop produces reviewed labels. These labels are append-only, versioned, and tracked with provenance. If a label changes, the history is preserved rather than overwritten. Model-generated labels are useful recommendations, but they do not become reference labels automatically. Humans adjudicate uncertain or high-risk cases, and those adjudicated labels become the reference set for evaluation.

The optimization loop improves prompts, routing, context usage, and candidate rules. It can evolve quickly, but it is evaluated against the reviewed reference set, not against labels produced by the same model it is trying to optimize. This distinction matters: A classifier that trains or validates itself on its own predictions can appear to improve while drifting away from the policy intent.

For quality control, we use a multi-panel judge – three independent LLM evaluations, each with a different prompt strategy. One classifies directly from evidence. One critiques the reasoning first, then classifies. One focuses exclusively on metadata signals, such as on-call, lineage, and semantic annotations, while ignoring names and descriptions. All three share a single judge model, a larger reasoning model deliberately different from the classifier model.

The three judges share one scaffold and differ only in how they are asked to reason. The skeleton below is illustrative, not the literal production prompts, but it shows the structure. Each judge receives the same masked evidence brief, the masking invariant still holds, and each returns a structured verdict.

# Shared scaffold (all three judges)
INPUT  = masked_evidence_brief   # pre-existing privacy label removed; masking invariant holds
OUTPUT = {label, rationale, confidence}
JUDGE_MODEL = larger reasoning model, deliberately != classifier model
 
# V1 - direct-from-evidence
verdict_1 = judge(brief, instruction="Classify the asset directly from the evidence.")
 
# V2 - critique-then-classify
verdict_2 = judge(brief, instruction="First critique the supporting and contradicting signals, then classify.")
 
# V3 - metadata-only
verdict_3 = judge(brief, instruction="Use ONLY metadata signals (on-call, lineage, semantic annotations). Ignore names and descriptions.")
 
# Aggregate
final_label = majority_vote(verdict_1, verdict_2, verdict_3)
Agreement = cohens_kappa(verdict_1, verdict_2, verdict_3)   # inter-rater reliability

Results aggregate by majority vote. We track panel agreement across the three judge framings as a stability signal, while Cohen’s kappa (κ) compares the judge consensus against the reference labels (or against the classifier output), providing a statistical signal about classification reliability. These kappa scores drive structured loop decisions: Continue when the system is healthy, WidenAudit when label noise is suspected, FreezeAndAudit when quality declines for two or more iterations, and DataProblem when labels or taxonomy appear fundamentally broken and the system should halt and escalate. This prevents the iteration loop from shipping regressions to production.

For imbalanced taxonomies, we use metrics that expose rare-class failures. Accuracy alone can be misleading: A classifier that labels everything as non-sensitive may look accurate if sensitive assets are rare. Matthews correlation coefficient, macro F1, per-class recall, balanced accuracy, and calibration checks give a more complete picture.

We also look for fragile decisions. One useful test is counterfactual masking: Remove one context field at a time and classify again. If the decision flips when a single weak signal disappears, the asset is flagged for review. The original prediction may still be correct, but the reasoning may be too brittle for confident automation.

When quality drops, the system should slow down or stop. That can mean widening the audit sample, freezing optimization, or escalating a taxonomy or labeling problem for human review. A learning system needs brakes, not just accelerators.

6.) Distill Stable Behavior Into Rules

Even a strong LLM classifier should not be the default enforcement path forever. This distillation (not autonomous decision-making) is where we concentrate the model’s value. Any rule that could change how sensitive data is protected is reviewed and approved by a person before it goes live.

As the system collects reviewed labels and decision traces, it can identify patterns that are stable enough to encode as deterministic rules. A rule might capture a high-precision semantic annotation, a reliable ownership and lineage combination, or a repeated pattern across a class of assets.

Candidate rules go through validation before they affect serving decisions. A typical flow looks like this:

• Propose a rule from stable context and label patterns.
• Test it against a held-out reviewed set.
• Run it in shadow mode on production-like traffic without changing serving behavior.
• Promote it only if quality, coverage, and regression checks clear the required gates.
• Retire or revise it if the pattern becomes stale or quality degrades.

Distillation operates in stages of increasing complexity:

Stage 1: Field-based rules. Extract single-field patterns (exact match, keyword, numeric range, value-set membership, namespace patterns), with a minimum support of two assets and minimum purity of 80%.These are candidate-mining thresholds for surfacing rules to evaluate, not promotion thresholds. Every candidate from any stage still has to clear holdout validation, a higher dev-precision bar, shadow mode, and human review where protection could change before it can serve. 

Stage 2: Composite rules. For uncovered categories, search for conjunctions (e.g., “on-call contains X AND semantic type is ACCOUNT_ID”) under stricter gates — 95% purity, 10 examples minimum, and a stability check on 50% subsamples. 

Stage 3 (optional): LLM-assisted rule generation. The model proposes custom conditions combining lineage depth with ownership patterns that manual heuristics miss, gated by rollout controls and default-off. Each candidate rule then proceeds through: holdout validation → blacklist if failed (bounded-TTL) → shadow mode (log, don’t apply) → promote to rules.yaml only if quality gates clear. Promoted rules shrink the LLM surface area.

The important principle is that deterministic rules should not quietly reduce protection. Rule promotion needs safeguards that are designed to catch regressions, especially for sensitive classes.

Validated rules are exported to Python, SQL, JSON, or Hack for deployment in production systems with zero LLM dependency. We manage these rollouts using compare-and-swap (CAS) semantics: We write immutable rule and prompt versions, then activate them via a lease-guarded compare-and-swap on the published pointer (atomic within our single-writer model). This ensures the production path remains a deterministic engine, while the LLM is reserved solely for novel assets that lack rule coverage.

This is what makes the hybrid approach sustainable. LLMs help the system learn. Deterministic rules help the system enforce.

7.) Automate the Right Things

Automation is necessary, but the boundary matters.

We automate context acquisition, evidence brief generation, candidate classification, evaluation runs, failure analysis, and candidate rule proposal. These are high-volume tasks where automation can reduce manual toil and make the process more consistent.

We keep human review in the places where judgment matters – ambiguous policy interpretation, reviewed reference labels, high-risk disagreements, and promotion decisions that could materially affect protection. This is a routing policy, not a prompt. 

A decision is escalated for human review when any of the following hold:

• Low calibrated confidence. The calibrated confidence falls below the auto-accept threshold, so the decision is not safe to ship automatically.
• Judge-panel disagreement. The three independent judges produce no clear majority, or inter-rater agreement (Cohen’s kappa) is low, a signal that the case is genuinely ambiguous.
• High-cost rare class. The candidate is a rare sensitive category where a false negative is expensive, so the asymmetric error cost warrants a human check even at moderate confidence.
• Fragile reasoning. Counterfactual masking flips the label when a single weak signal is removed.The prediction may still be right, but the reasoning is too brittle for confident automation.
• Protection-reducing rule promotion. A candidate rule would change enforcement for a sensitive class in a way that could reduce protection. Deterministic rules should not quietly weaken it.
• Controller escalation. The tuning controller enters Pausing or Diagnosing, indicating a quality concern or a fundamental labeling or taxonomy problem that a human must resolve.

That balance is deliberate. Privacy-aware infrastructure should not hide uncertainty. If the model, judge, or evaluation loop disagrees, the system should surface that disagreement as a useful signal. Sometimes the right answer is not a better prompt. Sometimes the right answer is clearer policy guidance, better labels, or a narrower taxonomy.

The best automation in this space does not replace people. It concentrates human attention on the hardest cases, records the reasoning, and turns stable learning into repeatable enforcement over time.

What We Learned

Context Quality Beats Prompt Quality

When classification stalls, it is tempting to keep tuning the prompt. In our experience, better context often matters more. Code resolution, lineage, ownership, and semantic annotations can change the decision space in a way prompt edits cannot.

The practical lesson is simple: Before asking whether the model needs a better instruction, ask whether it has the evidence a human reviewer would need. We saw this with a field named age in a caching pipeline. It was a cache TTL, not a person’s age, and prompt-only changes did not fix it reliably, adding code resolution and lineage did. Once the model could see that the field resolved to a TTL, the false positive went away.

Determinism Means Replayability

The goal is not to make an LLM produce the same text every time. The goal is to reproduce a decision later using the same versioned inputs, context, and logic.

That is why versioning matters. A useful decision trace should tell us what evidence was used, which rule or prompt version was active, and how the decision can be replayed during debugging, incident review, or audit support. In one review, we replayed a single past classification from its stored decision trace and the pinned context, rule, and prompt versions, and reconstructed exactly why the asset received the label it did, without rerunning the LLM.

Accuracy Alone Is Not Enough

For imbalanced taxonomies, accuracy can hide the failures that matter most. If a sensitive category is rare, a classifier can look good while missing too many examples of that category.

Balanced metrics, per-class recall, calibration checks, and review of false negatives are all part of the quality picture. No single metric carries the whole story. We saw a classifier that labeled almost everything non-sensitive show a high overall accuracy while its per-class recall on a rare sensitive category stayed low. Matthews correlation coefficient and macro F1 surfaced the gap that accuracy hid, and the misses became the cases we routed back for review.

Keep Recommendation Separate From Truth

Model-generated labels are useful, but they should not automatically become reference labels. The reference set needs reviewed provenance, and holdout evaluation should not be contaminated by the same model outputs being evaluated.

This separation adds friction by design. It is the friction that prevents a self-reinforcing loop from looking better while becoming less grounded. We saw the pattern directly. An optimization run scored against the same model’s earlier labels appeared to improve, but when we re-evaluated it against the frozen human-reviewed reference set, the apparent gains turned out to drift away from policy intent.

Coverage Is Not Correctness

Higher automation coverage is only useful if quality holds. A classifier can auto-resolve more assets while becoming less reliable on the cases that matter.

That is why coverage should be tracked alongside recall, precision, regression checks, and robustness tests. The goal is not to classify more assets automatically at any cost. It is to automate the cases that are stable enough to automate. In one case, promoting a broad rule lifted automation coverage but dropped shadow-mode per-class recall on a sensitive class. Because we track coverage alongside recall, we caught the regression and narrowed the rule before it reached serving.

Distillation Is the Production Model

LLMs are useful for ambiguity, cold start, and new patterns. Deterministic logic is better for the routine path where decisions need to be fast, explainable, and reproducible.

The sustainable model is a funnel: Let LLMs help discover and reason, then distill stable patterns into versioned rules that enforcement systems can run efficiently.

Self-Regulation Is Architectural, Not Operational

A learning system that does not know when to stop is a potential liability. We built a tuning controller that transitions through regimes:

• Observing (gathering signal). 
• Maintaining (healthy iteration).
• Conserving (gains slowing).
• Pausing (quality concerns).
• Diagnosing (halt for fundamental issues). 

In practice, the oscillation detector identifies stalled optimization, classifiers cycling between two candidate prompts without improving, and terminates them early, saving thousands of wasted classification calls per stalled run. This self-regulation was designed into the architecture from the start; retrofitting it would have been significantly harder.

Upcoming Directions

Three directions follow from this work:

1. Migrate legacy classifiers to this system, replacing ad-hoc heuristics with the full context-mesh + distillation pipeline.
2. Expand to other PAI workflows: The same pattern (context → LLM reasoning → distillation → deterministic enforcement) applies to lineage validation, purpose-boundary checking, and retention policy assignment.
3. Apply beyond privacy: Early experiments suggest these techniques generalize to agent observability and oversight, where the same tension exists between probabilistic reasoning and auditable enforcement.

AI-Native Products Raise the Bar for PAI

AI-native products raise the bar for privacy-aware infrastructure. They create new data modalities, faster iteration cycles, and more ambiguous signals. At the same time, privacy enforcement still needs decisions that are consistent, explainable, and reproducible.

Asset classification shows how to bridge that gap. Start with a clear contract. Build rich context. Use LLMs for novelty and ambiguity. Keep reviewed labels separate from model recommendations. Evaluate with metrics that expose rare-class failures. Distill stable behavior into deterministic, versioned rules.

That pattern lets the system learn from ambiguity without making ambiguity the foundation of enforcement.

The pattern also generalizes beyond our own use. A separate enforcement team compared this pattern against three alternatives head-to-head and chose it for their classification layer, independently of our work. In their evaluation, deterministic-first classification with LLM fallback produced more consistent, debuggable, and auditable decisions than end-to-end LLM approaches. Two teams independently arriving at the same trade-off (reasoning with LLMs, enforcing with rules) suggests a robust pattern.

The broader lesson is that privacy-aware infrastructure is not a tax on engineering. It is a driving force for better architecture: clearer contracts, richer context, stronger evaluation, safer publication, and systems that know when to ask for human judgment.

Acknowledgements

The authors would like to acknowledge the contributions of many members of the Privacy-Aware Infrastructure team who have played a crucial role in the work described here. In particular, we extend special thanks to Alex Basiuk, Dionisios Sotirios Krongos, Fanghao Song, Kartikey Sachdeva, and Loka Potnuru for their foundational contributions to classifier analysis, runtime feature migration, scanner hardening, false-positive reduction, and age-flow precision improvements — as well as the broader PAI team for context enrichment and evaluation.

We are also grateful to Dave Kurtzberg, Inchara Shivalingaiah, Juemin Wei, Nithya Arumugam, Zhe Wang, and team for independently validating the classification pattern within their autonomous remediation pipeline, to Jonathan Bergeron for sponsorship and support throughout, and to Deborah Davis for editorial guidance throughout.

The post Privacy-Aware Infrastructure in the AI-Native Era: An Asset Classification Case Study appeared first on Engineering at Meta.

This year marks Meta’s 10th consecutive year as a sponsor of the Python Software Foundation (PSF), the charitable organization dedicated to advancing, supporting, and protecting the open-source Python programming language and the community that sustains it. Python is one of the world’s most influential programming languages, and we use it across our engineering stack, from the backend of our apps and products like Instagram and Threads to cutting-edge AI research. 

We recognize the vital role the PSF plays in sustaining the language, nurturing its global community, and driving innovation. After a decade, it felt like the right moment to reflect on why we, as an organization of engineers, are committed to funding the PSF. By supporting the PSF, we aim to help ensure that Python remains robust, innovative and accessible for generations of engineers to come. We hope our involvement will inspire other individuals and organizations to join us in strengthening the foundation that supports so much of today’s technology.

The Importance of Python at Meta

Python is the most used programming language at Meta. It powers infrastructure across our most important products and initiatives and supports a wide range of teams across the company. Some of the core maintainers of Python are Meta engineers who have authored new features and Python Enhancement Proposals (PEPs) for the Python community. PyTorch, one of the world’s most widely-used machine learning frameworks, was originally developed at Meta in partnership with the community before being spun off into its own independent foundation. Meta also builds open-source Python developer tools to help developers write better quality, more performant Python. This includes projects like Pyrefly, an incredibly fast type checker and language server.

Supporting the continued growth and sustainability of Python is a natural fit for Meta’s technical vision. It will continue to play an important role in helping us achieve our goals as we invest further in AI, build new data-driven products, and further scale our infrastructure.

Why Meta Sponsors the Python Software Foundation

At Meta we understand that using open source software like Python comes with a shared responsibility to help ensure the language and its ecosystem remain healthy, secure, and innovative for everyone. Every product shipped, every model trained, and every insight generated with Python is made possible by the collective work of the open source community, backed up by the organizational support and infrastructure maintained by the PSF. For Meta, supporting the PSF is a strategic investment in the future of Python, and hence the long-term stability of our own technology stack. 

Our sponsorship of the PSF has helped fund impactful initiatives such as the Developer-in-Residence program, which employs full-time developers who are focused on improving the Python programming language and its ecosystem. This program has been transformative, allowing critical work to happen that would otherwise fall to overstretched volunteers or go unaddressed entirely.

PSF funding also goes towards strengthening the core infrastructure of the Python ecosystem, most notably the Python Package Index (PyPI), where our sponsorship has helped fund essential security enhancements. These improvements are vital for protecting the global Python community and ensuring that developers everywhere – including our own engineers – can safely share and consume packages.

Beyond purely technical investment, Meta’s support also helps fund educational programs and community events like PyCon US, where we’ve provided free and discounted passes to PyCon, supported workshops and summits, and contributed to fundraising efforts for groups like PyLadies. These investments help grow the Python community and foster the new talent that is essential for Python’s long-term sustainability.

In short, sponsorship of the PSF is a valuable investment in the tools and community that make our work possible.

How Can You Support the Python Software Foundation?

There are several ways you as an individual, or your organization as a whole, can contribute to the ongoing success and sustainability of the PSF:

• Make a one time donation: You can give any amount as a one off donation.
• Become a PSF member: By becoming a member you can vote in discussions on the direction of the PSF. There are different donation tiers available, including donating your time.
• Become a sponsor: For organizations looking to make a sustained impact, the PSF offers annual sponsorship tiers, each with increasing levels of recognition and benefits.

As an organization the most meaningful way for you to support the PSF is through annual sponsorship. Besides benefitting from the continued success of the Python language itself, there are a range of additional benefits depending on your sponsorship amount. Sponsors of the PSF receive public recognition, with their names and logos featured on the PSF website, in annual reports, and at major events. Sponsorship also provides valuable opportunities for community engagement, allowing organizations more opportunities to connect with the global Python community, participate in events, and demonstrate their commitment to open source. Higher-tier sponsors benefit from increased brand visibility through prominent logo placement and may be invited to speak or participate in special initiatives.

Thank You!

Finally, we want to say thank you to the Python community: the maintainers, contributors, educators, and advocates who make Python what it is today. Your passion and dedication are the foundation of Python’s success, and we’re proud to be able to support you, both as collaborators and sponsors.

Visit our website to learn more about Meta Open Source. You can also subscribe to our YouTube channel, or follow us on Facebook, Threads, Bluesky, LinkedIn, and X.

The post 10 Years of Meta’s Commitment to Python appeared first on Engineering at Meta.

Over the past several years, model capabilities and training dataset sizes have experienced exponential growth. During the past year or so, the time between new-frontier-model releases has gone down from months to weeks. Reliable and fast access to storage is important to both the speed and computational cost of this AI innovation. If AI is the brain, storage is the memory: Capability and speed are highly dependent on the size of memory and speed of retrieval.  

Yet while AI compute performance has roughly tripled every two years, storage and interconnect performance growth have been more modest. As a result, storage bottlenecks continue to be one of the primary contributors to GPU stalls for AI workloads, directly impacting expenditures and time to market. Aside from GPU utilization, storage architecture also directly impacts the speed of iteration in AI research; with GPUs increasingly becoming geo-distributed and dataset sizes increasingly becoming massive, researchers spend a significant amount of time ingesting and moving data across regions, thus impacting research velocity. In this blog post, we discuss how Meta’s BLOB-storage architecture evolved to address two primary challenges: maximizing GPU utilization and maximizing research velocity.

Storage Architecture Overview

Meta operates hundreds of exabyte-scale storage clusters that serve all of Meta’s external and internal products, including Facebook, Instagram, Reality Labs, Meta AI, Ads, Data Warehouse, and internal Databases. Our storage service exposes object storage, file systems, and block-device APIs, and these API abstractions are built on top of a horizontally scalable foundational block layer called Tectonic. The Tectonic layer is a regional, multi-tenant storage fabric that provides high durability and availability leveraging erasure-coding techniques, supports tiering across media types (e.g., HDD and flash), and manages smart placement of hot, cold, and warm data for efficient utilization of I/O across tenants. The BLOB-storage layers that operate on top of Tectonic expose a global, infinitely scalable storage fabric, and expose policies that let users make tradeoffs between durability and availability.

In a previous @Scale talk titled, “Training Llama: A Storage Perspective,” we discussed how Meta trained Llama directly over the Tectonic block layer by exposing an NFS-like FileSystem interface on top of it. While this architecture continues to be used widely within Meta, our modern training stack has been migrating slowly on top of the BLOB-storage interface, as is the case across the industry. This transition is motivated by the need for unified storage access to massive data lakes in the BLOB-storage layer as well as the need for high performance.

Maximizing GPU Utilization

Modern AI workloads are “data hungry” and have very different workload characteristics than traditional web applications: bursty and sustained high throughput, predictable and bounded pMax latencies, and variable I/O patterns. The focus for BLOB storage, in recent years, has largely shifted to maximizing GPU utilization.

Why Latency Matters

To see why bounded and low-pMax latencies are important, let’s consider model training. During that training, hundreds of thousands of GPUs iterate over vast amounts of data in storage multiple times (i.e., over multiple epochs), and the GPUs train datasets in batches. Periodically, after every certain number of steps or batches, the GPUs synchronize their state among themselves. If one GPU is slow, this step will slow down all GPUs as well as the entire training. 

Figure 1 shows a data-loading pipeline across two GPUs. The dataloader in every GPU host prefetches the next dataset batch, while the GPU is processing the current batch for maximum compute or I/O overlap. In the case of GPU1, the storage-fetch latency is well within bounds, so the GPU is never stalled waiting on I/O. In the case of GPU2, there are two instances where storage fetch exhibits high latency, stalling GPU. As a result of these stalls, the overall step-completion time is delayed.

Legacy BLOB-Storage Architecture Wasn’t AI-Ready

Over the years, BLOB storage evolved organically, adding layers on top of layers in a true service-oriented fashion. Many of these layers were stateful and maintained their own metadata stores. While these metadata-access latencies typically weren’t the bottleneck for the traditional use cases served by global HDDs, they were showstoppers for AI workloads with millisecond access to data in flash. Figure 2 shows the request flow for a typical getObject(“/bucket/path”) API. After the request arrives at the API server, the server does many metadata lookups across the namelayer, volumeslayer, and containerlayer before resolving the path to a set of (blockId, offset, size) tuples. Some of these lookups can cross regions, and it’s not uncommon for latencies to add up to hundreds of milliseconds; one slow response from any of the lookups was sufficient. After the lookups, the API server proxies the data from the Tectonic layer to the client.

While this architecture served conventional workloads well, the foundational assumptions that dictated design tradeoffs have since shifted. Some of these are:

• Performance and latency: As discussed, while latency needs for conventional workloads were modest, AI workloads demand predictable and bounded latencies all the way up to pMax.
  
• Reliability and durability: The legacy architecture was designed to be highly durable and available, even in the face of region outages; data and metadata were globally replicated by default. While AI workloads demand very high availability, the global-by-default design choice no longer holds.
  
• Cost efficiency: Legacy stack was built on top of HDDs and highly optimized for cost per byte. The IOPS demands for AI workloads necessitate flash, and in addition, the computational cost of storage becomes negligible relative to the computational cost of GPUs.
  
• Power efficiency: With GPUs, datacenters are increasingly power constrained rather than space constrained. Every kilowatt of power spent on storage is power not spent on GPUs. This is a new constraint with AI workloads.

In short, the tradeoff space has shifted enough for us to rethink the entire architecture.

Rebuilding the Foundation

As we set out to build the new foundation, we made the following major design choices:

• Unified metadata schema: We rewrote the metadata subsystem and collapsed the metadata spread across different layers into one unified and flat schema backed by ZippyDB. This paves the way for O(1) lookup to resolve paths to storage addresses, which is a step-function improvement.
• No dataplane proxy: We eliminated the dataplane proxy and built a fat client SDK that is capable of streaming bytes directly from storage servers to the clients. This helps with power-efficiency goals and also helps achieve higher throughput/lower latency.
• Regional deployment: The BLOB-storage stack is now lean with flexibility to be deployed as a regional or global service. We now deploy a regional BLOB-storage stack colocated with GPUs in every AI region.

Figure 3 shows the new request flow for getObject(“/bucket/path”). When the SDK on the client receives this API call, it now issues a getReadPlan(“/bucket/path”) request to the API server. The API server does O(1) lookup per chunk to the new metadata store to map the path to (blockId, offset, size) tuples. It then returns the ReadPlanResult to the SDK. The SDK has Tectonic BlockClient embedded within it, and so is now able to stream data from these blocks directly from Tectonic. With these changes, we have rebuilt the foundations and met the goal of adding zero overhead on top of Tectonic. By eliminating the data proxy, we also stay within budget for the power footprint.

Dealing With Spikes and Hot Spots

During data and checkpoint loading, AI workloads are known to access data concurrently across hundreds of GPUs. Subsets of data such as model weights are often “hot,” and events such as GPU restarts trigger sharp traffic spikes. With the foundations now fixed, our next problem was dealing with those spikes and hot spots. Luckily, the BLOB-storage layer has had experience dealing with hot spots over the years, so we adapted existing solutions to AI workloads here. Specifically, we employed two approaches:

• Distributed data cache: We leveraged the spare memory on the GPU hosts as a distributed data cache for frequently and concurrently accessed data. To achieve this, we reused components from Meta’s Owl subsystem: We integrated the peers in the Owl subsystem directly into the BLOB-storage client SDK so that all data access goes through this data cache. 
• Readplan metadata cache: Readplan refers to the mapping from path to storage address. We now cache the read-plan for frequently accessed BLOBs in a distributed-memory store similar to memcache. 

In practice we observe an average cache hit rate of 80% on the distributed data cache, and the read-plan cache provides 1-2 ms access to metadata. In essence, these simple mechanisms do three things: 

• Absorb the spikes and reduce the I/O requirements from storage.
• Solve the problem of metadata hot shards.
• Improve p50 and p99 latencies by serving from memory.

Protocol Optimizations

What we’ve discussed so far got us 80% of the way. We achieved the remaining 20% by identifying and fixing bottlenecks across the stack. Below are some noteworthy problems, though not an exhaustive list by any means:

• Laggards: One slow storage node contributing to tail latencies. This is a well-understood problem, and we resorted to hedged reads on the client side to mitigate this. 
• Egress spikes: During checkpoint events, it is common for the client to create sharp egress spikes. This in turn can cause congestion, timeouts, and retries, eventually stalling GPUs. We resolved this by building dynamic concurrency control on the client SDK to automatically tune parallelism based on application-level congestion signals. 

With all of the above, the new BLOB-storage stack is now capable of serving AI workloads without causing GPU stalls, adding negligible overhead on top of the Tectonic layer. Our next focus shifted to research.

Maximizing Research Velocity

GPUs are scarce and increasingly becoming geo-distributed; at the same time, training workloads need data colocated with GPUs for performance reasons. This creates an interesting challenge for researchers: They are now on the hook for ingesting and moving datasets across regions. 

At Meta, a typical training-job submission involves the following:

1. A researcher curates data from various sources, enriches them and persists them in BLOB storage.
2. The researcher picks a region where they want to run the job.
3. The researcher submits a data-ingestion job, which creates a snapshot of the training datasets onto the target region in a file format optimized for data loading from within the GPU host.
4. The researcher then waits for ingestion to finish; depending on the dataset size, that can take hours. 
5. The researcher submits their training job and monitors their run.
6. The researcher analyzes outputs, tweaks datasets, and iterates again, starting with Step 3.

Steps 2 through 4 can take hours and directly impact the speed of iteration for researchers. Ideally, we like our researchers’ time to be spent on tuning models, not waiting for storage. Currently, researchers copy snapshots before starting their jobs to colocate data with GPUs, which results in the most optimal performance. While this optimization for performance makes sense for large-scale training jobs that span weeks or months, the vast majority of jobs are much smaller; the researchers owning these jobs are more than willing to trade off occasional performance degradation for iteration speed.

And so, we needed a system where researchers are able to ingest data once and access data anywhere without thinking about regional boundaries. We needed a workflow that allows researchers to iterate in minutes and not hours. As we went back to the drawing board, the write-once, read-many characteristic of these datasets rang a bell. What if we think of storage as a disk in a planet-scale computer and borrow ideas from the operating-system world? When a Linux process running on a CPU core attempts to read a file from disk, the operating system transparently hydrates data on demand across the various layers of the cache—page cache in memory and L2 and L1 CPU caches. This intuition led to the architectural evolution in Figure 4:

The core idea is to leverage the various on-host and off-host storage resources as a tiered cache with global BLOB-storage fabric backed by HDDs as the ultimate source of truth. Specifically, we leverage the memory and flash on the GPU host as L1 and L2 caches. And we leverage the regional BLOB-storage fabric backed by flash as the L3 cache dataloader continues to access storage through the familiar BLOB-storage SDK. To effectively hide latencies and to simplify the data life cycle, we rely on the following:

• Dataloader prefetch: Dataloaders prefetch the next batch of datasets into memory while processing the current batch. This prefetch will surface as a read operation at the BLOB-storage SDK level.
• Deep prefetch: We expose an explicit prefetch() API as part of the BLOB-storage SDK. The dataloader will trigger explicit prefetch of the data needed during the next few minutes by invoking the prefetch() API in the background. This API triggers hydration of data from remote storage onto the local region L3 cache and also prewarms the metadata cache.
• Automatic data life cycle: Data in the L3 regional disaggregated flash tier is typically held for a configured period of time to allow reuse across epochs in a training cycle. We support custom eviction policies, including TTL and LRU policies. The eviction policies are also capacity/quota aware.

We saw rapid adoption of this new data-loading paradigm as soon as production rollout started, and we continue to support both of the data-loading paradigms in production today. To illustrate the impact in numbers, Figure 5 shows roughly the ingestion times before and after the rollout across all workloads:

In a world where new frontier models get released in weeks, this shift in the data-loading paradigm is a much-needed change to move even faster.

Key Takeaways

Modern AI workloads are data hungry, and storage plays an important role in both the computational cost and speed of innovation. Storage bottlenecks directly impact GPU utilization and computational cost, and in a world with geo-distributed GPUs, time spent on cross-region data ingestion directly impacts the speed of iteration in research. The BLOB-storage architecture at Meta was built to serve Meta’s family of apps, and we needed a step-function improvement in performance to serve AI workloads. This led to rethinking the entire architecture. By rebuilding the metadata subsystem and by adopting a tiered caching architecture with prefetching/on-demand hydration, we are able to meet the needs of today’s workloads effectively.

Future Work

We are continuously evolving storage at Meta to keep up with hardware evolution and workload demands. Some future work in this area will include: 

• Scaling storage to network limits.
• Supporting checkpointing without stalling GPUs at even higher scale.
• New challenges for inference workloads, which we are starting to tackle.

The post Meta’s AI Storage Blueprint at Scale appeared first on Engineering at Meta.

TL; DR

• At Meta’s scale, a few milliseconds of latency degradation can have a significant negative impact on ads performance. 
• When a Linux kernel upgrade risked regressing latency across Meta’s ad serving fleet, we turned to sched_ext — the upstream, BPF-based extensible scheduling framework — to build a scheduling policy customized to the Ads delivery workload.
• The result: a 28% reduction in ads retrieval stage tail(99th percentile) latency, 3.28 megawatts(MW) power saving, and a 1.1% increase in the number of ads ranked, proving that workload-specific scheduling optimization can directly drive business value.

Why Ads Latency Matters

Meta’s ads serving fleet handles more than 5 million requests per second on average at the serving platform entry point, which is over 400 billion per day across all monetized surfaces¹. Every millisecond shaved off the p99 latency makes the ads more relevant for people on our platforms, and better matches mean stronger ROI for advertisers.

This provides a real opportunity to reduce latency through workload-specific scheduling. That is why our Ads and Linux Kernel teams have been working together to build a scheduling policy customized to the ads delivery workload using sched_ext, the upstream, BPF-based extensible scheduling framework. Until now, we have been using the general-purpose schedulers typically integrated in the Linux kernel (CFS and EEVDF) that balance threads across CPUs with no understanding of the workload. However, here we know the purpose and importance of each thread. With sched_ext, we can encode this knowledge directly into the scheduler. Work that improves the p99 request latency is scheduled first, and everything else takes a back seat.

sched_ext at Meta

sched_ext is an open-source, BPF-based scheduler framework that officially entered kernel v6.12. We developed it by partnering with the authors of Google’s ghOSt to design a scheduler suitable for upstream Linux integration. It has already been deployed in several services at Meta, delivering meaningful reductions in scheduling latency.

While upgrading our fleet to the latest stable version of Linux (kernel v6.9) we observed that the new Earliest Eligible Virtual Deadline First (EEVDF) scheduler introduced in Linux kernel v6.6 was causing a latency regression which reduced the number of ads ranked in response. As a result, a subset of ads hosts were forced to remain on the older v6.4 kernel, creating technical debt and operational fragmentation.

Given its already strong performance, sched_ext was a great candidate to address these scheduling regressions.

Custom Scheduling with sched_ext

Sched_ext lets scheduler developers implement their preferred scheduling policy as a BPF program. When a host starts running the ads workload, an ads-optimized policy is applied. From that point on, the kernel calls into the BPF scheduler through a set of event-driven callbacks to handle common scheduling events, such as:

• Thread wake-up: choose a CPU when a thread becomes runnable.
• Enqueue: place a thread in a run queue.
• Dispatch: select the next thread when a CPU becomes idle.
• Idle transitions: respond to CPUs entering/leaving idle states.

At a high level, the policy soft-partitions CPUs into two pools, one for threads on the latency-critical request path and one for less latency-sensitive work. Which thread goes into which pool is part of the domain-specific knowledge encoded inside the policy. The size of each pool is adjusted dynamically using load-based heuristics. This approach tends to keep related work on the same CPUs over time, improving last-level cache (L3) locality and reducing costly DRAM access.

The policy is packaged as a user-space binary that loads the BPF program. That design makes experimentation and performance optimization much faster. To roll out a change, we can simply restart the scheduler process to unload the old policy and load the new one, without rebuilding or reinstalling the kernel.

Results and Impact

The initial launch took place to switch from kernel 6.4 with the CFS scheduler to kernel 6.9 with sched_ext on the largest ads serving server type. Based on the backtest experiment, the launch delivered:

• +1.1% on weighted-ads-ranked (metric for number of ads retrieved and ranked).
• 3.28 megawatts of power savings across the fleet.
• 28% reduction in service p99 latency on the ads retrieval path².

Compounding improvements. Two follow-on scheduler-policy updates, delivered as purely user-space changes, extended the win:

• Additional 60% reduction in service p99 latency.
• 18% reduction in timeout errors on the critical path.

This is a non-trivial win delivered with no dependency on kernel releases. Each follow-on iteration above shipped in days rather than months because the scheduler policy lives in user space as a BPF program. That cadence is what turned sched_ext from a “kernel upgrade unblocker” to a continuous-optimization platform for ads serving.

From Short-Term Fix to Strategic Asset

What started out as a targeted response to a very specific operational issue has turned out to be much more strategic, and widely applicable, than we originally anticipated. sched_ext delivers some key benefits to Meta:

A parallel and decoupled scheduler optimization path. Upstream Linux scheduling naturally evolves over time, sometimes in larger steps (such as the CFS-to-EEVDF transition), which can be disruptive to downstream consumers. sched_ext gives Meta the flexibility to continuously improve these custom schedulers alongside that evolution. We run and refine our own BPF-based scheduling logic, tailored to the unique demands of our production workloads, so our critical services stay optimized regardless of what happens upstream.

Independent deployment and reduced overheads. Scheduler improvements ship as BPF program updates, shipped in days rather than months. The resulting reduction in the cost of experimentation is transformative. Ideas that previously required a kernel patch and months of validation — local-cache-aware placement, ROI-based executor routing, NUMA-aware steering — become tractable iterations rather than major projects.

A shared industry asset. sched_ext was upstreamed into Linux v6.12, so the same mechanism Meta used here is now available to the entire Linux ecosystem. Any operator with a workload that doesn’t fit the general-purpose model — hyperscaler, cloud provider, embedded systems team — can ship workload-specific scheduling policies without forking the kernel.

Future Plans

sched_ext is already allowing us to see opportunities for further improvements in ads performance, by giving the application more fine-grained control over the behavior of the scheduler. For example, the ads services have important context about the relative importance of service requests, and are potentially able to signal to the scheduler when a thread starts working on an important request. When the scheduler receives this hint, it can take appropriate steps like increasing this thread’s scheduling slice or ensuring it’s always at the top of the queue.

Acknowledgments

Special thanks to Samuel Nair, Usama Arif, GP Musumeci, Praveen Alevoor, Ye Wang, and the broader Ads capacity efficiency and kernel team for their contributions and collaboration.

Ads Infra Leadership Team: Uladzimir Pashkevich, Varna Puvvada, Prabhakar Goyal, Neeraj Agrawal, Tak Yan, Liz Shepherd, Drew Lackman

The post Modernizing the Meta Ads Service With an Open-Source Kernel Scheduler appeared first on Engineering at Meta.

• Hierarchical Interest Representation is a research area for Meta Ads. We’re exploring an upstream representation layer over the universe of Ads entities – users, advertisers, products, services – learning unified embeddings that connect users’ inferred interests with the breadth of what advertisers offer in their deep funnel ads.
• The innovations in Hierarchical Interest Representation are an in-house transformer based graph learning with bias-aware attention and self-supervised cross-view distillation, learning multi-hierarchical interest representations across a large graph.
• Hierarchical Interest Representation blends real-world knowledge with engagement signals – multimodal advertiser and product content processed through LLMs enriches sparse interactions, enabling generalization to rare and unseen entities.
• Hierarchical Interest Representation outputs universal embeddings for ads entities and Bag-of-Meaning interest tokens that have the potential to power new personalization, retrieval, supervision, and specialized ranking architectures across the ads stack.
• Trained end-to-end on real Meta ads data at the scale of billions of interactions.

Hierarchical Interest Representation is an upstream representation layer designed to improve upon Meta’s deep funnel ranking optimization. It aims to connect businesses with the population of people on our platforms who carry the most genuine, latent interest in what they offer. The system is intended to function across Meta’s broader recommendation ecosystem, such as Meta’s Generative Ads Model (GEM), Andromeda, and the Adaptive Ranking Model, to advance deep funnel optimization.

People come to Meta’s apps and platforms to connect with people and content, expressing preference with every scroll. Engagement signals are used to understand both inferred and explicit interests and improve relevance of content across our platforms. Utilizing frontier AI to map latent interests from sparse engagement signals and aligning them with the vast landscape of advertiser offerings is a transformative approach to addressing the challenges of signal scarcity and driving up deep funnel ad performance.

The mission is to strengthen reasoning relationships throughout the landscape of ads entities – spanning users, businesses, and products – utilizing multi-hierarchical granularities that allow our models to navigate between stable, high-level interest anchors and the specialized, sparse signals of deep funnel intent. 

How Hierarchical Interest Representation Enhances Deep Funnel Optimization

Hierarchical Interest Representation pioneers a structural shift in representation modeling by navigating long-range graph topologies and distilling sparse engagement signals into unified interest clusters at various granularities. By fusing real-world knowledge with a semantic grasp of advertised products, it effectively strengthens the connection between ads and user intents.

Hierarchical Interest Representation encourages discovery-oriented ad experiences by extracting stable interest anchors from massive engagement datasets and grounding them in multi-modal world knowledge enrichment. This aims to enable the delivery of more relevant ad content to optimize deep funnel ads.

The Technical Challenges

User engagement with ads entities is naturally graph-structured: users and ads entities (advertisers, products, services, campaigns etc.) are nodes, and the activities and events connecting them are edges. At Meta’s scale, this is one of the largest graph networks in the industry. Learning the interest representation bears the following challenges:

User Inferred Signal Dynamics 

Meta provides users with tools to help tailor their experiences, like providing ‘Interested/Not interested’ feedback on posts they see. In addition, inferred interests based on engagement signals continue to play an important role for improving deep funnel ads. 

Large Networks With Sparse Connections: 

Every month, Meta’s ads network serves millions of ads, from millions of advertisers, to billions of people across our platforms. While this “vocabulary” is large, ad impression opportunities are limited and deep funnel user feedback is scarce.

Long-Range, Global Relationships

Given the sparsity of the individual connections in the deep funnel, it is useful to observe common patterns from long range, graph connected entities and users and encode into representation. Capturing long-range relationships within large graph networks is computationally demanding. Even as hardware capabilities scale, the pursuit of modeling accuracy necessitates the design of memory-efficient attention kernels and high-performance learning algorithms.

Introducing Hierarchical Interest Representation

Our latest research area is an upstream representation layer for learning universal, relational knowledge representations of users and ads entities. The representations capture users’ ads engagement patterns, absorb real-world world semantics, and cascade through multiple hierarchical granularities into latent-space projections at each level. Based on the graph data structure, four design properties drive the system end to end:

Dimension Reduction

Hierarchical Interest Representation projects a raw graph into a configurable super-graph where each super-node is a learned latent interest primitive. User-ad edges that are sparse at the raw graph become meaningfully denser at the primitive interest graph. Inherently, the primitive interest graph is more stationary and stable in vocabulary, even though the ads business is more dynamic.

Knowledge Enrichment

Hierarchical Interest Representation enriches heterogeneous entities, in particular advertiser and product types, with multimodal content features such as text, images, and video. These features are pulled from structured page metadata and advertiser catalog attributes and processed through vision or language models. This extra information complements existing engagement data. Instead of just knowing how users interact with something, it captures what that thing actually is. Even for entities it hasn’t seen before it understands the underlying businesses and products. 

Unified Relational Representation

Hierarchical Interest Representation learns thorough knowledge representation for users and entities, together with their latent interest primitives representation in a single metric space. It can infer entities’ mutual relationship and affinities across or within types. Using embedding operations, Hierarchical Interest Representation can determine primitive-to-primitive and cluster-to-cluster relationships. It can also estimate user proximity to interest primitives; how closely an ad/advertiser serves certain interest primitives; and what are the similar users, ads, and products that are closest neighbors. 

Multi-Hierarchical Granularities

Given the overall sparsity of deep funnel information, there is a trade-off on how to project into primitive interests. Dense and stable relationships usually imply a coarser and higher-level abstraction, while sparse and specific connection looks at finer level of abstraction. Hierarchical Interest Representation learns super graphs, which cascade through multiple hierarchical layers for this flexibility, accommodating ranking modeling architecture, personalization, or retrieval applications.

The Hierarchical Interest Representation Architecture

Hierarchical Interest Representation builds representations by combining world knowledge of advertisers and products with user direct temporal engagement data. Its LLM-inspired transformer architecture is applied to large-scale graphs, using sparse attention to capture long-range relationships. The system is designed to power various applications across the ads platform from retrieval to final stage ranking.

The following sections detail the architectural design of each component within the Hierarchical Interest Representation system.

Enriched Engagement Graph

Heterogenous Graph Structure

Hierarchical Interest Representation is built on a typed, weighted, time-decayed graph that unifies multiple entity types – users, ads, advertisers, campaigns, products, and pixels. These entities are connected by typed engagement edges including businesses creating ads artifacts and the users-ads interaction journey. Each edge carries its action type and timestamp that balances recency intent against long-term interest. This typed, heterogeneous structure is what lets a single downstream representation reason about someone’s lifestyle, an advertiser’s catalog reach, and the products that connect them – all in one unified space.

Scaling to Meta Production Data

The graph spans billions of users, entities and their interactions on a monthly basis, served both online for production freshness and offline for evaluation and rapid iteration. Each node carries rich initial embeddings fusing pretrained semantic features with behavioral statistics. Frequent nodes add a learnable ID embedding via deep hash embeddings, a small shared neural network applied to a vector of hashes of the node ID, keeping ID memory bounded as the vocabulary scales to tens of billions.

Hierarchical Encoder 

A transformer based hierarchical encoder is our modeling design to scale the representation learning at this gigantic graph size. It has considered several significant technical designs.

World Knowledge 

World knowledge serves as a means to connect the dots given, which is abundant on relatively stationary entities, such as advertisers, products, etc. We retrieve the multimodal information in summary text, images, and video and process it through a customized LLM inference engine that produces encoded feature inputs for our representation learning.

Structure Encoders 

The hierarchical encoder’s input layer combines rich complementary encoders. A node encoder fuses node-type, deep-hash-controlled node-ID, world knowledge features, and per-type metadata/features so each node enters the learning model with both its semantic identity and its real-world content. A position encoder applies position encoding over the sampled view from the graph to inject local topology with random walk, importance-prioritized strategies. An edge encoder prepares edge type, edge weight and temporal signals to flow into the attention learning mechanism.

Bias Composition 

Transformers measure pairwise relationships between tokens through attention. Graphs, by construction, encode pairwise relationships between nodes. These two ideas naturally complement one another. Graph-structural signals (such as node-type transitions along event edges and shortest-path distance for local connectivity) enter the model as attention biases that augment the query-key dot product in every layer. 

This is what makes the model topology-aware. Rather than treating a subgraph as a bag of nodes, the Hierarchical Encoder attends with explicit knowledge of how nodes are structurally related, capturing long-range relationships that standard message-passing tends to over-smooth.

Attention Kernel

Adding graph-structural bias to attention is normally expensive: the standard implementation materializes a full pairwise bias matrix, forcing a fallback from memory-efficient attention. We adopt FlexAttention, which computes each bias term on the fly so the pairwise matrix is never materialized. Variable-length subgraphs are packed into a single block-masked sequence, avoiding padding waste and cross-graph leakage. New bias terms slot in as small scoring rules, no low-level kernel work needed. The result is memory-efficient attention with full graph-structural awareness.

Training the Hierarchical Encoder

Cross-View Distillation

The Hierarchical Encoder is trained with self-supervision through a paired teacher-student scheme. For each anchor node, we sample a broad teacher view and a narrow student view, both passing through the Hierarchical Encoder. The student is trained to predict the same interest cluster as the teacher. Because the teacher sees a broader view of the graph, its prediction is more confident, giving the student a clear target to match. This extends supervision well beyond the small fraction of users who produce deep-funnel conversions, addressing much of the engagement sparsity that motivated Hierarchical Interest Representation. Sinkhorn-Knopp balanced assignment prevents single-cluster collapse, a known failure mode of self-supervised methods.

Engagement Prediction

Self-distillation discovers the latent interest primitives of the graph by teaching the model that different views of the same node should cluster together. Engagement prediction adds to the complementary supervised objective – given two node representations, an engagement type, and a time, predict whether a real engagement edge exists at that time. Together, they give Hierarchical Interest Representation both view-invariant structural priors and direct grounding in observed user behavior, turning it into a general-purpose scoring function across the delivery stack.

Online Graph Infrastructure 

The raw heterogeneous graph lives on Meta’s online graph engine. The same data source powers both training and online serving, ensuring the same distribution across both. For training, subgraph fetches and node-feature reads are pipelined with GPU compute. Multiple workers prepare upcoming batches in parallel while the model trains on the current one, hiding data-loading latency behind the forward and backward passes. During the testing phase, this achieved a 30x wall-clock speedup over the synchronous baseline and kept GPU model FLOPs utilization high. Bit-exact reproducibility is preserved end to end, so infrastructure migrations and checkpoint recovery never silently shift model behavior.

Causality

To prevent information leakage, training and evaluation strictly respect the temporal order of events. The graph engine applies a cutoff timestamp on each call and masks any node or edge dated after that point.

Edges are split chronologically into train, validation, and test windows. Batches are shuffled only within fixed time chunks, giving the optimizer gradient diversity without crossing temporal boundaries. The model only learns from information that would have been available at the time, so accuracy gains reflect real learning rather than future leakage.

Tokenization

Bag-of-Meaning Tokens

Continuous universal embeddings are powerful for ranking, but they are not naturally suited for inverted-index retrieval, set aggregation, or human interpretation. We discretize them into Bag-of-Meaning (BoM) tokens – a compact, unordered vocabulary of interest concepts produced by composite quantization. A user is represented by the BoM tokens describing their interests. An ad or advertiser is represented by the BoM tokens of the interests they serve.

Activating Hierarchical Interest Representation Across the Delivery Stack

Hierarchical Interest Representation is designed to plug learned interest representations into the ads delivery stack to eventually improve deep funnel ranking across existing components such as GEM, Andromeda, and the Adaptive Ranking Model. For example, BoM tokens augment engagement-based personalization and supervision with user latent interest, and enable fast, compact recall for retrieval with inverted index lookup. Hierarchical Interest Representation’s multi-hierarchical structure supports specialized architectures, such as Mixture of Experts generative distribution learning routed by super-interest categories, and enables hierarchical reasoning over stable interest anchors for user-ad affinity.

The Future of Hierarchical Interest Representation

We will continue to advance training scaling efficiency, embedding freshness, knowledge compression, and memory-efficient attention kernels for greater representational expressiveness. We are also investigating parameter-efficient, objective conditional fine-tuning of the upstream Hierarchical Interest Representation, enabling segment-level specialization for heterogeneous deep funnel optimization objectives on top of a shared encoder.

Acknowledgements

We would like to thank Sammy Bald, Shaoqing Yuan, Chuan Hu, Chris Hayduk, Wenfan Xu, Liang Xu, Piyush Dak, Ikuhiro Ihara, Ashish Kalbhor, Manjari Jha, Priya Krishnamurthy, Xulei Liu, JC Lyu, Yuqi Bi, Vivek Bitla, Krishna Poola, Jiayin Ge, Santanu Kolay, Matt Steiner, Sandeep Pandey, Gunjit Singh, Marvin Kim, Karan Jindal, Krishi Suresh Kumar, Mehul Parsana, Manveer Kaur, Hua Liu and the entire team behind the development and production of the Hierarchical Interest Representation Project.

The post Exploring Hierarchical Interest Representation For Meta Ads Deep Funnel Optimization appeared first on Engineering at Meta.