RSS Feeds

Returns for breaches of licence conditions rose 31% in the first quarter of 2026, adding to pressure on prison places

More people were recalled to prison than were released during the first three months of 2026, the first time on record the “recall rate” has surpassed 100% in England and Wales.

Between January and March 2026, 12,977 people were released from prison while 13,193 people were returned to prison for breaching their licence conditions, up 31% from last year, new Ministry of Justice (MoJ) data showed.

Exclusive: Electoral Commission urged to look into donations made by mother of convicted fraudster

• Revealed: George Cottrell transferred $2m to his mother days before her donations to Reform UK

Labour has accused Reform UK of being “up to its neck in sleaze” as it called on the Electoral Commission to urgently investigate substantial donations made to the party by the mother of a convicted fraudster.

Bridget Phillipson, Labour’s new chair, called on the commission to examine whether any rules had been broken, after revelations in the Guardian that George Cottrell transferred more than $2m to his mother in the days before her donations to the party in 2024.

PM tells reporters the government cannot ignore the energy resource ‘when people are struggling’

Andy Burnham has said he will take a “pragmatic” approach to oil and gas drilling in the North Sea, saying the government could not ignore the potential energy resources it holds.

The prime minister’s comments came in response to a question about his conversation with Donald Trump on his first day in No 10. Trump had said Burnham told him he would “open up” North Sea oil, something the UK summary of the call made no mention of.

How Airbnb teams build trustworthy Generative AI products by treating evaluation as a first-class engineering discipline; not an afterthought.

By: Rohit Girme, Dan Miller, Mia Zhao, Lifan Yang, Clint Kelly

Introduction

Generative AI breaks a lot of the assumptions that used to hold true for software testing. Unlike traditional software, LLM outputs are non-deterministic, and “correct” is subjective. Because so much judgment is involved, you often need an AI to evaluate an AI, which introduces its own potential failure modes. Making matters more complicated, a single interaction with an LLM can chain retrieval, reasoning, tool calls, and generation, each of which can fail independently.

At Airbnb, we build LLM-powered features across our product, with recent launches including review highlights, AI customer support, smart communication features for guests and hosts, and more. Behind the scenes, we also use AI to help us spot trends and understand what’s working, guiding where we improve the product next.

Each product team may have its own evaluation criteria, process, workflows, etc. However, these are built on top of some common foundations and principles. An infrastructure team provides tooling and best practices, incorporating learnings across domains so that they are shared with everyone building products at Airbnb.

In this article, we wanted to share some of these best practices and learnings with the broader engineering community. Please note that the recommendations here are not intended to be prescriptive; there is no one-size-fits all approach when it comes to running evals.

1. Foundation

Evaluating LLM-based systems is challenging work, and this should be planned for at the outset. Without a deliberate strategy, three things tend to happen:

• False confidence: A generic “helpfulness” metric scores well, you ship, but it didn’t capture the failure mode people actually hit.
• Undetected regressions: A prompt change subtly degrades a dimension you weren’t measuring.
• Wasted effort: You build a scaled eval pipeline for metrics that don’t correlate with outcomes.

Expect to spend a meaningful share of your total project effort on evaluation. This is not unnecessary overhead, it’s how you build products that actually work.

1.1 The one rule

When in doubt, look at your data. Manually reviewing your data and building an intuition for what counts as success is always the starting point we recommend to teams. Build your prototype, and run it through 100 examples (synthetic is fine). Then read the outputs. Read the traces and find the model’s mistakes. Categorize them and build an eval.

This single habit will do more for your product quality than any framework, tool, or methodology in this document.

1.2 Eval-driven development

Formalized, that habit becomes eval-driven development (EDD), the GenAI analogue of test-driven development. Rather than predicting every failure upfront, EDD builds the infrastructure and habits to discover, encode, and continuously test for failure modes as they appear. It also forces stakeholders to externalize what “good” means, which shapes the product roadmap.

Five principles anchor EDD:

1. Define goals and gates upfront. What are you optimizing for? What must be true before you ship? These answers may not be clear right away; you might discover them as part of your data exploration.
2. Let real errors guide your metrics. Co-develop them with cross-functional partners based on observed failures. Don’t invent them in a vacuum.
3. Keep your evaluator set small and sharp. 3–5 well-calibrated LLM-as-judge evaluators beat 20–30 noisy ones. Each should target one specific correctness dimension.
4. Appoint a decision-maker. While what constitutes correctness should be a team discussion, people will sometimes disagree. Include a final (human) decision-maker who makes the ultimate call on what constitutes good vs. bad system behavior.
5. Collaborate continuously. Have your product partner regularly answer: “Is X better or worse than Y?” and “What’s actually wrong with this output?”

2. The three evaluation methods

Every evaluation you run will use one or a combination of these three methods.

Layer 1: Programmatic checks (fast, low resource — catches obvious failures)

                         ↓

Layer 2: LLM-as-a-Judge (nuanced - catches quality issues)

                         ↓

Layer 3: Human evaluation (high resource - validates edge cases, 
 calibrates the stack)

2.1 Programmatic & heuristic metrics

Deterministic, code-based checks that don’t require an LLM call should be your first filter, catching obvious failures before you send anything to a judge or human labeler.

✅ Do: Use structured outputs (JSON schemas) to ensure strict typing.

❌ Don’t: Rely on prompt instructions alone to format data. This breaks downstream data pipelines.

2.2 LLM-as-judge (Virtual judges)

Use a stronger LLM to evaluate another LLM’s output against a carefully designed rubric. This is how you assess nuanced qualities e.g. tone, coherence, faithfulness, relevance, at a fraction of the resources needed for human evaluation.

Rubric design matters. Ambiguity is the enemy. Something like “Is the provided explanation readable and up to our standards?” isn’t likely to be effective — if a human can’t apply the rubric consistently, an LLM certainly can’t.

Here is a simplified example of a single virtual judge’s rubric:

Score the readability of listing explanations. A good explanation sounds
like a friendly travel agent: warm but professional, 
simple, natural, grammatically complete.

Score 1 if it reads cleanly. 
Score 0 if it has ANY of these problems:

- Tone: too formal/jargony, too casual
("awesome vibes"), too salesy ("amazing!"), or robotic.

- Internal terms: never use internal terminology.

- Formatting: no quotation marks, no bullets, no fragments. End every
explanation with a period - never "!" or "?".

- Grammar: use articles/determiners/prepositions for natural flow
("this home has a pool", "close to downtown"). In a series, use the
article once then drop it: "a backyard, grill, and kitchen" - not
repeated, not omitted entirely.

- Complexity: plain words over jargon ("pool" not "aquatic recreation
area"; "near" not "proximate").

Examples:
- "Host mentions a pool and hot tub available near downtown." → 1
- "The listing mentions a pool!" → 0 (internal term "listing"; ends in "!")
- "This domicile encompasses aquatic amenities." → 0 (complex words; jargon)

Return ONLY:
{
"reason": "<list of [error_type, explanation] tuples as a string, or []>",
"score": <1 or 0>
}

2.2.1 Calibration: Making your virtual judge trustworthy

A virtual judge that hasn’t been calibrated is worse than no judge at all, because it gives you false confidence. Here are the calibration steps we recommend:

1. Create a golden dataset of 50–100 examples. This MUST include bad examples (not just good ones).
2. Run your virtual judge against the golden set.
3. Measure agreement. Target percentages in the high 80s-90s. Possible options to measure disagreement are Cohen’s kappa or Krippendorff’s alpha. (Perfect agreement isn’t achievable — even humans disagree.)
4. Analyze disagreements. Refine the prompt and update your few-shot examples. Then re-run the loop until you hit the target agreement.
5. Recalibrate periodically as failure modes evolve.

2.3 Human evaluation

Human judgment remains the gold standard for ground truth, high-stakes domains, and resolving disagreements between automated evaluators.

2.4 Evaluation scenarios and recommended methods

Overall, the rule of thumb is to start with 20–100 rows labeled by subject-matter experts. Move to a scaled annotation workforce only when the rubric is rock-solid and volume is the bottleneck.

And if your experts disagree on a label, stop. Solve human disagreement before automating anything.

3. Evaluating agentic systems

Agentic systems involve multi-step reasoning, tool calling, branching logic and intermediate state transitions. Evaluating only the final output is insufficient: a correct final answer can mask a broken reasoning path, wrong tool parameters, or an inefficient trajectory.

Therefore, you will need to evaluate across three layers:

To achieve this, you can take advantage of the fact that an agent generally pushes traces and spans under an application root. This contains information about the type of agent, the sub agent if invoked, input/output of the agent, tools invoked if any, and more. These traces can be written out to an observability platform or persistent storage.

Then, you can use DFS or another type of tree traversal to reconstruct the trace in memory. This lets you ensure certain subagents were invoked at the right time, the agent called the right tools, etc. And you can scope your evaluation to specific agents/subagents.

4. A practical walkthrough

Here’s what the full process looks like end-to-end, using a fictionalized and simplified version of a real use case.

Scenario: You’re building an AI assistant that answers questions about a travel platform’s support policies.

Step 1: Explore & discover. Run 100 inputs through your prototype and read every output. You find:15 responses generated policy details not in the source documents (faithfulness issue); 8 correct but too verbose (conciseness); 5 refused valid questions (over-refusal); 3 had broken JSON (format).

Step 2: Build evals. Add programmatic checks for JSON validity and length bounds. Write a virtual judge for faithfulness (separate prompt, different model, chain-of-thought) and another for conciseness. Have your PM or subject matter expert label 60 examples, including failures, as a golden set.

Step 3: Calibrate & iterate. Your faithfulness virtual judge agrees with the PM 78% of the time. Not good enough. Analysis reveals the judge is penalizing accurate paraphrases as “unfaithful.” Update the rubric and add few-shot examples. Agreement jumps to 88%. Improve the retrieval step; faithfulness failures drop significantly.

NOTE: Here, we find that when iterating on models and prompts, it’s best to fix one variable at a time. First fix the model and vary the prompt, then fix the prompt and vary the model, then fix both and vary the serving configuration. At each stage, virtual judge results narrow the candidate pool. Then, you can improve the virtual judge(s) using samples from the top candidates. The evaluators and the candidates sharpen each other until both stabilize.

Step 4: Scale & monitor. Scale evaluation across 5,000 examples. Set up production monitoring: sample 5% of live de-identified traffic daily, run programmatic checks + virtual judges, and surface flagged outputs for human review. A weekly PM review closes the loop, with new failure modes introducing new evals and subsequent system improvements.

NOTE: We sample live traffic continuously using privacy-preserving techniques. All data undergoes robust de-identification prior to human review, and usage is strictly purpose-limited to safety and quality assurance, aligning with Airbnb Privacy Principles.

Key takeaways

1. Look at your data. Read outputs and traces before building anything else.
2. Avoid generic metrics. Build evaluators for your product’s real failure modes.
3. Start with 50–100 rows. Fail fast, iterate cheaply.
4. One evaluator per dimension. No “God evaluators.”
5. Calibrate to high 80s-90s% agreement before trusting your Virtual Judge at scale.
6. Use all three methods. Programmatic, Virtual Judge, and human as layered defenses.
7. Include bad examples in your Gold Set. You can’t test discernment without them.
8. Evaluate the system, not just the model. Test retrieval, tool calls, the full pipeline. For agents, evaluate the trajectory, not only the final answer.
9. Mirror evals in production. Pre-production metrics are not one-and-done.
10. Evaluation is a team sport. Evaluation is about shaping what product success looks like, and that takes contributions from many people. The teams that succeed with AI aren’t the ones with the best models, they’re the ones with the best communication and clearest product vision.

If this type of work interests you, check out some of our related positions!

Acknowledgments

We would like to thank Tania Myronivska, Haozhen Ding, and Sebastian Wickenburg for their thoughtful feedback and contributions to this guide, Jisheng Liang and John Hewson for their guidance and insights, and Min Yi and Yi Li for their constant support.

We would also like to thank Evelyn Xu for their support in authoring this post during their time at Airbnb.

All product names, logos, and brands are property of their respective owners. All company, product, and service names used in this website are for identification purposes only. Use of these names, logos, and brands does not imply endorsement.

Eval-driven development: Lessons from evaluating GenAI at scale was originally published in The Airbnb Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

A mistake on a routine task can cost you a minute. A mistake on a large refactor or a debugging trail spanning months of commit history can cost far more, as it compounds silently over hundreds of exchanges. By the time you catch it, every step built on top of it needs unwinding, too. That's the difference between work that rewards speed and high-complexity work that requires getting it right the first time.

Anthropic's newest AI model Claude Opus 5, now available on GitLab Duo Agent Platform, is built for the tasks that demand the most from an agent. With Opus 5, your engineering team can trust agents with more complex, critical work. In GitLab's internal evaluation, Opus 5 resolved 93.3% of benchmark tasks, a 20.3-point improvement over Opus 4.8's 73.0% resolution rate.

"The teams getting the most value from AI agents can hand over their hardest, highest-stakes work and trust the reasoning holds up from first step to last."

— Stuart Moncada, VP, AI Product Management, GitLab

Reasoning that holds up under complexity

Some teams hesitate when delegating their most challenging work to an agent because the mistakes are costly to unwind. On long-running, high-complexity tasks, an agent's reasoning has to persist from start to finish. With Opus 5’s deeper reasoning, more of your complex tasks, like multi-file features and larger refactors, resolve correctly the first time, helping cut down on the time you spend diagnosing and re-prompting failed runs. Teams running GitLab Duo Agent Platform with Opus 5 can expect to see fewer partial patches, with more of the work coming back ready to merge.

That reliability extends to completeness as well. In GitLab’s internal testing, Opus 5 completed 100% of the tasks it attempted, matching Opus 4.8’s completion rate. The edge is in what they produce: more of Opus 5's solutions are verified correct, putting Opus 5's resolution rate at 93.3%, against Opus 4.8's 73.0%.

One task in GitLab's evaluation called for mockable SSO login support in a CLI authentication flow, a change spanning five files, including new exported types and configuration fields. Several other models tested produced no attempt at a fix. Opus 5 built the full implementation, committed it, and opened a merge request.

You can expect the same precision in code review. Opus 5 flags real bugs and produces few false positives, so your team can stay focused on genuine vulnerabilities and spend less time filtering noise.

If your workload runs several agents at once, you waste less time untangling conflicts between them. Opus 5 keeps subagents coordinated, ensuring they stay out of each other's work. Writer-verifier patterns catch problems between agents before they reach you, with one agent checking another's output before it's accepted. Teams running longer, more autonomous sessions with more agents in parallel see the strongest results.

For cost-sensitive workloads running multiple parallel agents, GitLab Credits usage caps let you set a hard limit on spend, so parallel work never runs beyond what you've budgeted.

Speed keeps pace with depth

On GitLab's hardest benchmark tasks, Opus 5 pairs reliability with speed. At the 95th percentile, the slower tail of its runs, Opus 5 finished 2.2% faster than Opus 4.8 (768 seconds vs. 784.98 seconds) and 21.9% faster than Sonnet 4.6 (768 seconds vs. 982.57 seconds). Reliability and speed move together. For your team, that means more predictable turnaround, even on your longest runs.

Choose the right model for the task at hand

The right model depends on the task in front of you, not a single org-wide policy. Sonnet-class models handle the bulk of day-to-day development work: fast, affordable, and dependable for what most teams run constantly. Turn to Opus 5 when the work demands deeper reasoning: the hardest debugging, the largest refactors, the decisions you don't want to rework.

You set that choice directly in your GitLab instance through model selection. Whichever model you choose, it runs inside the same infrastructure: the context layer, policy checks, and audit trail that cover every model on GitLab Duo Agent Platform.

Put Opus 5 to work

Claude Opus 5 is available now on GitLab Duo Agent Platform and, like other models, runs on GitLab Credits. New to Duo Agent Platform? Start a free trial today. Already a GitLab Premium or Ultimate subscriber? Turn on Duo Agent Platform and use the GitLab Credits included with your subscription.

Both MPs have had the parliamentary whip restored following separate independent disciplinary processes

The MPs Diane Abbott and Joani Reid have been readmitted into Labour and had the parliamentary whip restored, just under two weeks into Andy Burnham’s premiership.

The party confirmed that this followed separate independent disciplinary processes, and Labour said the leadership played no role in either decision.

This week GitLab signed the Open Weights and American AI Leadership letter, joining a long list of other technology companies that support a strong, open AI ecosystem.

The letter argues that open weights spur innovation, give customers greater control, and provide an important path to AI safety and security. In addition to being a policy position we share, it’s core to how we think about agentic engineering: Teams do their best work when they can choose the right model for the job.

As the intelligent orchestration platform for DevSecOps that enables speed with control for agentic software engineering, GitLab prioritizes customer choice by orchestrating the software lifecycle and supporting multiple models across a team’s workflow.

Empowering customers to choose their AI models

For many organizations, there is an emerging interest in having governed access to best-in-class foundation and open weight models. GitLab supports both.

Foundation models often lead on general-purpose capability, while open weight models can provide benefits for customer control over cost, deployment, and data residency. Our goal is to help customers combine them as needed.

One of the most critical decisions corporate leaders make is how to protect software and strategic IP against security, privacy, and competitive threats. Organizations shouldn't be locked into one cloud or one AI model provider.

GitLab is the only platform that's cloud neutral and AI model neutral. That choice only holds up if the model market stays open. Open weight models give development teams the choice of where to run their AI models — in air-gapped environments if necessary — while keeping control of their code.

Our stance

Like policymakers, we want a safe, secure AI ecosystem, and view openness as an important part of achieving it. We support policies that preserve the ability to develop, distribute, and use open weight models subject to focused, risk-based safeguards and well-targeted tools for addressing genuine misuse. These kinds of interventions can play a meaningful role in fostering a robust ecosystem in which multiple model providers — open and proprietary — can compete on the merits, to the benefit of innovation, security, and customer choice.

Related links

• GitLab AI Transparency Center
• Securing AI together: GitLab's partnership with security researchers
• Why enterprise independence matters more than ever in DevSecOps

Forty hectares of Oaken Wood could be razed after it was identified as potential site of expansion by Reform-run council

It was at the confluence of seven forest paths that Allison Sweetman came to a stop. Trees rose on all sides, save for a recently coppiced section to the north-east. The noon heat had silenced most birds, but the quiet was peaceful.

“This is what they call Seven Wents,” she said. For centuries, the people of Kent had travelled this way, from East Malling to Barming, from Ditton to Teston, for church, commerce and friends. “It’s on maps going back 400 years.”

Boom in migrating moth sightings put down to hot weather, strong winds and food scarcity in Europe

He had just sat down to dinner in the garden when a speedy visitor whizzed by – a “grey lump with orange wings”, recalled Luke Burstow, from Sussex. It was a hummingbird hawk-moth, a flying insect that migrates to the UK from continental Europe and north Africa.

“I was like: ‘Wow!’,” said the software project manager. “I didn’t even know they existed.”

From Leicestershire to Carmarthenshire, neighbourhoods are struggling with wave after wave of houseflies. Why are they suffering while others are spared? And is there no alternative to nets and zappers?

Francesca Davies’s living room is absolutely teeming with flies. You can’t enter the contact centre worker’s home without a cluster of insects rising from the sofa or the floor to buzz around your face. Davies and her family live in West Heath, a suburb of the Cheshire town Congleton. Davies’s husband, Alex Andrew, a 29-year-old maintenance engineer, grew up in the market town and he used to think of West Heath as a desirable area, with its well-maintained houses and access to green space. When he and Davies were able to buy a new-build here in 2019 via a shared ownership scheme, they felt incredibly lucky. That is, until they moved in, and realised that every summer swarms of flies descend on their neighbourhood, an issue that is “getting worse”, Davies says.

At the moment, the 27-year-old swats flies out of her young daughters’ bedroom every night before she puts them to bed, and keeps doors and windows closed or covered with fly nets as much as possible. She and Andrew have even bought an industrial fly killer (like the ones often found in fish and chip shops) for their kitchen, as it is impossible to start cooking without attracting even more insects. The pair have become used to the sound of flies “getting zapped throughout the night”, Davies says.

Raja is normally a celebration of the start of the monsoon, but in the eastern state of Odisha, the rains came so late the seeds were sown under clear skies

Kasturi Gamango and Pramila Sabar cannot contain their giggles as they explain one of the big attractions of the Raja festival, which they attended a few days earlier along with 20 other farmer families in Laxmipur, in Odisha’s Ganjam district.

“We do no work for three days during the festival,” they say. “Our husbands cook and clean while we enjoy our free time.”

Author of books behind forthcoming film starring Viola Davis and Idris Elba appears to have had disagreement with cast member Amandla Stenberg

Tomi Adeyemi, the author of Children of Blood and Bone, has spoken further on the difficult process behind adapting her novel for this big screen, calling it “the worst thing I have ever had to live through”.

In a five-minute video on TikTok, the author said she left the set of the Paramount production “hyperventilating and sobbing”, adding: “I never want to hear about this project again”.

Diggers are working at what campaigners call a place where ‘human life, biodiversity and the environment will be sacrificed’

On the morning of 16 February, dozens of police officers arrived at San Francisco Ángulo, a small rural community an hour south of El Salvador’s capital. They had come, residents were told, to escort heavy machinery belonging to Cyeemsal, a Mexican-owned company hired to begin construction of a landfill on a site the community considers sacred.

Félix Laínez, 58, president of the San Francisco Ángulo community association, had been warned that this would happen. The community has lost a series of legal appeals against the court rulings authorising the work, and construction is continuing. “It is clear that we will find no support in any state institution,” says Laínez, who has little hope of success in their final constitutional appeal, which is now before the supreme court.

Both tech companies beat Wall Street predictions on revenue in their second quarters

Apple and Amazon both released their second quarter earnings on Thursday, a test of investor confidence in major tech companies amid growing concerns over exorbitant AI spending.

Apple revealed quarterly revenue of $109.4bn, beating Wall Street expectations of $108.65bn in revenue. It also reported $2.02 earnings per share, driven by sales of its marquee products such as iPhones and laptops.

President says he has no objection to pulling his former lawyer’s name until dissenting Republicans are out of office

• US politics live – latest updates

Donald Trump said on Thursday he may “temporarily” pull the nomination of Todd Blanche to serve as attorney general, but would keep him in the role in an acting capacity until two Republican senators objecting to his confirmation leave office next year – an extraordinary escalation of an intra-party fight over an agreement to create a $1.8bn slush fund and give the president tax immunity.

Two Republican senators – John Cornyn of Texas and Thom Tillis of North Carolina – have refused to back Blanche’s nomination until they receive written confirmation from the justice department that it is not moving forward with a widely criticized agreement creating a $1.8bn fund to compensate people claiming they were targets of political weaponization and granting the president, his family and business entities broad immunity from past tax investigations.

Figures show hot, dry, windy conditions in EU are 43% more severe than the average over last 20 years

Fire weather in the EU has broken records for severity for this time of year, data shows, as firefighters race to extinguish blazes across the Mediterranean.

Inflamed by carbon pollution, the hot, dry and windy weather across the EU from the start of the year until the end of July has been 43% more severe than the average for the same period over the last two decades, data from the European Forest Fire Information Service (Effis) shows.

The champions are capable of shattering every record in a temporary aligning of the planets, the elusiveness of pure perfection

Treble-20. “Ohh, he couldn’t.” Treble-19. “Oh, you spiteful young man.” Bull. “You spiteful person! That is ridiculous!” As ever, Wayne Mardle – commentating on the World Matchplay final for Sky Sports on Sunday night – put it most succinctly of all.

The great individual champions require a vindictive streak. The ability not just to harness their own strength, but to tessellate it with their opponent’s moment of greatest vulnerability. To sense weakness, recognise doubt and pounce mercilessly upon it. To nurture and desire the very same qualities we try to extinguish and discourage in our children. To find a 167 checkout, when your opponent is sitting on double-10, in a major final.

• Family says Smith faced ‘dementia caused by CTE’

• College Football Hall of Famer played 10 NFL seasons

Former San Diego Chargers linebacker and College Football Hall of Famer Billy Ray Smith Jr has died. He was 64.

Smith’s family said in a statement Wednesday that Smith died following a bout with CTE-caused dementia.