RSS Feeds

Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. In every issue, I cover Big Tech and startups through the lens of senior engineers and engineering leaders. Today, we cover one out of four topics from last week’s issue of The Pulse. Full subscribers received the article below eight days ago. If you’ve been forwarded this email, you can subscribe here.

It’s been an absolutely wild few days, with security breaches and major security vulnerabilities being exposed worldwide across tech. Many incidents took place on Tuesday and Wednesday of this week. We’ll start with a brief summary, then look closer at three of them:

1. Intermediate CA certificate compromised: eight private 384-bit keys issued by eMudhra – which is an intermediate Certificate Authority (CAs) – leaked, meaning that servers in server-to-server communication can face man-in-the-middle (MITM) attacks. Any internet communication not going through modern browsers (that already patched this) could be risky.
2. PHP: Unpatched servers risk remote code execution (RCE) and opening up access for attackers. Around 70% of all websites run on PHP and unpatched systems are at risk. Two critical security patches have been released.
3. Railway: customer secrets stored on the service were leaked and some have been decrypted. Secrets exposed can include access to production systems, enabling attackers to potentially access customer systems.
4. GitLab: Unpatched GitLab servers are at risk of an attacker gaining full access. A critical security patch for RCE has been released.
5. Node.js: Unpatched servers are vulnerable to remote execution attacks. A new release fixes 10 critical vulnerabilities (CVEs), including one that allows for RCE, and another fix revokes the compromised eMudhra intermediate certificate.
6. Go: a new release is out, adding a mitigation for the compromised eMudhra intermediate certificate.
7. TiDB: attackers gained near-total access to the complete control plane systems of the managed TiDB provider, PingCAP, for a whole month (!!). The attacker released 10TB of customer-sensitive data, including from Booking.com, Airbnb, Pinterest, Databricks, and more such as SQL queries that customers ran, billing, contacts, and source code for PingCAP’s systems The breach lasted from 9 Sept until 7 Oct, but was only disclosed this week. PingCAP claims no customer data stored inside TiDB was accessed, but I find it hard to see how the company will avoid losing security-conscious customers. This incident shows that the absence of adequate access monitoring and alerting can be an open invitation for attacks lasting weeks.

With that, let’s get into the first three incidents in the list.

eMudhra CA compromise

Certificate authorities (CAs) are organizations that vouch for identity on the internet. When a browser, phone, or server pings a site like chase.com or revolut.com, intermediate CAs and digital signatures verify that you really are talking with the bank’s server, and not a fake one set up by an attacker with the purpose of stealing users’ credentials. This is where the HTTPS protocol comes into play, with root CAs, intermediate CAs and the TLS handshake.

The whole system is built on a chain of trust and verification: we trust the root CAs and their certificates, and so we can trust the signatures of the intermediate CAs. The assumption is that both the root CA and intermediate CAs are trustworthy and verifiable.

But eMudhra broke this trust in the most blatant way: its private keys turned out to be single-digit numbers. A private key is a 384-bit key, which means that it would be a number between 0 and 2^384, (a number with 116 digits!)

Except, the key turned out to be the number “4.” Yes, that’s right: the number four. What on earth was eMudhra doing?! It’s as if they chose random numbers between 0 and 384, not 0 and 2^384! It is quite literally comical: in 2007, an XKCD comic described exactly this choice:

And the other keys were also not that much better: 56, 66, 115, 156, 230 and 237! It’s as if eMudhra generated an 8-bit private key, not a 384-bit one.

eMudhra explaining how these keys were generated (emphasis mine):

“We have done some preliminary assessment, there appears to be a flaw in the KGC [Key Generation Ceremony] script that was used on that specific day to create these ECC issuers. As for these ECC keys [elliptic-curve cryptography key pairs], our review of the KGC, assessment of the code and view of HSM [Hardware Security Module] keys so far indicate that a) a set of original ECC keys were created onboard as expected b) an approved deviation to our usual KGC to test a wrap/unwrap was introduced as these keys were meant for export for this instance c) a flaw in the code resulted in the second pair of keys being created onboard which were weaker d) these second pair of weaker keys were used by mistake to create the CSR [Certificate Signing Request] (because the key labels were similar) and signed by our root e) the CSR from these weaker keys were also sent over to Sectigo for cross signing (we verified the files sent over to Sectigo)

The RSA issuers went through a different path as per our assessment of the code and are therefore not impacted.

Some additional context - these issuers were meant for export to our US and European data centers to solve data sovereignty issues for some of our customers (but never ended up operationally in production except for one).

We are still assessing the code to see steps where this process went wrong.”

I have so many questions:

• How does eMudhra test generated keys? There seems to have been no automated testing of the key in place at all, in order to discover that these numbers were not “random” from a flaw in the key generation ceremony
• How did eMudhra not notice this on day one, when the Key Generation Ceremony went wrong? If this was an automated process, the script would’ve changed from one day to the next, which would raise questions. If the process was fully manual, it explains how the mistake would be missed. And if it was manual, then why? Also, where are the detailed audits for such an important process?
• The certificates with the easy-to-guess private keys were generated on 30 April 2025, so why did eMudhra not spot the mistake for 18 months?

Carelessness in the creation of intermediate certificates risks reputational damage to eMudhra’s business as a certificate authority. Google Chrome announced it is evaluating whether to continue with the company:

“Due to the seriousness of this incident, the Chrome Root Program is closely investigating the ongoing inclusion of eMudhra’s remaining CA certificates in the Chrome Root Store and will share further updates in public forums (e.g., the Common CA Database Public list) as our evaluation progresses.”

Given eMudhra’s sole business is in issuing digital certificates, being dropped by Chrome as a CA could be crippling. eMudhra is a publicly traded company on the Indian stock market, and two days before the breach was revealed its market cap was close to $500M. After details of its incompetence in creating certificates emerged, the company lost 40% of its value in two days.

PHP exploit: AI model beats attackers to finding critical vulnerability

The PHP situation is interesting because the “good guys” in this case (the security researchers) found a critical vulnerability in the framework before attackers did. From OpenAI:

“Over the past two months, we led a coalition of maintainers, researchers, and security providers to address critical vulnerabilities in PHP, an open-source programming language used to run websites and web applications. Calif [a respected security team] identified a critical flaw using GPT‑5.6 Sol as part of our Patch the Planet⁠(opens in a new window) initiative, with Wiz [a security company] also independently discovering and reporting both vulnerabilities. PHP’s maintainers led the fixes and release, with our teams and partners like Wiz, Cloudflare, and others contributing research, technical support, and coordination to prepare protections ahead of disclosure.”

On the surface, this is great news! Critical vulnerabilities were found and can now be patched, thereby closing the door before attackers can exploit them. Right..?

The problem of critical vulnerabilities for open source software is that systems which aren’t immediately patched can be exploited. When the PHP team released the fix for these two vulnerabilities, attackers got details about the vulnerabilities: the code itself shows where the attack can be inititiated! In the case of PHP, that means attackers can now easily execute remote code on any PHP server that is as yet unpatched.

Around 70% of all websites run PHP servers, which is a massive attack surface. We can be confident that many sites won’t update their PHP versions for days, weeks, months, or even years.

This particular vulnerability was so severe that before the security patch, the National Security Agency (NSA) in the US issued guidance urging websites to immediately patch upon release, or take applications fully offline until patched.

Exploits began hours before the fix was announced. It was known in advance that a major vulnerability fix was coming for PHP, and somehow, the vulnerability leaked from the group that disclosed and fixed it. Cloudflare was in the group that worked on this problem and created tooling to monitor and block exploit attempts. From Cloudflare CEO Matthew Prince:

Amazon detected exploit attempts days before the fix. From AWS:

“Amazon Threat Intelligence identified a PHP2Shell exploitation attempt dating to Oct. 17, 2026, in MadPot telemetry, four days before public disclosure of CVE-2026-109618 and CVE-2026-109619. The exploit attempt originated from an IP address associated with a commercial anonymization network. While the vulnerability was under embargo, the targeting of a honeypot from an anonymized IP address suggests the activity was conducted by a threat actor, though it remains unclear how the actor acquired exploit information prior to disclosure.”

Attackers now have AI agents to automate and speed up attacks. Now that the vulnerability is known, attackers can simply spin up agents, instructing them to attempt to exploit servers that have not applied the patch. This probably explains why Cloudflare saw millions of exploit attempts just a few hours after the patch was released.

Railway’s customer secrets leak

Elsewhere on the same day, developer-first cloud provider Railway was hacked and the environment variables (secrets) of its customers were exposed. From the company:

“Our investigation to date indicates that an unauthorized third party may have accessed encrypted environment variables and, for some of them, obtained the decrypted contents. Environment variables may contain secrets such as API keys or database and registry credentials.”

Immediately after detecting the breach, Railway took decisive action by notifying customers and the public, and pausing all deployments.

Pausing deployments created a huge headache for customers, who could not rotate their keys without crashing their systems running on Railway. Also from Railway:

“We strongly recommend that you immediately roll all environment variables, including any API keys, stored in Railway, as well as any private registry credentials, and review logs for those services for unusual activity.

All deployments are paused while we investigate. Until builds are restored, you will not be able to roll out new versions. We apologize for this inconvenience. Follow this incident to be notified when builds resume.”

This is contradictory: rotating environment variables – say, a database access token – means that the application using the old variable stops working until a redeploy. However, Railway has paused deploys! So, customers of Railway had two choices:

• Rotate keys (the responsible step) and deliberately crash their application or take it offline (irresponsible)
• Don’t rotate and hope their key was not leaked – (irresponsible) – and let their application continue to operate (responsible)

Of course, it’s clear why Railway is pausing deployments after the environment variables leaked, and the team must ensure that new variables added by customers do not leak.

In the end, it took Railway close to 20 hours to resolve the problem and allow deployments again. During that time, they unfortunately gave security-conscious customers an impossible choice of bricking their service while rotating keys, or ignoring security best practices to keep services running.

Can we expect future ‘deluges’ of security incidents?

AI labs did warn of a looming “security tsunami” fueled by the technology in May 2026, as per CNBC:

“Dario Amodei warned Tuesday [5 May 2026] that artificial intelligence has created a narrow window for the world’s tech firms, governments and banks to fix tens of thousands of software vulnerabilities found by his company’s latest model.

That AI model, Mythos, was previewed last month along with the disclosure that it had unearthed decades-old vulnerabilities in crucial software.

Since AI models from geopolitical adversary China are “maybe six to 12 months” behind the Anthropic product, there is “roughly that amount of time” to fix these issues, Amodei said.

“The danger is just some enormous increase in the amount of vulnerabilities, in the amount of breaches, in the financial damage that’s done from ransomware on schools, hospitals, not to mention banks,” Amodei said.

Anthropic has limited Mythos to a few partner companies because of concerns about what criminals or adversarial nations could do with it.”

Around that time, I talked with “AI-pilled” devs – some with access to the Mythos model – who told me the same: they expect the security landscape to become very bad in the coming months, with record numbers of systems being breached. One engineer said he’s moving many of his secrets like 2FA backup phrases off cloud storage services and onto paper because he expects cloud storage services to also be breached.

This week, it has seemed like the prediction in May has come to fruition, much of it done with AI’s help. A few days ago, on 17 October, OpenCode was breached and its customer secrets – AI API keys – stolen. Cofounder, Dax Raad, shared that the company has been under constant attack, in his reply to Railway:

Attacks much easier & defense much harder with AI

I sense the balance of power between the good and bad actors in this space is shifting because of AI. Let’s see what it took for an attacker to get access to a system before AI existed, via a critical vulnerability exposure (CVE) – disclosed or not:

With AI, finding critical vulnerabilities is easier for everybody, including bad actors who can automate attacks as well, and do so easily. The change:

It gets worse: before AI, it took quite some skill to be an effective attacker: you needed to be a good-enough coder, understand concepts like buffer overflows, and also learn techniques for executing remote code execution attacks. With AI, none of that applies. Venture capitalist, Deedy Das, described it as scary that nontechnical people now have the tools to easily exploit known CVEs like the PHP one:

So, let me update the diagram to account for the drastic lowering of the bar to becoming an effective attacker:

And we see all of this playing out in practice, starting with publicly disclosed CVEs skyrocketing:

We may need to urgently rethink software security for all systems:

• Automated patching: manual monitoring for patches on a daily/weekly basis will become as irresponsible as applying patches months or years after release. Of course, automatic patching has its own risks, such as if a vendor providing a patch is breached, or a patch causes an outage, etc.
• Expect attackers to use day-zero CVEs. Attackers will increasingly have access to undisclosed vulnerabilities and will access systems externally, or even execute code remotely.
• Ongoing automated security monitoring to be non-negotiable. To detect security breaches, you need a decent level of automated logging and monitoring. This needs to be in place sooner rather than later. The TiDB incident this week shows what happens when a company allows an attacker to freely access their system for a month.
• Defenders will have to use more AI. Attackers are gaining advantage by using AI agents, so defenders will have to do the same. There are no best practices in this field, but there are likely to be lessons in how companies like OpenCode, Railway, and PingCAP are responding – if they share their approaches.
• Securing services and software will become more expensive. Continuously running AI agents to monitor for and mitigate breaches costs money, as do security engineers who are “AI-native.” This all increases the cost of security engineering. We previously covered security engineering basics and did a podcast on code security.

... and the security incidents continue

Since publishing the original article on 22 October, there have been more:

• The Brotli remote execution CVE: Brotli is a compression library used at Node.js, Bun, Envoy, and others. The flaw allows an attacker to take over a service. Details
• Another PHP remote code execution CVE. Just two days after the high-impact PHP2Shell patches, PHP warned of another important CVE, for which a fix is coming. PHP versions will need to be updated fast again.
• Linux CVEs posted publicly. Several unpatched Linux kernel vulnerabilities were accidentally posted on the public kernel mailing list. Details.
• Python Pillow CVE: a vulnerability inside the Python Pillow image library lets an attacker issue OS commands. Details
• … and others I’ve doubtless missed.

At this rate, we might start getting “vulnerability fatigue” and become numb to critical security vulnerabilities unless they affect frameworks our applications use. To avoid that, it’ll be necessary for the industry to improve at coming up with and implementing security practices which work in a security environment where AI speeds up the discovery, publication, and exploitation of vulnerabilities.

Read the full issue of The Pulse this is from, or check out this week’s The Pulse. This week’s issue covers:

1. Anthropic kills its coding interview. A team at the leading frontier AI company gets zero signal from DSA interviews and so has removed it from the engineering recruitment process. The rest of the company, and other AI-native startups, could follow.
2. Details on incident.io’s AI spend and ROI. The resiliency startup has slashed the amount it spends on AI; down from $4,500 per dev, per month, in August, to $1,600 now.
3. Amazon formally incorporates AI usage into SDE and SDM expectations. AI has been placed at the heart of software engineering after the company found a correlation between AI usage and productivity. Also, software development engineers (SDEs) and managers (SDMs) must be close to customers.
4. Microsoft issues new exit offers to engineers. The tech giant has offered 60,000 employees the opportunity to quit by the end of the year. But the exit package is not particularly tempting, and so the number of people who take it will surely be in the thousands, not tens of thousands.

Sign up now! Sign up now! Sign up now? Sign up now!

While it is customary for fans of top-flight clubs with delusions of grandeur to cope with Fizzy Cup exits by pretending the competition is beneath them, this is no longer an option that is open to supporters of Newcastle United. Winning the trophy sponsored by a low-calorie, fruit-flavoured energy drink brand from Thailand may only rank fifth in the domestic pecking order behind the Premier League, FA Cup and both transfer windows, but nobody will ever forget how much it meant to hundreds of thousands of overjoyed Geordies when their team did exactly that the season before last to end a trophy drought dating back to the Jurassic era. And with a quarter-final trip to the tough-tackling League Two bullyboys of Fleetwood up for grabs during their match against serial trophy-dodgers Everton last night, the travelling Toon Army must have fancied their chances of being in the shake-up again.

I clearly remember the first time I laughed out hard and loud, as a 12-year-old, reading Three Men in a Boat by Jerome K Jerome. Tonight, 50 years later, after a stretch of horrendous bad years and months (and for my mental health, I exclude the struggles of Tottenham), I read the Football Daily column. Its is often quite witty, but this time, reading about some penalty kick gone awry … I know, I have a simplistic sense of humour, but I laughed so hard and cried at the same time, that my family wondered what happened. Thanks for a cathartic experience. Words” – L Json.

As a Spurs supporter who saw my first game at White Hart Lane when I was five years old – almost eight decades ago – I can only express my relief that attention has drifted away from the continual ridicule of my team. The fact that a certain club that should return to south of the Thames managed to lose at Highbury (ironically), that lot from down Kings Road and Manchester United are all hogging the headlines for all the wrong reasons just makes life sweeter. It’s time for my second nap of the day now, so I’ll sign off” – Stewart McGuinness.

This is an extract from our daily football email … Football Daily. To get the full version, just visit this page and follow the instructions. 

Some experts express concern about ‘lack of transparency’ as agency removes warnings on three estrogen patches

Three more patches for treating menopause will no longer carry black-box warnings about cardiovascular disease, breast cancer and dementia, officials at the US Food and Drug Administration (FDA) said on Friday.

The FDA did not convene an advisory committee and offered few details on the scientific basis for the decision in the press briefing. The news follows the removal of black-box warnings from six menopause treatments in February, as patients across the US struggle with shortages of the patches.

• ‘We have made these mistakes … you can get punished’

• Dowman lost appeal against red card at Fleetwood

Mikel Arteta has warned Max Dowman that he “is not in Disneyland” and must learn from his red card against Fleetwood after Arsenal’s appeal against his three-match suspension was rejected.

The 16-year-old became the youngest player ever to be sent off in the League Cup on Tuesday as the Premier League champions crashed out to the League Two side Fleetwood. Arteta said he was “very disappointed” that the Football Association rejected the appeal against his red card and felt Dowman was lucky not to suffer a broken foot from “reckless tackles”.

In the eight years since my graphic novel on death published, I’ve learned readers of all ages want to talk about loss and preserve their family’s stories

Becoming an author in my late 50s was unexpected, the result of my daughter Hallie’s own existential anxiety in her early 20s.

Fearing my untimely demise, she begged me to write down advice that would cover all of life’s bases going forward. I agreed, but put the project off for years; it wasn’t until she cornered me with a laptop on vacation that I gave in. She asked countless questions, and I found, to my surprise, that I had plenty to say. Hallie helped shape the manuscript, and then spent a year painting its warmly emotional gouache illustrations.

I lost a parent (or other loved one, including spouse or beloved pet) and your book helped me through.

I gave the book to my kids and wrote my own notes in the margins.

I can’t bear the thought of my mom or dad dying, but they don’t want to talk about it. I’m hoping that sharing the book with them will start some conversations.

Всем привет! Я Дмитрий Шихаев, маркетолог для IT-компаний. Сейчас работаю с крупными IT-заказчиками, помогаю выводить технологические продукты на рынок и развиваю направление ABM-маркетинга в России.

Но путь к этому был далеко не сладким. Когда-то мы с товарищем Гошей, у которого за плечами были базовые знания BASIC и опыт работы с конструктором BAS, решили практически без денег и опыта создать собственный софт для массовой отправки качественных персонализированных деловых писем.

Сегодня разработчику стыдно вспоминать эту историю, поэтому его лицо скрыто в контенте, а имя изменено. Он вырос в сильного специалиста и мы работаем вместе до сих пор.

А тогда бюджет у нас был как у студента за неделю до стипендии, техническая команда состояла из двух калек - маркетолога без знаний IT-внутрянки и начинающего автоматизатора, зато амбиции были как у всего состава Кремниевой долины.

Independent candidate Bill Hill is threatening to unseat Trump ally Nick Begich with a message of affordability

As Republicans fight to maintain their narrow majority in Congress, they face a unique challenge in Alaska – not from a Democrat, but from an independent commercial fisher and first-time candidate running to flip the state’s lone House seat.

Bill Hill, who is an Alaska Native and has worked in education and construction, has positioned himself as a working-class candidate who will fight to make life more affordable for people like himself and his family.

US citizen Leo, who has clashed with Donald Trump over several issues, submits his vote by post in Chicago

Pope Leo has voted in the US midterm elections – the first time a sitting pontiff is known to have cast a ballot in any political election at any time in history.

Leo, who has clashed with the US president, Donald Trump, over several issues, submitted his vote by post in Chicago, where he was born, the Vatican said on Friday. Leo became the first American leader of the Catholic church last year after the death of Pope Francis.

На первый баг ушло десять часов и пять гипотез, ни одна не подтвердилась. На второй — ещё десять часов и семь попыток. А потом каждый закрылся за минуты: в одном случае хватило строчки в дампе состояния, в другом — первого запуска с самодельным обработчиком падений.

Три истории о гонках в M:N-рантайме языка Nova: что сломалось, как я это искал и что в итоге нашло баг. Объясняю простыми словами, знать устройство рантайма не нужно, а в конце — чек-лист для отладки гонок в многопоточном коде.

Daniel Jones held on $3m bail after studio art chair Pipo Nguyen-Duy and Tran Thi Phong found dead at home

Oberlin College was shaken on Friday after an award-winning professor of photography who had worked at the Ohio liberal arts college for nearly 30 years was found shot and killed on Wednesday, officials said.

Pipo Nguyen-Duy, 64, and his wife, Tran Thi Phong, 56, were found shot to death in their home.

Взять готовое или написать свое? Вечный спор в любом бизнесе, который остро встает при выборе буквально чего угодно. И биллинга в частности.

На старте идея написать свою биллинговую систему может показаться невероятно привлекательной. Ведь это просто, быстро и дешевле — у нас и так есть команда разработчиков, зачем платить вендору? К тому же, порой очень хочется получить ту самую гибкость, которой, как кажется, на первый взгляд, нет в коробочных решениях.

Но так ли это на самом деле? В этой статье посмотрим, почему «сделаем сами лучше и быстрее» — не всегда оптимальное решение, когда дело касается биллинга (и дело не  только в российском законодательстве) и во что на самом деле обходится поддержка самописной тарификации.

It influenced The Simpsons and sparked the careers of Eugene Levy, Catherine O’Hara, John Candy and many others. Now this extraordinary Canadian series is finally getting its due

SCTV – short for Second City Television – is one of most influential comedy shows ever created. In the 1970s and 80s, while North American comedy fans worshipped at the altar of Saturday Night Live, the comedy nerds loved SCTV. The Canadian sketch show was weird. It was dark. It was multilayered and years ahead of its time.

Dave Thomas, head writer and an original cast member, was there from the start. “Conan O’Brien said that he used to watch and he thought, ‘This show is being written specifically for me,’” Thomas tells me. “There were references in it that he was absolutely sure only he got. We wrote at the top of our reference level. And the idea was to see if the audience would meet us there.”

Прогресс мчится невероятными темпами. Еще в сентябре запуск 125-миллиардной MoE-модели на домашнем ПК казался забавой для владельцев серверных стоек. А уже к концу октября ее спокойно крутят на RTX 5070, при этом 321-миллиардная модель уже работает на одиночной RTX 3090 с 64 ГБ ОЗУ.

Давайте разберемся, за счет чего возможен такой рывок и какую реальную скорость выдает новый подход на практике.

#мыпомогаемЛизаАлерт

Вместе мы можем больше: «ЛизаАлерт» на связи или что меняет спутниковый интернет в поисках пропавших людей.

К вечеру становится понятно, что человек действительно пропал...