{{< admonition type=“tip” >}} This article was first published as part of a substack experiment, I reproduced it here. {{< /admonition >}}

Hey everyone, welcome to Week 2!

Last week, I talked about the risks of a single AI agent. But what happens when you put multiple AIs together to work as a team?

These are called Multi-Agent Systems, or MAS. Think of it like going from managing one employee to managing an entire department. Suddenly, things get a lot more complex. The agents have to talk to each other, share information, and coordinate their actions.

And just like with a human team, this is where new problems can start. The risks don't just add up; they multiply.

Here are a couple of big new threats that pop up when AIs work in teams:

The “Bad Teammate” problem (rogue agents)

This is when a malicious or hacked AI agent joins the team. Because it's trusted by the other agents, it can fly under the radar and cause a lot of damage.

Imagine an HR team with AI agents. A “rogue” agent could get access to the payroll system and start giving fake salary increases to an attacker's account. Because the other agents trust it, the fraudulent payments get approved.

The “Gossip and Rumors” problem (agent communication poisoning)

This is when an attacker messes with the communication between the agents. They inject false information into the conversation, which then spreads like a rumor through the whole team.

One agent might ask another, “Is this transaction approved?” The attacker intercepts the message and makes the second agent see a fake “Yes.” This can cause a chain reaction of bad decisions, all based on one piece of bad information.

The good news? There are special playbooks, like the MAESTRO framework, designed specifically to find these kinds of team-based security holes.

The key takeaway is this: managing a team of AIs is a whole different ballgame. You have to worry not just about each individual agent, but how they trust and talk to each other.

My reading list is a bit shorter this week, mostly because I’ve fallen down a deep, deep 3D printing rabbit hole. (My desk is now covered in very handy 3d printed tools for the printer itself and one glorious OctoRocktopus).

Still, between prints, I managed to find some absolute gems. This week's theme seems to be the practical, sometimes harsh reality of AI adoption, mixed with some fascinating policy decisions in the open-source world.

Here’s what I’ve been reading.

The State of AI: Hype vs. Reality

It feels like we're in a bit of a reality-check moment with AI. The hype around all-powerful AI agents is clashing with the messy truth of actually getting them to work inside a real business.

• The Truth About AI Agents Only a Practitioner Can Tell You by Chris Tyson. This piece was a breath of fresh air. Tyson cuts through the marketing fluff to explain why most companies are nowhere near ready for the AI agent revolution everyone is promising. A must-read for anyone in a leadership position.

• Build and Host AI-Powered Apps with Claude – No Deployment Needed by Anthropic. This is a genuinely cool development. Anthropic is letting people build and host small AI apps directly on Claude, and the users pay for the API calls. It’s like being able to launch a web app without ever having to think about servers. A clever solution to the deployment problem for smaller projects.

• Apple Research Is Generating Images with a Forgotten AI Technique by Marcus Mendes. It turns out Apple is digging through AI's old record collection. This article looks at how their researchers are reviving a forgotten AI technique for generating images, suggesting there's still gold in those older methods.

• MCP Is Eating the World—-and It's Here to Stay by Stainless. A great opinion piece on the Model Context Protocol (MCP). The author argues that while MCP isn't some revolutionary breakthrough, its strength is its simplicity and timing. It just works, and that’s why it’s probably going to stick around for a long time.

Drawing Lines in the Sand: Policy & Open Source

This week saw some interesting lines being drawn in the world of open source. It’s fascinating to see major projects grapple with the legal and ethical questions around AI.

• Docs: Define Policy Forbidding Use of AI Code Generators by qemu. The QEMU project made a bold move, officially banning contributions from AI code generators. Their reasoning? The legal and licensing implications are still a complete mess, and they’re choosing to play it safe.

• Libxml2's “No Security Embargoes” Policy by Joe Brockmeier. In a similar spirit of radical transparency, this post outlines why the libxml2 project discloses security issues immediately, with no embargo period.

• Microsoft Dependency Has Risks by Miloslav Homer. A good reminder of the old wisdom: don't put all your eggs in one basket. This piece explores the risks that come from relying too heavily on a single company's ecosystem.

Miscellaneous Finds

And now for a few other interesting things that crossed my screen this week.

• Games Run Faster on SteamOS than Windows 11, Ars Testing Finds by Kyle Orland. The team at Ars Technica did some testing and found that Valve's free Linux-based SteamOS actually gets better frame rates on the Lenovo Legion Go than Windows 11 does. A fun win for the open-source world.

• Massive Biomolecular Shifts Occur in Our 40s and 60s by Rachel Tompa. Just when you thought aging was a steady, gradual decline, Stanford Medicine researchers found that our bodies go through huge biological shifts around age 40 and 60. A fascinating, if slightly unnerving, read.

• The Offline Club. Feeling overwhelmed by all this tech? I found the perfect antidote. This is a community that hosts events around the world designed to help people unplug, disconnect from their devices, and reconnect with each other. I'm definitely intrigued.

{{< admonition type=“tip” >}} This article was first published as part of a substack experiment, I reproduced it here. {{< /admonition >}}

{{< backlink 20250627-introducing-agents “Last time” >}}, we learned that AI agents are like smart assistants that can think, remember, and most importantly, do things on their own.

That autonomy is what makes them so powerful. But it also creates some brand-new, frankly scary, security problems. Today I’m going to look at two of the biggest ones: Memory Poisoning and Tool Misuse.

Memory poisoning

So, what is Memory Poisoning?

The threat: The best way to think about this is like gaslighting an AI. It’s when an attacker deliberately feeds an AI false information over and over, until the AI starts to believe that information is true. Once that bad “memory” is planted, the agent will start making bad decisions based on it.

It's not about tricking the AI just once. It’s about corrupting its memory over time.

• Imagine a travel agent AI. An attacker keeps telling it, “By the way, chartered flights are always free.” If the AI hears this enough, it might save that “fact” to its memory. The next thing you know, it's letting people book expensive private flights without paying. Ouch.
• Or think about a team of customer service AIs. If one agent gets its memory corrupted with a fake, overly generous refund policy, it could then share that bad information with the other agents. Suddenly, the whole team is giving out wrong refunds, all based on one corrupted memory.

How to prevent it: You basically have to become a fact-checker for your AI.

• Constantly scan the AI’s memory for weird or unusual data.
• Only allow trusted sources to make changes to its long-term memory.
• Keep different user sessions separate. This stops one bad actor from poisoning the well for everyone else.

Tool misuse

This next one is just as important.

The threat: Remember how I said agents can use “tools” like sending emails or browse the web? Tool Misuse is when an attacker tricks an agent into using one of its tools for something harmful.

It’s like giving your assistant a company credit card (a “tool”). They have permission to use it for work. But a trickster could convince your assistant to use that card to buy a bunch of stuff for them instead. The assistant isn't evil, it's just being tricked into using its power the wrong way. This is often called a “Confused Deputy”attack. The AI is the deputy with power, but it's being confused by a malicious user.

• An attacker could trick an AI into using its email tool to start sending spam or leaking private data. This happened to Github, where the agent was tricked to leak private repositories.
• Or they could find a flaw in a shopping agent's logic that lets them skip the “payment” step entirely.

How to prevent it: It all comes down to having strict rules for every tool.

• Set clear limits. Be very specific about what tools the AI can use, when it can use them, and what it can do with them.
• Use a sandbox. This is a classic security move. Let the AI use its tools in a “sandbox”—a safe, isolated environment where it can't accidentally cause any real damage.
• Keep good logs. Track every single time a tool is used. If you see something strange, like an AI suddenly trying to send 1,000 emails, you can shut it down quickly.

These two threats show us that an agent's greatest strengths—its memory and its ability to act—can also be its biggest weaknesses if they're not protected.

{{< admonition type=“tip” >}} This article was first published as part of a substack experiment, I reproduced it here. {{< /admonition >}}

Hey everyone, let's keep going!

So far, I've covered the basics of AI security and some specific problems like Prompt Injection. Today, I’m talking about the next big thing: AI Agents.

You might be wondering, “What's an AI Agent?” and how is it different from the AI chatbots we already know?

Think of it like this. A chatbot is like asking a librarian a question. They find the information and give it to you. An AI Agent is like hiring a super-smart personal assistant. You don't just ask it a question; you give it a goal.

It's not just a chatbot; it's a doer.

You can tell it, “Plan a weekend trip to the beach for me,” and it will figure out all the steps on its own. It's designed to be autonomous; to make its own decisions and take action to get the job done.

What Makes an AI Agent Tick?

These agents have a few key abilities that make them so powerful.

• They can think and plan. An agent can take a big, messy goal and break it down into a series of smaller, common-sense steps. It can even look back at what it has done, learn from its mistakes, and change its plan.
• They have a memory. Agents can remember what you've talked about before. This helps them keep track of what's going on and learn from past actions, making them much smarter over time.
• They can use tools. This is the really big one. Agents can take action in the real world by using “tools.” These tools can be anything: Browse a website, running a search, doing calculations, or even writing and executing computer code.

So, Where's the Risk?

That last part, the ability to take action and use tools, is what makes these agents so useful. But it's also what makes them risky.

The very thing that makes them powerful, their autonomy, is also their biggest weakness. When you give an AI the power to act on its own, you create new security risks that we've never had to deal with before. Problems like:

• Memory Poisoning: What if an attacker messes with the agent's memory to trick it later?
• Tool Misuse: What if someone tricks the agent into using its tools for something harmful?

These aren't just theories. Frameworks like LangChain and CrewAI make it easier than ever for developers to build these agents, so we're going to see them everywhere.

There are many other threats in the Agent landscape, a study performed by Antrophic found that AI agents, when faced with replacement or an inability to achieve a goal might resort to blackmail or leak confidential information to competitors.

Understanding how they work is the first step to protecting against the new risks they bring.

Stay tuned, because next time we’re going to look at the attacks in more detail. That's when things get really interesting.

{{< admonition type=“tip” >}} This article was first published as part of a substack experiment, I reproduced it here. {{< /admonition >}}

Alright, welcome back to our chat about AI security!

On Monday, I looked at the big picture. Today, I’m zooming in on two specific problems that pop up all the time. These are straight from the official OWASP Top 10 list of big risks for AI, so they're definitely ones to watch.

Let's dive into Prompt Injection and Sensitive Information Disclosure.

Prompt injection

So, what on earth is prompt injection?

The threat: Imagine you have a super helpful robot assistant. A prompt is just the instruction you give it. But with prompt injection, a trickster hides a secret, malicious instruction inside a normal-looking one.

It’s like telling your robot: “Please get me a coffee, and oh, by the way, also give me the keys to the secret vault.” The robot is so focused on following instructions that it might just do it. The sneaky part can even be hidden in an image or a file, not just text.

The result? The AI could be tricked into:

• Leaking secret information.
• Giving an attacker access to tools they shouldn't have.
• Changing important content without permission.

To prevent these issue, you can't just put up one wall; you need a few layers of defense.

• Be specific. Tell the AI exactly what kind of answer you expect from it. The clearer your rules, the harder it is for the AI to get tricked.
• Give the AI less power. This one is huge. The AI should only have access to the bare minimum it needs to do its job (the principle of least privilege). Think of it like an intern—you wouldn't give them the keys to everything on their first day.
• Get a human's approval. If the AI is about to do something high-risk, like deleting a file or sending money, a human should always have to click “approve.” Always.
• Keep it separate. Treat any information from outside sources as untrusted. Put a clear wall between what a user asks and the secret data the AI can access.

Sensitive information disclosure

The threat: This one is a bit more straightforward. It’s when an AI accidentally blurts out information that should have been kept private.

I'm talking about things like customer names and addresses, company financial data, or even bits of the AI's own secret source code. The AI is designed to be helpful, but sometimes it's too helpful and shares things it shouldn't.

How to prevent it:

• Don't share secrets with the AI. The easiest way to stop a secret from getting out is to never tell it to the AI in the first place. Only give the model access to data that is absolutely necessary.
• Teach your users. Remind people who use the AI not to type personal or confidential information into the chat box. A little training goes a long way.
• Be honest about data. Have a clear, simple policy about what data you collect and how you use it. And, most importantly, give people an easy way to say “no, thanks” and opt out of their data being used.

Both of these threats really highlight something important. We can't just focus on old-school hacking anymore. We have to understand how the conversation with an AI can be twisted and misused.

Want to try tricking an AI as a game? Try out Gandalf, a fun game on LLM security where you trick Gandalf to provide you his password.

{{< admonition type=“tip” >}} This article was first published as part of a substack experiment, I reproduced it here. {{< /admonition >}}

Welcome to Day 1 of my guide to the important topic of Generative AI (GenAI) and Large Language Model (LLM) security.

LLMs are powerful AI systems that are being used more and more in business. They offer amazing new abilities, but they also create new security problems and risks. Old cybersecurity methods, which mainly focused on stopping hackers from breaking into computers, are not enough to protect these new systems.

Why AI security is different and important

The fast growth of LLMs has created new risks for data security. These advanced AI systems have special weaknesses. This means we need new ways to test them and protect them.

Here are the key differences and challenges:

• Prompt injection: Attackers use tricky instructions to make the AI do something it shouldn't. Such as showing weird recipes in your vibe-coded app.
• Data leakage: The AI might accidentally share secret information. This happens to the best of us, just ask Microsoft.
• Hallucinations: The AI gives wrong information but sounds very sure it is correct. Even when you are a lawyer, AI might hallucinate.
• Agentic vulnerabilities: These are complex attacks on smart AI “agents” that use many tools and make their own decisions. This was demonstrated by Github’s agent leaking private repositories.
• Supply chain risks: Problems can come from the different steps used to create and update the AI models.

Unlike normal computer programs, AI models can sometimes act in ways we don't expect. This is especially true when they face new situations or attacks. The results are not simply “right” or “wrong.” So, we must watch them closely and decide what level of error is acceptable.

AI security is needed for every step, from start to finish. This includes collecting data, training the model, testing it, using it, and finally, turning it off. We need a complete plan that covers everything.

Helpful Guides and Methods

To handle these new threats, experts have created several guides and methods:

• OWASP Top 10 for LLM Applications: This is a famous list of the top 10 security risks for LLM applications. It is made by a community of experts to give basic advice for using LLMs safely (link).
• GenAI Red Teaming: This is like a “fire drill” for AI. Experts pretend to be attackers to find weaknesses in the AI system's security and safety. This helps find problems before real attackers do (link).
• LLMSecOps Framework: This method helps add security into every stage of building and using an LLM. It makes sure that security is a part of the whole process, not just an extra step at the end (link).
• MAESTRO Threat Modeling Framework: MAESTRO is a special method to study and find security risks in advanced AI systems where multiple AI “agents” work together. It helps security teams find and fix unique problems in these complex systems (link).

These guides give practical advice for everyone who builds and protects AI systems, including developers, system designers, and security experts. In the coming posts I will explore this vast new landscape together with you.

A little later then usuals. Yesterday I was at the Dutch ComicCon, and I forgot to post. Here is my reading of last week.

The Real Impact of AI

I think we’re all wondering about the deeper effects of weaving AI into our daily lives. This week, I found a few articles that really made me stop and think. The first was a standout study from MIT that suggests using tools like ChatGPT for writing could lead to a kind of “cognitive debt.” They literally measured brain activity and found that relying on AI can cause the parts of our brain responsible for deep thinking to become under-engaged. It's a fascinating and slightly worrying idea.

On a much darker note, I read a tragic story about a man's mental health crisis that became dangerously entangled with his conversations with an AI. It’s a powerful reminder that we're still grappling with the very human consequences of this technology.

My Reading List:

• Your Brain on ChatGPT: A must-read MIT study on how AI might be creating a 'cognitive debt.' The summary article from TIME is a bit quicker to get through.
• A Tragic Story (Content Warning): A heavy but important piece from Rolling Stone about the unforeseen human cost when AI and a mental health crisis collide.
• The AI Drawbridge is Going Up: A sharp argument that the AI world is becoming less open, much like the web did before it.
• How Llama 3.1 Remembers Harry Potter: A look at an AI's massive recall ability and the major copyright questions it raises.
• Andrej Karpathy on the New Software: A short but thought-provoking piece from Y Combinator on how software development itself is changing.
• AI in Dutch Schools: For my Dutch readers, a look at how the educational system is thinking about AI in testing.
• Vibecoding & Google Translate: A weirdly interesting post on what translation can teach us about culture.

AI Security & Development: A Messy Frontier

This is where things get really interesting for me. The intersection of AI, development, and security is a wild west right now. Simon Willison perfectly captured the danger with what he calls the “lethal trifecta” for AI agents: giving an AI access to private data, letting it browse untrusted content (the internet), and allowing it to talk to the outside world. It’s a recipe for disaster.

This isn't just theory, either. Another article reported that LLM agents are shockingly bad at tasks that require confidentiality, failing basic tests in a simulated CRM environment. And from the developer’s perspective, I saw two sides of the coin: Miguel Grinberg explained why these AI coding tools just aren't working for him, while Simon Willison shared how an AI-generated library became his first open-source project.

My Reading List:

• The Lethal Trifecta for AI Agents: Simon Willison’s essential breakdown of the core security risk of today's AI agents.
• How LLMs Could Be Insider Threats: Anthropic research on how an AI can be prompted to perform blackmail or espionage. Spooky stuff.
• LLM Agents Flunk Confidentiality Tests: A report from The Register on just how unreliable agents are with sensitive info.
• Hacking AI Agents: A practical guide from Joseph Thacker on finding and exploiting vulnerabilities in AI applications.
• Curl vs. AI: A great interview with the creator of Curl on dealing with the flood of low-quality, AI-generated bug reports.
• Why AI Coding Tools Don't Work For Me: An honest take from Miguel Grinberg on the current limitations of AI coding assistants.
• My First AI-Generated Library: The fascinating counter-story from Simon Willison on building a complete library with AI help.
• Security Best Practices from Model Context Protocol & a look at its potential threats from CyberArk.
• Free Local AI Security Checks: A cool tool I found on Hacker News for checking AI-generated code in VSCode.
• Interviews with Tanya Janca: Two great posts from her, one an interview with Laura Bell Main and the other a collection of her fantastic conference talks.

Open Source News

It was a big week for open-source drama and discoveries. The headline was definitely the massive malware network found hiding on GitHub—a stark reminder to be careful out there. On a brighter note, I read about a new Linux phone being built with open-source hardware right here in the EU.

• Massive Malware Network on GitHub: The story of how Klarrio uncovered a huge, malicious network on the world's biggest code platform.
• A New EU-Made Linux Phone: An interview with the folks at Liberux about their open-source hardware phone.
• Open Source Can't Coordinate: A short but insightful post on the challenges of getting things done in open-source projects.

Dev Tools I'm Eyeing

I'm always on the lookout for tools that can make my workflow a little better. This week, a keyboard-centric setup for VSCode + Neovim caught my eye, along with a tool for smarter git squash commands.

• VSCode + Neovim Setup: A guide for anyone who wants a powerful, keyboard-first development environment.
• Git-Smart-Squash: A tool that looks like it could really clean up my git workflow.
• Harper Grammar Checker: A free, open-source, and privacy-focused alternative to other grammar tools.
• On Software Complexity: A meta-analysis of three different ways to think about what makes software “complex.”
• Also, I've just been keeping up with David Heinemeier Hansson's blog. Always a good read.

And Finally, Something Completely Different...

To cleanse the palate after all that heavy reading on AI risk and malware, here’s a fantastic video on how to make Gözleme, the amazing Turkish flatbread snack. Enjoy!

{{< admonition type=“tip” >}} This article was first published as part of a substack experiment, I reproduced it here. {{< /admonition >}}

Hey everyone,

Let's be honest. This new wave of generative AI is moving incredibly fast. One minute we're asking it to write a poem, and the next, AI “agents” are being built to act on their own.

I've been working in tech for a long time, but some of the security risks I'm seeing are… different. They're strange, new, and frankly, a little scary.

Your old cybersecurity playbook? It’s not going to cut it. Trying to use old security methods on these new AIs is like trying to put a bike lock on a cloud. The problems are just in a different dimension.

What is not from another dimension? Receiving this entire 12 post series in your mailbox…

So, I decided to put together a guide. For the next three weeks, I'm going to walk you through the security risks of this new AI world. I'll look at the real threats and, more importantly, how to deal with them. On Monday, Wednesday, Friday and Saturday a bite-sized newsletter drops which gets you up to speed on a single topic. A quick read and plenty of discussion at the coffee machine (or in slack if you are home)!

Here’s what you can expect in the coming 12(!) posts:

• Week 1: Getting a Handle on the Basics Why is securing an AI so different from a regular app? We'll jump right into the most common weak spots, like tricking an AI into doing something it shouldn't (Prompt Injection) or making it spill secrets it's supposed to keep. Then, we’ll talk about AI agents—what happens when AI starts doing things on its own?
• Week 2: When Things Get Weird This is where it gets really interesting. We’ll look at what happens when AIs team up and their problems multiply. We'll cover AI Hallucinations (what happens when an AI just makes stuff up) and how that can cause a total mess. We'll also dig into scary stuff like an AI's goals being hijacked by a bad actor.
• Week 3: Building a Defense That Actually Works It's not all doom and gloom! We’ll spend this week focused on solutions. I'll show you how to protect your data when working with AI and what “Red Teaming” an AI looks like. (Hint: It’s about trying to break your own stuff to find the flaws first). We'll also look at some cool new tools and frameworks designed to keep AI systems safe. This series is for you if you're a developer, a security pro, or just curious about what's really going on under the hood of AI.

If you’ve been looking for a straightforward guide to the real security challenges of AI, this is it.

The first post is coming this monday. If you know anyone who should be part of this conversation, now would be a great time to share this with them.

Software Engineering

In my feed the opening talk by DHH at Rails World 2024 popped up, most notably due his stance on the reduction of complexity in running an online business. He promotes running your own (virtual) hardware, reducing build pipelines and not using Platform as a Service providers (#nopaas). Watch it below.

{{< rawhtml >}} {{< /rawhtml >}}

It really interested me, for my hobby projects I don't have a lot of time and I would like the experience to be as smooth as butter. Years ago I wrote Rails based web applications, so the release of Rails 8 with this introduction made me curious how Rails development is nowadays. Spent a weekend working on a small project and it is pretty darn good I must say.

AI Stuff

Threats and stupidity

Tim Bray talked about AI Angst [2], how the world seems to struggle with using AI and feel threatened by it. At the same time we are full into the time of AI Agents with cool projects to track their effectiveness. As it is still possible to leak private data using AI agents (echoleak) [25] and AI agents are wiping your computer when stuff becomes too hard [6], it seems we are still some ways off the safe application of AI agents. Most AI applications seem to be some type of “fraud” as well, such as calory counting apps [7]. Just because you stick AI into it, doesn't make it better.

I highly recommend reading Neil Madden's review of the AI code written by Cloudflare for their new OAuth library [12]. The process they used is well documented, so we can see exactly where the AI stopped being able to generate the required code and needed human interaction. Most interesting point of this review is that Neil is into security, and this is a security library and, shocker, the AI failed at safe application of security. Luckily the humans of Cloudflare are excellent coders and know their stuff!

There are good applications as well of course, such as Honeycomb finding that computers can work faster then humans [16]. Or having experience developers use AI to do something new, such as build an iOS app [14].

Apple, in the meantime, dropped a major paper “Shojaee, Mirzadeh & Alizadeh et al. (2025) The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity, arXiv.org.” [11], which identifies that current reasoning models are using patterns from the past to build up thoughts and are not really reasoning. This resulted in a lot of discussion [13], but the paper seems to hold.

A new repository was launched, vibesec, which holds AI rules for various programming languages/models.

Closing

I really should get a better workflow going. Currently my reading goes into Zotero and then on sunday I just categorize the items correctly. Perhaps I can make something that will build this post during the week, as I read it... how do you do it?

The complete list

{{< rawhtml >}}

{{< /rawhtml >}}

Tech in general

I learned that most of the layoffs in the US are not so much about AI taking jobs. Sure, there are bound to be a bunch of people that are no longer employed because their jobs was easily replaced by a system, but there is more then meets the eye. In “The hidden time bomb in the tax code that's fueling mass tech layoffs” explores the tax rule that was changed under Trump-I, section 174, which basically no longer allows companies to write-off R&D effort in the current fiscal year.

Security in general

Some really neat attacks or attack vectors:

• Weaponizing Dependabot: Pwn Request at its finest :: Learn how Dependabot can be co-opted to exploit some sensitive workflows, through the Confused Deputy Problem and branch name injections.

AI

New models of interest

• Black Forrest Lab – Flux.1 Kontext for image manipulation

General News

Lauren Weinstein reported that OpenAI was ordered to store logs of all conversations with ChatGPT, even the private and “do not use for training” data. The original article was by arstechnica.

Antirez wrote a nice opinion post on why thy think humans are still beter then AI at coding.

In the same light, Cloudflare released an oauth library “mostly” written by AI. Max Mitchell went through the github history and found that without human involvement we would not have this library. Granted, 95% of the code seems generated, but it would not work without humans.

A note by Cloudflare: To emphasize, this is not “vibe coded”. Every line was thoroughly reviewed and cross-referenced with relevant RFCs, by security experts with previous experience with those RFCs. I was trying to validate my skepticism. I ended up proving myself wrong.

As AI is able to understand complex papers much easier then humans, Reuven Cohen posted that he used Perplexity AI to read a paper on secretly tracking human movement through walls using standard WiFi routers (Geng et al.). It took the AI less then 1 hour to implement the paper in an application.

Sonia Mishra wrote a very nice piece on The Rise of Vibe Coding: Innovation at the Cost of Security, I highly recommend anyone thinking about using {{< backlink “vibe-coding” “vibe-coding” >}} to check it out.

Security issues

• AI-hallucinated code dependencies become new supply chain risk by Bill Toulas
• Claude seems to have learned how to bypass restrictions set by the Cursor IDE. It was not allowed to use mv and rm, so it wrote a shell script to do it for it and executed it.
• VectorSmuggle :: A comprehensive proof-of-concept demonstrating vector-based data exfiltration techniques in AI/ML environments. This project illustrates potential risks in RAG systems and provides tools and concepts for defensive analysis.

Model Context Protocol

The world is going nuts about Model Context Protocol.

• Have you ever wanted to write a MCP server in bash? Now you can – mcp-server-bash-sdk.

Attacks

A list of interesting attack vectors or stories:

• Exploiting Model Context Protocol (MCP) – Demonstrating Risks and Mitigating GenAI Threats :: Cato CTRL built 2 PoCs showing how the Model Context Protocol (MCP), the “USB-C” for GenAI, can be exploited for attacks. GenAI threats are evolving fast.
• GitHub MCP Exploited: Accessing private repositories via MCP :: We showcase a critical vulnerability with the official GitHub MCP server, allowing attackers to access private repository data. The vulnerability is among the first discovered by Invariant's security analyzer for detecting toxic agent flows.
• Remote Prompt Injection in GitLab Duo Leads to Source Code Theft
• MCP Security Exposed: What You Need to Know Now :: explores the security issues with MCP, very American in its writing, be warned.
• Poison everywhere: No output from your MCP server is safe :: In this blog post, we’ll briefly explore MCP and dive into a Tool Poisoning Attack (TPA), originally described by Invariant Labs. We’ll show that existing TPA research focuses on description fields, a scope our findings reveal is dangerously narrow. The true attack surface extends across the entire tool schema, coined Full-Schema Poisoning (FSP). Following that, we introduce a new attack targeting MCP servers — one that manipulates the tool’s output to significantly complicate detection through static analysis. We refer to this as the Advanced Tool Poisoning Attack (ATPA).
• Top 10 MCP Security Risks You Need to Know – Prompt Security :: Discover the top 10 security risks in Model Context Protocols (MCPs). Learn how attackers exploit prompt injection, tool misuse, and more.

Defense

A company announced itself, Spawn Systems, that is promoting its product MCP Defender, a firewall type system to shield you of MCP abuse. There is very little information. From the Github history the first commit was May 28th, and the entire thing seems to be {{< backlink “vibe-coding” “vibe coded” >}}, I would not yet trust this project.