The Aspiring Nerd

Welcome to Curated Finds: a handpicked collection of the most interesting, insightful, and inspiring links I've come across.Dive in and explore what captured my interest.

There’s a lot of talk about how MCP isn’t secure, but I think most people don’t realize just how easy it is to trick LLMs. Simon Willison gives a solid overview of the main risks, which he calls the “lethal trifecta”.

The lethal trifecta of capabilities is:

  • Access to your private data — one of the most common purposes of tools in the first place!
  • Exposure to untrusted content — any mechanism by which text (or images) controlled by a malicious attacker could become available to your LLM
  • The ability to externally communicate in a way that could be used to steal your data (I often call this “exfiltration” but I’m not confident that term is widely understood.)

The core issue is that LLMs are great at following instructions, but they don’t distinguish between legit ones and malicious ones.

LLMs follow instructions in content. This is what makes them so useful: we can feed them instructions written in human language and they will follow those instructions and do our bidding.

The problem is that they don’t just follow our instructions. They will happily follow any instructions that make it to the model, whether or not they came from their operator or from some other source.

He digs into MCP specifically:

The problem with Model Context Protocol—MCP—is that it encourages users to mix and match tools from different sources that can do different things.

Many of those tools provide access to your private data. Many more of them—often the same tools in fact—provide access to places that might host malicious instructions.

And yeah, there’s no easy fix.

Here’s the really bad news: we still don’t know how to 100% reliably prevent this from happening.

Plenty of vendors will sell you “guardrail” products that claim to be able to detect and prevent these attacks. I am deeply suspicious of these: If you look closely they’ll almost always carry confident claims that they capture “95% of attacks” or similar… but in web application security 95% is very much a failing grade.

Quoting Devansh:

Fine-tuning advanced LLMs isn’t knowledge injection — it’s destructive overwriting. Neurons in trained language models aren’t blank slates; they’re densely interconnected and already encode crucial, nuanced information. When you fine-tune, you risk erasing valuable existing patterns, leading to unexpected and problematic downstream effects. Instead, use modular methods like retrieval-augmented generation, adapters, or prompt-engineering — these techniques inject new information without damaging the underlying model’s carefully built ecosystem.

Source: codinginterviewsmadesimple.substack.com

Debugging can be a confusing and emotionally unpleasant process. Writing code involves building neat mental models and fitting them together in satisfying ways. Everything is nicely labeled and accounted for. When something goes wrong, it doesn’t just break your code - it also breaks this comforting illusion of control and understanding.

Source: seangoedecke.com

I’ve always been pretty careful about what I put on the internet, especially photos, and definitely pictures of my kids. I’ve made a point to keep where I live and hang out under wraps to protect my privacy. If I were famous (or had anything serious to hide) I’d be seriously worried about how good models are getting these days.

Watching OpenAI’s new o3 model guess where a photo was taken is one of those moments where decades of science fiction suddenly come to life. (…)

It’s also deeply dystopian. Technology can identify locations from photographs now. It’s vitally important that people understand how easy this is—if you have any reason at all to be concerned about your safety, you need to know that any photo you share—even a photo as bland as my example above—could be used to identify your location.

As is frequently the case with modern AI, the fact that this technology is openly available to almost anyone has negative and positive implications. As with image generation, it’s important that people can see what this stuff can do first hand. Seeing this in action is a visceral lesson in what’s now possible.

Source: simonwillison.net

I’ve had this convo at work a bunch—LLMs are becoming a commodity, getting easier and cheaper by the day. But I never really thought about the flip side. What if coding is becoming the commodity instead? 🤯

 “The first generation of AI-powered products (often called “AI Wrapper” apps, because they “just” are wrapped around an LLM API) were quickly brought to market by small teams of engineers, picking off the low-hanging problems. But today, I’m seeing teams of domain experts wading into the field, hiring a programmer or two to handle the implementation, while the experts themselves provide the prompts, data labeling, and evaluations. For these companies, the coding is commodified but the domain expertise is the differentiator.”

Source: Drew Breunig - dbreunig.com

Really excited about the possibilities with MCP—but is it actually ready for safe use in production? Not so sure. Elena Cross put together a great summary of the potential security issues.

“MCP is not secure by default. And if you’ve plugged your agents into arbitrary servers without reading the fine print — congrats, you may have just opened a side-channel into your shell, secrets, or infrastructure.”

Source: elenacross7.medium.com

I’ve always liked the idea of owning my own little patch of grass on the internet. From my first homepage full of animated GIFs in the late ’90s to this very blog, I’ve always treasured having a space that’s free of corporate influence and greed. Hardly anyone reads it—but at least it’s mine.

Matt Webb, Interview for People and Blogs:

Blogging is small-p political again, today. It’s come back round. It’s a statement to put your words in a place where they are not subject to someone else’s algorithm telling you what success looks like; when you blog, your words are not a vote for the values of someone else’s platform.

Source: manuelmoreale.com (via Simon Willison)

I’ve always been a tinkerer and a hacker, and I love building things. Unfortunately, I’m not a developer (even though I know the basics of JavaScript and Python), so I’m often stuck in my tracks by my lack of skill to fully build or prototype the ideas I have. It’s either too complex or too time-consuming.

For the last year or two, I’ve used LLMs and generative AI tools to prototype and validate early concepts. My stack is pretty simple: I chat with Claude (using Artifacts) or ChatGPT, edit existing code with Cursor, or go further by building full-fledged web apps with platforms like Bolt.new and Lovable.

Andrej Karpathy sums it up the best:

It’s not too bad for throwaway weekend projects, but still quite amusing. I’m building a project or webapp, but it’s not really coding - I just see stuff, say stuff, run stuff, and copy paste stuff, and it mostly works.

I also like how Geoffrey Huntley  describes the impact on those of us working in SaaS and development:

People with ideas+unique-insight can get concepts to market in rapid time and be less dependent on needing others expertise as the worlds knowledge is now in the palms of everyone’s hands.

Technologists are still required, perhaps it’s the ideas guys/gals who should be concerned as software engineers now have a path to bootstrap a concept in every white collar industry (recruiting, law, finance, finance, accounting, et al) at breakneck speed without having to find co-founders.

(…)

If you’re a high agency person, there’s never been a better time to be alive…

While I see myself as an idea guy who can just do things, I still wouldn’t feel comfortable shipping all of this to production without some oversight or feedback from someone who actually knows what they’re doing.

Source: ghuntley.com (via Simon Willison)

You often hear that AI will replace us all and that Skynet will take over, but Anthropic’s new initiative should provide a clearer view of how LLMs impact the economy and labor market. There’s a lot of uncertainty, anxiety, and misunderstanding around AI’s capabilities, so it’s crucial that regulations are based on actual data. I really appreciate them open-sourcing the dataset, but I won’t lie—knowing that all my conversations might potentially end up in research data is a little unnerving, even if it’s anonymized.

These kinds of longitudinal analyses can give us new insights into AI and the job market. For example, we’ll be able to monitor changes in the depth of AI use within occupations. If it remains the case that AI is used only for certain tasks, and only a few jobs use AI for the vast majority of their tasks, the future might be one where most current jobs evolve rather than disappear. We can also monitor the ratio of automation to augmentation, providing signals of areas where automation is becoming more prevalent.

Our research gives data on how AI is being used, but it doesn’t provide policy prescriptions. Answers to questions about how to prepare for AI’s impact on the labor market can’t come directly from research in isolation; instead, they’ll come from a combination of evidence, values, and experience from broad perspectives.

Source: anthropic.com

As Michael Scott would say, ‘Well, well, well… how the turntables!’ OpenAI seems upset that another AI company is training on its data without permission. The article also provides a gentle introduction to distillation.

It is, as many have already pointed out, incredibly ironic that OpenAI, a company that has been obtaining large amounts of data from all of humankind largely in an “unauthorized manner,” and, in some cases, in violation of the terms of service of those from whom they have been taking from, is now complaining about the very practices by which it has built its company.