Code May Be 'Free' Now, But Verification Isn't

by Ryan (Founder)

AI has made producing code dramatically cheaper.

Features that once took days or weeks to build can now be built in minutes to hours, and entire applications can be assembled by people who needed a developer to achieve the same just a few years ago.

Tools like Lovable, Replit, Cursor, Claude, and Bolt have compressed the gap between having an idea in your head and turning it into functional, real-world software.

What hasn’t been reduced to the same degree is the work required to verify that software.

If your goal is to take an app to production (meaning, it doesn’t just live on your computer locally—you’ve deployed it to the internet for others to access), you still need to verify quite a few things before you can consider it "safe:"

• You need to know whether authorization is correct
• Whether users can access data they shouldn’t be able to see
• Whether secrets are exposed
• Whether API routes trust input they shouldn’t trust
• Whether dependencies have known vulnerabilities
• Whether privileged actions are actually restricted

AI can certainly help with that work, but it doesn’t remove the need for those things to be scrutinized for accuracy and safety. Ideally by an experienced developer who knows what to look for versus just randomly prompting the AI until an app "feels" correct; even if it’s hiding serious issues under the hood.

This creates a mismatch between the cost of producing software versus the cost of being confident it works properly and securely.

An app can look finished long before anyone has determined whether or not it’s safe to ship. Creating an account works, the database connects, Stripe handles payments fine, emails show up as expected, and deployment to production succeeds.

Those are useful signals in isolation, but they only tell you that the expected path (a.k.a the "happy path") is functioning. They do not tell you much about what happens when your app is used in a way you didn’t intend.

A recent paper examining security and privacy problems reported by developers using LLM-native development environments describes this gap well.

The researchers argue that these tools are “shifting the primary developer role from code synthesis to verification.”

While AI has substantially reduced the effort required to arrive at an implementation, it hasn’t removed the need for someone to establish whether that implementation deserves to be trusted.

The study was based on 446 manually verified posts describing security or privacy problems developers experienced with LLM-native IDEs, drawn from a much larger pool of Reddit threads.

The problems extended well beyond an LLM occasionally generating vulnerable code. Developers in the study reported unauthorized file operations, unexpected execution, destructive actions, opaque data flows, sensitive information exposure, and failures involving the broader system around the model.

The researchers found that many of these issues came from system-level design choices such as extensive project context, insufficient control over autonomous actions, and the way generated output is ultimately executed.

Prompts and skills alone won’t fix this

AI-assisted development increasingly happens through various instructions: your prompt, skill files, MCP connections, etc.

You prompt the tool and tell it that users should only see their own records, that an admin action should stay private, or that an API key should never reach the browser.

Those instructions describe intent, but intent is not the same thing as enforcement.

The real security boundaries exist at the code level in things like your database policy, server-side authorization checks, route handlers, secrets-management, or whatever mechanism actually controls access inside of your app.

Even if you’ve written a hyper-detailed set of skills or prompts, you’re still grappling with probability. There’s no guarantee an LLM will "listen" to every instruction you give it.

This is expected. LLMs are non-deterministic. This means that, where the code it’s writing for you is deterministic (does exactly the same thing every time), LLMs are the opposite. They run on probabilistic math to decide their response to you. That means no two answers will be 100% identical unless the output requested necessitates it.

The paper contains a particularly useful example of this problem in the wild.

A developer had configured GitHub Copilot to ignore a .env file containing secrets. The researchers attempted to reproduce the problem themselves and found that, after repeated debugging of a database connection, Copilot read the file despite the configured access restrictions. The study also found user-specified constraint violations among the reported security issues.

What can we learn from this? A rule written for an AI/LLM is not the same as a strict control enforced by the app itself.

Think of it this way: "users should only see their own data" is a requirement. If your app is using something like Firebase or Supabase, row-level security or a correctly implemented authorization check is the actual control mechanism.

Contrast this with “never expose this secret” in a skill file or prompt. That’s an instruction. Keeping that secret inaccessible to the client (browser/device) is the actual control mechanism, not the instruction itself.

This distinction is familiar if you’re used to working with code directly. But AI-driven development makes it much easier to mistake a clearly expressed intention for an implemented guarantee.

Not to burst your bubble, but: assuming expressed intent is equivalent to a guarantee is how vibe coders can cook themselves fast.

It may be a surprise to you when the LLM ignores an instruction, but that’s exactly what it’s designed to do (factor in context and try to "guess" probabilistically what a valid response would be).

The researchers describe the accumulation of unverified AI output (any code that hasn’t been inspected for accuracy and safety) as "silent security debt."

This kind of debt doesn’t tend to announce itself during normal use. A missing authorization check on the backend doesn’t necessarily break the UI as you click through the app. An overly-permissive database policy may never be noticed by the person who built the app. An API key can be exposed to attackers while every visible feature continues to work.

The software can feel finished because what’s visible is working, while the parts that determine how trustworthy that software is remain largely neglected.

Code was the original cost sink, now it’s verification

This changes the economics of software development. For a long time, writing the software itself was one of the most expensive parts of the process. If a feature took several days to build, a significant amount of time and attention was naturally spent thinking through how it was supposed to work as it was being implemented.

Writing code and understanding it were still independent tasks technically speaking, but they were tightly coupled in a way that forced the developer to reason through the system and how it worked as they built it.

AI weakens and in many cases, eliminates, that relationship. This is intoxicating because you can implement features faster without having to reason through their who, what, where, when, and why.

That makes these tools insanely powerful, but it also means that just because you can use an LLM to spit out code and review it for you, that shouldn’t be considered equivalent to taking the time to review the code yourself, or, have an expert verify it on your behalf.

The behavior of developers in the study reinforces this point.

Manual verification of AI-generated code was one of the most commonly discussed mitigation strategies, alongside version control, configuration restrictions, protection of sensitive files, sandboxing, and other familiar security controls.

The researchers suggest a good default posture is “treating AI output as untrusted input.”

This is wise. Before LLMs (and it’s still 100% true today), developers used the axiom "never trust the client" (don’t rely on the browser or device side of the app to be secure or trustworthy—always assume it isn’t).

Relying on defaults like those is far more useful than having another argument about whether AI-generated code is inherently good or bad.

This is important for people who are new to building software.

AI can help someone with little-to-no experience create surprisingly sophisticated software without understanding every detail of the generated code. That’s great as far as democratizing software development goes, but it also creates an obvious blind spot.

You cannot reliably look for a bug or security hole you don’t know exists, and a working UI can’t tell you whether the implementation under the hood has or will respect every security boundary you’ve defined.

Don’t skip verification

The takeaway here is that while AI might make software development cycles faster, it doesn’t remove the need for verification. If you’re inexperienced (or under-experienced), you can mistake a working app for a production-ready app. Writing code may have gotten cheaper, but knowing whether the system actually does what you intend has gotten more expensive.

As you build your app, don’t just assume because it "works on my machine" that magically translates to "it works."

And if you’re going to ship something to production that deals with sensitive data or taking money from others, it’s worth investing in an expert review of your work. Skipping that step is the difference between having a hit app that can grow steadily, versus an app plagued with harmful issues that derail progress entirely.