AI Doomerism: The Risk That Actually Reaches Your Business

Most AI risk writing is about the end of the world. Very little of it is about the thing that will actually go wrong in your company this quarter, which is an agent with credentials doing something plausible and incorrect at three in the morning.

Both conversations use the word "safety". Only one of them has a work item attached.

The gap between the debate and the deployment

The public argument is about capability: whether a sufficiently advanced system develops goals of its own. It is a real research question and I am not qualified to settle it. It is also almost entirely disconnected from the decision a business owner is making, which is whether to give a language model permission to send email, edit files, move money, or write to a production database.

That second decision does not depend on how the capability debate resolves. The failure modes are available today, they are mundane, and they have names:

• An agent reads a web page that contains text written to be read by an agent, and follows it.
• A retry loop runs 400 times against a metered API because nothing capped it.
• A summarisation step quietly drops the one row that mattered, and the output looks fine.
• A tool call succeeds, the verification step is missing, and the system reports success.
• A model writes a partially complete file, and the next step reads it as finished.

None of these require the model to want anything. They require it to be confidently wrong while holding a valid credential, which is the normal operating condition.

Why prompts are the wrong place to put a boundary

The standard mitigation is an instruction: never modify files outside the project directory, always ask before sending. This works until the input contains something more persuasive than your instruction, and the whole point of the system is that it processes inputs you did not write.

A rule expressed in the prompt is a request. A rule expressed in code is a boundary.

I built Sentinel to test how far that distinction goes in practice. It sits between a local coding model and the filesystem, and the checks — path validation, approval requirements, secret redaction — run in Python, outside the model, before any action is taken. The model cannot negotiate with them, because it is not being asked.

That design has a cost worth stating plainly: it is not OS-level isolation, and it is not complete protection against prompt injection. A determined attacker who can run code has more options than a path check can cover. What it does buy is that the common failures become structurally impossible rather than statistically unlikely, and every action leaves a hash-linked audit entry you can read afterwards.

The three questions worth asking

Before an agent touches anything you care about:

What can it do without asking? Write this list down. If it cannot be written down, the answer is "everything", and you have found your first problem. In practice most useful agents need a much smaller permission set than they are given, because permissions are granted once during setup and never revisited.

How would you know it failed? Silent success is the expensive failure mode. A task that reports "done" without a verification step has not told you anything. In ai-os every task carries a command that must pass before the task counts as complete, and a missed heartbeat escalates to a person rather than retrying quietly. That is not sophisticated; it is just the difference between monitoring and hoping.

What does the audit trail actually contain? After something goes wrong, you need to reconstruct what happened. If your logs contain prompts and responses but not the actions taken and their results, you will be able to see what the model said and not what it did.

What this means for buying decisions

If a supplier's answer to safety is a description of their prompt engineering, you have learned something. Ask instead which controls are enforced outside the model, what happens when a verification step fails, and what they have deliberately chosen not to let the system do.

A vendor who cannot name a limitation has not thought about one. The AI agent production readiness checklist covers the rest of the ground, and if you already have an agent running and want that evidence assembled, an AI systems review is the scoped version of this article.

The extinction question can stay open. The permission question is due before you deploy.