The thesis: The next important AI infrastructure shift may not be a better chatbot. It may be a new class of low-latency models that can make semantic decisions cheaply enough to sit inside high-frequency software loops.
Software makes an enormous number of decisions that require more context than a rule or simple classifier can handle—but do not require a frontier model to generate paragraphs of text.
Examples include:
- block or allow this request;
- escalate or close this alert;
- which tool should an agent call;
- is this transaction suspicious;
- what should this NPC do next;
- should this workflow continue or stop.
Today, many of these workloads fall into an awkward gap:
rules / classical ML → ? → frontier LLM
The emerging category we care about sits in that middle layer.
Recent systems such as TypeSafe's Jev are one implementation of this idea: unstructured state goes in; a bounded decision, score, class, or probability comes out. The model does not need to generate prose first and then have software parse the answer back into a control signal.
The important investment question is not whether any single model wins.
It is whether semantic decision-making becomes cheap enough to embed continuously inside software.
1. The missing layer in the inference stack
Traditional rules and classifiers remain unbeatable when the problem is stable, narrow, and well specified.
Frontier LLMs dominate when open-ended reasoning, generation, or flexible instruction following matters.
But many production workloads need something in between:
| Layer | Strength | Limitation |
|---|---|---|
| Rules / classical ML | Extremely cheap and fast | Brittle as context becomes more semantic |
| Decision models | Cheap contextual classification, scoring, routing | Narrower than general-purpose reasoning |
| Frontier LLMs | Flexible reasoning and generation | Higher latency and cost |
That middle layer matters most when the same kind of semantic judgment has to happen millions or billions of times.
An early independent test of Jev—a recent example of this model class—showed the direction of the latency advantage, although the sample was small and not a general capability benchmark.
The point is not the exact multiple.
The point is that removing autoregressive text generation changes the cost and latency envelope of a decision.
2. Scale changes the economics
At low volume, inference cost barely matters.
At large scale, it becomes architecture.
Illustrative scenario using Jev's published price as one current reference point for the emerging category. The chart is not a forecast for the entire category.
At 500 input tokens per decision, one billion decisions would cost roughly:
- $21k at $0.042 / million input tokens;
- $100k at $0.20 / million;
- $5M at $10 / million;
before any output-token charges.
The implication is straightforward:
A model that is merely “cheaper” at 100 calls may become an entirely different product primitive at 1 billion calls.
3. Cybersecurity may be the cleanest beneficiary
Modern security software is already a giant real-time decision engine.
Every request, identity, endpoint event, tool call, packet, and alert creates another opportunity to ask:
normal or suspicious? allow or block? escalate or ignore?
This is exactly where expensive LLM reasoning is attractive semantically but often difficult economically.
Datadog
Datadog's AI Guard already evaluates AI interactions synchronously and monitors the latency and token cost of those checks.
That makes observability and security one of the clearest near-term beneficiaries of cheaper semantic decision models.
The potential advantage is not replacing existing deterministic security systems.
It is pushing richer semantic analysis into workflows where running a frontier model on every event would be too expensive or too slow.
The limitation is equally important:
In a public 2,000-email phishing benchmark, the low-cost decision-model example was much cheaper and faster than Claude Haiku 4.5—but also materially less accurate.
This is a useful reminder that type-safe output is not the same thing as correct output.
The technology becomes valuable only where the cost / latency / accuracy trade-off is appropriate for the workflow.
4. Cloudflare: intelligence closer to the edge
Cloudflare is interesting because it can benefit both as a user and as an infrastructure provider.
Its products already make enormous numbers of automated security and routing decisions.
As semantic inference becomes cheaper, more contextual intelligence can potentially move directly into:
- bot detection;
- abuse prevention;
- API security;
- request routing;
- model routing;
- agent security;
- edge application logic.
At the same time, Cloudflare operates Workers AI and AI Gateway.
If application architectures move from:
one AI call per user request
toward:
many tiny AI decisions inside each software transaction
then Cloudflare can potentially monetize the traffic itself.
The relevant metric is therefore not simply inference cost.
It is:
cost per decision × number of decisions
5. Hyperscalers: lower unit cost, higher demand elasticity
For AWS, Azure, and Google Cloud, cheaper decision models create two opposing effects.
First-order: less compute is required for a given inference task.
Second-order: many more inference tasks become economical.
If semantic inference becomes cheap enough, developers may stop thinking in terms of “AI features” and start embedding model calls throughout ordinary software execution.
A SaaS product that makes one model call per session could eventually make hundreds.
A security system that sends only ambiguous alerts to an LLM could evaluate every event.
An agent that asks a frontier model to reason through every branch could instead use cheap decision models for routing, verification, and control—escalating only the hard cases.
For hyperscalers, the key question is therefore demand elasticity.
Efficiency is not automatically bearish for compute demand.
It may expand the addressable workload faster than it reduces compute per inference.
6. Games: from AI dialogue to continuous AI behavior
Gaming makes this architectural distinction unusually intuitive.
Most game characters do not need to generate dialogue every frame.
They need to decide:
- where to move;
- whether to attack;
- whether to hide;
- whom to trust;
- whether something unusual happened;
- what objective matters next.
NVIDIA and KRAFTON's work on PUBG Ally already separates fast reflex behavior from slower language-model reasoning.
A cheaper semantic decision layer creates an obvious middle tier:
deterministic behavior → semantic decision model → expensive reasoning / generation
This could be the architecture that makes large populations of context-aware NPCs practical.
The opportunity is not one brilliant AI character.
It is hundreds of moderately intelligent characters running continuously.
7. Agents may become the largest hidden workload
Agents perform many invisible decisions for every visible output:
- Which tool?
- Which model?
- Is this result sufficient?
- Should I search again?
- Is this action safe?
- Does this require approval?
- Has the task completed?
- Should the system escalate?
Using a frontier model for every branch increases latency and cost.
Using rules for every branch makes the system brittle.
Decision models create a potential control layer between the two.
A future agent stack may look less like one giant model and more like a distributed system:
frontier model → workflow → many cheap semantic decisions → tools / state → occasional escalation
That architecture could reduce expensive reasoning per completed task while increasing the total number of intelligent decisions dramatically.
8. Public-company exposure
| Company / group | Potential exposure | Why it matters |
|---|---|---|
| Cloudflare (NET) | Cost + infrastructure | Edge decisions, AI Gateway, Workers AI |
| Datadog (DDOG) | Cost + product demand | Inline AI evaluation and agent observability |
| Cybersecurity vendors | Product expansion | Semantic triage, policy, tool-call and identity decisions |
| AWS / Azure / Google Cloud | Volume elasticity | Lower cost per decision may unlock much larger inference volume |
| NVIDIA | Workload expansion | More local and real-time AI inference |
| Game publishers / engines | New product surface | Large-scale context-aware NPC behavior |
| Agent platforms | Cost + latency | Routing, judging, verification and control |
The strongest near-term cases are likely to be businesses where:
- semantic inference already sits in the critical path;
- decisions happen at very high frequency;
- latency or cost currently limits deployment.
What we would watch
The category is still early.
Current systems vary significantly in accuracy, latency, training method, and task scope. Vendor benchmarks should not be treated as proof that one architecture dominates.
The more durable question is structural:
Does a persistent model layer emerge between classical ML and frontier LLMs?
If the answer is yes, the implication is much broader than any single model company.
AI moves from something software occasionally calls to something software continuously does.
That would lower the cost of intelligence for existing workloads—but the larger effect may be the creation of entirely new ones.
Sources
- TypeSafe AI — Introducing System One Models & Jev https://typesafe.ai/blog/introducing-system-one-models-and-jev
- TypeSafe workflow evals https://evals.typesafe.ai/
- Every — Mini-Vibe Check: TypeSafe's Jev Judged Everything I've Written in 0.7 Seconds https://every.to/also-true-for-humans/mini-vibe-check-typesafe-s-jev-judged-everything-i-ve-written-in-0-7-seconds
- Public Jev phishing benchmark https://github.com/anisselbd/jev-phishing-bench
- Datadog — Evaluating AI Guard to improve quality and control cost https://www.datadoghq.com/blog/llm-observability-at-datadog-security/
- Cloudflare — The AI engineering stack we built internally https://blog.cloudflare.com/internal-ai-engineering-stack/
- Microsoft — Scaling AI with 8 to 20x energy efficiency https://www.microsoft.com/en-us/microsoft-cloud/blog/2026/06/15/scaling-ai-with-8-to-20x-energy-efficiency/
- NVIDIA — How KRAFTON Built PUBG Ally https://developer.nvidia.com/blog/how-krafton-built-pubg-ally-a-co-playable-character-powered-by-nvidia-ace/