The efficiency trap: tokens, TOON, and the real availability question

Lori Mac VittieDistinguished Engineer and Chief Evangelist

Today’s push toward ultra-efficient inference, token compression, and clever encoding schemes tempts us to chase performance and cost as if availability will quietly fall in line. It will not. Availability does not care about how clever your encoding looks in a benchmark slide.

Recently, the token-efficiency trend lit up social feeds with excitement around Token-Oriented Object Notation (TOON), a format designed to reduce tokens for inference because pricing is based on tokens. Then the parodies started, which got treated as serious innovations and spread as “the death of JSON,” proving once again that sarcasm does not travel well across text, and hype always outruns nuance.

The real story here is not format wars. It is the same operational challenge we have wrestled with for decades: balancing performance, availability, and reliability with cost. Hard costs are measured in compute and network. Soft costs show up in user trust, brand damage, operator workload, and churn. Anyone who has ever capacity-planned a holiday weekend knows this balancing act is not new.

With AI inference, however, availability now includes correctness. A system that returns answers quickly but inaccurately is not “available;” it is malfunctioning. Ask any operator who has rolled back a change at 3 a.m. while clutching lukewarm coffee and reconsidering career choices.

One size does not fit all

This is why TOON deserves a realistic, not romantic, perspective, much like any compression technique. Some may recall that while HTTP compression worked great on text-based content, it did nothing for images like JPG that were compressed by default. Today the distinction remains the same: structured data is not instructions.

TOON is designed to optimize structured data and claims strong token savings with slightly improved retrieval accuracy. For some, that might be useful. But structure is not the whole picture. Inference and agent-driven systems are not limited to database records and API responses. They deal with instructions, intent, and reasoning paths. Saving tokens on data is not the same as improving decisions.

Consider a simple JSON instruction to an agent:

{
"action": "create_ticket",
"priority": "high",
"summary": "User cannot access their account",
"user_id": 84713
}

In TOON, that compresses to:

action: create_ticket
priority: high
summary: User cannot access their account
user_id: 84713

Nice and compact. Unfortunately, this is where the benefit stops. The agent still must interpret what action means in context. Does “create ticket” require confirmation, location, assignment, categorization, routing, or correlation? TOON saves space but does not improve reasoning clarity.

A 2025 arXiv paper (2510.14453) offers a different tactic: natural-language tool output. Their results showed more than 18% accuracy improvement with more than 30% token reduction. Why? Because models are trained primarily on natural language, not custom miniature data languages.

Using their approach, we send:

Create a high priority support ticket for user 84713. The issue is: User cannot access their account.

Fewer tokens than JSON, more clarity than TOON, and aligned with how models already think. This is not free-form creative writing. It is template-bounded instruction phrased as natural language. This has the added benefit of fewer tokens and it is easier to detect manipulation, i.e. it has unintended security benefits.

The lesson is simple: not all inputs are equal, and one format cannot serve all use cases. Compression formats like TOON may shine when passing large, structured data sets. Natural-language templates may win when giving instructions to reasoning models. Pretending one solution fits everything is how outages are born.

Token count is important but not a top-line success metric

Token count is not a top-line success metric for AI. It is one variable in a multi-variable trade that includes performance, reliability, and availability. Increasingly, we must consider cost, too. You can optimize one without improving the overall system and occasionally make it worse. History is full of engineers who chased the most visible metric instead of the most important outcome.

Delivering inference must balance the operational and business legs of the stool:

• Performance: how quickly the system responds
• Availability: whether it responds correctly
• Reliability: whether it responds consistently
• Cost: whether the business can sustain it at scale

Shrink tokens, but not clarity. Improve speed, but not at the expense of reasoning. Optimize cost, but not by externalizing it into retries, escalations, or manual review.

The industry will almost certainly land on multiple formats used situationally. TOON for structured data. Natural-language templates for instruction. JSON for compatibility. And likely others for specialized workloads.

We are still early and anyone claiming final victory is chasing attention, not engineering. The goal is not choosing the smallest or newest format. The goal is choosing the one that delivers dependable results.

If savings reduce trust, you are not improving the system; you are simply moving the failure downstream.

About the Author

Lori Mac VittieDistinguished Engineer and Chief Evangelist

More blogs by Lori Mac Vittie