Your company rolled out an internal AI assistant for the support team. The pilot went great. Three months later, the monthly LLM API invoice lands at €43,000, and nobody in finance saw it coming. The team used it more than expected, conversations grew longer, and the model you picked was priced for precision, not volume. This is not a hypothetical. It is exactly the scenario that pushed SAP, a company running AI across hundreds of internal use cases, to introduce a strict three-tier token controlling system.
Token costs are the silent budget killer of enterprise AI adoption. They are easy to ignore during a proof of concept and punishingly hard to reverse once hundreds of employees integrate an LLM into their daily workflow. SAP's response is a governance blueprint that any organisation running LLMs at scale can adapt, and it starts with one uncomfortable truth: if you do not put a hard limit on token consumption, the consumption will find the limit of your budget instead.
The Token Cost Explosion Nobody Budgeted For
When a company first experiments with large language models, the math looks harmless. A single question to GPT-4o might consume 300 input tokens and 150 output tokens. At $2.50 per million input tokens and $10 per million output tokens, that call costs roughly $0.00225. Multiply by 50 daily queries per employee and 200 employees, and you are still at just $22.50 per day. The finance team signs off without a second thought.
The problem is that real-world usage never stays that flat. Conversations grow longer because users discover that follow-up questions get better answers. Teams start uploading documents for summarisation, pushing input tokens into the thousands per request. Somebody enables a "thinking" mode or switches to a more expensive model because it sounds smarter. A single RAG pipeline with a 4,000-token context window and five retrieved chunks can burn 20,000 tokens per query without anyone noticing. Within a quarter, that $22.50 per day becomes $1,500 per day, and the bill arrives three weeks after the spending happened.
SAP saw this pattern across its organisation. The company had encouraged broad AI adoption, but without central visibility and spending controls, token consumption became a runaway expense line. Their answer was not to kill AI projects. It was to make every token spend deliberate, visible, and pre-approved according to impact.
Inside SAP's Three-Tier Token Controlling System
SAP's new governance framework divides every AI initiative into three budget tiers. The exact thresholds are internal, but the structure is what matters, and it maps cleanly onto how any mid-size or large enterprise can think about AI cost risk.
Tier 1: Small-scale, auto-approved. Projects with a predictable, low token footprint fall into this bucket. Think of a single-department chatbot that handles 200 internal queries a day, each under 1,000 tokens. SAP sets a monthly token budget ceiling, likely in the range of €5,000 to €10,000 in equivalent API cost. As long as the project stays under that ceiling, no additional approval is needed. The team can iterate quickly, but they are still monitored. If consumption spikes 20% month-over-month, an automated alert flags it for review.
Tier 2: Mid-scale, departmental approval required. When a project needs more tokens, say a cross-team RAG system that processes 5,000 support tickets per month with large context windows, it crosses into Tier 2. Here the budget might sit between €10,000 and €50,000 per month. The project owner must submit a short business case to a department head or AI steering committee: expected token volume, model choice, cost per query, and the business metric it moves. Approval is not a one-time event. Quarterly reviews check whether actual spend matches the forecast, and the project can be downgraded to Tier 1 or escalated to Tier 3 if reality diverges.
Tier 3: Enterprise-wide, board-level sign-off. Anything that touches multiple business units, customer-facing products, or a token budget above roughly €50,000 per month requires a full investment case. SAP's board wants to see a quantified ROI, reduced support tickets, faster document processing, revenue uplift, tracked against token spend. The project gets a dedicated cost centre, and the CFO's office receives a monthly variance report.
The system is not about bureaucracy for its own sake. It forces the same discipline on AI spending that companies already apply to cloud infrastructure, software licensing, and headcount. The difference is that cloud costs are visible in near real-time through AWS Cost Explorer or Azure Cost Management. LLM token costs, left to default billing consoles, are often a single lump sum on a credit card statement 30 days after the fact. SAP's tiers close that visibility gap.
How Token Costs Actually Work, A Quick Primer
To build your own version of this governance, you need to understand what you are actually paying for. A "token" is a chunk of text that a language model processes, roughly 0.75 words in English, but it varies by model and language. Every prompt you send and every word the model generates consumes tokens. Pricing is per 1,000 or 1,000,000 tokens, and different models have wildly different rates.
Here are real prices as of mid-2026 for the models most enterprises use:
| Model | Input (per 1M tokens) | Output (per 1M tokens) | |-------|----------------------|------------------------| | GPT-4o | $2.50 | $10.00 | | GPT-4o-mini | $0.15 | $0.60 | | Claude 3.5 Sonnet | $3.00 | $15.00 | | Claude 3 Haiku | $0.25 | $1.25 | | Gemini 1.5 Pro | $1.25 | $5.00 | | Gemini 1.5 Flash | $0.075 | $0.30 |
The 40x price gap between GPT-4o-mini and Claude 3.5 Sonnet for output tokens is the difference between a €500 monthly bill and a €20,000 one for the same query volume. Most enterprise use cases do not need the most expensive model for every request. A tiered governance system lets you route simple classification or extraction tasks to a cheap model and reserve the expensive one for complex reasoning where it actually changes the outcome.
Worked Example: Estimating Your Monthly LLM Bill
Let us make this concrete. Suppose your customer support team builds an AI assistant that handles 300 conversations per day. Each conversation averages 4 back-and-forth turns. Each user message is about 150 tokens; each model response is about 250 tokens. You also include a system prompt of 800 tokens and 1,200 tokens of retrieved knowledge base chunks per conversation.
Daily token consumption per conversation:
- Input: system prompt (800) + 4 user messages (4 × 150 = 600) + 4 retrieved chunks (1,200) + 3 previous assistant responses included as context (3 × 250 = 750) = 3,350 input tokens
- Output: 4 assistant responses × 250 = 1,000 output tokens
Daily total for 300 conversations: 1,005,000 input tokens, 300,000 output tokens. Monthly (22 business days): ~22.1 million input tokens, ~6.6 million output tokens.
Using GPT-4o: 22.1M × $2.50/1M = $55.25 input + 6.6M × $10/1M = $66.00 output → $121.25 per month. Cheap. Using Claude 3.5 Sonnet: 22.1M × $3.00 = $66.30 + 6.6M × $15.00 = $99.00 → $165.30 per month. Still fine.
Now add one "small" change: the team decides to include the full conversation history instead of just the last three turns, pushing input tokens to 8,000 per conversation. Monthly input jumps to ~52.8M tokens. GPT-4o cost becomes $52.8M × $2.50 = $132 + $66 output = $198/month. Still okay? Now imagine 10 departments run similar assistants, and two of them switch to a model with a 128k context window and start uploading entire PDFs. The monthly bill crosses €50,000 without a single person making a deliberate "expensive" decision. That is the dynamic SAP's controlling system is designed to catch.
A Python Script to Calculate and Monitor Token Costs
You can build a lightweight cost estimator that runs against your LLM provider's usage API. Here is a Python script that pulls daily token consumption from the OpenAI API and projects monthly cost, with an alert threshold.
import openai
from datetime import datetime, timedelta, timezone
# Configuration
OPENAI_API_KEY = "your-key"
DAILY_BUDGET_USD = 50.0 # alert if daily cost exceeds this
MODEL_PRICING = { # per 1M tokens
"gpt-4o": {"input": 2.50, "output": 10.00},
"gpt-4o-mini": {"input": 0.15, "output": 0.60},
}
client = openai.OpenAI(api_key=OPENAI_API_KEY)
def get_daily_usage():
yesterday = (datetime.now(timezone.utc) - timedelta(days=1)).strftime("%Y-%m-%d")
usage = client.usage.retrieve(date=yesterday)
return usage.data
def calculate_cost(usage_data):
total_cost = 0.0
for item in usage_data:
model = item.model
if model in MODEL_PRICING:
input_tokens = item.input_tokens
output_tokens = item.output_tokens
cost = (input_tokens / 1_000_000) * MODEL_PRICING[model]["input"] + \
(output_tokens / 1_000_000) * MODEL_PRICING[model]["output"]
total_cost += cost
return total_cost
daily_usage = get_daily_usage()
daily_cost = calculate_cost(daily_usage)
monthly_projection = daily_cost * 22 # business days
print(f"Yesterday's token cost: ${daily_cost:.2f}")
print(f"Projected monthly cost (22 days): ${monthly_projection:.2f}")
if daily_cost > DAILY_BUDGET_USD:
# In production, send a Slack/Teams alert here
print(f"ALERT: Daily spend ${daily_cost:.2f} exceeds ${DAILY_BUDGET_USD} threshold!")
This script is deliberately simple. In a real governance setup, you would run it as a cron job, push metrics to a dashboard (Grafana, Datadog, or a custom internal tool), and trigger approval workflows when a project exceeds its assigned tier budget for two consecutive days. The key is that the alert fires within 24 hours, not 30 days.
Building Your Own Token Cost Governance (Without SAP's Budget)
You do not need SAP's internal ERP integration to implement the three-tier model. The core components are a usage data source, a budget registry, an approval workflow, and an alerting pipeline. Here is how to assemble them with tools your team likely already has.
Step 1: Centralise usage data. OpenAI, Anthropic, and Google all provide usage APIs. If you route LLM calls through a proxy like LiteLLM or a custom API gateway, you can aggregate token counts across providers in one place. Store daily usage per project or cost centre in a simple database table.
Step 2: Assign every AI project a budget tier. Define thresholds that match your organisation's risk appetite. A realistic starting point for a 500-person company:
- Tier 1: up to €2,000/month, auto-approved, monitored weekly.
- Tier 2: €2,000, €15,000/month, department head approval required, reviewed monthly.
- Tier 3: above €15,000/month, CTO/CFO approval, quarterly ROI review.
Adjust thresholds based on your average deal size and AI maturity. The absolute numbers matter less than the principle that nobody can silently burn five figures a month.
Step 3: Automate tier enforcement. When a Tier 1 project's 7-day rolling average exceeds 80% of its budget, send a warning to the project owner. If it exceeds the budget for two consecutive weeks, automatically escalate it to Tier 2 and lock further usage until the department head approves the increase. This sounds harsh, but it prevents the exact scenario SAP faced: a well-intentioned team accidentally running up a €50,000 bill because "the model was just so useful."
Step 4: Make costs visible to the people who generate them. A dashboard that shows each team's daily token spend, ranked, does more for cost discipline than any policy document. When a product manager sees their chatbot costing €380/day while the neighbouring team's equivalent costs €42/day because they use a cheaper model and shorter prompts, optimisation becomes self-motivated.
Where This Model Breaks, And How to Fix It
SAP's three-tier system is not a silver bullet. It solves the visibility and accountability problem, but it introduces tensions that every enterprise must manage.
Innovation throttling. If every new AI idea requires a business case before it can spend more than €2,000, teams stop experimenting. The fix is a dedicated "sandbox" budget pool outside the tier system, say €10,000/month company-wide, that any team can draw from for prototypes, capped at €500 per experiment and time-boxed to four weeks. This keeps the governance tight on production systems without killing the exploration that finds the next big use case.
Token counting is not exact across models. GPT-4o and Claude 3.5 Sonnet use different tokenizers. The same English sentence can be 10 tokens for one and 12 for the other. Your budget thresholds need a 10-15% tolerance band, and your monitoring should use each provider's own token count, not a cross-model approximation.
Hidden costs beyond tokens. Fine-tuning jobs, embedding storage, and vector database queries all add cost that a pure token budget misses. A complete governance model wraps these into the same tier structure. If a project fine-tunes a model and then serves it, the training cost amortised over the expected query volume should be part of the Tier 2 or 3 business case.
Shadow AI. Teams that find the approval process too slow will expense their own ChatGPT Enterprise seats or put a personal credit card on an API key. SAP's system works because it is coupled with procurement controls, the company simply blocks expensing of unapproved AI services. Smaller companies can achieve the same by making the approved gateway the only path to production models and educating teams that the gateway is fast for Tier 1 requests.
5 Best Practices for Enterprise Token Cost Control
-
Set budget tiers before you have a cost problem. Retroactively telling a team their €30,000/month assistant needs board approval creates resentment. Define tiers during the pilot phase, when costs are still small and the conversation is collaborative.
-
Route requests to the cheapest model that does the job. Classification, entity extraction, and simple Q&A rarely need GPT-4o or Claude Sonnet. Use GPT-4o-mini or Claude Haiku as the default, and let the application escalate to a stronger model only when confidence scores drop. This single practice often cuts token costs by 60-80% without any user-perceptible quality loss.
-
Monitor daily, not monthly. A monthly invoice is a post-mortem. A daily Slack message that says "Project X spent €1,200 yesterday, 90% of its Tier 2 weekly run rate" lets the team adjust prompts or switch models before the month ends. For a deeper dive on prompt-level optimisation, our guide on slashing production RAG costs by 80% through prompt caching and context optimisation walks through concrete techniques that complement governance with engineering.
-
Tie token budgets to business outcomes. A Tier 3 approval should require the project owner to name the KPI they are moving, support ticket deflection rate, document processing time, conversion uplift, and commit to a quarterly review where token spend is compared against that KPI's movement. Projects that cannot show impact get downgraded or shut down.
-
Build a lightweight approval workflow, not a committee. If a Tier 2 escalation takes two weeks and three meetings, teams will route around it. A well-designed system lets a department head approve a budget increase from a Slack message with a one-sentence justification, as long as the spend stays within the department's overall AI budget envelope.
From Governance to Execution
SAP's move is a signal that enterprise AI has crossed from "experiment freely" to "operate responsibly." The companies that build token governance now will be the ones that can afford to scale AI across every function. The ones that do not will eventually hit a brutal boardroom conversation where AI goes from strategic priority to frozen budget line.
Implementing a three-tier system with real-time monitoring and automated enforcement is engineering work. It requires integrating provider APIs, building dashboards, wiring up approval logic, and maintaining it as models and prices change. Teams that would rather focus on their core product often bring in a partner to build this infrastructure as a fixed-scope project. ProjectMakers has built custom AI cost monitoring and governance tooling for mid-size and enterprise clients, including real-time dashboards that map token spend directly to business units and trigger approval workflows, all part of our AI solutions practice.
Start with a single number: pull your last three months of LLM API invoices and plot the trend. If the line is going up faster than your AI adoption rate, you already need the tier system. If you have not looked at the invoices yet, today is the right day.
Source: Tokens: SAP will Ausgaben für KI deutlich einschränken
