AI Tokens Overview

Large Language Models (LLMs) do not process raw text, words, or letters directly. They process numerical arrays. A token is the fundamental atomic building block that a tokenizer converts text into before feeding it to the neural network.

• Not whole words: A single token can be a whole word, a sub-word piece, a prefix, a suffix, single punctuation, whitespace, or a single character.

• Rule of Thumb: token is approximately 0.75 words (or) roughly 4characters. Each line is approximately 9-10 tokens.

• Vocabulary Size: Modern tokenizers (like OpenAI's o200k_base or Anthropic's Claude tokenizer) maintain a fixed lookup vocabulary of around 100,000 to 200,000 distinct tokens.

Both the context and actual question/instruction are combined and counted together as Input Tokens (also referred to as Prompt Tokens or Ingress Tokens). From the model's perspective, there is no physical operational boundary between "context" and "prompt." Everything sent to the API before the model starts generating its response is treated as a single continuous sequence of tokens. Both the old user prompts (the questions) and the old model responses (the answers) are packaged together as ingress (input) context for your latest prompt. 

To clarify the naming and direction of tokens:

1. When a past turn happens:

   • Your question is counted as ingress (input).

   • The model's generated answer is counted as egress (output).

2. When you send a new prompt later in that same chat:

   • Everything that came before - both your past questions/prompts AND the model's past answers/responses - is bundled up, re-sent to the API, and counted as ingress (input) context for the current turn.

If you want to prevent your old conversation context from being sent - saving your token allowance, lowering costs, and protecting your data privacy - you have three direct options:

1. The Simplest Fix: Start a New Chat Thread

The easiest way to drop old context is to simply click "New Chat" or open a fresh window every time you switch topics.

• Why it works: A brand-new thread sends an empty chat history array to the API. Only your system instructions and your single current prompt are counted as ingress tokens.

2. Turn Off Chat History / Training Saving

If you are using consumer web apps (like ChatGPT or Claude Pro), you can go into settings and disable chat history or workspace saving.

• Why it works: While the app still carries context within an active session so you can talk to it, turning off history prevents your chats from being stored long-term on their servers or used for model training. (Note: This stops them from saving your data, but within a single active scrolling chat, the app will still re-send the transcript to maintain continuity).

3. Use API Parameters Like temperature or Stateless API Calls (For Developers)

If you are interacting with models programmatically via code or API:

• Don't append history arrays: Write your API calls so that you only pass the system prompt and the single current user string, omitting the messages array history entirely.

• The Trade-off: The model will treat every single prompt as if it's talking to you for the very first time. It will have total amnesia about anything you typed two minutes prior.

  
  

A typical API payload contains three main components - all of which are summed up as Input Tokens:

JSON

[
  {"role": "system", "content": "You are a senior systems architect..."},  <-- Context / System Persona
  {"role": "user",   "content": "Here is the codebase: [5,000 lines...]"},  <-- Context / Reference Data
  {"role": "user",   "content": "Refactor this function to be async."}    <-- Actual Prompt / Task
]

What Counts Towards Your Input Token Customer Bill:

1. System Instructions: The persona, behavioral constraints, safety guidelines, and formatting schemas.

2. Conversation History: In a multi-turn chat, every previous message from both you and the assistant re-sent in the payload counts as input tokens for that turn.

3. Reference Data / Context: Documents, pasted code, uploaded PDFs, CSVs, web search results, database schema dumps, or retrieved RAG snippets.

4. The Task / Instruction: Your specific question or command.

5. Structural Overhead: Special control tokens used by the library (e.g., <|im_start|>user, \n, role tags) that tell the model where roles start and end.

Why API Billing Separates Token Types:

• Input Tokens (Prompt Context): Computed in parallel in a single forward pass through GPU/TPU matrices.Cheaper to process.

• Output Tokens (Generation): Generated auto-regressively one single token at a time, where each new token requires a full pass through the entire model. This takes significantly more GPU compute time, making output tokens 3x to 4x more expensive.

• Reasoning / Thinking Tokens: Newer reasoning models generate internal scratchpad tokens during processing.Even if these thinking tokens are hidden from the final chat interface, they count against the output token budget and are billed at full output token rates.

Input vs. Output Pricing Differential:

Input tokens are vastly cheaper than output tokens - typically 3x to 5x cheaper (e.g., $2.50 per million input tokens vs.$10.00 per million output tokens). This is why loading up a prompt with detailed context is far more cost-effective than letting the model execute long, iterative generation loops.

What Drains Your Pro Allowance the Fastest?

When you chat with an LLM, the model processes the entire conversation history with every new prompt to maintain context. This means input tokens accumulate exponentially.

Re-Sending Conversation History:

If you are 20 messages deep into a single chat thread, sending a simple question like "Can you rephrase that last point?"forces Claude to process:

• Ingress (Input): All previous 20 turns + uploaded files + your latest prompt.

• Egress (Output): The new short response.

Because ingress tokens grow linearly with chat length, long ongoing chats will burn through your 5-hour Pro limit in just a few messages.

Amazon Kiro:

Amazon Kiro scales the final credit deduction using a Per-Model Multiplier.The multiplier scales from 0.05x to 4.4x+ depending on the reasoning horsepower of the model handling your prompt:

Model Selected in Kiro	Credit Multiplier	How it affects your 1,000 Credits
Fast / Open-Weight Models(e.g., Qwen, MiniMax, Haiku)	0.05x – 0.6x	A heavy egress code generation consumes only 0.1 to 0.3 credits. Yields 3,000+ interactions/mo.
Mid-Tier Models (e.g., Claude Sonnet, GPT-5.6 Terra)	1.0x – 2.2x	A normal prompt-and-code output burns roughly 1 to 2 credits. Yields 500 to 1,000 interactions/mo.
Frontier Reasoning Models(e.g., Claude Opus, GPT-5.6 Sol)	4.0x – 4.4x	The exact same code generation burns 4 to 5 credits per turn. Yields 200 to 250 interactions/mo.

Core Differences: Open-Weight / Fast vs. Standard Baseline:

Feature / Dimension	Open-Weight / Fast Tier	Standard Baseline Tier
Model Nature	Publicly downloadable weights (e.g., DeepSeek, Llama, Qwen, Mistral) or ultra-fast distilled/flash models.	Proprietary closed-source models hosted directly by major labs (e.g., Claude Sonnet, GPT-4o / GPT-5.6 Terra).
Primary Purpose	High-throughput, bulk task execution, low-latency completions.	General-purpose, multi-turn reasoning, and complex coding/spec generation.
Cost & Token Pricing	Extremely Low (~$0.10 to $0.80 per 1M tokens). In platforms like Kiro, these carry discount multipliers (e.g., 0.05x to 0.25x).	Moderate / Standard (~$2.50 to $15.00 per 1M tokens). In platforms like Kiro, these represent the 1.0x to 1.5x baseline multiplier.
Inference Speed (TPS)	Very High (typically 100 to 200+ tokens/second).	Moderate (typically 30 to 80 tokens/second).
Best Used For	Refactoring small code files, writing unit tests, running terminal commands, formatting JSON, basic triage.	Architectural design, complex feature implementations, multi-file code editing, and debugging.