Special Token Design
Define and apply typed special tokens (system, user, assistant, tool_call, tool_result) consistently — the model's instruction-following depends on the exact delimiters it was trained on.
Intent & Description
🎯 Intent
LLMs learn conversation structure from the special tokens present during fine-tuning. Misusing or omitting them at inference breaks the model’s ability to follow its own format — degrading output quality silently.
📋 Context
A chat model fine-tuned with specific role delimiters (e.g. <|im_start|>system, <|im_start|>user) expects those exact tokens at inference. Calling the model with raw text, wrong delimiters, or custom invented tokens means it can’t locate the system prompt boundary, user query, or assistant turn — and instruction following degrades.
💡 Solution
Study the model’s official chat template and reproduce it exactly using tokenizer.apply_chat_template() (HuggingFace) or the documented format. Define explicit typed roles for every message boundary. For tool-calling models, use the documented tool_call and tool_result token types — not ad-hoc JSON embedded in user messages. Never invent special tokens at inference time that the model wasn’t trained to recognize.
Real-world Use Case
📌 TL;DR
Use the model’s exact chat template — every special token in its documented position. The model learned to follow instructions based on those delimiters; deviating silently degrades quality.
Advantages
- Correct structure the model was trained to expect — maximizes instruction-following quality
- Role separation makes multi-turn context unambiguous to the model
- Documented chat templates are reproducible and model-version-stable
Disadvantages
- Chat templates are model-specific and change between versions — must be tracked per deployment
- Wrong chat template degrades output with no error signal — it just looks worse
- Custom fine-tuning with different special tokens requires updating all downstream inference code