30 lines
1.2 KiB
Markdown
30 lines
1.2 KiB
Markdown
|
# Task 5: Calibrate token estimates
|
||
|
|
|
||
|
|
**Plan**: reusable-impact-tooling
|
||
|
|
**Status**: DONE (hook now extracts actual token counts from transcript usage fields; falls back to heuristic; weights cache reads at 10% for energy estimates)
|
||
|
|
**Deliverable**: Updated estimation logic in `pre-compact-snapshot.sh`
|
||
|
|
|
||
|
|
## What to do
|
||
|
|
|
||
|
|
1. The current heuristic uses 4 bytes per token. Claude's tokenizer
|
||
|
|
(based on BPE) averages ~3.5-4.5 bytes per token for English prose
|
||
|
|
but varies for code, JSON, and non-English text. The transcript is
|
||
|
|
mostly JSON with embedded code and English text.
|
||
|
|
|
||
|
|
2. Estimate a better ratio by:
|
||
|
|
- Sampling a known transcript and comparing byte count to the token
|
||
|
|
count reported in API responses (if available in the transcript).
|
||
|
|
- If API token counts are present in the transcript JSON, use them
|
||
|
|
directly instead of estimating.
|
||
|
|
|
||
|
|
3. The output token ratio (currently fixed at 5% of transcript) is also
|
||
|
|
rough. Check if the transcript contains `usage` fields with actual
|
||
|
|
output token counts.
|
||
|
|
|
||
|
|
4. Update the script with improved heuristics or direct extraction.
|
||
|
|
|
||
|
|
## Done when
|
||
|
|
|
||
|
|
- Token estimates are within ~20% of actual (if verifiable) or use
|
||
|
|
actual counts from the transcript when available.
|