sprawl token-size
📌 1. The Target Problem
When teams accumulate unmanaged .cursorrules, CLAUDE.md copies, alwaysApply: true rules, and sprawling MCP server schemas, agents burn 30,000+ tokens on Turn 1 ("Context Rot"). Without a deterministic static auditor, engineering teams cannot quantify how much context weight is loaded unconditionally (Always-On Tokens) versus deferred until needed (Lazy-Loaded Tokens), nor can they enforce hard context token budgets in CI/CD pipelines.
⚙️ 2. Usage & Options
sprawl token-size [target_dir] [--detailed] [--max-tokens INT]
Options & Flags
target_dir: Optional workspace path to audit (defaults to current working directory).--detailed: Displays a per-file breakdown sorted by token weight, showing full relative artifact paths, symlink deduplication, and loading behavior notes.--max-tokens INT: CI/CD budget gate; displaysCI Budget Gate: PASSorFAILand exits with code1(SprawlError) if Turn-1 Always-On tokens exceedINT.--json: Emits a machine-readable JSON audit payload (includingoptimization_hints) for CI pipelines and telemetry dashboards.
🔍 3. How it Works (Under the Hood)
- Deterministic Stdlib BPE Estimator (
estimate_tokens): Uses a zero-dependency lexical + UTF-8 byte hybrid heuristic (<5msexecution,<5%variance fromcl100k_base) without requiringtiktokenor network access. - Symlink Deduplication across IDE Adapters: Inspects
AGENTS.md,GEMINI.md,CLAUDE.md,.cursorrules,.windsurfrules,.clinerules, and.github/copilot-instructions.md. Symlinked bindings created bysprawl bindadd0duplicate tokens (symlink -> AGENTS.md (0 dup tokens)), whereas unlinked physical file copies are flagged (duplicate file (use sprawl bind symlink)). - 4-Layer Always-On (Turn 1) vs. Lazy-Loaded (On-Demand) Split:
- Governance Rules (
.agents/rules/): Rules withalwaysApply: trueoractivation: alwayscount toward Turn-1 Always-On Tokens (always-on (loaded every turn)); glob-scoped or model-decision rules count toward Lazy-Loaded Tokens (on-demand (loaded when relevant)). - Agent Skills (
.agents/skills/): Only the YAML frontmatternameanddescriptioncatalog entry counts toward Turn-1 Always-On Tokens; the full markdown body is classified under Lazy-Loaded Tokens (index on Turn 1 + body on-demand). - MCP Schemas (
mcp_config.json): Serializes activemcpServersdefinitions into Turn-1 Always-On Tokens.
- Actionable Optimization Hints (
build_optimization_hints): Automatically detects and ranks Turn-1 token waste (duplicate physical IDE manifests,alwaysApply: truerules exceeding 500 tokens, and oversized root manifests exceeding 3,500 tokens) with copy-pasteable remediation steps.
🩺 4. Troubleshooting
- CI pipeline fails with
Context budget exceeded: Turn-1 Always-On token count is X tokens - Resolution: Follow the
Actionable Ways to Cut Turn-1 Tokensrecommendations printed in the audit panel, or runsprawl token-size --detailedto inspect individual artifacts. Convert domain-specific rules toalwaysApply: falsewith path globs, or runsprawl bind --forceto replace physical IDE rule copies with zero-cost symlinks.