sprawl token-size

Diagnostics

Statically audits Always-On (Turn 1) vs. Lazy-Loaded (On-Demand) context token weight across manifests, rules, skills, and MCP server schemas (<100ms).


📌 1. The Target Problem

1. The Target Problem

"How much of my AI agent's context window is consumed before I even type my first prompt, and how do I stop bloated rules and duplicate IDE configs from degrading reasoning quality?"

When teams accumulate unmanaged .cursorrules, CLAUDE.md copies, alwaysApply: true rules, and sprawling MCP server schemas, agents burn 30,000+ tokens on Turn 1 ("Context Rot"). Without a deterministic static auditor, engineering teams cannot quantify how much context weight is loaded unconditionally (Always-On Tokens) versus deferred until needed (Lazy-Loaded Tokens), nor can they enforce hard context token budgets in CI/CD pipelines.


⚙️ 2. Usage & Options

sprawl token-size [target_dir] [--detailed] [--max-tokens INT]

Options & Flags

  • target_dir: Optional workspace path to audit (defaults to current working directory).
  • --detailed: Displays a per-file breakdown sorted by token weight, showing full relative artifact paths, symlink deduplication, and loading behavior notes.
  • --max-tokens INT: CI/CD budget gate; displays CI Budget Gate: PASS or FAIL and exits with code 1 (SprawlError) if Turn-1 Always-On tokens exceed INT.
  • --json: Emits a machine-readable JSON audit payload (including optimization_hints) for CI pipelines and telemetry dashboards.

🔍 3. How it Works (Under the Hood)

  1. Deterministic Stdlib BPE Estimator (estimate_tokens): Uses a zero-dependency lexical + UTF-8 byte hybrid heuristic (<5ms execution, <5% variance from cl100k_base) without requiring tiktoken or network access.
  2. Symlink Deduplication across IDE Adapters: Inspects AGENTS.md, GEMINI.md, CLAUDE.md, .cursorrules, .windsurfrules, .clinerules, and .github/copilot-instructions.md. Symlinked bindings created by sprawl bind add 0 duplicate tokens (symlink -> AGENTS.md (0 dup tokens)), whereas unlinked physical file copies are flagged (duplicate file (use sprawl bind symlink)).
  3. 4-Layer Always-On (Turn 1) vs. Lazy-Loaded (On-Demand) Split:
  • Governance Rules (.agents/rules/): Rules with alwaysApply: true or activation: always count toward Turn-1 Always-On Tokens (always-on (loaded every turn)); glob-scoped or model-decision rules count toward Lazy-Loaded Tokens (on-demand (loaded when relevant)).
  • Agent Skills (.agents/skills/): Only the YAML frontmatter name and description catalog entry counts toward Turn-1 Always-On Tokens; the full markdown body is classified under Lazy-Loaded Tokens (index on Turn 1 + body on-demand).
  • MCP Schemas (mcp_config.json): Serializes active mcpServers definitions into Turn-1 Always-On Tokens.
  1. Actionable Optimization Hints (build_optimization_hints): Automatically detects and ranks Turn-1 token waste (duplicate physical IDE manifests, alwaysApply: true rules exceeding 500 tokens, and oversized root manifests exceeding 3,500 tokens) with copy-pasteable remediation steps.

🩺 4. Troubleshooting

  • CI pipeline fails with Context budget exceeded: Turn-1 Always-On token count is X tokens
  • Resolution: Follow the Actionable Ways to Cut Turn-1 Tokens recommendations printed in the audit panel, or run sprawl token-size --detailed to inspect individual artifacts. Convert domain-specific rules to alwaysApply: false with path globs, or run sprawl bind --force to replace physical IDE rule copies with zero-cost symlinks.

🔗 Related Resources & Tutorials