toon token savings by joestein-ssc · Pull Request #2574 · IBM/mcp-context-forge

joestein-ssc · 2026-01-29T19:05:07Z

📝 Summary

TOON Encoder Plugin - Converts JSON tool results to TOON (Token-Oriented Object Notation) format for reduced token consumption when sending responses to LLM agents/GPUs.

Problem

When using MCP Gateway as a proxy for multiple MCP servers (e.g., 10+ servers), the JSON responses from tool invocations consume significant tokens. For users running local inference or paying per-token for API calls, this represents both a performance bottleneck and cost concern.

Solution

A tool_post_invoke plugin that converts JSON tool results to TOON format before returning to the agent, achieving 30-70% token reduction.

Expected Token Savings

Data Type	JSON Size	TOON Size	Savings
Simple object	150 bytes	80 bytes	~47%
Array of 10 objects	2 KB	600 bytes	~70%
Nested data	5 KB	2 KB	~60%

Verified Example

JSON (231 chars):
{"users": [{"id": 0, "name": "user0", "active": true}, ...]}

TOON (109 chars):
users:
  [5]{active,id,name}:
   true,0,user0
   true,1,user1
   ...

Actual Savings: 52.8% (122 bytes saved)

Files Added

plugins/toon_encoder/
├── __init__.py              # Package exports
├── toon.py                  # Pure Python TOON encoder/decoder (~400 LOC)
├── toon_encoder.py          # Plugin implementation (~250 LOC)
├── README.md                # User documentation
└── review.md                # PR review notes

tests/unit/plugins/toon_encoder/
├── __init__.py
├── test_toon.py             # TOON encoder/decoder tests (~300 LOC)
└── test_toon_encoder.py     # Plugin integration tests (~350 LOC)

specs/toon-encoder/          # Design specification
├── README.md
├── design.md
├── implementation.md
├── testing.md
└── pr-template.md

🏷️ Type of Change

🧪 Verification

Check	Command	Status
Lint suite	`make lint`	✅
Unit tests	`make test`	✅
Coverage ≥ 90%	`make coverage`	✅

Plugin-specific tests:

pytest tests/unit/plugins/toon_encoder/ -v

✅ Checklist

Code formatted (make black isort pre-commit)
Tests added/updated for changes
Documentation updated (if applicable)
No secrets or credentials committed

📓 Notes

Design Decisions

Why a Plugin (vs. Core Gateway Change)?

Non-breaking - Existing deployments unaffected; opt-in behavior
Configurable - Per-tool filtering, size thresholds, format markers
Reversible - Can disable without code changes
Follows patterns - Uses established tool_post_invoke hook like json_repair, output_length_guard

Why Pure Python (vs. Rust FFI)?

No external dependencies - Works in any deployment environment
Maintainable - Team can debug/extend without Rust toolchain
Sufficient performance - Encoding is O(n) string manipulation; benchmarked at <1ms

Safety Features:

Only converts if TOON is actually smaller than JSON
Graceful error handling with skip_on_error: true fallback
Format marker (annotations.format: "toon") for downstream parsing
Configurable size thresholds to prevent edge cases

Usage

Add to plugins/config.yaml:

plugins:
  - name: "ToonEncoder"
    kind: "plugins.toon_encoder.toon_encoder.ToonEncoderPlugin"
    hooks: ["tool_post_invoke"]
    mode: "enforce"
    priority: 900
    config:
      min_size_bytes: 100

Agent Integration Required

When enabling TOON encoding, agents must update their system prompts to understand TOON format. See plugins/toon_encoder/README.md for complete system prompt examples.

TOON Format Reference

JSON	TOON
`{"name": "alice", "age": 30}`	`name: alice` `age: 30`
`[{"id":1},{"id":2}]`	`[2]{id}:` `1` `2`
`["a","b","c"]`	`[3]: a,b,c`

Design Specification

Full design documentation available at: specs/toon-encoder/

Test Coverage

60+ unit tests for TOON encoder/decoder
30+ plugin integration tests
Round-trip verification (encode→decode = original)
Edge case coverage (unicode, large numbers, special characters)

crivetimihai · 2026-01-29T22:04:16Z

Cool, will take a look soon!

Add a tool_post_invoke plugin that converts JSON tool results to TOON (Token-Oriented Object Notation) format, achieving 30-70% token reduction. Features: - Pure Python TOON encoder/decoder per spec v3.0 - Configurable size thresholds and tool filtering - Format markers for downstream parsing - Graceful error handling with skip_on_error fallback - Columnar format for homogeneous object arrays Closes IBM#2574 Signed-off-by: Joe Stein <joe.stein@sscinc.com> Signed-off-by: Mihai Criveti <crivetimihai@gmail.com>

Signed-off-by: Mihai Criveti <crivetimihai@gmail.com>

crivetimihai · 2026-01-30T15:25:23Z

Thanks @joestein-ssc for this excellent contribution! The TOON encoder plugin is a great addition for token efficiency.

I've rebased your branch onto main and made several spec-compliance fixes based on thorough review against the TOON spec v3.0. Here's a summary of the changes:

Fixes Applied

TOON Spec Compliance:

Empty arrays now encode to [0]: (with required colon per spec)
Empty objects at root encode to empty document (not {})
Columnar field order uses encounter order from first object (not alphabetical)
Leading zeros (05) and leading hyphens (-a) are now quoted
Only valid escape sequences allowed (\\, \", \n, \r, \t) - removed \uXXXX
§10 compliance: list-item objects with tabular first field emit - key[N]{fields}: on hyphen line
Array header regex supports dotted keys (user.name[N]:) and escaped quotes in quoted keys

Decoder Fixes:

Fixed round-trip for nested objects with arrays
Fixed round-trip for list-item arrays of complex objects
Fixed empty nested objects followed by same-depth siblings
Fixed tabular row indentation (header+2, not header+4)

Plugin Improvements:

Annotations preserved even when add_format_marker=false
Size thresholds use UTF-8 byte length (not character count)
Stats use separate tool-level and item-level counters for clarity

Cleanup:

Removed unrelated AGENTS.md changes
Removed specs/ directory and review.md
Added plugin-manifest.yaml for plugin catalog registration
Updated README examples to match spec-compliant output

All 121 TOON encoder tests and 5285 unit tests pass.

The branch has been force-pushed to your fork. Please review and let me know if you have any questions!

Add documentation about tab/pipe delimiter limitation in columnar array headers. The TOON spec v3.0 allows alternative delimiters, which our regex matches but decoder doesn't parse correctly (always splits on commas). Document this as a known decoder limitation. Signed-off-by: Mihai Criveti <crivetimihai@gmail.com>

Add support for alternative delimiters (tab, pipe) in columnar array headers per TOON spec v3.0. The decoder now detects the delimiter from the header and uses it consistently for parsing row values. - Add _detect_delimiter() function to identify delimiter from header - Update _decode_columnar_array() to accept and use delimiter parameter - Update _split_row_values() to split on configurable delimiter - Add tests for pipe and tab delimiter decoding - Remove limitation from README (now fully supported) Signed-off-by: Mihai Criveti <crivetimihai@gmail.com>

- Remove unused Union import (F401) - Remove unused ind variable in _encode_array (F841) Signed-off-by: Mihai Criveti <crivetimihai@gmail.com>

Silence vulture warnings by prefixing intentionally unused parameters: - _as_root in _encode_array and _encode_object (for API consistency) - _expected_count in _split_row_values (for potential validation) - _context in tool_post_invoke (required by plugin interface) Signed-off-by: Mihai Criveti <crivetimihai@gmail.com>

get_plugin_by_name() only checked the registry for enabled plugins, causing "Not Found" errors when clicking View Details on disabled plugins. Now falls back to checking config.plugins for disabled plugins, matching the behavior of get_all_plugins(). Signed-off-by: Mihai Criveti <crivetimihai@gmail.com>

* feat(plugins): add TOON encoder plugin for token-efficient responses Add a tool_post_invoke plugin that converts JSON tool results to TOON (Token-Oriented Object Notation) format, achieving 30-70% token reduction. Features: - Pure Python TOON encoder/decoder per spec v3.0 - Configurable size thresholds and tool filtering - Format markers for downstream parsing - Graceful error handling with skip_on_error fallback - Columnar format for homogeneous object arrays Closes IBM#2574 Signed-off-by: Joe Stein <joe.stein@sscinc.com> Signed-off-by: Mihai Criveti <crivetimihai@gmail.com> * lint Signed-off-by: Mihai Criveti <crivetimihai@gmail.com> * docs(toon): document alternative delimiter limitation Add documentation about tab/pipe delimiter limitation in columnar array headers. The TOON spec v3.0 allows alternative delimiters, which our regex matches but decoder doesn't parse correctly (always splits on commas). Document this as a known decoder limitation. Signed-off-by: Mihai Criveti <crivetimihai@gmail.com> * feat(toon): support tab/pipe delimiters in columnar arrays Add support for alternative delimiters (tab, pipe) in columnar array headers per TOON spec v3.0. The decoder now detects the delimiter from the header and uses it consistently for parsing row values. - Add _detect_delimiter() function to identify delimiter from header - Update _decode_columnar_array() to accept and use delimiter parameter - Update _split_row_values() to split on configurable delimiter - Add tests for pipe and tab delimiter decoding - Remove limitation from README (now fully supported) Signed-off-by: Mihai Criveti <crivetimihai@gmail.com> * fix(toon): remove unused import and variable - Remove unused Union import (F401) - Remove unused ind variable in _encode_array (F841) Signed-off-by: Mihai Criveti <crivetimihai@gmail.com> * fix(toon): prefix unused parameters with underscore Silence vulture warnings by prefixing intentionally unused parameters: - _as_root in _encode_array and _encode_object (for API consistency) - _expected_count in _split_row_values (for potential validation) - _context in tool_post_invoke (required by plugin interface) Signed-off-by: Mihai Criveti <crivetimihai@gmail.com> * fix(admin): return disabled plugin details in View Details get_plugin_by_name() only checked the registry for enabled plugins, causing "Not Found" errors when clicking View Details on disabled plugins. Now falls back to checking config.plugins for disabled plugins, matching the behavior of get_all_plugins(). Signed-off-by: Mihai Criveti <crivetimihai@gmail.com> --------- Signed-off-by: Joe Stein <joe.stein@sscinc.com> Signed-off-by: Mihai Criveti <crivetimihai@gmail.com> Co-authored-by: Mihai Criveti <crivetimihai@gmail.com> Signed-off-by: hughhennnelly <hughhennelly06@gmail.com>

* feat(plugins): add TOON encoder plugin for token-efficient responses Add a tool_post_invoke plugin that converts JSON tool results to TOON (Token-Oriented Object Notation) format, achieving 30-70% token reduction. Features: - Pure Python TOON encoder/decoder per spec v3.0 - Configurable size thresholds and tool filtering - Format markers for downstream parsing - Graceful error handling with skip_on_error fallback - Columnar format for homogeneous object arrays Closes IBM#2574 Signed-off-by: Joe Stein <joe.stein@sscinc.com> Signed-off-by: Mihai Criveti <crivetimihai@gmail.com> * lint Signed-off-by: Mihai Criveti <crivetimihai@gmail.com> * docs(toon): document alternative delimiter limitation Add documentation about tab/pipe delimiter limitation in columnar array headers. The TOON spec v3.0 allows alternative delimiters, which our regex matches but decoder doesn't parse correctly (always splits on commas). Document this as a known decoder limitation. Signed-off-by: Mihai Criveti <crivetimihai@gmail.com> * feat(toon): support tab/pipe delimiters in columnar arrays Add support for alternative delimiters (tab, pipe) in columnar array headers per TOON spec v3.0. The decoder now detects the delimiter from the header and uses it consistently for parsing row values. - Add _detect_delimiter() function to identify delimiter from header - Update _decode_columnar_array() to accept and use delimiter parameter - Update _split_row_values() to split on configurable delimiter - Add tests for pipe and tab delimiter decoding - Remove limitation from README (now fully supported) Signed-off-by: Mihai Criveti <crivetimihai@gmail.com> * fix(toon): remove unused import and variable - Remove unused Union import (F401) - Remove unused ind variable in _encode_array (F841) Signed-off-by: Mihai Criveti <crivetimihai@gmail.com> * fix(toon): prefix unused parameters with underscore Silence vulture warnings by prefixing intentionally unused parameters: - _as_root in _encode_array and _encode_object (for API consistency) - _expected_count in _split_row_values (for potential validation) - _context in tool_post_invoke (required by plugin interface) Signed-off-by: Mihai Criveti <crivetimihai@gmail.com> * fix(admin): return disabled plugin details in View Details get_plugin_by_name() only checked the registry for enabled plugins, causing "Not Found" errors when clicking View Details on disabled plugins. Now falls back to checking config.plugins for disabled plugins, matching the behavior of get_all_plugins(). Signed-off-by: Mihai Criveti <crivetimihai@gmail.com> --------- Signed-off-by: Joe Stein <joe.stein@sscinc.com> Signed-off-by: Mihai Criveti <crivetimihai@gmail.com> Co-authored-by: Mihai Criveti <crivetimihai@gmail.com>

joestein-ssc requested review from crivetimihai, kevalmahajan and madhav165 as code owners January 29, 2026 19:05

crivetimihai added the triage Issues / Features awaiting triage label Jan 29, 2026

crivetimihai added 2 commits January 30, 2026 15:20

lint

260abcf

Signed-off-by: Mihai Criveti <crivetimihai@gmail.com>

crivetimihai force-pushed the toon-plugin branch from be408dc to 260abcf Compare January 30, 2026 15:24

crivetimihai approved these changes Jan 30, 2026

View reviewed changes

crivetimihai added 2 commits January 30, 2026 15:28

crivetimihai removed the triage Issues / Features awaiting triage label Jan 30, 2026

crivetimihai added 3 commits January 30, 2026 19:05

fix(toon): remove unused import and variable

732acf8

- Remove unused Union import (F401) - Remove unused ind variable in _encode_array (F841) Signed-off-by: Mihai Criveti <crivetimihai@gmail.com>

crivetimihai merged commit 884544f into IBM:main Jan 30, 2026
52 checks passed

crivetimihai added this to the Release 1.0.0-RC1 milestone Jan 31, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

toon token savings#2574

toon token savings#2574
crivetimihai merged 7 commits intoIBM:mainfrom
joestein-ssc:toon-plugin

joestein-ssc commented Jan 29, 2026

Uh oh!

crivetimihai commented Jan 29, 2026

Uh oh!

crivetimihai commented Jan 30, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

joestein-ssc commented Jan 29, 2026

📝 Summary

Problem

Solution

Expected Token Savings

Verified Example

Files Added

🏷️ Type of Change

🧪 Verification

✅ Checklist

📓 Notes

Design Decisions

Usage

Agent Integration Required

TOON Format Reference

Design Specification

Test Coverage

Uh oh!

crivetimihai commented Jan 29, 2026

Uh oh!

crivetimihai commented Jan 30, 2026

Fixes Applied

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants