Multi-LLM Networking & Peer Mesh

Why Distributed Inference

Maxim's agent pipeline calls the LLM multiple times per cycle: perception, memory consolidation, goal reasoning, execution planning, and statistical review. On a single machine with one GPU, these calls compete for the same inference server. When the model is large or the context window is deep, that bottleneck caps the cycle rate.

Distributed inference solves this by letting peer machines contribute their compute. A laptop on the same network, a desktop in another room, or a cloud VM halfway around the world can all send inference requests to the machine running the GPU. The leader handles the model; peers handle everything else.

Architecture Overview

The networking layer sits between the LLM router and the inference backend. On the leader, a reverse proxy accepts authenticated requests and forwards them to the local llama-cpp-server. On peers, the router is configured to point at the leader instead of localhost.

Both the leader's own agent loop and remote peers are independent HTTP clients of the same llama-cpp-server. The LeaderProxy authenticates and rate-limits peer traffic before it reaches the backend. The leader's own requests go directly to :8100, bypassing the proxy entirely.

Roles

Every Maxim instance operates in one of three roles. The role determines how the LLM router resolves inference endpoints and whether the proxy/tunnel subsystems activate.

LeaderProxy (Phase 7a)

The LeaderProxy is a stdlib-only reverse proxy that listens on port 8099 and forwards authenticated requests to the local llama-cpp-server on port 8100. It uses only Python's http.server and urllib.request — no third-party dependencies.

Debug Endpoints

The proxy exposes four debug endpoints for operational visibility. All require the same Bearer token as inference requests.

Admission Control (Phase 7b)

The inference server can only handle so many concurrent requests before latency degrades or VRAM is exhausted. Admission control prevents overload by rejecting excess traffic early, before it reaches the backend.

Environment Variables

On the peer side, the LLM router's retry logic handles 429s gracefully: it reads Retry-After, backs off, and retries. From the agent's perspective, the request is simply slower — the pipeline does not crash.

Lane Metrics (Phase 8)

The WorkerPool routes inference requests through capability tiers: large (14B+ GPU), medium (7B CPU/GPU), and small (1.7B CPU). Functions declare which tier they need via a FunctionRouter with fallback chains. Per-tier performance counters track throughput and latency in real time.

The MetricsRegistry is a singleton shared between the agent runtime and the LeaderProxy. Both write to the same counters, giving a unified view of local and remote load. These metrics feed into maxim doctor, which flags lanes with elevated failure rates or latency spikes.

# View lane metrics from the proxy debug endpoint
curl -s -H "Authorization: Bearer $MAXIM_API_KEY" \
  https://maxim.yourdomain.com/v1/debug/metrics | python -m json.tool

# Example output
{
  "lanes": {
    "infer":  { "p50_ms": 142, "p99_ms": 890, "fail_pct": 0.3, "tok_per_sec": 48.2 },
    "review": { "p50_ms": 98,  "p99_ms": 520, "fail_pct": 0.0, "tok_per_sec": 31.7 },
    "record": { "p50_ms": 67,  "p99_ms": 310, "fail_pct": 0.1, "tok_per_sec": 22.4 }
  },
  "uptime_s": 3847,
  "total_requests": 1294
}

System Heartbeat

The heartbeat subsystem runs as a daemon thread that samples system vitals every 10 seconds. It provides early warning when hardware resources are constrained or the agent loop has stalled.

Enabling the Heartbeat

# Enable heartbeat logging
export MAXIM_HEARTBEAT=1

# Or enable lane trace (which also enables heartbeat)
export MAXIM_LANE_TRACE=1

# Heartbeat output in the log
[heartbeat] gpu=72% vram=5.1/8.0GB cpu=2.4 ram=61% disk=42GB loop=+0.8s lanes=3/0/1

Stall detection is particularly useful for debugging distributed setups. If the agent loop blocks on a remote inference call that the leader has rate-limited, the heartbeat will flag the idle gap before the user notices the pause.

Peer Setup

Setting up a peer takes three steps: install Maxim on the peer machine, connect to the leader, and verify the link.

1. Connect to the Leader

# On the peer machine
maxim peer connect https://maxim.yourdomain.com/v1

# This prompts for the API key (generated on the leader with `maxim tunnel key rotate`)
# and writes ~/.config/maxim/peer.yml

2. Verify Connectivity

# Quick connectivity test (runs from the peer, no full agent runtime needed)
maxim peer test https://maxim.yourdomain.com/v1

# Expected output:
#   Connecting to https://maxim.yourdomain.com/v1 ...
#   Auth ............ OK (Bearer token accepted)
#   Heartbeat ....... OK (proxy alive, 1294 requests served)
#   Inference ....... OK (model loaded, 48 tok/s)
#   Latency ......... 23ms round-trip

3. Run with Tracing

# Enable lane tracing to see which requests go remote
export MAXIM_LANE_TRACE=1
maxim --language-model mistral-7b

# Trace output shows remote routing:
# [lane:infer] POST /v1/chat/completions -> remote (23ms RTT, 142ms total)

endpoint: https://maxim.yourdomain.com/v1
api_key: mk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
timeout_s: 30
retry_max: 3
verify_ssl: true

Troubleshooting

Most networking issues fall into a few categories. Start with maxim doctor — it checks tunnel status, proxy reachability, and auth validity automatically.

For deeper diagnostic procedures, see docs/troubleshooting/ in the repository.

What's Next

The networking layer has a clear progression from the current manual-setup model toward automatic discovery and intelligent routing.