Embodiment — Maxim Docs | Sensor-Entity-Modulator Protocol

The SEM Protocol

Core Insight

Hardware varies wildly — cameras, joints, wheels, grippers, IMUs, swords, NPCs. Rather than building monolithic abstractions per robot type, every interactive thing is described as a composable triple: an Entity (the thing), its Sensors (how you read it), and its Modulators (how you change it).

Entity

The physical or virtual thing. Entities compose into trees: arm → elbow → wrist → gripper. Each entity is self-describing.

Examples: joint, camera, wheel, sword, NPC, door

Sensor

Reads state from an entity. One sensor = one readable quantity. Returns structured SensorReading with value, unit, and timestamp.

Examples: angle, temperature, durability, trust, frame

Modulator

Changes state of an entity. Exposes named affordances — each with typed parameters, description, and timeout.

Examples: rotate_angle, slash, speak, restart, sharpen

Auto-Generated Tools

When an entity tree loads from YAML, tools are automatically created and registered. No hand-written tool classes per hardware component.

Tool Name	Type	Description
sense_shoulder	EntitySense	Read all sensors on shoulder
read_shoulder_angle	SensorRead	Read angle sensor
shoulder_rotate_angle	Affordance	Rotate shoulder joint
rusty_sword_slash	Affordance	Slash at a target
grim_ferryman_speak	Affordance	Say something to the ferryman

Name collisions are handled by progressive prefixing: shoulder_rotate_angle → robot2_shoulder_rotate_angle if two robots share the same entity name.

Cerebellum: Forward Models

Biological Inspiration

The biological cerebellum stores forward models that predict sensory consequences of motor commands. Climbing-fiber complex spikes carry prediction error; massive microcircuit specialization enables fast, accurate motor control without conscious thought.

Maxim's Cerebellum stores lightweight predictors per (entity, modulator, affordance, param_bucket). Each learns via Rescorla-Wagner prediction error:

expected += α × (actual − expected)

Confidence < 0.3 → LLM fallback (teaches the Cerebellum)
Confidence ≥ 0.3 → cached prediction (no LLM call)
High variance → LLM fallback (uncertain predictions need grounding)

The LLM is a teacher, not a per-tick oracle. After enough observations, the Cerebellum handles predictions deterministically. In testing, LLM calls drop from 100 to ≤40 over 100 actions.

Motor Programs

A reach-and-grasp isn't three separate LLM decisions. It's one motor program — an ordered sequence of SEM actions that fires as a unit. Programs crystallize when the agent repeats the same sequence 3+ times for the same goal.

Example: reach_forward Step 1: shoulder.motor.rotate_angle(degrees=45, speed=1.0) Step 2: elbow.motor.rotate_angle(degrees=30, speed=1.0) Crystallized after 3 repetitions. Confidence: 0.82 Known risks: overextension if shoulder.angle > 160 at start

Triple-Indexed Registry

The ProgramRegistry indexes programs in three directions:

By goal — "I want to reach forward" → matching programs
By entity — "I'm holding a sword" → all programs involving swords
By affordance — "I want to slash" → programs for sword, axe, arm, claws

Pain-Gated Execution

Each step has an optional pain gate — a sensor threshold that aborts the program before damage occurs. Gates tighten by 10% after each painful execution. The PainBus is subscribed for real-time mid-sequence interrupts.

Motor Engrams

Biological Inspiration

Hippocampal-cerebellar interactions are well-documented: the hippocampus provides contextual scaffolding for motor learning ("where and when did I learn this movement?"), while the cerebellum stores the procedural knowledge itself.

Motor engrams are ephemeral cross-system traces linking motor programs to situational context:

Cerebellum stores the how (motor program steps)
Hippocampus stores the when/where/what (contextual episode)
The engram links them via the associative graph

Engrams form only on significant outcomes — pain > 0.3, surprising results (RPE > 0.3), or novel programs. Routine successes don't need episodic context. Engrams decay after ~2 days unless reinforced, using the standard hippocampal consolidation cycle.

The Cyclical Feedback Loop

Cross-System Loop Cerebellum → Hippocampus: Motor program outcome → engram captured Hippocampus → Cerebellum: Engram recall → context-dependent gating Hippocampus → NAc: Engram triggers causal observation NAc → Cerebellum: Prediction vetoes/greenlights program Hippocampus → SCN: Temporal index → rhythmic patterns SCN → Cerebellum: Time-of-day threshold adjustment

Virtual Entities: Beyond Robotics

The SEM protocol is hardware-agnostic. A sword is just an Entity with sensors (durability, sharpness) and modulators (slash, parry) backed by NarrativeModulator instead of hardware. The same cognitive stack that learns about robot joints also learns about swords, NPCs, and doors.

Rusty Sword

Sensors: durability, sharpness, weight

Modulators: slash, parry, throw, sharpen, repair

Failure: shatter (durability < 0.1), dulled (sharpness < 0.15)

Grim Ferryman (NPC)

Sensors: trust, mood, health

Modulators: speak, offer_payment, threaten, punch

Failure: hostility (trust < 0.1), refusal (mood < -0.5)

The Cerebellum learns "swinging a damaged sword at a stone golem reduces durability by ~0.15" the same way it learns "rotating an elbow at 90°/s increases strain by ~0.1." Same Rescorla-Wagner update, same forward model, same pain triggers.

Composable Failure Modes

Six base failure modes: overextension, overheating, strain, fatigue, impact, exhaustion. Custom failures compose from these without taxonomy explosion:

Composed Failure: Tennis Elbow composes: [strain, fatigue] trigger: all: - {field: strain, op: ">", value: 0.6} - {field: fatigue, op: ">", value: 0.5} pain_intensity: 0.5 persistent: true recovery_condition: {field: fatigue, op: "<", value: 0.2}

Failures route through the existing PainBus and ToolPainBridge. NAc learns (affordance, entity_state) → failure causal links. Persistent failures stay active until recovery conditions are met. All failure state persists across sessions.

Architecture

YAML Spec ──→ Entity Tree ──→ Auto-Generated Tools ──→ Agent │ │ ├──→ Failure Triggers ──→ PainBus ──→ NAc (causal learning) │ │ ├──→ ATL Body Concepts ──→ Grounded LLM predictions │ │ ├──→ Cerebellum ──→ Forward models + Motor programs │ │ └──→ Hippocampus ──→ Motor engrams (context-dependent gating)

Module	Purpose
embodiment/sem.py	Core protocols: Sensor, Modulator, Entity, FailureMode
embodiment/spec.py	YAML loader, SpecSensor/SpecModulator stubs
embodiment/tool_bridge.py	Auto-tool generation with collision detection
embodiment/body.py	Embodiment runtime, failure eval, vital drift, prompt state
embodiment/cerebellum.py	Forward models, motor program registry, engram formation/recall
embodiment/motor.py	MotorProgram, MotorStep, ProgramRegistry (triple-indexed)
embodiment/engrams.py	MotorEngram, salience computation, formation logic
embodiment/program_executor.py	Step-by-step runner, pain gates, PainBus interrupt
embodiment/llm_backend.py	LLM + Narrative sensor/modulator backends
embodiment/percepts.py	EmbodimentPerceptSource (1Hz polling, demand mode)
embodiment/atl_integration.py	Auto-register ATL body_part concepts

The SEM Protocol

Core Insight

Entity

Sensor

Modulator

Auto-Generated Tools

Cerebellum: Forward Models

Biological Inspiration

Motor Programs

Triple-Indexed Registry

Pain-Gated Execution

Motor Engrams

Biological Inspiration

The Cyclical Feedback Loop

Virtual Entities: Beyond Robotics

Rusty Sword

Grim Ferryman (NPC)

Composable Failure Modes

Architecture

All Chapters