Mnemosyne

A sensor-fusion runtime that hot-plugs new input modalities into a frozen trained core. A new sensor is integrated by fine-tuning only a small adapter (~22 K params) while the core and fuser stay frozen, so adding a modality does not require retraining or redeploying the model.

• Runtime: CPU-only; no GPU required. Inference under 200 MB RAM (int8-quantized).
• Requirements: Python + PyTorch; the GGUF C loader (c_loader/) builds with any C compiler.
• Determinism: single seed (mnemosyne.SEED = 1337) seeds torch/numpy/random; data windows derive from (seed_base, sample_index), so identical seeds reproduce identical runs.

Install & train

pip install torch   # CPU build is sufficient

# Phase-1 warm start: train core + fuser + two known modalities (rf, spectrogram)
python -m mnemosyne.train          # writes checkpoints/

# Evaluate a never-seen modality end to end
python -m mnemosyne.demo --eval-radar
python -m mnemosyne.demo --eval-gesture

Core API

FuserLoom

The whole fusion system: backbone + fuser + ports + output head.

from mnemosyne.loom import FuserLoom

loom = FuserLoom(
    dim=64,         # core latent dim
    d_enc=32,       # encoder output dim per port
    num_classes=5,
    core_vhdus=3,
    state_size=32,
)

Fused output = head(backbone(fuser(fused_feature_stream))), where fused_feature_stream is the gate-weighted sum over active ports concatenated with the backbone's temporal read of the same stream.

Port registry

Each modality is a ModalityPort (encoder → dimension sandbox). Built-in encoders:

from mnemosyne.ports import build_port

port = build_port("radar", dim=64, d_enc=32)  # -> ModalityPort

Adding a novel modality (QuickSwap)

from mnemosyne.quickswap import add_novel_port, quicklearn, quickremove, quickadd_and_learn

# One-call version:
report = quickadd_and_learn(loom, "radar", steps=250, lr=5e-3, device="cpu")

# Step by step:
add_novel_port(loom, "radar")     # fresh random encoder + mixing gate
quicklearn(loom, "radar",
           steps=250,             # gradient steps (fuser-only fine-tune)
           lr=5e-3,
           drop_p=0.9,            # known-sensor dropout probability
           seed_offset=0)         # batch-stream offset for reproducibility

quickremove(loom, "radar")        # remove a port; other ports untouched

quicklearn trains only the target port's parameters plus its mixing gate; the backbone, fuser, and other ports stay frozen. Sensor dropout unplugs random subsets of known sensors per batch so the port must actually learn the new stream.

Custom modality ports

To integrate your own sensor, provide an encoder whose forward emits (batch, seq_len, d_enc) frames and wrap it as a port:

import torch.nn as nn
from mnemosyne.ports import ModalityPort
from mnemosyne.fuser import DimensionSandbox

class MySensorEncoder(nn.Module):
    def __init__(self, d_in, d_enc):
        super().__init__()
        self.net = nn.Sequential(
            nn.Conv1d(d_in, 32, 3, padding=1), nn.GELU(),
            nn.AdaptiveAvgPool1d(1),
        )
        self.proj = nn.Linear(32, d_enc)

    def forward(self, x):            # x: (batch, seq_len, d_in)
        b, t, d = x.shape
        h = self.net(x.reshape(b * t, d, 1)).squeeze(-1)
        return self.proj(h).reshape(b, t, -1)

encoder = MySensorEncoder(d_in=my_width, d_enc=loom.d_enc)
adapter = DimensionSandbox(loom.d_enc, loom.dim)
port = ModalityPort("mysensor", encoder, loom.d_enc, loom.dim)
loom.add_port(port, known=False)   # unknown provenance -> novel

Then run quicklearn(loom, "mysensor", ...) with windows from your sensor. Synthetic generators for the built-in modalities live in mnemosyne.sensors (gen_rf_window, gen_spectrogram_window, gen_radar_window, gen_vision_window, gen_gesture_window, gen_batch) if you need reference shapes.

Persistence: GGUF bundle

import torch
from mnemosyne.gguf import write_gguf, read_gguf

write_gguf(
    "checkpoints/fusion.gguf",
    loom,
    fused_embedding=torch.zeros(loom.dim),
    metadata={"name": "fusion", "seed": 1337},
)
data = read_gguf("checkpoints/fusion.gguf")   # dict of tensors + metadata KV

Bundle contents:

• fused_embedding (dim,) — current fused semantic vector
• codebook (num_classes, dim) — frozen retrieval targets
• per-port tensors {name}.{param} — int8 weights when available, fp16 otherwise
• metadata KV block: name, seed, known[], novel[], class_names[], port_info[] (name, d_in, d_enc, params), memory

A dependency-free reader (c_loader/mnemosyne_loader.c, stdio-only) reads the same files; build with make -C c_loader. Bundle size ≈ 94 KB for the default configuration.

Quantized inference

from mnemosyne.eval import quantize_model

qloom = quantize_model(loom)       # int8 copy of the whole loom
# same forward pass through dequantized weights:
out = qloom(windows)

Measured cost: ~6–7 ms/sample, ~520 KB of weights, accuracy unchanged versus fp32 on the reference tasks.

Enclave (licensing API)

Session budgeting and license enforcement live inside the model weights (enclave_state buffers persist with the model). Relevant entry points:

python -m mnemosyne.enclave_cli issue --kind dev --seats 4 --out seat.mnport   # issuer side
python -m mnemosyne.enclave_cli install seat.mnport                            # consumer side
python -m mnemosyne.enclave_cli status                                         # inspect state

from mnemosyne.enclave import Enclave

enclave = Enclave(loom)
enclave.install_port("seat.mnport")   # validates form, HMAC tag, checksum, seat binding, lifespan
print(enclave.status())               # dict: kind, seats, lifespan, governor state

Programmatic issuance for site-side integrations: mnemosyne.enclave.issue_port(kind="dev"|"prod", seats=N, seat_id=..., ...) returns canonical single-line .mnport text; consumers may pass it via mnemosyne.enclave.install_from_text(text, enclave). Port text is byte-canonical — re-wrapped or indented copies are rejected.

Module map