EngramEngramDocs

Architecture

System design, component overview, and data flows for the Engram subnet.

Overview

System diagram
┌──────────────────────────────────────────────────────────────┐
│                       Bittensor Chain                        │
│               (metagraph · weight setting · TAO)             │
└─────────────────────┬──────────────────────┬─────────────────┘
                      │                      │
              ┌───────▼──────┐    ┌──────────▼──────┐
              │  Validator   │    │      Miner       │
              │              │    │                  │
              │ • challenge  │───▶│ • FAISS index    │
              │ • score      │    │ • embedder       │
              │ • set weights│◀───│ • proof service  │
              └──────────────┘    └──────────┬───────┘
                                             │
                                   ┌─────────▼────────┐
                                   │   engram-core     │
                                   │   (Rust / PyO3)   │
                                   │ • CID generation  │
                                   │ • HMAC proofs     │
                                   └──────────────────┘

Components

Miner (neurons/miner.py)
An aiohttp HTTP server that exposes /ingest and /query endpoints. Embeds text using OpenAI or sentence-transformers, stores vectors in a FAISS IVF-flat index, and responds to validator challenges with HMAC proofs.
Validator (neurons/validator.py)
Issues IngestSynapse and QuerySynapse calls to all registered miners, measures recall accuracy and latency, then sets on-chain weights proportional to composite scores every 600 seconds.
engram-core (Rust/PyO3)
A Rust extension module compiled with PyO3. Handles CID generation (SHA-256 over float32 bytes) and HMAC-SHA256 storage proof generation/verification.
SDK (engram/sdk/)
EngramClient — a zero-dependency Python HTTP client. Also provides LangChain VectorStore and LlamaIndex BasePydanticVectorStore adapters.
FastAPI backend (engram-web/api/)
A thin FastAPI bridge between the Next.js frontend and the live subnet. Caches metagraph data for 30 seconds. Deployed at api.theengram.space.
Frontend (engram-web/)
Next.js app with a live subnet dashboard, query playground, miner leaderboard, and this docs site. Deployed on Vercel at theengram.space.

Data flows

Ingest flow:

bash
Client SDK
  → POST /ingest {text, metadata}
  → Miner embeds text (OpenAI / sentence-transformers)
  → engram-core derives CID from embedding
  → FAISS index stores vector + CID
  → Response: {cid: "v1::a3f2b1..."}

Query flow:

bash
Client SDK
  → POST /query {query_text, top_k}
  → Miner embeds query text
  → FAISS ANN search → top-K nearest neighbors
  → Response: [{cid, score, metadata}, ...]

Validation flow:

bash
Validator
  → Pick random CID from ground truth corpus
  → Send IngestSynapse to miner
  → Verify returned vector matches ground truth (cosine sim > 0.99)
  → Record latency and proof success
  → Repeat for all registered miners
  → Compute composite scores → set on-chain weights

Storage layer

Miners use FAISS IVF-flat for in-memory vector storage with persistence to disk. The index is saved to FAISS_INDEX_PATH after each ingest session and loaded on startup.

Note
The Rust engram-core module is optional — the Python fallback is used automatically if the Rust build is unavailable. Build it with pip install maturin && cd engram-core && maturin develop --release for production performance.
Previous
Protocol Reference
engram docs · v0.1edit on github →