Embedding Sets

Embedding models and vector configuration.

Embedding Sets

Status: Production Version: 0.3.0 Schema Migrations:

  • `20260117000000_embedding_sets.sql` - Initial implementation
  • `20260201500000_full_embedding_sets.sql` - Full sets, MRL, auto-embed

Overview

Embedding Sets enable tiered semantic search with focused embedding collections. They allow power users to create specialized semantic search indexes optimized for specific domains, projects, or use cases while maintaining backward compatibility with the global default search behavior.

Concepts

What is an Embedding Set?

An embedding set is a named collection of notes with dedicated vector embeddings optimized for semantic search within a specific domain or context. Each set maintains its own embedding index, allowing focused semantic queries that ignore irrelevant content.

Default Behavior

By default, all semantic search operates on the "default" embedding set, which contains all notes. This maintains backward compatibility and provides the expected global search behavior without configuration.

Set Types (v0.3.0)

Embedding sets can be one of two types:

Filter Sets (Default)

Filter sets share embeddings from the default embedding set. When you search a filter set, it filters results from the pre-existing default embeddings.

  • Advantages: No additional storage, instant creation
  • Limitations: Same embedding model/dimensions as default set

Full Sets

Full sets maintain their own independent embeddings with dedicated configuration.

  • Advantages: Custom embedding model, MRL dimension truncation, auto-embed rules
  • Limitations: Additional storage, embedding generation time

Example:

{
  "name": "Fast Search",
  "set_type": "full",
  "truncate_dim": 256,
  "auto_embed_rules": {
    "on_create": true,
    "on_update": true
  }
}

Matryoshka Representation Learning (MRL)

MRL-enabled models encode information hierarchically, allowing embeddings to be truncated to smaller dimensions while preserving quality.

Supported Models:

  • `nomic-embed-text` (dims: 768, 512, 256, 128, 64)
  • `mxbai-embed-large-v1` (dims: 1024, 512, 256, 128, 64)

Trade-offs:

DimensionStorage ReductionQuality Loss
64-dim12×~3-5%
128-dim6×~2%
256-dim3×~1%
Full1×0%

See `docs/content/embedding-model-selection.md` for detailed guidance.

Query Templates

Set `query_template` on embedding-set create or update to match the query instruction used by an external producer. It must contain exactly one literal `{text}` placeholder and be at most 2048 UTF-8 bytes; invalid values return an HTTP 400 problem response. The field is returned by the set detail endpoint. Omitting it preserves existing query behavior; on update, omission or `null` keeps the current value. Use `{text}` for an explicit identity template.

For Qwen3-Embedding served by an OpenAI-compatible Text Embeddings Inference server, configure the registered provider and an embedding config with model `Qwen/Qwen3-Embedding-8B`, storage `dimension: 1024`, `provider_dimension: 4096`, and the producer's model revision `1d8ad4ca9b3dd8059ad90a75d4983776a23d44af`. Create a full set referencing that config with (the config ID must reference the configuration just created):

{
  "name": "Qwen3 company embeddings",
  "model": "Qwen/Qwen3-Embedding-8B",
  "dimension": 1024,
  "provider_dimension": 4096,
  "provider": "custom",
  "provider_config": {"provider_id": "openai"}
}

Point that registered OpenAI-compatible route at the TEI service, then create the set:

{
  "name": "Company profiles",
  "slug": "company-profiles",
  "set_type": "full",
  "embedding_config_id": "uuid-from-config-response",
  "vector_source": "external",
  "query_template": "Instruct: Identify the industry, market and business model of the company described in the text\
Query:{text}"
}

The template is applied once, before every provider's raw embedding call, including OpenAI-compatible, Ollama, and other registry routes. A set template supersedes any built-in Ollama query prefix. Documents are not wrapped with the query template. Query and internally generated document vectors retain the first 1024 dimensions and are then L2-normalized, matching the producer's `truncate-first-1024-then-l2-normalize` pipeline, including for cosine indexes.

Changing the template changes the non-secret stored embedding contract fingerprint; refresh internally managed embeddings after changing it. Existing sets without a template retain byte-identical fingerprints. Query preparation fails closed: if a stored template cannot be applied, search falls back to FTS with `degraded: true`, `degradation.effective_mode: "fts"`, and stable `degradation.code: "embedding_query_template_unsupported"`. Contextual revision jobs use the same set-aware query preparation for the archive's default set. Preparation failures are logged and preserve Phase 1 with an explicit skip reason in the job result.

Body chunks remain searchable for every set, including after changing `vector_source` to `external`. Profile rows are searchable only for sets with `vector_source: "external"`; profiles restored into internal sets retain the existing body-only search behavior.

`dimension` is the storage/index dimension and stays capped at 2000 for `vector` or 4000 for `halfvec`. Optional nullable `provider_dimension` must be at least `dimension` and at most 16384. It explicitly enables raw-output truncation and L2 normalization for both queries and generated documents and participates in the contract fingerprint. Omission preserves legacy provider-native behavior for template-less sets, including fingerprints and refresh decisions. Imported external vectors already have the storage dimension. MCP config create/update accepts `provider_dimension`, and config read/list returns it.

Stored vectors written before the query-template change are not rewritten. L2 rescaling after truncation is ranking-neutral under cosine search.

Full-v1 Knowledge Shards preserve `query_template`, `vector_source` and `provider_dimension` using schema `2.2.0` when any extended contract is present. Explicit older export versions reject contracts they would discard. Older shards restore NULL query templates/raw dimensions and internal write ownership. Body chunks stay searchable under either ownership; profile vectors are searchable only when `vector_source` is `external`. The new shard tuple requires consumer qualification before cross-repository release claims.

Auto-Embed Rules

Full embedding sets can automatically manage embedding lifecycle:

{
  "on_create": true,         // Generate on note creation
  "on_update": true,         // Regenerate on note update
  "update_threshold_percent": 10.0,  // Min change to trigger update
  "max_embedding_age_secs": 86400,   // Max age before refresh
  "priority": 5,             // Job queue priority (higher = sooner)
  "batch_size": 10,          // Batch size for bulk operations
  "rate_limit": 100          // Max embeddings per minute
}

Use Cases

Domain-Specific Knowledge Bases:

  • Separate machine learning research from software architecture notes
  • Isolate project-specific documentation from general knowledge
  • Create topic-specific indexes (medical, legal, technical)

Project Isolation:

  • Create per-project embedding sets for focused project search
  • Prevent cross-contamination between unrelated projects
  • Maintain project-specific semantic relationships

Temporal Segmentation:

  • Shard historical notes in separate sets
  • Focus search on recent content only
  • Maintain different retention policies per set

Access Control Foundation:

  • Prepare for future multi-tenant features
  • Isolate sensitive content in separate indexes
  • Enable selective sharing of knowledge domains

Architecture

Database Schema

embedding_set Table

Stores embedding set metadata and configuration:

  • Identity: name, slug, description, purpose, usage_hints, keywords
  • Membership: mode (auto/manual/mixed), criteria (JSON)
  • Configuration: embedding_config_id reference
  • Index Status: index_status, index_type, last_indexed_at
  • Statistics: document_count, embedding_count, index_size_bytes
  • Lifecycle: is_system, is_active, auto_refresh, refresh_interval
  • Agent Metadata: agent_metadata (JSON for AI discovery)

embedding_set_member Table

Many-to-many relationship between sets and notes:

  • Primary Key: (embedding_set_id, note_id)
  • Membership Type: auto, manual_include, manual_exclude
  • Audit: added_at, added_by

embedding_config Table

Embedding configuration profiles:

  • Model Settings: model, dimension
  • Chunking: chunk_size, chunk_overlap
  • Index Settings: hnsw_m, hnsw_ef_construction, ivfflat_lists

embedding Table Extension

Existing embedding table extended with:

  • embedding_set_id: Foreign key to embedding_set

Modes

Auto Mode

Notes automatically included based on criteria evaluation. Criteria evaluated on:

  • Set creation
  • Manual refresh via API
  • Scheduled auto-refresh (if enabled)

Criteria Options:

  • `include_all: true` - Include all notes (default set behavior)
  • `tags: ["ml", "research"]` - Notes with ANY of these tags
  • `collections: [uuid1, uuid2]` - Notes in ANY of these collections
  • `fts_query: "machine learning"` - Notes matching FTS query
  • `created_after: "2024-01-01T00:00:00Z"` - Date range filters
  • `created_before: "2024-12-31T23:59:59Z"`
  • `exclude_archived: true` - Skip archived notes (default)

Manual Mode

Notes only included by explicit add_members API calls. No automatic membership evaluation.

Use Cases:

  • Curated collections requiring human judgment
  • One-off specialized indexes
  • Testing or experimental sets

Mixed Mode

Combines auto criteria with manual additions/exclusions. Notes added via criteria can be manually removed. Additional notes can be manually added beyond criteria matches.

Use Cases:

  • Auto-populated sets with manual overrides
  • Exclude specific notes from auto-matched set
  • Include outliers not matching criteria

Index Status

pending

Initial state after set creation. Index needs to be built.

building

Background job actively building embeddings and index.

ready

Index is current and available for search.

stale

Membership changed (new notes added). Index needs rebuild.

disabled

No index built. Used for very small sets or manual-only collections.

API Reference

Base URL

All endpoints use the base path: `/api/v1/embedding-sets`

External Profile Embedding Runs

External embedding sets can import entity-profile vectors generated by another system. The import is keyed by the producer's `source` and `entity_id`; callers do not need Fortemi note IDs. Fortemi resolves or creates the backing note through the source-addressed upsert contract, then stores the profile vector in the target set without queuing embedding jobs.

Streamed Upload Format

The primary import path is a streamed NDJSON upload:

POST /api/v1/embedding-sets/{slug}/runs
Content-Type: application/x-ndjson

Line 1 is the manifest object and must include `"type": "manifest"`, `run_id`, `previous_run_id`, `space_id`, `created_at`, `template_version`, `counts`, and `body_sha256`:

{"type":"manifest","run_id":"run-2026-10-09","previous_run_id":null,"space_id":"...","created_at":"2026-10-09T04:00:00Z","template_version":"profile-v1","counts":{"profiles":1000,"deletions":0},"body_sha256":"..."}

Following lines are row objects with `"type": "profile"` or `"type": "deletion"`. `body_sha256` is lowercase SHA-256 hex over each non-manifest, nonblank row line in stream order: the exact UTF-8 JSON bytes for the row as parsed before the line ending, followed by one canonical `\ `. The manifest line is not included. `FORTEMI_VECTOR_IMPORT_MAX_LINE_BYTES` bounds a single uploaded line and defaults to 1 MiB; the request has no import-specific total cap beyond the server's normal streaming body limits.

Profile rows include `entity_id`, `source`, `profile_hash`, optional `profile_text`, `template_version`, `space_id`, `dims`, `embedding`, and optional `metadata`. When `profile_text` is supplied, `profile_hash` must equal `sha256(template_version + "\ " + profile_text)` as lowercase hex. Deletion rows include `entity_id`, `source`, and `run_id`; they remove only that entity's profile vector in the set.

Ordering And Idempotency

Runs are applied in a chain. The target set's last applied `run_id` must match the incoming `previous_run_id`; the first run uses `previous_run_id: null`. Re-posting an already applied `run_id` is a no-op and returns the persisted report with `status: "already_applied"`.

Rows with a bad space ID, dimension, non-finite value, non-unit L2 vector for L2-normalized spaces, or profile-hash mismatch are rejected individually and reported. Manifest count failures, `body_sha256` mismatch, unsupported Parquet files, internal target sets, and out-of-order runs fail the whole run.

Server-Side Folder Import

The legacy JSON path import is retained for operator-controlled environments only. It requires an `admin` scope credential and is disabled unless `FORTEMI_VECTOR_IMPORT_ROOT` is set. The requested path may be absolute under that root or relative to it; Fortemi canonicalizes the root and requested path and rejects parent traversal or symlinks that resolve outside the root.

A run folder contains:

  • `manifest.json`
  • `profiles/.jsonl` or `profiles/.ndjson`
  • optional `deletions.jsonl` or `deletions.ndjson`

`manifest.json` includes `run_id`, `previous_run_id`, `space_id`, `created_at`, `template_version`, `counts`, and `files`. Each file entry declares `path`, lowercase `sha256`, and `rows`. The importer verifies every checksum and row count before applying rows.

Bulk-Load Procedure

For large imports:

1. Create or update the target embedding set with `vector_source: "external"`. 2. Set `defer_index_build` while loading large runs. 3. Stream the run:

POST /api/v1/embedding-sets/{slug}/runs
Content-Type: application/x-ndjson

{"type":"manifest",...}
{"type":"profile",...}
{"type":"deletion",...}

Parquet producers should convert to NDJSON and stream the result. See `scripts/vector-import/parquet_to_ndjson.py` for a pyarrow reference converter.

4. If an operator must import a server-local run folder, set `FORTEMI_VECTOR_IMPORT_ROOT` and post a confined path with an admin token:

POST /api/v1/embedding-sets/{slug}/runs
Content-Type: application/json

{ "path": "customer-profiles/run-2026-10-09" }

5. Inspect reports:

GET /api/v1/embedding-sets/{slug}/runs
GET /api/v1/embedding-sets/{slug}/runs/{run_id}

6. Build the vector index after the batch:

POST /api/v1/embedding-sets/{slug}/build-index

The MCP `import_embedding_run` tool uses the admin-only confined path import. Parquet import remains reserved; current builds support streamed NDJSON, and Parquet producers or loaders convert with pyarrow or an equivalent tool before uploading.

List Embedding Sets

GET /api/v1/embedding-sets

Returns all active embedding sets with summary statistics.

Response:

[
  {
    "id": "019b76da-a800-7001-8000-000000000001",
    "name": "Default",
    "slug": "default",
    "description": "Primary embedding set containing all notes",
    "purpose": "Provides semantic search across the entire knowledge base",
    "document_count": 1523,
    "embedding_count": 8947,
    "index_status": "ready",
    "is_system": true,
    "keywords": ["all", "general", "default"],
    "model": "nomic-embed-text",
    "dimension": 768
  },
  {
    "id": "a1b2c3d4-...",
    "name": "ML Research",
    "slug": "ml-research",
    "description": "Machine learning research papers and notes",
    "purpose": "Focused semantic search for ML concepts and techniques",
    "document_count": 237,
    "embedding_count": 1456,
    "index_status": "ready",
    "is_system": false,
    "keywords": ["machine-learning", "research", "papers"],
    "model": "nomic-embed-text",
    "dimension": 768
  }
]

Get Embedding Set

GET /api/v1/embedding-sets/:slug

Returns detailed information about a specific embedding set.

Parameters:

  • `slug` (path) - URL-friendly set identifier (e.g., "default", "ml-research")

Response:

{
  "id": "a1b2c3d4-...",
  "name": "ML Research",
  "slug": "ml-research",
  "description": "Machine learning research papers and notes",
  "purpose": "Focused semantic search for ML concepts",
  "usage_hints": "Use for queries about neural networks, deep learning, optimization",
  "keywords": ["ml", "research", "papers", "neural-networks"],
  "mode": "auto",
  "criteria": {
    "tags": ["machine-learning", "research"],
    "exclude_archived": true
  },
  "embedding_config_id": "019b76da-a800-7001-8000-000000000001",
  "document_count": 237,
  "embedding_count": 1456,
  "index_status": "ready",
  "index_size_bytes": 11239424,
  "is_system": false,
  "is_active": true,
  "auto_refresh": true,
  "agent_metadata": {
    "created_by_agent": "claude-opus-4.5",
    "rationale": "Separate ML content for focused research queries",
    "performance_notes": "Best for conceptual ML queries, not code"
  },
  "created_at": "2024-01-15T10:30:00Z",
  "updated_at": "2024-01-20T14:22:00Z"
}

Create Embedding Set

POST /api/v1/embedding-sets

Creates a new embedding set. Background job automatically queued to build index.

Request Body:

{
  "name": "Project Alpha",
  "slug": "project-alpha",
  "description": "Documentation and notes for Project Alpha",
  "purpose": "Focused search for project-specific knowledge",
  "usage_hints": "Use for queries about Project Alpha architecture, requirements, decisions",
  "keywords": ["project-alpha", "project", "alpha"],
  "mode": "auto",
  "criteria": {
    "tags": ["project-alpha"],
    "collections": ["uuid-of-project-alpha-collection"],
    "exclude_archived": true
  }
}

Fields:

  • `name` (required) - Display name for the set
  • `slug` (optional) - URL-friendly identifier (auto-generated from name if omitted)
  • `description` (optional) - Short description of the set's contents
  • `purpose` (optional) - Detailed purpose for AI agent decision-making
  • `usage_hints` (optional) - When and how to use this set
  • `keywords` (optional) - Discovery keywords for AI agents
  • `mode` (optional) - Membership mode: "auto", "manual", "mixed" (default: "auto")
  • `criteria` (optional) - Auto-membership criteria (required for auto/mixed mode)
  • `embedding_config_id` (optional) - Custom embedding config (defaults to global default)

Response:

Returns the created embedding set (same structure as GET).

Update Embedding Set

PATCH /api/v1/embedding-sets/:slug

Updates embedding set metadata. Cannot rename or deactivate system sets.

Request Body:

{
  "description": "Updated description",
  "purpose": "Updated purpose",
  "usage_hints": "New usage guidelines",
  "keywords": ["updated", "keywords"],
  "is_active": true,
  "auto_refresh": false,
  "mode": "mixed",
  "criteria": {
    "tags": ["new-tag", "another-tag"],
    "exclude_archived": true
  }
}

All fields are optional. Only provided fields are updated.

Response:

Returns the updated embedding set.

Delete Embedding Set

DELETE /api/v1/embedding-sets/:slug

Permanently deletes an embedding set. Cannot delete system sets (e.g., "default").

Response:

204 No Content

List Set Members

GET /api/v1/embedding-sets/:slug/members?limit=50&offset=0

Lists notes that are members of the embedding set.

Query Parameters:

  • `limit` (optional) - Maximum members to return (default: 50)
  • `offset` (optional) - Pagination offset (default: 0)

Response:

[
  {
    "embedding_set_id": "a1b2c3d4-...",
    "note_id": "e5f6g7h8-...",
    "membership_type": "auto",
    "added_at": "2024-01-15T10:30:00Z",
    "added_by": null
  },
  {
    "embedding_set_id": "a1b2c3d4-...",
    "note_id": "i9j0k1l2-...",
    "membership_type": "manual_include",
    "added_at": "2024-01-16T14:22:00Z",
    "added_by": "[email protected]"
  }
]

Add Set Members

POST /api/v1/embedding-sets/:slug/members

Manually adds notes to an embedding set. Marks index as stale.

Request Body:

{
  "note_ids": [
    "e5f6g7h8-i9j0-k1l2-m3n4-o5p6q7r8s9t0",
    "u1v2w3x4-y5z6-a7b8-c9d0-e1f2g3h4i5j6"
  ],
  "added_by": "claude-opus-4.5"
}

Response:

{
  "count": 2
}

Remove Set Member

DELETE /api/v1/embedding-sets/:slug/members/:note_id

Removes a note from an embedding set. Also removes the note's embeddings from this set.

Response:

204 No Content

Refresh Embedding Set

POST /api/v1/embedding-sets/:slug/refresh

Re-evaluates criteria and updates membership for auto/mixed mode sets. Queues background jobs to rebuild embeddings for new members.

Response:

{
  "added": 15
}

Returns count of newly added members.

Search with Embedding Sets

GET /api/v1/search?q=neural+networks&mode=semantic&set=ml-research

Performs semantic search restricted to a specific embedding set.

Query Parameters:

  • `q` (required) - Search query
  • `mode` (optional) - "hybrid", "fts", "semantic" (default: "hybrid")
  • `set` (optional) - Embedding set slug (default: "default")
  • `limit` (optional) - Maximum results (default: 20)

Response:

Same as standard search, but results only from the specified set. Fortémi generates the query vector with the set's embedding provider, model, and effective dimension. If the set uses an OpenAI-compatible registered route, set `provider_config.provider_id` to that registry ID. Provider endpoints and credentials remain registry-owned.

If the provider is unavailable or returns a vector with the wrong dimension, the request executes FTS instead and returns `"degraded": true` with a stable `degradation.code`. It never silently labels the fallback as semantic.

Background embedding jobs use the same effective contract. Each stored vector records a non-secret SHA-256 contract fingerprint covering provider, model, dimension, normalization policy, and embedding-set identity. Provider credentials and endpoints are not fingerprint inputs. A provider response with the wrong model, vector count, response indexes, or dimension is rejected before existing embeddings are replaced.

MCP Tools

The MCP server provides tools for AI agents to discover and use embedding sets.

list_embedding_sets

Lists all available embedding sets for discovery.

Input Schema:

{}

Output:

Array of embedding set summaries (same as API GET /embedding-sets).

Usage:

const sets = await mcpClient.callTool("list_embedding_sets", {});

get_embedding_set

Gets detailed information about a specific embedding set.

Input Schema:

{
  "slug": "ml-research"
}

Output:

Full embedding set details (same as API GET /embedding-sets/:slug).

Usage:

const set = await mcpClient.callTool("get_embedding_set", {
  slug: "ml-research"
});

create_embedding_set

Creates a new embedding set.

Input Schema:

{
  "name": "Project Beta",
  "slug": "project-beta",
  "description": "Project Beta documentation",
  "purpose": "Focused search for Project Beta",
  "usage_hints": "Use for Project Beta queries",
  "keywords": ["project-beta", "beta"],
  "mode": "auto",
  "criteria": {
    "tags": ["project-beta"],
    "exclude_archived": true
  }
}

Output:

Created embedding set.

Usage:

const set = await mcpClient.callTool("create_embedding_set", {
  name: "Research Papers 2024",
  mode: "auto",
  criteria: {
    tags: ["research", "papers"],
    created_after: "2024-01-01T00:00:00Z",
    exclude_archived: true
  }
});

list_set_members

Lists notes in an embedding set.

Input Schema:

{
  "slug": "ml-research",
  "limit": 50,
  "offset": 0
}

Output:

Array of set members.

Usage:

const members = await mcpClient.callTool("list_set_members", {
  slug: "ml-research",
  limit: 100
});

add_set_members

Adds notes to an embedding set.

Input Schema:

{
  "slug": "ml-research",
  "note_ids": ["uuid1", "uuid2"],
  "added_by": "claude-opus-4.5"
}

Output:

{
  "count": 2
}

Usage:

const result = await mcpClient.callTool("add_set_members", {
  slug: "ml-research",
  note_ids: [noteId1, noteId2],
  added_by: "claude-opus-4.5"
});

remove_set_member

Removes a note from an embedding set.

Input Schema:

{
  "slug": "ml-research",
  "note_id": "uuid"
}

Output:

{
  "success": true
}

Usage:

await mcpClient.callTool("remove_set_member", {
  slug: "ml-research",
  note_id: noteId
});

refresh_embedding_set

Refreshes an embedding set by re-evaluating criteria.

Input Schema:

{
  "slug": "ml-research"
}

Output:

{
  "added": 15
}

Usage:

const result = await mcpClient.callTool("refresh_embedding_set", {
  slug: "ml-research"
});

search_notes with Embedding Sets

Search within a specific embedding set.

Input Schema:

{
  "query": "transformer architecture",
  "mode": "semantic",
  "set": "ml-research",
  "limit": 20
}

Output:

Search results restricted to the specified set.

Usage:

const results = await mcpClient.callTool("search_notes", {
  query: "attention mechanisms in transformers",
  mode: "semantic",
  set: "ml-research"
});

Criteria Configuration

Criteria Structure

Criteria is a JSON object with the following optional fields:

{
  "include_all": false,
  "tags": ["tag1", "tag2"],
  "collections": ["uuid1", "uuid2"],
  "fts_query": "search terms",
  "created_after": "2024-01-01T00:00:00Z",
  "created_before": "2024-12-31T23:59:59Z",
  "exclude_archived": true
}

Field Semantics

include_all

Type: boolean Default: false

If true, includes all non-archived notes. Typically used only for the default set.

Example:

{
  "include_all": true,
  "exclude_archived": true
}

tags

Type: array of strings Default: []

Includes notes with ANY of the specified tags (OR logic).

Example:

{
  "tags": ["machine-learning", "deep-learning", "research"]
}

Matches notes tagged with "machine-learning" OR "deep-learning" OR "research".

collections

Type: array of UUIDs Default: []

Includes notes in ANY of the specified collections (OR logic).

Example:

{
  "collections": [
    "a1b2c3d4-e5f6-4747-8888-999999999999",
    "b2c3d4e5-f6a7-4848-9999-000000000000"
  ]
}

fts_query

Type: string Default: null

Includes notes matching the full-text search query.

Example:

{
  "fts_query": "neural network optimization"
}

Uses PostgreSQL FTS with English language configuration.

created_after

Type: ISO 8601 datetime string Default: null

Includes notes created after the specified timestamp.

Example:

{
  "created_after": "2024-01-01T00:00:00Z"
}

created_before

Type: ISO 8601 datetime string Default: null

Includes notes created before the specified timestamp.

Example:

{
  "created_before": "2024-12-31T23:59:59Z"
}

exclude_archived

Type: boolean Default: true

If true, excludes archived notes from the set.

Example:

{
  "tags": ["research"],
  "exclude_archived": false
}

Criteria Combination Logic

Multiple criteria are combined with AND logic:

{
  "tags": ["research", "papers"],
  "collections": ["uuid1"],
  "created_after": "2024-01-01T00:00:00Z",
  "exclude_archived": true
}

Matches notes that:

  • Have tag "research" OR "papers", AND
  • Are in collection "uuid1", AND
  • Were created after 2024-01-01, AND
  • Are not archived

Example Configurations

Recent Work

{
  "created_after": "2024-11-01T00:00:00Z",
  "exclude_archived": true
}

Project-Specific

{
  "tags": ["project-alpha"],
  "collections": ["project-alpha-collection-uuid"],
  "exclude_archived": true
}

Topic Archive

{
  "tags": ["machine-learning", "deep-learning"],
  "created_before": "2024-01-01T00:00:00Z",
  "exclude_archived": false
}

Meeting Notes

{
  "fts_query": "meeting notes attendees action items",
  "tags": ["meeting"],
  "exclude_archived": true
}

Agent Metadata

Agent metadata provides structured information for AI agents to discover and use embedding sets effectively.

Metadata Structure

{
  "created_by_agent": "claude-opus-4.5",
  "rationale": "Why this set was created",
  "performance_notes": "When this set performs well/poorly",
  "related_sets": ["set-slug-1", "set-slug-2"],
  "suggested_queries": [
    "example query 1",
    "example query 2"
  ]
}

Field Descriptions

created_by_agent

Type: string Optional: yes

Identifies the AI agent or tool that created the set.

Example: "claude-opus-4.5", "auto-discovery-service"

rationale

Type: string Optional: yes

Explains why this set was created and what problem it solves.

Example: "Separate ML research from general software notes to reduce noise in ML-specific queries"

performance_notes

Type: string Optional: yes

Documents performance characteristics or usage patterns.

Example: "Best for conceptual ML queries. Use 'ml-code' set for implementation queries."

Type: array of strings Optional: yes

Lists related or complementary embedding sets.

Example: ["ml-code", "ml-datasets", "ml-papers"]

suggested_queries

Type: array of strings Optional: yes

Example queries that work well with this set.

Example:

[
  "transformer attention mechanisms",
  "backpropagation optimization",
  "gradient descent variants"
]

Discovery Pattern

AI agents should use agent metadata to:

1. List available embedding sets 2. Filter by keywords matching query domain 3. Check suggested_queries for query pattern match 4. Consider performance_notes for set selection 5. Explore related_sets for comprehensive coverage

Example Agent Logic:

// User query: "How do transformers work?"
const sets = await listEmbeddingSets();

// Filter by keywords
const candidates = sets.filter(s =>
  s.keywords.some(k =>
    ["ml", "machine-learning", "transformers", "neural-networks"]
      .includes(k)
  )
);

// Check suggested queries
const bestMatch = candidates.find(s =>
  s.agent_metadata?.suggested_queries?.some(q =>
    q.toLowerCase().includes("transformer")
  )
) || candidates[0];

// Search in best matching set
const results = await searchNotes({
  query: "transformer architecture",
  set: bestMatch.slug
});

Implementation Details

Database Triggers

update_embedding_set_stats()

Automatically updates document_count and embedding_count when:

  • Notes added to or removed from set
  • Embeddings created or deleted for set

trigger_update_set_stats()

Fires on INSERT/UPDATE/DELETE to embedding_set_member.

trigger_update_embedding_set_stats()

Fires on INSERT/UPDATE/DELETE to embedding table when embedding_set_id is involved.

Functions

get_default_embedding_set_id()

Returns the UUID of the default embedding set (fast lookup).

get_default_embedding_config_id()

Returns the UUID of the default embedding config.

update_embedding_set_stats(set_id UUID)

Manually updates statistics for a specific set. Called by triggers and after bulk operations.

Index Status Management

Index status transitions:

pending -> building -> ready
ready -> stale (when membership changes)
stale -> building -> ready (on refresh)

Status updated by:

  • Background jobs (building)
  • API operations (pending, stale)
  • Refresh operations (stale -> building)

Background Jobs

create_embedding_set Job

Queued automatically on set creation. Builds initial embeddings and index.

refresh_embedding_set Job

Queued on manual refresh API call or scheduled auto-refresh. Re-evaluates criteria and rebuilds index.

build_set_index Job

Queued after membership changes. Builds pgvector HNSW or IVFFlat index.

Migration Safety

The migration maintains backward compatibility:

  • Existing embeddings migrated to default set
  • Default set marked as system (cannot delete)
  • All existing notes added as default set members
  • No changes to search behavior without explicit set parameter

Best Practices

Set Design

Keep sets focused:

  • Specific domain or project
  • Clear membership criteria
  • Distinct from other sets

Avoid over-segmentation:

  • Too many small sets reduce discovery
  • Prefer fewer, well-defined sets over many narrow ones
  • Consider merged set with good criteria vs. multiple tiny sets

Document set purpose:

  • Clear description and purpose
  • Usage hints for when to use this set
  • Keywords for AI agent discovery

Criteria Design

Use tag-based criteria for flexibility:

{
  "tags": ["project-alpha", "architecture"],
  "exclude_archived": true
}

Combine tags and collections for precision:

{
  "tags": ["research"],
  "collections": ["ml-research-collection-uuid"],
  "exclude_archived": true
}

Use date ranges for temporal segmentation:

{
  "tags": ["meeting"],
  "created_after": "2024-01-01T00:00:00Z",
  "exclude_archived": true
}

Performance

Index status matters:

  • Only search ready indexes
  • Monitor stale status and refresh periodically
  • Disable indexes for very small sets (< 10 notes)

Set size considerations:

  • Small sets (< 100 notes): Manual mode acceptable
  • Medium sets (100-1000): Auto mode with specific criteria
  • Large sets (> 1000): Consider sub-segmentation

Refresh frequency:

  • Enable auto_refresh for active sets
  • Set appropriate refresh_interval (default: 1 day)
  • Manual refresh after bulk note imports

Agent Usage

Set discovery:

// List sets and filter by domain
const sets = await listEmbeddingSets();
const mlSets = sets.filter(s =>
  s.keywords.includes("machine-learning")
);

Query routing:

// Route query to appropriate set
const queryDomain = detectDomain(userQuery); // "ml", "project-alpha", etc.
const set = sets.find(s => s.keywords.includes(queryDomain)) || sets.find(s => s.slug === "default");

const results = await searchNotes({
  query: userQuery,
  set: set.slug
});

Fallback strategy:

// Search specific set first, fall back to default
let results = await searchNotes({ query, set: "ml-research" });
if (results.length === 0) {
  results = await searchNotes({ query, set: "default" });
}

Future Enhancements

Planned Features

Cross-set search:

  • Search multiple sets simultaneously
  • Weighted combination of results
  • De-duplication across sets

Set hierarchies:

  • Parent-child set relationships
  • Inheritance of criteria
  • Automatic propagation of updates

Custom embedding models per set:

  • Domain-specific embedding models
  • Different dimensions per set
  • Model performance tracking

Set performance analytics:

  • Query success rate per set
  • Average result quality
  • Usage statistics

AI-driven set creation:

  • Automatic set discovery from usage patterns
  • Suggested sets based on query analysis
  • Dynamic criteria optimization

Migration Path

Current schema supports future enhancements:

  • `agent_metadata` field extensible for new agent features
  • `criteria` JSONB allows new filter types
  • `embedding_config` separation enables per-set models
  • `index_type` field supports alternative index algorithms

Troubleshooting

Check index status:

GET /api/v1/embedding-sets/:slug

If `index_status` is not "ready", wait for background job to complete or check job queue.

Verify membership:

GET /api/v1/embedding-sets/:slug/members

Ensure expected notes are members.

Refresh set:

POST /api/v1/embedding-sets/:slug/refresh

Empty Search Results

Check criteria matches notes:

List members to verify criteria is matching expected notes.

Verify embedding set parameter:

Ensure search query uses correct `set` parameter.

Check embedding status:

Verify notes in set have embeddings:

SELECT COUNT(*) FROM embedding WHERE embedding_set_id = 'set-uuid';

Stale Index

Manual refresh:

POST /api/v1/embedding-sets/:slug/refresh

Enable auto-refresh:

PATCH /api/v1/embedding-sets/:slug
{
  "auto_refresh": true
}

Criteria Not Matching

Test criteria with direct SQL:

SELECT n.id, n.title
FROM note n
LEFT JOIN note_revised_current nrc ON nrc.note_id = n.id
WHERE n.deleted_at IS NULL
  AND n.id IN (SELECT note_id FROM note_tag WHERE tag_name = 'your-tag')
  AND (n.archived IS FALSE OR n.archived IS NULL);

Adjust criteria based on results.

Performance Issues

Check set size:

Very large sets (> 10,000 notes) may have slow index builds.

Monitor index type:

HNSW indices faster for large sets than IVFFlat.

Consider set segmentation:

Split very large sets into smaller focused sets.

References

Database Schema

  • Migration: `/migrations/20260117000000_embedding_sets.sql`
  • Repository: `/crates/matric-db/src/embedding_sets.rs`
  • Models: `/crates/matric-core/src/models.rs`

API Implementation

  • Routes: `/crates/matric-api/src/main.rs` (lines 268-289)
  • Handlers: `/crates/matric-api/src/main.rs` (lines 1285-1397)

MCP Implementation

  • Tools: `/mcp-server/index.js` (lines 1007-1165)
  • Hybrid Search: `/docs/hybrid-search.md` (when available)
  • API Reference: `/docs/api-reference.md` (when available)
  • MCP Integration: `/docs/mcp-integration.md` (when available)