Operations Overview

Day-to-day operational procedures.

Operations and Deployment Guide

This guide covers deployment, operations, and troubleshooting for Fortémi using Docker.

System Overview

Deployment: Docker bundle (all-in-one container)
Components: PostgreSQL 18 + pgvector + PostGIS, Rust API, Node.js MCP server
Ports: 3000 (API), 3001 (MCP)
Data: PostgreSQL data in Docker volume `matric-pgdata`

1. Initial Setup 2. Deployment Procedures 3. Container Management 4. Database Operations 5. MCP Server Operations 6. Monitoring and Health Checks 7. Troubleshooting 8. Backup and Recovery 9. Configuration 10. Multi-Memory Operations

Initial Setup

Prerequisites

Docker and Docker Compose
Nginx (for reverse proxy)
Domain with SSL certificate

First-Time Deployment

# 1. Clone repository
git clone https://github.com/fortemi/fortemi.git
cd Fortémi

# 2. Start container (creates database)
docker compose -f docker-compose.bundle.yml up -d

# 3. Wait for initialization (first run takes ~60 seconds)
docker compose -f docker-compose.bundle.yml logs -f

# 4. Configure environment
cat > .env <<EOF
ISSUER_URL=http://localhost:3000
FORTEMI_ALLOW_LOCAL_ISSUER=true
EOF

# 5. Restart with configuration
docker compose -f docker-compose.bundle.yml down
docker compose -f docker-compose.bundle.yml up -d

# 6. Verify (MCP credentials are auto-registered on startup)
curl http://localhost:3000/health
curl http://localhost:3001/.well-known/oauth-protected-resource

Nginx Configuration

Configure nginx to proxy to the container. Set `FORTEMI_TRUSTED_PROXY_CIDRS=127.0.0.1/32,::1/128` when this same-host nginx is the immediate Fortemi peer. The edge must overwrite client forwarding fields, as below, and Fortemi should listen only on a private or loopback interface. Unset `FORTEMI_TRUSTED_PROXY_CIDRS` means forwarded metadata is ignored.

server {
    listen 443 ssl http2;
    server_name your-domain.com;

    ssl_certificate /path/to/cert.pem;
    ssl_certificate_key /path/to/key.pem;

    # API routes
    location / {
        proxy_pass http://localhost:3000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_set_header X-Forwarded-Protocol "";
        proxy_set_header Forwarded "";
    }

    # WebSocket and SSE support for real-time events
    location /api/v1/ws {
        proxy_pass http://localhost:3000;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_read_timeout 86400;
    }

    location /api/v1/events {
        proxy_pass http://localhost:3000;
        proxy_set_header Host $host;
        proxy_set_header Connection '';
        proxy_http_version 1.1;
        chunked_transfer_encoding off;
        proxy_buffering off;
        proxy_cache off;
        proxy_read_timeout 86400;
    }

    # MCP routes
    location = /mcp {
        proxy_pass http://localhost:3001/;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_set_header X-Forwarded-Protocol "";
        proxy_set_header Forwarded "";
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }

    location /mcp/ {
        proxy_pass http://localhost:3001/;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_set_header X-Forwarded-Protocol "";
        proxy_set_header Forwarded "";
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }
}

Deployment Procedures

Standard Update Workflow

# 1. Pull latest code
git pull origin main

# 2. Backup database (recommended)
docker exec Fortémi-matric-1 pg_dump -U matric matric > backup_$(date +%Y%m%d_%H%M%S).sql

# 3. Rebuild and restart
docker compose -f docker-compose.bundle.yml build
docker compose -f docker-compose.bundle.yml down
docker compose -f docker-compose.bundle.yml up -d

# 4. Verify
curl http://localhost:3000/health
docker compose -f docker-compose.bundle.yml logs --tail=50

Critical Rules

1. Backup before major updates - Database migrations run automatically on container start 2. Check logs after restart - Verify migrations applied successfully 3. Test health endpoint - Confirm API is responding

Rollback Procedure

If deployment fails:

# 1. Stop container
docker compose -f docker-compose.bundle.yml down

# 2. Restore database from backup
docker compose -f docker-compose.bundle.yml up -d
sleep 30  # Wait for PostgreSQL to start
docker exec -i Fortémi-matric-1 psql -U matric -d matric < backup_YYYYMMDD_HHMMSS.sql

# 3. Checkout previous version
git checkout <previous-commit>

# 4. Rebuild with old code
docker compose -f docker-compose.bundle.yml build
docker compose -f docker-compose.bundle.yml down
docker compose -f docker-compose.bundle.yml up -d

# 5. Verify
curl http://localhost:3000/health

Container Management

Common Commands

# Status
docker compose -f docker-compose.bundle.yml ps

# Logs (follow)
docker compose -f docker-compose.bundle.yml logs -f

# Logs (last N lines)
docker compose -f docker-compose.bundle.yml logs --tail=100

# Restart
docker compose -f docker-compose.bundle.yml restart

# Stop
docker compose -f docker-compose.bundle.yml down

# Start
docker compose -f docker-compose.bundle.yml up -d

# Rebuild
docker compose -f docker-compose.bundle.yml build

# Shell access
docker exec -it Fortémi-matric-1 /bin/bash

Full Reset (Wipes Database)

docker compose -f docker-compose.bundle.yml down -v
docker compose -f docker-compose.bundle.yml up -d

Database Operations

Tenant-scoped read-only psql

Since the ADR-090 forced-RLS migration (`20260824010000`), direct SQL against `note`, `archive_registry`, and other guarded tables requires an explicit tenant. Use the checked-in read-only recipe from the same release checkout. It resolves the memory name through the scoped registry, verifies the schema and RLS role, and counts live notes.

# DATABASE_URL selects an authorized SQL login subject to RLS.
# Use your normal password file/service configuration; do not embed passwords.
psql "$DATABASE_URL" -X -v ON_ERROR_STOP=1 \
  -v tenant_id=00000000-0000-0000-0000-000000000000 \
  -v archive_name=public \
  -f scripts/sql/read-only-notes.sql

For personal installations, the all-zero UUID is the reserved local tenant; `public` is the initial memory name. Use the actual memory name for a non-default archive (the same name used with `X-Fortemi-Memory`), not a guessed schema name. A changed default memory does not change this recipe's explicit selection. For hosted installations, obtain the tenant UUID and archive name you are authorized to inspect from the operator's tenant inventory. Do not use the reserved local UUID to inspect hosted customer data. Binding a UUID is not an authorization check: a SQL login able to set this parameter must only be given to trusted operators, with database/schema/table access restricted appropriately.

The recipe's essential sequence, all on one connection, is:

BEGIN READ ONLY;
SET LOCAL app.current_tenant = '00000000-0000-0000-0000-000000000000';
-- Resolve and verify the authorized archive first; public is the initial archive.
SET LOCAL search_path = public;
SELECT current_user, current_setting('app.current_tenant'), current_schema();
SELECT count(*) AS live_notes FROM note WHERE deleted_at IS NULL;
COMMIT;

The downloadable recipe uses `set_config(..., true)`, equivalent to `SET LOCAL`, and safely quotes the registry's schema name. `note` is singular. The canonical live-note predicate is `deleted_at IS NULL`; `archived = true` does not mean soft-deleted. Add `archived = false` only when deliberately excluding archived notes from the count.

`SET LOCAL` ends at commit/rollback. It must share the transaction and connection with the query; a separate `psql -c` invocation, later pooled session, or a statement outside a transaction does not inherit it. Bind scope for every transaction. `-X` ignores local psql startup customizations and `ON_ERROR_STOP` makes SQL/script failures exit nonzero. A read-only transaction rejects writes.

With an RLS-subject role, absent tenant context fails when a guarded policy is evaluated (an unset setting or invalid/empty UUID); an empty relation may yield no rows without evaluating the policy. A valid wrong tenant cannot see another tenant's rows. In this recipe, an absent/invisible archive fails the one-row registry check before any note query. A visible archive with no live notes returns zero. Treat zero as a scoped result, never proof that the database is empty. The recipe also refuses a superuser or `BYPASSRLS` role and refuses a schema whose note table lacks forced RLS.

Connection

For an interactive scoped read, connect with `psql "$DATABASE_URL" -X -v ON_ERROR_STOP=1`, then run the complete transaction above, or use `\i scripts/sql/read-only-notes.sql` after setting both recipe variables. For the bundle, run psql inside the service with the appropriate SQL login and pipe the recipe into the same invocation:

docker compose -f docker-compose.bundle.yml exec -T fortemi \
  psql -U <SQL_LOGIN> -d matric -X -v ON_ERROR_STOP=1 \
  -v tenant_id=00000000-0000-0000-0000-000000000000 \
  -v archive_name=public < scripts/sql/read-only-notes.sql

Common Queries

# Database size
docker exec Fortémi-matric-1 psql -U matric -d matric -c \
  "SELECT pg_size_pretty(pg_database_size('matric'));"

# Table sizes
docker exec Fortémi-matric-1 psql -U matric -d matric -c "
SELECT relname AS table_name, pg_size_pretty(pg_total_relation_size(relid)) AS size
FROM pg_catalog.pg_statio_user_tables
ORDER BY pg_total_relation_size(relid) DESC;"

# Live-note count: run scripts/sql/read-only-notes.sql with tenant and archive variables above.

# Active connections
docker exec Fortémi-matric-1 psql -U matric -d matric -c \
  "SELECT count(*) FROM pg_stat_activity WHERE datname = 'matric';"

Maintenance

Routine scoped reads above are distinct from privileged whole-database backup and maintenance. Use the designated maintenance/backup identity for the commands below and the backup procedure; do not grant `BYPASSRLS` to the runtime role or disable RLS to make read scripts work. `VACUUM`, `REINDEX`, and materialized-view refreshes do not belong in `BEGIN READ ONLY`. A scoped SELECT/export is not a complete database backup; backup privileges and restore verification are tracked separately in Gitea #1115 and #727.

# Vacuum analyze (weekly recommended)
docker exec Fortémi-matric-1 psql -U matric -d matric -c "VACUUM ANALYZE;"

# Refresh embedding set stats
docker exec Fortémi-matric-1 psql -U matric -d matric -c \
  "REFRESH MATERIALIZED VIEW embedding_set_stats;"

# Reindex (if query performance degrades)
docker exec Fortémi-matric-1 psql -U matric -d matric -c "REINDEX DATABASE matric;"

MCP Server Operations

The MCP server runs automatically inside the Docker bundle on port 3001.

Verify MCP Configuration

# Check OAuth protected resource metadata
curl http://localhost:3001/.well-known/oauth-protected-resource

# Expected response:
# {
#   "resource": "http://localhost:3000/mcp",
#   "authorization_servers": ["http://localhost:3000"],
#   ...
# }

Claude Code Integration

Project `.mcp.json`:

{
  "mcpServers": {
    "fortemi": {
      "url": "http://localhost:3001"
    }
  }
}

MCP Health Check

curl http://localhost:3001/health

Monitoring and Health Checks

Health Endpoints

# API health
curl http://localhost:3000/health

# MCP health
curl http://localhost:3001/health

Real-Time Event Monitoring

Fortémi provides real-time event streaming for live job and note monitoring. See Real-Time Events for full documentation.

SSE for live job monitoring:

# Stream all events (useful for monitoring job processing)
curl -N http://localhost:3000/api/v1/events

Events include `QueueStatus` (every 5s), `JobQueued`, `JobStarted`, `JobProgress`, `JobCompleted`, `JobFailed`, and `NoteUpdated`.

Webhook alerts for failures:

Set up a webhook to receive alerts when jobs fail:

# Create webhook for failure alerts
curl -X POST http://localhost:3000/api/v1/webhooks \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://your-alerting-service.com/webhook",
    "events": ["JobFailed"],
    "secret": "<WEBHOOK_SECRET>"
  }'

Legacy WebSocket for anonymous local dashboard integration:

When Fortemi is explicitly running with `REQUIRE_AUTH=false` and `I_UNDERSTAND_NO_AUTH=true`, connect to `ws://localhost:3000/api/v1/ws` for legacy local dashboard updates. Send `"refresh"` to trigger an immediate queue status broadcast. Auth-required or hosted deployments should use `/api/v1/events`; `/api/v1/ws` returns `410 Gone`.

Container Health

# Docker health status
docker inspect Fortémi-matric-1 --format='{{.State.Health.Status}}'

# Recent health check results
docker inspect Fortémi-matric-1 --format='{{json .State.Health}}' | jq

Log Analysis

# All logs
docker compose -f docker-compose.bundle.yml logs

# Errors only
docker compose -f docker-compose.bundle.yml logs 2>&1 | grep -i error

# Since specific time
docker compose -f docker-compose.bundle.yml logs --since "1h"

Troubleshooting

Container Won't Start

# Check logs
docker compose -f docker-compose.bundle.yml logs

# Check if port is in use
ss -tlnp | grep -E '3000|3001'

# Verify Docker is running
docker ps

MCP Authentication Fails

Symptom: "Protected resource URL mismatch" error

Cause: Missing or incorrect `ISSUER_URL` in `.env`

Fix:

# Create/update .env with local issuer settings
echo "ISSUER_URL=http://localhost:3000" >> .env
echo "FORTEMI_ALLOW_LOCAL_ISSUER=true" >> .env

# Restart container
docker compose -f docker-compose.bundle.yml down
docker compose -f docker-compose.bundle.yml up -d

# Verify
curl http://localhost:3001/.well-known/oauth-protected-resource

Symptom: MCP returns "unauthorized" even with valid token

Cause: MCP credentials invalid or auto-registration failed

Fix:

# Restart — credentials are auto-registered on startup
docker compose -f docker-compose.bundle.yml down
docker compose -f docker-compose.bundle.yml up -d

Database Connection Errors

# Check PostgreSQL is running inside container
docker exec Fortémi-matric-1 pg_isready -U matric

# Check database exists
docker exec Fortémi-matric-1 psql -U matric -l

# Verify required extensions
docker exec Fortémi-matric-1 psql -U matric -d matric -c "SELECT extname, extversion FROM pg_extension WHERE extname IN ('vector', 'postgis');"

Slow Performance

# Run vacuum
docker exec Fortémi-matric-1 psql -U matric -d matric -c "VACUUM ANALYZE;"

# Check for long-running queries
docker exec Fortémi-matric-1 psql -U matric -d matric -c "
SELECT pid, state, query_start, query
FROM pg_stat_activity
WHERE state = 'active' AND datname = 'matric';"

Out of Disk Space

# Check Docker disk usage
docker system df

# Clean unused images
docker image prune -a

# Check volume size
docker system df -v | grep matric

Backup and Recovery

Manual Backup

# Backup to local file
docker exec Fortémi-matric-1 pg_dump -U matric matric > backup_$(date +%Y%m%d_%H%M%S).sql

# Verify backup
ls -lh backup_*.sql | tail -1
head -50 backup_*.sql | tail -1

Restore from Backup

# Stop and start fresh container (preserves volume)
docker compose -f docker-compose.bundle.yml down
docker compose -f docker-compose.bundle.yml up -d

# Wait for PostgreSQL
sleep 30

# Restore
docker exec -i Fortémi-matric-1 psql -U matric -d matric < backup_YYYYMMDD_HHMMSS.sql

# Verify
curl http://localhost:3000/health

Automated Backup Script

#!/bin/bash
# backup-matric.sh
BACKUP_DIR="/path/to/backups"
DATE=$(date +%Y%m%d_%H%M%S)
BACKUP_FILE="$BACKUP_DIR/matric_$DATE.sql"

mkdir -p "$BACKUP_DIR"
docker exec Fortémi-matric-1 pg_dump -U matric matric > "$BACKUP_FILE"
gzip "$BACKUP_FILE"

# Keep only last 7 days
find "$BACKUP_DIR" -name "matric_*.sql.gz" -mtime +7 -delete

echo "Backup completed: ${BACKUP_FILE}.gz"

Add to crontab for daily backups:

0 2 * * * /path/to/backup-matric.sh

Configuration

Environment Variables Reference

All environment variables are optional unless marked as required. The API reads these values at startup.

Server Configuration

Variable	Default	Description	Example
`HOST`	`0.0.0.0`	HTTP server bind address	`127.0.0.1`
`PORT`	`3000`	HTTP server port	`8080`
`DATABASE_URL`	`<DATABASE_URL>`	PostgreSQL connection string	`<DATABASE_URL>`
`FILE_STORAGE_PATH`	`/var/lib/matric/files`	Directory for file attachments	`/mnt/storage/files`

Rate Limiting

Variable	Default	Description	Example
`RATE_LIMIT_ENABLED`	`true`	Enable rate limiting	`false`
`RATE_LIMIT_REQUESTS`	`100`	Max requests per period	`1000`
`RATE_LIMIT_PERIOD_SECS`	`60`	Rate limit period in seconds	`300`
`FORTEMI_QUOTA_REDIS_URL`	None	Required shared quota store for hosted multi-tenant mode	`redis://quota.internal:6379/0`
`FORTEMI_QUOTA_REQUESTS`	`600`	Hosted request limit per fixed identity window (`1..1000000`)	`1200`
`FORTEMI_QUOTA_WINDOW_SECS`	`60`	Hosted fixed-window duration (`1..86400`)	`60`

The `FORTEMI_QUOTA_*` settings are an internal hosted preview. CE does not connect to this Redis store. Hosted startup fails when the URL is missing, invalid, or unavailable; runtime store loss fails authenticated admission closed with a non-cacheable HTTP 503.

CORS Configuration

Variable	Default	Description	Example
`ALLOWED_ORIGINS`	`https://your-domain.com,http://localhost:3000`	Comma-separated CORS origins	`https://app.example.com,https://staging.example.com`

Logging

Variable	Default	Description	Example
`RUST_LOG`	`info`	Tracing filter directives; debug/trace are explicit protected diagnostic modes	`info` or `matric_api=debug,info`
`LOG_FORMAT`	`text`	Log output format (`text` or `json`)	`json`
`LOG_FILE`	(none)	Path to log file (enables file logging)	`/var/log/matric/api.log`
`LOG_ANSI`	(auto-detected; off for files)	Strict optional ANSI override (`true`/`false` or `1`/`0`) for text logs	`false`

Invalid `LOG_FORMAT`, `LOG_ANSI`, or `RUST_LOG` values fail startup. When debug/trace is explicitly enabled, use the protected diagnostic sink and redaction controls defined by the `docs/architecture/hosted-telemetry-classification.md` contract (#974); do not treat verbose output as an ordinary hosted default.

OAuth / MCP

Variable	Required	Default	Description	Example
`ISSUER_URL`	Yes	`http://HOST:PORT`	External OAuth issuer URL	`https://memory.example.com`
`MCP_CLIENT_ID`	Yes	(none)	OAuth client ID for MCP server token introspection	`mm_abc123`
`MCP_CLIENT_SECRET`	Yes	(none)	OAuth client secret for MCP server	`<MCP_CLIENT_SECRET>`
`MCP_BASE_URL`	No	`${ISSUER_URL}/mcp`	MCP protected resource URL	`https://memory.example.com/mcp`

Ollama Backend (Primary AI Backend)

Variable	Default	Description	Example
`OLLAMA_BASE`	`http://127.0.0.1:11434`	Ollama API base URL	`http://host.docker.internal:11434`
`OLLAMA_EMBED_MODEL`	`nomic-embed-text`	Ollama embedding model name	`mxbai-embed-large`
`OLLAMA_GEN_MODEL`	`qwen3.5:9b`	Ollama generation model name	`llama3.2`
`OLLAMA_EMBED_DIM`	`768`	Embedding vector dimension	`1024`
`OLLAMA_HOST`	`http://localhost:11434`	Ollama host for model discovery	`http://ollama:11434`

OpenAI Backend (Alternative AI Backend)

Variable	Default	Description	Example
`OPENAI_API_KEY`	(none)	OpenAI API key	`<OPENAI_API_KEY>`
`OPENAI_BASE_URL`	`https://api.openai.com/v1`	OpenAI API base URL	`https://api.openai.com/v1`
`OPENAI_EMBED_MODEL`	`text-embedding-3-small`	OpenAI embedding model	`text-embedding-3-large`
`OPENAI_GEN_MODEL`	`gpt-4o-mini`	OpenAI generation model	`gpt-4o`
`OPENAI_EMBED_DIM`	`1536`	OpenAI embedding dimension	`3072`
`OPENAI_TIMEOUT`	`60`	OpenAI request timeout (seconds)	`120`
`OPENAI_SKIP_TLS_VERIFY`	`false`	Skip TLS certificate verification	`true`
`OPENAI_HTTP_REFERER`	(none)	HTTP Referer header for OpenAI requests	`https://example.com`
`OPENAI_X_TITLE`	(none)	X-Title header for OpenAI requests	`My App`

Advanced Inference Configuration

Variable	Default	Description	Example
`MATRIC_INFERENCE_DEFAULT`	`ollama`	Default inference backend (`ollama` or `openai`)	`openai`
`MATRIC_OLLAMA_URL`	`http://localhost:11434`	Ollama URL (alternative to `OLLAMA_BASE`)	`http://ollama:11434`
`MATRIC_OLLAMA_GENERATION_MODEL`	`qwen3.5:9b`	Ollama generation model	`llama3.2`
`MATRIC_OLLAMA_EMBEDDING_MODEL`	`nomic-embed-text`	Ollama embedding model	`mxbai-embed-large`
`MATRIC_OPENAI_URL`	`https://api.openai.com/v1`	OpenAI URL	`https://custom-proxy.example.com/v1`
`MATRIC_OPENAI_API_KEY`	(none)	OpenAI API key	`<OPENAI_API_KEY>`
`MATRIC_OPENAI_GENERATION_MODEL`	`gpt-4o-mini`	OpenAI generation model	`gpt-4o`
`MATRIC_OPENAI_EMBEDDING_MODEL`	`text-embedding-3-small`	OpenAI embedding model	`text-embedding-3-large`
`MATRIC_GEN_TIMEOUT_SECS`	`120`	Generation request timeout (seconds)	`180`
`MATRIC_EMBED_TIMEOUT_SECS`	`30`	Embedding request timeout (seconds)	`60`

Job Worker

Variable	Default	Description	Example
`WORKER_ENABLED`	`true`	Enable background job worker	`false`

Graph Quality Pipeline

Variable	Default	Description	Example
`GRAPH_NORMALIZATION_GAMMA`	`1.0`	Score normalization exponent applied before SNN	`0.8`
`GRAPH_SNN_K`	(system default)	Nearest neighbors per node for SNN computation	`10`
`GRAPH_SNN_PRUNE_THRESHOLD`	(system default)	Minimum SNN score to retain an edge	`0.1`
`GRAPH_PFNET_Q`	(system default)	PFNET pathfinder metric space parameter	`2`
`GRAPH_COMMUNITY_RESOLUTION`	(system default)	Louvain community granularity (higher = more communities)	`1.0`
`GRAPH_STRUCTURAL_SCORE`	(system default)	Weight for structural vs. similarity scores in edge ranking	`0.5`
`EMBED_CONCEPT_MAX_DOC_FREQ`	(system default)	TF-IDF document frequency cutoff for concept filtering in embeddings	`0.8`

Real-Time Events

Variable	Default	Description	Example
`MATRIC_EVENT_BUS_CAPACITY`	`256`	Event bus broadcast channel capacity	`1024`
`MATRIC_WEBHOOK_TIMEOUT_SECS`	`10`	Webhook HTTP request timeout (seconds)	`30`
`MATRIC_MAX_BODY_SIZE_BYTES`	`2147483648`	Global request-body ceiling (2 GB for database backups); attachment files remain bounded by `MATRIC_MAX_UPLOAD_SIZE_BYTES`	`1073741824`

Full-Text Search (FTS)

Variable	Default	Description	Example
`FTS_WEBSEARCH_TO_TSQUERY`	`true`	Enable websearch syntax (OR, NOT, phrase)	`false`
`FTS_TRIGRAM_FALLBACK`	`true`	Enable trigram search for emoji/symbols	`false`
`FTS_BIGRAM_CJK`	`true`	Enable bigram search for CJK text	`false`
`FTS_SCRIPT_DETECTION`	`true`	Auto-detect query language script	`false`
`FTS_MULTILINGUAL_CONFIGS`	`true`	Enable language-specific text search configs	`false`

Redis Cache

Variable	Default	Description	Example
`REDIS_ENABLED`	`true`	Enable Redis caching for eligible explicit FTS searches	`false`
`REDIS_URL`	`redis://localhost:6379`	Redis connection URL	`redis://redis:6379/0`
`REDIS_CACHE_TTL`	`300`	Eligible FTS result cache TTL in seconds (5 minutes)	`600`

Backup Operations

Variable	Default	Description	Example
`BACKUP_DEST`	`/var/backups/matric-memory`	Backup destination directory	`/mnt/backups`
`BACKUP_SCRIPT_PATH`	`/usr/local/bin/backup-matric.sh`	Path to backup script	`/opt/scripts/backup.sh`
`MAX_MEMORIES`	`10`	Max memory archives (scale per hardware tier: 10/50/200/500)	`50`

PostgreSQL (Bundle Deployment Only)

Variable	Default	Description	Example
`POSTGRES_USER`	`matric`	PostgreSQL superuser	`postgres`
`POSTGRES_PASSWORD`	`matric`	PostgreSQL password	`secure_password`
`POSTGRES_DB`	`matric`	PostgreSQL database name	`matric_prod`

Configuration Precedence

Environment variables are read with the following precedence:

1. System environment variables (highest priority) 2. `.env` file (loaded via `dotenvy::dotenv()`) 3. Hard-coded defaults in `crates/matric-core/src/defaults.rs`

AI Features Configuration

AI features (embedding generation, auto-titling, AI revision) require either Ollama or OpenAI to be configured.

Using Ollama (local/self-hosted):

1. Install and run Ollama on your host machine 2. Pull required models:

   ollama pull nomic-embed-text
   ollama pull qwen3.5:9b

3. Configure Docker to access Ollama:

   # For Docker Desktop (macOS/Windows)
   OLLAMA_BASE=http://host.docker.internal:11434

   # For Linux with Ollama on the same host (compose supplies host-gateway)
   OLLAMA_BASE=http://host.docker.internal:11434

Configure Ollama to listen only on Docker's resolved host-gateway address, as described in Ollama Connectivity. 4. Add to `.env` or uncomment in `docker-compose.bundle.yml`:

   OLLAMA_BASE=http://host.docker.internal:11434
   OLLAMA_EMBED_MODEL=nomic-embed-text
   OLLAMA_GEN_MODEL=qwen3.5:9b

Using OpenAI:

Add to `.env`:

OPENAI_API_KEY=<OPENAI_API_KEY>your-key...
OPENAI_EMBED_MODEL=text-embedding-3-small
OPENAI_GEN_MODEL=gpt-4o-mini

Verify AI Features:

# Create a test note and request embedding generation
curl -X POST http://localhost:3000/api/v1/notes \
  -H "Content-Type: application/json" \
  -d '{"content": "Test note for embedding generation"}'

# Check job queue for embedding job
curl http://localhost:3000/api/v1/jobs?type=embedding

# View logs for embedding/generation errors
docker compose -f docker-compose.bundle.yml logs | grep -i "ollama\|openai\|embedding"

Common AI Issues:

Symptom	Cause	Fix
Embedding jobs stuck	Ollama not reachable	Set `OLLAMA_BASE` env var
Auto-titling not working	No LLM configured	Configure Ollama or OpenAI
"connection refused" errors	Wrong Ollama host	Use `host.docker.internal` for Docker Desktop

Modifying Configuration

# Edit .env for external URLs
nano .env

# Edit docker-compose for container settings
nano docker-compose.bundle.yml

# Apply changes
docker compose -f docker-compose.bundle.yml down
docker compose -f docker-compose.bundle.yml up -d

Search Degradation Monitoring

Semantic and hybrid search preserves availability during an embedding-provider failure by executing FTS and returning `degraded=true`. Each fallback also emits a `search.embedding_degraded` audit event and a warning log with bounded, redacted metadata:

stable `reason_code`;
requested and effective search modes;
provider and model lengths plus a model fingerprint;
configured, backend-declared, and actual vector dimensions when known;
whether an embedding set was requested.

The event does not contain query text, archive names, provider URLs, API keys, or raw model identifiers. Common reason codes are `embedding_backend_unavailable`, `embedding_request_failed`, `embedding_response_empty`, `embedding_dimension_mismatch`, and `embedding_contract_unavailable`.

Treat sustained events as an inference-routing incident. Verify the active embedding provider, the target set's embedding configuration, provider reachability, and vector dimensions. Degraded responses bypass the search cache, so provider recovery takes effect on the next request.

Embedding jobs resolve their provider at execution time rather than retaining one startup backend. Stored embedding rows include `contract_fingerprint` for freshness and reindex decisions. Legacy rows may have a null fingerprint until they are regenerated. Cardinality, response-index, model, and dimension mismatches fail the job before the replacement transaction, preserving the previous vectors.

Multi-Memory Operations

Fortemi memory archives use separate schemas. Select both the tenant and memory for operator reads; schema selection alone does not supply tenant context.

Schema Monitoring

To list authorized memory schemas, run this query inside the same read-only, tenant-bound transaction described above:

SELECT name, schema_name, is_default, schema_version
FROM public.archive_registry ORDER BY created_at;

To compare table counts, use the same transaction:

SELECT ar.name, ar.schema_version,
  (SELECT count(*) FROM information_schema.tables
   WHERE table_schema = ar.schema_name) AS actual_tables
FROM public.archive_registry ar;

Database size, catalog relation sizes, and connection counts are administrative metrics; they may span tenants and are not tenant-scoped note counts.

Per-Memory Backup

Schema-only dumps are maintenance artifacts and can omit shared records and attachment bytes. Use the backup procedure with the approved backup identity for recovery. Do not loop over an unscoped `archive_registry` query or treat a schema-only dump as a complete memory backup.

Per-Memory Maintenance

# Vacuum specific memory schema
docker exec fortemi-matric-1 psql -U matric -d matric -c "
SET search_path TO archive_work_2026, public;
VACUUM ANALYZE note; VACUUM ANALYZE embedding;"

# Reindex specific memory
docker exec fortemi-matric-1 psql -U matric -d matric -c "
REINDEX SCHEMA archive_work_2026;"

Troubleshooting Multi-Memory

Issue	Cause	Resolution
"relation does not exist"	Schema auto-migration hasn't run	Access memory to trigger migration
Schema version mismatch	Check `archive_registry.schema_version` vs actual table count	Review migration logs
"Memory not found" 404	Check `X-Fortemi-Memory` header value matches `archive_registry.name` exactly	Header value is case-sensitive
Performance degradation with many schemas	Each schema adds minimal overhead	VACUUM must run per-schema

Quick Reference

Daily Operations

# Check status
docker compose -f docker-compose.bundle.yml ps
curl http://localhost:3000/health

# View logs
docker compose -f docker-compose.bundle.yml logs --tail=50

Weekly Maintenance

# Vacuum database
docker exec Fortémi-matric-1 psql -U matric -d matric -c "VACUUM ANALYZE;"

# Backup
docker exec Fortémi-matric-1 pg_dump -U matric matric > backup_$(date +%Y%m%d).sql

# Trigger graph quality maintenance (normalize → SNN → PFNET → Louvain → diagnostics)
curl -X POST http://localhost:3000/api/v1/graph/maintenance

Emergency Procedures

# Quick restart
docker compose -f docker-compose.bundle.yml restart

# Full restart
docker compose -f docker-compose.bundle.yml down
docker compose -f docker-compose.bundle.yml up -d

# Restore from backup
docker exec -i Fortémi-matric-1 psql -U matric -d matric < latest_backup.sql

Resources

Repository: https://github.com/fortemi/fortemi
Operators Guide: operators-guide.md
MCP Documentation: mcp-server/README.md
Real-Time Events: real-time-events.md