CI/CD Pipeline

Gitea Actions workflow and deployment automation.

CI/CD Pipeline

Overview

Fortémi uses Gitea Actions for continuous integration and deployment. The pipeline is configured in `.gitea/workflows/ci-builder.yaml` and runs on self-hosted runners.

Release targets:

Internal: Container registry - Development builds
Public: GitHub Container Registry (`ghcr.io/fortemi/fortemi`) - Production releases

The documentation site has separate Pagenary workflows: `.gitea/workflows/docsite-build.yml` validates docs changes, and `.gitea/workflows/docsite-deploy.yml` publishes `docs.fortemi.com/server` on release tags or manual dispatch. See Documentation Site Publishing for the package version, tenant registry, local commands, and deployment details.

Self-Hosted Runner Security

The `matric-builder` runner controls the host Docker daemon through its Unix socket, so it should be treated as privileged access to the runner host. It is the default label for build, test, container, and publish jobs. The `titan` and `gpu` labels are reserved for hardware or local-service access and should not be used for publish jobs unless the workflow has a specific host-exposure review. Socket-backed jobs run only on a dedicated trusted runner host or VM; a read-only socket mount does not constrain Docker API control, and untrusted fork workloads must not share that runner. Rootless Docker, a mutually authenticated remote BuildKit service, or an ephemeral VM is preferred when the job does not need the host daemon.

Secret-bearing jobs must not run for `pull_request` events. The CI lint job runs `scripts/ci/verify-release-job-guards.py`, which fails if a workflow that accepts pull requests contains a `${{ secrets.* }}` job without a job-level guard excluding pull-request execution. Keep the explicit guards in the workflow even when a job also appears to be protected by branch or tag conditions.

Package and registry publish posture:

Internal Gitea registry/package publishing uses `BUILD_REPO_TOKEN`. Keep this

token limited to Fortemi package and release publishing, and do not reuse it as a general administrator token.

Public GHCR and GitHub release publishing uses `GH_PUBLISH_TOKEN`. The

expected minimum scopes are `write:packages` and `contents:write`; private repository targets may also require repository access.

Runner registration tokens, PATs, deploy keys, and registry tokens must stay

out of the repository and out of runner labels/config examples. Rotate any token that appears in logs or shell history.

Package-distribution readiness is not proven by these docs alone. A release

gate still needs a publish-and-consume verification run for the target package registry before enterprise/private package readiness is claimed.

See `build/RUNNER_SETUP.md` for runner registration, Docker-socket isolation, and label configuration details. This is an internal CI trust boundary, not guidance for the user-facing Fortemi bundle; bundle daemon access remains separately governed by issue #937.

Pipeline Stages

1. Lint (runs on: matric-builder)

Validates code quality and formatting:

- cargo fmt --all -- --check      # Enforce consistent formatting
- cargo clippy --all-targets --all-features -- -D warnings  # Catch common mistakes

Triggers: On push to main, on pull requests to main, on version tags (`v*`)

Exit criteria: All formatting rules pass, no clippy warnings

The lint job also rejects tracked editor/source backup artifacts ending in `.bak`, `.orig`, or `~`. Recover prior source through git history instead of keeping stale copies beside active code. Any intentional compatibility fixture must be an exact path in `scripts/ci/source-backup-artifacts.allowlist` with a rationale. The check uses tracked paths, not broad `backup` keywords, so product backup code and runtime database backups in operator data directories are outside this rule; runtime outputs must still never be committed as repository source.

2. Build & Unit Test (runs on: matric-builder)

Compiles and runs the test suite with a dedicated PostgreSQL container (`build/Dockerfile.testdb`). The testdb image includes `max_locks_per_transaction=256` to support parallel archive schema tests (each archive create/drop acquires locks on ~41 tables + indexes).

- cargo build --release --workspace
- cargo test --package matric-jobs --test worker_integration_test -- --test-threads=1
- cargo test --workspace --exclude matric-jobs
- cargo test --doc

Dependencies: Requires `lint` job to pass

Test Coverage:

matric-core (unit tests)
matric-db (repository tests with real PostgreSQL)
matric-search (hybrid search tests)
matric-inference (mock tests; real tests in integration-test job)
matric-jobs (worker integration tests, run serially)
matric-api (API endpoint tests)

3. Build Docker Image (runs on: matric-builder)

Creates the Docker image for testing:

- docker build -t Fortémi:test .

Dependencies: Requires `build` job to pass

4. Test Container (runs on: matric-builder)

Deploys the built image in an isolated Docker network and runs API tests:

Starts PostgreSQL with pgvector
Starts Redis for caching
Runs database migrations
Starts the API container
Executes `scripts/container-api-tests.sh`

Dependencies: Requires `build-image` job to pass

5. Integration Tests (runs on: titan)

Tests AI/ML functionality with Ollama (GPU-enabled):

- nvidia-smi                    # Verify GPU access
- curl http://localhost:11434/api/tags  # Verify Ollama available
- cargo test --package matric-inference --features integration

Dependencies: Requires `build` job to pass

Environment:

OLLAMA_HOST: http://localhost:11434
MATRIC_INFERENCE_DEFAULT: ollama
GPU access for embedding generation

Timeout: 30 minutes

6. Main Validation and Test Handoff (runs on: matric-builder)

Ordinary `main` pushes stop after the local container, GPU, security, contract, and unit-test gates pass. They do not authenticate to either container registry and do not build or publish remote development images.

After the validation graph succeeds, `main-validation` dispatches the comprehensive test workflow at the exact verified commit. Registry publication is reserved for versioned release refs.

7. Publish Release (runs on: matric-builder)

Publishes release images to internal registry on version tags:

Tags published:

`{version}` - Semantic version (e.g., `2026.2.0`)
`latest` - Latest stable release
`bundle-{version}`, `bundle-latest` - All-in-one images

`latest` and `bundle-latest` are mutable convenience aliases. Release verification records immutable digest references from the versioned tags:

VERSION=2026.6.1
docker pull ghcr.io/fortemi/fortemi:${VERSION}
docker pull ghcr.io/fortemi/fortemi:bundle-${VERSION}
docker image inspect ghcr.io/fortemi/fortemi:${VERSION} --format '{{index .RepoDigests 0}}'
docker image inspect ghcr.io/fortemi/fortemi:bundle-${VERSION} --format '{{index .RepoDigests 0}}'

Production deployments should pin `ghcr.io/fortemi/fortemi@sha256:...` references in deployment records instead of relying on mutable tag values.

Triggers: Version tags only (`v*`)

Note: Sidecar images (GLiNER, pyannote) are released independently — see Sidecar Image Workflows below.

8. Publish to GitHub (ghcr.io)

Publishes release images to GitHub Container Registry for public distribution:

IMAGE: ghcr.io/fortemi/fortemi
Tags: {version}, latest, bundle-{version}, bundle-latest

Dependencies: Requires `test-container` and `integration-test` jobs to pass

Triggers: Version tags only (`v*`)

9. Finalize Gitea and GitHub Releases

Creates the internal Gitea and public GitHub releases in one finalizer with:

Changelog extracted from `CHANGELOG.md`
Installation instructions for Docker
Quick start commands

Before release creation, `verify-ghcr-release` uses a clean Docker client with no registry credentials to prove both versioned images are publicly pullable. It also verifies that `latest` aliases resolve to the versioned digests, the API and bundle platform sets match policy, and both amd64 images carry the release commit and version labels.

Dependencies: Requires `publish-release`, `publish-github`, and `verify-ghcr-release` to pass. Consolidating both release entries into one tag-only job prevents branch builds from scheduling redundant release tasks.

Sidecar Image Workflows

GLiNER and pyannote sidecar images are released independently from the main Fortémi images. They change infrequently and are expensive to build (ML model downloads), so they have their own workflows.

The native `matric-api` desktop sidecar uses `.gitea/workflows/publish-sidecar.yml`. Every main-branch build publishes an append-only release named `sidecar-<12-char-commit>`, with the three platform binaries, `SHA256SUMS.txt`, and an in-toto/SLSA provenance statement. The workflow verifies an existing immutable identity and fails if its target, checksums, provenance, or asset bytes differ.

`sidecar-latest` is a mutable discovery pointer. Consumers must resolve it to the immutable commit-qualified release, pin that release URL, and verify the checksum manifest and provenance statement. A rolling URL is not a trust anchor and may legitimately serve different bytes after a new main build.

The committed documentation seed at `docker/seed-data/fortemi-docs.shard` has a sibling provenance receipt. CI runs `scripts/ci/verify-docs-shard-freshness.py` to verify the archive digest, byte length, manifest version, server image, and workspace release baseline. When `scripts/ci/rebuild-shard-in-ci.sh` is used to refresh the seed, update the receipt in the same commit and propagate the exact server-generated artifact to downstream conformance suites.

build-gliner.yaml

Trigger	Tags
`sidecar-gliner-v*` tag push	`:gliner`, `:gliner-latest`, `:gliner-{version}`
Manual (`workflow_dispatch`)	Runs only when dispatched against a matching release tag

build-pyannote.yaml

Trigger	Tags
`sidecar-pyannote-v*` tag push	`:pyannote`, `:pyannote-latest`, `:pyannote-{version}`
Manual (`workflow_dispatch`)	Runs only when dispatched against a matching release tag

Releasing a sidecar

# Release GLiNER version 1
git tag -a sidecar-gliner-v1 -m "sidecar-gliner-v1: update model to gliner_large-v2.1"
git push origin sidecar-gliner-v1

# Release pyannote version 1
git tag -a sidecar-pyannote-v1 -m "sidecar-pyannote-v1: initial GHCR release"
git push origin sidecar-pyannote-v1

Both workflows push to the internal Gitea registry and GHCR simultaneously. Every image publisher then reads the manifest back from each registry and uploads a digest receipt. See Container Release Evidence for the family matrix, control status, artifact format, and verification commands.

Self-Hosted Runners

The pipeline uses two self-hosted runners:

matric-builder (host-daemon-backed)

Lint, Build, Test
Docker image builds
Container testing
Registry publishing

titan (GPU-enabled)

Integration tests with Ollama
Requires GPU for embedding generation

Secrets Required

BUILD_REPO_TOKEN

Used for internal container registry authentication:

echo "${{ secrets.BUILD_REPO_TOKEN }}" | docker login $REGISTRY -u ${{ gitea.actor }} --password-stdin

GH_PUBLISH_TOKEN

Used for GitHub Container Registry and GitHub Releases. Create a GitHub Personal Access Token (classic) with these scopes:

`write:packages` - Push images to ghcr.io
`contents:write` - Create releases
`repo` - Full repository access (for private repos)

echo "${{ secrets.GH_PUBLISH_TOKEN }}" | docker login ghcr.io -u fortemi --password-stdin

Setup instructions: 1. Go to GitHub → Settings → Developer settings → Personal access tokens → Tokens (classic) 2. Generate new token with required scopes 3. Add to Gitea repository → Settings → Secrets → Add secret named `GH_PUBLISH_TOKEN` 4. The PAT must belong to a member of the `fortemi` GitHub organization

Triggers

Push to main

on:
  push:
    branches: [main]

Runs full pipeline including internal dev publish. Does NOT publish to GitHub.

Pull Requests

on:
  pull_request:
    branches: [main]

Runs lint, build, test-container, and integration-test. Does NOT publish.

Version Tags

on:
  push:
    tags: ['v*']

Runs full pipeline including:

Internal registry publish (release images)
Gitea release creation
GitHub Container Registry publish
GitHub release creation

Release Process

1. Update the workspace and MCP package versions, lockfiles, changelog and release announcement. Regenerate contract metadata and the documentation seed from the matching API build. 2. Commit with the commit-signing key and deliver to `main`; wait for that exact source's CI and comprehensive tests to pass. 3. From a clean checkout of current `origin/main`, run `tools/release/cut-tag.sh YYYY.M.PATCH --dry-run`, then `tools/release/cut-tag.sh YYYY.M.PATCH -m "release message"`. 4. The helper selects the OpenBao-custodied release key and runs `tools/ci/verify-signed-tag.sh`. The required release fingerprint is `9292EFCBB0EA41BECEEFDAFA9C1B8CE0E0E09C33`; a valid signature from the commit key does not satisfy this gate. 5. Push only the verified tag: `git push origin refs/tags/vYYYY.M.PATCH`. 6. Wait for the builder, tag validation, registry publication, comprehensive tests, native binary/mirror publication and suite-platform gates. Independently verify immutable image digests and native checksums/provenance. An existing tag or release entry alone does not establish completion.

Preserve failed published tags. Correct the problem in a new patch release; never move an immutable release tag to replace rejected evidence. Version 2026.9.8 was rejected for using the commit key; 2026.9.9 is its corrective release.

Local Development

To match CI checks locally:

# Format code
cargo fmt --all

# Check formatting
cargo fmt --all -- --check

# Run clippy
cargo clippy --all-targets --all-features -- -D warnings

# Run tests
cargo test --workspace
cargo test --doc

# Build release
cargo build --release --workspace

Docker Images

Four image variants are published on release:

API-only (`fortemi/fortemi:{version}`)

Requires external PostgreSQL database
Suitable for Kubernetes/container orchestration
Smaller image size

Bundle (`fortemi/fortemi:bundle-{version}`)

All-in-one with embedded PostgreSQL
No external dependencies
Suitable for quick starts and single-node deployments

GLiNER (`fortemi/fortemi:gliner-{version}`)

Zero-shot named entity recognition sidecar
CPU-only, no GPU required
Also built on `build/gliner/**` changes via `build-gliner.yaml`

pyannote (`fortemi/fortemi:pyannote-{version}`)

Speaker diarization sidecar
GPU-accelerated (CPU fallback available)
Requires HuggingFace token for gated model download
Also built on `build/pyannote/**` changes via `build-pyannote.yaml`

Troubleshooting

Clippy Warnings Block CI

If clippy fails, the pipeline stops. Fix locally:

cargo clippy --all-targets --all-features -- -D warnings
# Fix all warnings, then commit

Formatting Issues

cargo fmt --all
git add .
git commit -m "style: apply cargo fmt formatting"

Integration Tests Timeout

GPU runner has 30-minute timeout. If Ollama tests exceed this:

Check Ollama service health on titan runner
Verify model availability: `curl http://localhost:11434/api/tags`
Check for GPU memory issues: `nvidia-smi`

GitHub Push Fails

Verify the `GH_PUBLISH_TOKEN` secret:

Token must have `write:packages` scope
Token must not be expired
Token owner must be a member of the `fortemi` organization with push access

GitHub Release Creation Fails

Check if release already exists (HTTP 422 response)
Verify token has `contents:write` or `repo` scope
Check GitHub API rate limits

Future Enhancements

[ ] Add test coverage reporting
[ ] Cache cargo dependencies between runs
[ ] Multi-architecture Docker builds (arm64, armv7)
[ ] Security scanning (cargo-audit, trivy)
[ ] Automatic SBOM generation