Release Process

CalVer versioning, tagging, and release workflow.

Release Process

This document describes the release process for Fortémi. The executable release policy is `.aiwg/release.config`; stable tags must use `tools/release/cut-tag.sh` and its distinct OpenBao-backed release signing key.

Versioning

Fortémi uses CalVer (Calendar Versioning):

  • Format: `YYYY.M.PATCH`
  • Examples: `2026.1.0`, `2026.1.5`, `2026.12.0`
  • No leading zeros - npm/cargo semver rejects them
  • PATCH resets to 0 each month
  • Git tags use `v` prefix: `v2026.1.0`

Release Checklist

Pre-Release

  • [ ] A green release validation run: ci-builder `workflow_dispatch` on the exact

candidate commit on `main`, plus its comprehensive `test.yml` handoff. Record both run IDs and the candidate SHA in the release PR/notes before `tools/release/cut-tag.sh`.

  • [ ] A green `docsite-build` `workflow_dispatch` on the same candidate (strict

links and asset checks). Ordinary main pushes do not build the docsite; the tag-only chain reaches `docsite-deploy` only after release.

  • [ ] Linting clean: `cargo clippy --all-targets --all-features -- -D warnings`
  • [ ] Format check: `cargo fmt --all -- --check`
  • [ ] Dependency policy: `cargo deny check advisories bans licenses sources`
  • [ ] Documentation contract: `DOCS_CONTRACT_MODE=blocking npm run docs:contract -- --profile=hosted_strict`
  • [ ] External issuer tests: `scripts/test-mcp-external-issuer.sh` and

`scripts/test-keycloak-reference-realm.sh` pass with disposable Docker stacks. The Keycloak test imports the shipped realm and claim policy, checks PKCE/client credentials, audiences, role ceilings, MCP and metadata, and rejects secret leakage in logs.

  • [ ] CI pipeline green on main branch
  • [ ] Documentation updated for new features
  • [ ] No critical open issues blocking release
  • [ ] If the release touches the Intel overlay or provider routing:

`./scripts/smoke-intel-vllm.sh` passes (see Intel Arc / vLLM Deployment)

Dispatch release validation

After the version bump is merged, fetch main and dispatch the exact candidate. Use a Gitea token with workflow-dispatch permission (do not record the token):

git fetch origin main
CANDIDATE_SHA=$(git rev-parse origin/main)
jq -n --arg sha "$CANDIDATE_SHA" \
  '{ref: "main", inputs: {source_sha: $sha}}' \
  > /tmp/fortemi-release-validation.json
curl --fail-with-body --silent --show-error -X POST \
  -H "Authorization: token ${GITEA_TOKEN:?Set a workflow-dispatch token}" \
  -H 'Content-Type: application/json' \
  'https://git.integrolabs.net/api/v1/repos/Fortemi/fortemi/actions/workflows/ci-builder.yaml/dispatches' \
  --data-binary @/tmp/fortemi-release-validation.json

Dispatch the docsite build on the same candidate (it accepts no inputs, so run it while `main` is still the candidate commit):

curl --fail-with-body --silent --show-error -X POST \
  -H "Authorization: token ${GITEA_TOKEN:?Set a workflow-dispatch token}" \
  -H 'Content-Type: application/json' \
  'https://git.integrolabs.net/api/v1/repos/Fortemi/fortemi/actions/workflows/docsite-build.yml/dispatches' \
  --data '{"ref":"main"}'

Alternatively, open Actions → CI → Run workflow, select `main`, enter the full 40-character candidate commit in `source_sha`, and run it. In Actions, record the ci-builder run ID/URL and the comprehensive Tests run ID/URL that its handoff creates with `test_category=all` and the same `source_sha`. Wait for both runs to finish green. Record `candidate SHA`, `ci-builder run ID/result`, and `test.yml run ID/result` in the release PR/notes; an accepted dispatch or a skipped run is not validation.

The stale-source guard requires the candidate to remain the current main tip at admission and at handoff. If main advances, validate the new candidate again; a green superseded run with a skipped handoff does not satisfy the release gate. `delivery.require_ci_green` also applies to post-merge main CI and its test handoff. Fix red main forward on a new branch.

A main dispatch runs lint, audits, the Rust release build and tests, the API/hosted images, container checks, kind Pod Security and GPU integration, then hands off to the comprehensive Tests run. It publishes nothing: the publication, signing, GHCR verification and finalization jobs are tag-only.

Some release build steps still run only on tags, inside the publication jobs: the complete knowledge-shard matrix with remote verification, multiarch API/hosted/MCP/bundle builds, docs-shard regeneration on the release image, arm64 health and both-architecture hosted smoke checks, and per-platform SBOMs. Until the no-publish dry-run mode in Fortemi/fortemi#1251 lands, those are first exercised by the tag run; a failure there requires a new patch version.

Version Bump

1. Set the workspace version in `Cargo.toml` and refresh the corresponding workspace package entries in `Cargo.lock`. 2. Set the same version in `mcp-server/package.json` and `mcp-server/package-lock.json`. MCP version alignment is required for every server release. Set `version` and `appVersion` in `deploy/helm/fortemi/Chart.yaml` to the same version; the Helm chart lint job fails when they differ. 3. Regenerate and check the AsyncAPI YAML/checksum, then immediately regenerate and check the event corpus, including metadata-only version bumps:

   scripts/ci/asyncapi-contract.sh generate
   scripts/ci/asyncapi-contract.sh check
   scripts/ci/asyncapi-event-fixtures.sh generate
   scripts/ci/asyncapi-event-fixtures.sh check

Commit the YAML, checksum, fixture manifest, producer receipt and all generated event fixtures together. The required core tests check the committed corpus's exact bytes and inventory against the raw YAML, runtime schemas and typed events. A stale corpus fails even when all 48 payload files are unchanged. 4. Move the selected `CHANGELOG.md` entries into the dated release section and update comparison links. Add `docs/releases/vYYYY.M.PATCH-announcement.md`.

Create Release

Commit the reviewed release files with the configured OpenBao-backed commit signing authority. Ordinary commits and release tags use distinct keys:

git add Cargo.toml Cargo.lock mcp-server/package.json mcp-server/package-lock.json \
  deploy/helm/fortemi/Chart.yaml \
  contracts/asyncapi/asyncapi.yaml contracts/asyncapi/asyncapi.sha256 \
  contracts/asyncapi/fixtures/manifest.json contracts/asyncapi/fixtures/events/ \
  contracts/asyncapi/producer-event-fixture-receipt.json \
  CHANGELOG.md docs/releases/vYYYY.M.PATCH-announcement.md
git -c user.signingkey=62297562B1C7053088F405DB0117DAAA677A5BF2 \
  -c gpg.program=tools/git/gpg-from-openbao.sh commit -S \
  -m "chore: release vYYYY.M.PATCH"
git push origin main

After both recorded release validation runs pass, use a clean checkout at the exact current `origin/main` commit. If the working checkout contains unrelated files, use an isolated clean checkout and preserve the original work. The tag wrapper checks workspace/MCP versions, changelog, announcement, clean status and `origin/main`, then signs and verifies with the published release key:

tools/release/cut-tag.sh YYYY.M.PATCH -m "vYYYY.M.PATCH - Release title"
git push origin vYYYY.M.PATCH

Then push the same commit and tag to the GitHub backup mirror:

git push github main vYYYY.M.PATCH

`core.sshCommand` is set to `tools/git/ssh-route.sh`. It sends `origin` (Gitea) traffic through the OpenBao-held deploy key and `github` traffic through the operator's own GitHub SSH keys, so neither remote needs per-command overrides. The routing values live in the local `aiwg.vault.*` Git config and are listed in `.aiwg/aiwg.config` under `remotes.transport`. Run `tools/git/push-origin.sh --check` to confirm the Gitea key resolves and is accepted before pushing.

The wrapper accepts stable CalVer versions. It does not replace CI, lockfile or publication verification. Do not bypass it with a raw annotated tag or publish unrelated local tags. A failed gate leaves the release incomplete.

Shared-database integration and coverage test cases run serially. Hosted-role checks inspect the entire archive catalog, and schema-changing fixtures and child processes cannot be isolated by separate Rust mutexes. Tests that issue concurrent requests internally still exercise that concurrency. Run reproductions against a fresh disposable database; never point them at an operational database.

After a pushed tag fails a required gate, preserve it and its failure evidence. Fix and validate the cause, then cut a new patch version. Do not move the old tag or waive coverage to finish publication.

CI/CD Automation

When you push the tag, the CI pipeline automatically:

1. Runs all tests (lint, unit tests, integration tests, container tests) 2. Publishes Docker images to both registries:

  • Internal: `ghcr.io/fortemi/fortemi:{version}`
  • Public: `ghcr.io/fortemi/fortemi:{version}`

3. Creates releases on both Gitea and GitHub with:

  • Changelog extracted from `CHANGELOG.md`
  • Docker installation instructions
  • Quick start commands

Docker image tags published:

  • `{version}` - Specific version (e.g., `2026.2.0`)
  • `latest` - Latest stable release
  • `bundle-{version}` - All-in-one image with embedded PostgreSQL
  • `bundle-latest` - Latest bundle image

Mutable `latest` tags are convenience aliases only. Release verification must resolve and record immutable digest references for versioned images:

VERSION=2026.6.1
docker pull ghcr.io/fortemi/fortemi:${VERSION}
docker pull ghcr.io/fortemi/fortemi:bundle-${VERSION}
docker image inspect ghcr.io/fortemi/fortemi:${VERSION} --format '{{index .RepoDigests 0}}'
docker image inspect ghcr.io/fortemi/fortemi:bundle-${VERSION} --format '{{index .RepoDigests 0}}'

Use `ghcr.io/fortemi/fortemi@sha256:...` references in production deployment records and rollback plans. The release jobs upload registry-derived receipts that bind version and mutable tags to these digests. Verify and retain them as described in Container Release Evidence.

The Gitea and GitHub release entries are finalized together only after the tag-only `verify-ghcr-release` job repeats those checks with an anonymous Docker configuration and verifies the published image revision/version labels. This checks public container availability and keeps both entries under the same container publication gate. Comprehensive `test.yml` runs afterward and gates native binary publication. An entry and images can therefore exist while the release remains incomplete. Require both configured workflows to pass and verify the native assets, checksums and provenance before declaring release completion.

Sidecar images (GLiNER, pyannote) are released independently with their own tags. See CI/CD docs for details.

Dispatch versus tag job audit (#1250)

The ci-builder push trigger selects `main` only. A `v*` push first runs `build-builder.yaml`, which dispatches ci-builder with the tag ref and immutable `source_sha`; the tag column below describes that chained ci-builder run. These job guards are unchanged by #1250. All jobs also require successful `needs` unless their guard explicitly uses `always()`.

On a main dispatch without `source_sha`, `CHAIN_SOURCE_SHA` is the dispatch's `github.sha`. With it, that exact commit is fetched. Both take the same job path if the SHA is still main's tip; an old SHA fails admission as superseded, and heavy jobs/handoff skip. Without `source_sha` the dispatch uses the shared concurrency group; with it, it uses the per-run group (#1234).

Guards (verbatim; G0 denotes the implicit default):

  • G0: `success() (implicit)`
  • G1: `needs.admit-source.outputs.current == 'true' && (github.event_name != 'pull_request')`
  • G2: `needs.admit-source.outputs.current == 'true'`
  • G3: `startsWith(github.ref, 'refs/tags/v')`
  • G4: `startsWith(github.ref, 'refs/tags/v') && needs.publish-release.result == 'success' && needs.publish-github.result == 'success'`
  • G5: `startsWith(github.ref, 'refs/tags/v') && needs.publish-github.result == 'success'`
  • G6: `startsWith(github.ref, 'refs/tags/v') && needs.publish-release.result == 'success' && needs.publish-github.result == 'success' && needs.verify-ghcr-release.result == 'success' && needs.sign-release-images.result == 'success'`
  • G7: `needs.admit-source.outputs.current == 'true' && (github.ref == 'refs/heads/main' && github.event_name != 'pull_request')`
  • G8: `needs.admit-source.outputs.current == 'true' && (always() && github.event_name != 'pull_request' && ((github.ref == 'refs/heads/main' && needs.main-validation.result == 'success') || (startsWith(github.ref, 'refs/tags/v') && needs.finalize-releases.result == 'success')))`
  • G9: `always()`
ci-builder jobGuardMain dispatch, with/without current `source_sha`Tag chain
`runner-capacity`G0YesYes
`verify-release-ref`G0Yes; signature check is a no-opYes; verifies signed tag
`sync-github-source`G1YesYes
`admit-source`G0Checks current main tipAdmitted (no main-tip check)
`build-testdb`G2YesYes
`lint`G2YesYes
`knowledge-shard-matrix`G2YesYes
`helm-chart`G2YesYes
`audit`G2YesYes
`deny`G2YesYes
`auth-consumer`G2YesYes
`mcp-lockfile-sync`G2YesYes
`mcp-server-tests`G2YesYes
`build`G2YesYes
`build-image`G2YesYes
`hosted-image-smoke`G2YesYes
`pod-security-restricted`G2YesYes
`test-container`G2YesYes
`integration-test`G2YesYes
`publish-release`G3SkippedYes
`publish-github`G3SkippedYes
`sign-release-images`G4SkippedYes
`verify-ghcr-release`G5SkippedYes
`finalize-releases`G6SkippedYes
`main-validation`G7YesSkipped
`handoff-tests`G8After main-validationAfter finalize-releases
`cleanup-run-images`G9AlwaysAlways

The comprehensive main handoff runs `runner-capacity`, `admit-source`, `build-testdb`, `fast-tests`, `integration-tests`, `coverage`, `slow-tests`, `validate-intel-overlay`, `test-summary` and cleanup. Its `handoff-sidecar` is strictly tag-only. Native Linux/macOS release-asset builds therefore still run only in the tag chain; they need their dedicated native runners and publication handoff and are not represented as pre-tag native-platform qualification.

Post-Release

  • [ ] Verify the canonical Gitea release and required assets
  • [ ] Verify the mirrored release appears on GitHub Releases
  • [ ] Verify Docker images on ghcr.io
  • [ ] Record immutable Docker digest references for `{version}` and `bundle-{version}`
  • [ ] Update any deployment configurations
  • [ ] Pull new image and restart production service if needed:
  docker pull ghcr.io/fortemi/fortemi:bundle-${VERSION}
  docker compose -f docker-compose.bundle.yml up -d
  • [ ] Smoke test production endpoints:
  curl http://localhost:3000/health

Release Documentation

Each release should have documentation in these locations:

LocationPurpose
`CHANGELOG.md`Technical changelog with highlights table
Gitea ReleasePublic release notes with install instructions

CHANGELOG.md Format

## [YYYY.M.PATCH] - YYYY-MM-DD

### Highlights

| What Changed | Why You Care |
|--------------|--------------|
| Feature A | Benefit description |

### Added
- New features

### Changed
- Changes to existing features

### Fixed
- Bug fixes

### Deprecated
- Features to be removed

### Removed
- Removed features

### Security
- Security fixes

Hotfix Releases

For urgent fixes:

1. Create fix on main branch 2. Increment PATCH: `2026.1.0` → `2026.1.1` 3. Abbreviated changelog entry 4. Follow normal release process

Monthly Releases

For planned releases at month boundaries:

1. Reset PATCH to 0 2. New month number: `2026.1.5` → `2026.2.0` 3. Comprehensive changelog entry

Breaking Changes

For breaking API changes:

1. Document in CHANGELOG.md under dedicated section 2. Provide migration guide in release notes 3. Consider deprecation period before removal 4. Update API version path if major incompatibility

Rollback Procedure

If a release has critical issues:

# 1. Revert to previous version in production
git checkout v2026.1.0  # previous version
cargo build --release
sudo systemctl restart matric-api

# 2. Create hotfix release
# Follow hotfix procedure above

Authentication Migration Guide

When enabling authentication on an existing deployment:

Pre-Migration Checklist

  • [ ] Deploy latest version with `REQUIRE_AUTH=true` (default)
  • [ ] Register OAuth clients for all integrations: `POST /oauth/register`
  • [ ] Create API keys for CLI/automation: `POST /api/v1/api-keys`
  • [ ] Distribute credentials to all clients/users
  • [ ] Test authentication with a sample request:
  curl -H "Authorization: Bearer <ACCESS_TOKEN>" http://localhost:3000/api/v1/notes

Enable Authentication

1. Set `REQUIRE_AUTH=true` in `.env` 2. Restart: `docker compose -f docker-compose.bundle.yml up -d` 3. Verify public endpoints still work: `curl http://localhost:3000/health` 4. Verify auth is enforced: `curl http://localhost:3000/api/v1/notes` (should return 401)

Rollback

For local sidecar/dev rollback only, set both `REQUIRE_AUTH=false` and `I_UNDERSTAND_NO_AUTH=true` in `.env` and restart to disable auth.

Automation Status

Current automation (implemented):

  • [x] Docker image publishing on tag push (ghcr.io and internal registry)
  • [x] Multi-architecture (`linux/amd64`, `linux/arm64`) API, MCP and bundle images
  • [x] GitHub Release creation with changelog
  • [x] Gitea Release creation with changelog

Planned improvements:

  • [ ] Automatic CHANGELOG generation from commits
  • [ ] Release notification webhooks