Documentation

validate

Checks fixture YAML structure and schema without executing any cases. Exits 0 on success, 2 on violations. With no paths provided, scans the current directory (respecting .agentcarousel-ignore).

# Validate a single fixture
agc validate fixtures/my-agent/cases.yaml

# Strict mode — warnings become errors (recommended in CI)
agc validate fixtures/my-agent/cases.yaml --strict

# Emit SARIF 2.1.0 for GitHub code scanning
agc validate --format sarif > results.sarif

# Validate all fixtures in the project
agc validate

test

Runs cases with mock generation. No live LLM calls and no API keys required. Results are written to the local history database. Mock mode is the default.

# Run all fixtures (mock mode, default)
agc test fixtures/my-agent/cases.yaml

# Filter to smoke cases only
agc test fixtures/my-agent/cases.yaml --filter-tags smoke

# Use a custom mock directory
agc test fixtures/my-agent/cases.yaml --mock-dir mocks/

# Stop on the first case failure
agc test fixtures/my-agent/cases.yaml --fail-fast

# Machine-readable output for CI
agc test fixtures/my-agent/cases.yaml --format json

eval

Runs cases with optional judge scoring. Defaults to mock mode (no API keys needed). Switch to --execution-mode live to call a real LLM. Multiple runs (--runs N) let you sample variance and detect flakiness.

# Mock eval (default — no API keys needed)
agc eval fixtures/my-agent/cases.yaml

# Live eval with LLM-as-judge
export ANTHROPIC_API_KEY=...
export OPENAI_API_KEY=...
agc eval fixtures/my-agent/cases.yaml \
  --execution-mode live \
  --model gpt-4o \
  --judge \
  --judge-model claude-haiku-4-5-20251001 \
  --evaluator all \
  --runs 3

# Filter to judge-tagged cases only
agc eval fixtures/my-agent/cases.yaml \
  --evaluator judge --judge --filter-tags judge

# Custom HTTP endpoint (non-standard model providers)
agc eval fixtures/my-agent/cases.yaml \
  --execution-mode live --model custom \
  --generator-endpoint http://localhost:8080/v1/chat

lint

Checks fixture quality beyond schema validation: smoke-tag coverage, judge-case descriptions, rubric weight sums, and bundle compliance fields. Exits 0 when no issues are found; 2 on errors.

# Check all fixtures
agc lint fixtures/

# Check a single file
agc lint fixtures/my-agent/cases.yaml

# Exit non-zero on warnings too (useful in CI)
agc lint fixtures/ --error-on-warn

init

Scaffolds a new skill or agent fixture directory under fixtures/<name>/. Always scaffold from agc init rather than writing from scratch.

# Scaffold a skill fixture
agc init --skill my-agent

# Scaffold an agent fixture
agc init --agent my-agent

Creates cases.yaml, prompt.md, bundle.manifest.json, and an empty golden/ directory.

report

Inspect persisted runs from the local history database: list runs, show case-level details, or diff two runs side-by-side.

# List recent runs
agc report list

# List with a limit
agc report list --limit 10

# Show details for a specific run
agc report show <RUN-ID>

# Render a local evidence directory or run.json file
agc report show ./reports/evidence-packs/my-agent/

# Diff two runs
agc report diff <RUN-ID-A> <RUN-ID-B>

stats

Shows historical pass-rate trends, per-case flakiness, and latency percentiles from the local run history database. Useful for tracking quality over time and spotting unreliable cases.

# Overview across all skills (last 50 runs)
agc stats

# Filter to a specific skill
agc stats --skill my-agent

# Analyse more history, machine-readable output
agc stats --limit 100 --format json

export

Packages a completed run into an evidence bundle: a .tar.gz archive with an optional minisign attestation signed by a human domain expert.

# Export a specific run by id
agc export <RUN-ID>

# Export to a specific path
agc export <RUN-ID> --out ./my-bundle.tar.gz

# Export the 5 most recent runs into a directory
agc export --last 5 --out-dir ./evidence/

# List recent runs to find the id
agc report list --limit 5

compliance

Scores run history against bundled OSCAL control catalogs and renders per-control attestation reports. Cases are mapped to controls via tags (for example fda-samd:fda-samd-medical-device-reporting). A control is reported satisfied only with three or more cases and effectiveness ≥ 0.80; anything less is partial evidence or a gap. Available frameworks: nist-ai-rmf, eu-ai-act, iso-42001, hipaa, fda-samd, nist-800-171, nist-800-172, nist-800-207.

# Per-control attestation report (Markdown)
agc compliance report --framework fda-samd --skill my-agent

# OSCAL Assessment Results JSON (machine-readable)
agc compliance report --framework hipaa --oscal > hipaa.oscal.json

# All frameworks, one file each
agc compliance report --framework all --out ./reports/

# Controls with no coverage, plus remediation advisories
agc compliance gaps --framework eu-ai-act

# Generate fixture cases pre-tagged with control IDs
agc compliance generate --skill my-agent \
  --tag nist-ai-rmf:measure-1.1 --count 3

The OSCAL Assessment Results artifact is also written into every agc export evidence tarball, and agc metrics --framework <id> scopes the compliance metrics table to a single framework.

bundle

Manages fixture bundles: distributable archives combining a manifest, fixture files, and mocks.

# Pack a bundle (updates sha256s in manifest, writes .tar.gz)
agc bundle pack fixtures/my-agent

# Pack to an explicit output path
agc bundle pack fixtures/my-agent --out my-bundle.tar.gz

# Verify bundle integrity (dir, manifest file, or .tar.gz)
agc bundle verify my-bundle.tar.gz

# Pull bundle manifest and artifacts from the registry
agc bundle pull my-agent-1.0.0 \
  --url https://api.agentcarousel.com

# Pull and immediately verify
agc bundle pull my-agent-1.0.0 \
  --url https://api.agentcarousel.com --verify

publish

Registers a bundle and ingests evidence runs in one step. Reads the registry write token from config or the AGENTCAROUSEL_API_TOKEN environment variable. If a prompt.md file exists in the bundle directory, it is sent to the registry and stored as the agent's system prompt.

# Publish bundle + most recent matching run
agc publish fixtures/my-agent \
  --url https://api.agentcarousel.com

# Dry run — resolve values without making API writes
agc publish fixtures/my-agent \
  --url https://api.agentcarousel.com --dry-run

# Submit all matching local runs (newest first)
agc publish fixtures/my-agent \
  --url https://api.agentcarousel.com --all-runs

# Limit number of runs when using --all-runs
agc publish fixtures/my-agent \
  --url https://api.agentcarousel.com --all-runs --limit 5

trust-check

Queries the registry for the current trust state of a bundle. By default requires trusted state; use --min-trust to lower the threshold. Optionally verifies a local attestation against a minisign public key.

# Query trust state (exits non-zero if below --min-trust, default: trusted)
agc trust-check my-agent-1.0.0 \
  --url https://api.agentcarousel.com

# Accept stable or higher
agc trust-check my-agent-1.0.0 \
  --url https://api.agentcarousel.com \
  --min-trust stable

# Verify with a local attestation file + minisign public key
agc trust-check my-agent-1.0.0 \
  --url https://api.agentcarousel.com \
  --attestation ./attestation.json \
  --minisign-pubkey ./minisign.pub

doctor

Checks your environment, configuration, and fixture setup for common issues in one pass: API keys, config file, history DB, fixtures directory, and JSON schema.

# Full environment check
agc doctor

# Machine-readable output
agc doctor --json

update

Checks GitHub for a newer release of the CLI and installs it in-place using an atomic rename. Uses the same install path as the running binary.

# Check and install if available
agc update

# Check availability without installing
agc update --check

completions

Prints a shell completion script to stdout for bash, zsh, or fish.

# Zsh
agc completions zsh > ~/.zsh/completions/_agc

# Bash
agc completions bash > /etc/bash_completion.d/agc

# Fish
agc completions fish > ~/.config/fish/completions/agc.fish

Configuration

The CLI resolves configuration in this order, with later sources taking precedence:

1. --config <path>: explicit path flag
2. ./agentcarousel.toml: project-level config
3. ~/.config/agentcarousel/config.toml: user-level config

Run history is stored in a local SQLite database:

Override with AGENTCAROUSEL_HISTORY_DB=/path/to/history.db.

Fixture Format

Fixtures are YAML files named cases.yaml inside their skill directory. The authoritative schema is schemas/skill-definition.schema.json in the repository. Always scaffold from agc init rather than writing from scratch.

Top-level fields

Case fields

Minimal example

schema_version: 1
skill_or_agent: hello-skill
cases:
  - id: hello-skill/happy-path
    description: Responds to a greeting
    tags: [smoke, happy-path]
    input:
      messages:
        - role: user
          content: "Say hello"
    expected:
      tool_sequence: []
      output:
        - kind: contains
          value: "hello"

Output assertion kinds

Case Tags

Tags let you filter which cases run. Use --filter-tags smoke on PRs for fast feedback and run all tags on main or nightly.

Evaluators

Select the evaluator that requires the least complexity. Escalate only when a simpler one cannot express the assertion.

Development Process

Standard fixtures should go from blank intake to passing CI in under two hours. Follow these phases in order.

Quick self-check commands

# 1. Schema + rules validation (strict mode)
agc validate fixtures/my-agent/cases.yaml --strict

# 2. Fixture quality check
agc lint fixtures/my-agent/cases.yaml

# 3. Mock test run
agc test fixtures/my-agent/cases.yaml

# 4. Eval with rubric (if rubric items exist)
agc eval fixtures/my-agent/cases.yaml

Contributing

Contributions are welcome. The process is designed to keep fixtures aligned with released CLI behavior and to prevent scope creep.

1. Open a GitHub Issue first: before writing any YAML, open an issue titled Fixture: <skill-or-agent-id> and fill in the intake checklist from CONTRIBUTING.md.
2. Follow the development process: complete all six phases, including passing validate --strict, lint, and test.
3. Security: never include real API keys, credentials, or PII in fixture inputs, mock responses, or expected outputs. See SECURITY.md .

Authentication

Public read endpoints require no credentials. Write endpoints require a bearer token.

# Set the token before publishing
export AGENTCAROUSEL_API_TOKEN=<your-registry-write-token>

agc publish fixtures/my-agent \
  --url https://api.agentcarousel.com

Endpoints

Base URLs

GET /health

Liveness check. Returns { "ok": true }.

GET /v1/bundles

Lists all registered bundles. Returns bundle_id, bundle_version, trust_state, description, domain, created_at, and last_run_at.

GET /v1/bundles/{bundleId}/trust-state

Returns the public trust state for a bundle. No authentication required.

const bundleId = "my-agent-1.0.0";
const res = await fetch(
  `${baseUrl}/v1/bundles/${encodeURIComponent(bundleId)}/trust-state`
);
const trust = await res.json();

GET /v1/bundles/{bundleId}/file

Fetches a bundle artifact by path. The special path prompt.md serves the agent's system prompt as text/plain (stored as a DB column, not object storage). All other paths serve fixture/mock bytes from object storage.

# Fetch the agent's system prompt
GET /v1/bundles/my-agent-1.0.0/file?path=prompt.md

# Fetch a fixture file
GET /v1/bundles/my-agent-1.0.0/file?path=cases.yaml

POST /v1/bundles

Register or resolve a bundle. Idempotent. Accepts application/json or multipart/form-data. Pass a prompt field in multipart to store the agent's system prompt.

const res = await fetch(`${baseUrl}/v1/bundles`, {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
    "Authorization": `Bearer ${token}`,
  },
  body: JSON.stringify({
    bundle_id: "org/my-agent",
    bundle_version: "1.0.0",
    risk_tier: "medium",
  }),
});

POST /v1/runs

Ingest a run evidence tarball produced by agc export. Accepts multipart/form-data with an evidence field containing the .tar.gz.

const form = new FormData();
form.append("evidence", evidenceFile); // .tar.gz blob
form.append("registry_bundle_id", "my-agent-1.0.0");

const res = await fetch(`${baseUrl}/v1/runs`, {
  method: "POST",
  headers: { "Authorization": `Bearer ${token}` },
  body: form,
});

Trust State

Every registered bundle has a trust state. New bundles start at Experimental and advance as qualifying runs accumulate.