Evaluate your app with planted truth

Create a company in the Fictix Console, plant a supported issue, run your detector against the company records, and submit its findings for a score. The company-key REST flow below uses the company you already created in the Console.

1. Prepare the company and keys

Follow the Quickstart to create a synthetic test company and get its slug, API origin, and company key. Use the matching production or staging origin throughout.

In Settings → Company → Company API keys, choose Read and modify for the server-side evaluation harness. Planting scenarios and creating test matrices/runs require write; reading company records requires read. Give the application being evaluated a separate Read only key. Keep both keys on their respective servers, using FICTIX_PROJECT_API_KEY in each process. Neither process needs a copied Console session or an admin key.

The commands below use the harness key and the three environment variables from the quickstart.

2. Choose and plant one supported case

Read the library before choosing a case:

curl --fail-with-body -sS \
  "$FICTIX_API_ORIGIN/api/v1/orgs/$FICTIX_PROJECT_SLUG/scenarios" \
  -H "X-API-Key: $FICTIX_PROJECT_API_KEY"

Check the template's entity_types_affected and parameter definitions. duplicate-expense-fraud plants duplicate vendor bills. The current library does not include a duplicate-customer-invoice template.

Initialize simulation state, then pause company activity while you evaluate. Starting an existing simulation returns its current state; it does not create another company.

curl --fail-with-body -sS -X POST \
  "$FICTIX_API_ORIGIN/api/v1/orgs/$FICTIX_PROJECT_SLUG/simulation/start" \
  -H "X-API-Key: $FICTIX_PROJECT_API_KEY" \
  -H "Content-Type: application/json" \
  --data-raw '{"seed":42,"dials":{"trickle_rate":"paused"}}'

curl --fail-with-body -sS -X POST \
  "$FICTIX_API_ORIGIN/api/v1/orgs/$FICTIX_PROJECT_SLUG/simulation/pause" \
  -H "X-API-Key: $FICTIX_PROJECT_API_KEY"

Plant two duplicate bills with no amount variation:

curl --fail-with-body -sS -X POST \
  "$FICTIX_API_ORIGIN/api/v1/orgs/$FICTIX_PROJECT_SLUG/simulation/scenarios" \
  -H "X-API-Key: $FICTIX_PROJECT_API_KEY" \
  -H "Content-Type: application/json" \
  --data-raw '{"scenarios":[{"id":"duplicate-expense-fraud","params":{"count":2,"amount_variance_pct":0}}]}'

Inject once. If the connection fails before you receive a response, check the company and scenario state before repeating a write. Scenarios consume the company's scenario quota.

Runnable duplicate-bill example

Download the customer integration example, configure the quickstart's three server-side variables, and use its exact bill-repeat detector:

npm run evaluate:bills
npm run evaluate:bills -- --submit

The first command reads bills and prints findings. The second also creates a matrix and submits one test run using your Read and modify harness key. It prints the actual score and run ID. It uses only company-key REST routes, without fetching the planted-truth manifest. The example compares exact amounts, matching the zero-variance case above. You can use this command instead of manually performing steps 3–5 below.

3. Run your application against the records

Use the application's read-only key to page /data/bills, or load an export into your own application. Preserve each record's Fictix id.

Your detector should inspect business fields such as vendor, bill number, currency, amount, and dates. Keep planted-truth manifests outside the detector. Collect the actual record IDs it identifies:

{
  "detected": [
    {
      "entity_type": "bill",
      "entity_id": "replace_with_an_id_your_detector_found",
      "finding": "Repeated bill number and amount for the same vendor"
    }
  ]
}

Save this as detector-findings.json. Use real IDs from your detector; the placeholder above is not a detection. An empty detected array is a valid no-findings result.

4. Create a test matrix

With the harness key, create a matrix for the case you planted:

curl --fail-with-body -sS -X POST \
  "$FICTIX_API_ORIGIN/api/v1/orgs/$FICTIX_PROJECT_SLUG/test-matrices" \
  -H "X-API-Key: $FICTIX_PROJECT_API_KEY" \
  -H "Content-Type: application/json" \
  --data-raw '{"name":"Duplicate bill evaluation","scenarios":[{"scenario_id":"duplicate-expense-fraud","severity":"high","required":true}]}' \
  -o matrix.json

The response contains matrix_id. Build a submission using that returned ID:

jq --arg matrix_id "$(jq -r '.matrix_id' matrix.json)" \
  '{matrix_id:$matrix_id,detected:.detected}' detector-findings.json > test-run.json

5. Submit findings and read the score

curl --fail-with-body -sS -X POST \
  "$FICTIX_API_ORIGIN/api/v1/orgs/$FICTIX_PROJECT_SLUG/test-runs" \
  -H "X-API-Key: $FICTIX_PROJECT_API_KEY" \
  -H "Content-Type: application/json" \
  --data-binary @test-run.json \
  -o result.json

Read overall.overall_recall, overall.overall_precision, overall.overall_f1, and overall.matrix_passes for the aggregate result. results contains scenario-level recall, precision, missed needles, and false positives. Retain run_id, manifest_hash, and your application/model version so you can compare later runs against the same evidence. Test runs consume the company's test-run quota.

Retrieve a previous run with GET /api/v1/orgs/{slug}/test-runs/{run_id}. To list a matrix's history, use GET /api/v1/orgs/{slug}/test-runs?matrix_id={matrix_id}; the matrix ID is required. Inspect that history before retrying an uncertain submission.

A no-findings submission should record missed planted issues. HTTP 401/403, invalid JSON, or a failed detector request are integration failures rather than successful evaluation results.

Use Console Test Lab with your detector endpoint

You can also create a scenario in Scenarios, then open Test Lab, enter your detector URL, and choose Run tests. The Console handles the matrix and scoring requests with your signed-in session.

The Console sends an HTTP POST from your browser to the detector:

{
  "mode": "fictix-test-run",
  "project_slug": "your-company-slug",
  "project_id": "your-company-project-id",
  "scenario_ids": ["duplicate-expense-fraud"]
}

Return HTTP 200 with a JSON detected array in the format shown above. The endpoint must respond within 30 seconds. HTTPS is supported; local development also permits HTTP on localhost or 127.0.0.1.

For localhost tests in Chrome, allow the Console site's local network permission when prompted. If you blocked it, enable local network or loopback access in the site's browser permissions and retry. A blocked request can time out before your detector receives it.

Allow the calling Console origin through CORS: https://console.fictix.ai for production or https://staging.fictix.ai for staging. Handle the OPTIONS preflight and allow POST with the Content-Type header. The Console sends no detector authorization header and never forwards your Fictix company key. Your detector server supplies its own read-only company key when reading records.

Approved demo-job integrations

Fictix-approved management integrations can use the separate demo-job API and its test-matrix/run routes. Those routes use Fictix-supplied management credentials. Console customers can use the company-key workflow above.

Load data into your app

Catalog