Test matrices and runs

Test matrices define what an application should detect. Test runs submit what the application actually detected. Fictix scores the submitted findings against planted truth.

Base path:

/api/v1/demo-jobs/{job_ref}

Create test matrix

POST /api/v1/demo-jobs/{job_ref}/test-matrices

Example body:

{
  "name": "Accounting QA",
  "scenarios": [
    {
      "scenario_id": "ghost-employee",
      "severity": "critical",
      "required": true
    }
  ]
}

Use a matrix to define the expected detection contract for a product workflow.

List test matrices

GET /api/v1/demo-jobs/{job_ref}/test-matrices

Get test matrix

GET /api/v1/demo-jobs/{job_ref}/test-matrices/{matrix_id}

Delete test matrix

DELETE /api/v1/demo-jobs/{job_ref}/test-matrices/{matrix_id}

Create test run

POST /api/v1/demo-jobs/{job_ref}/test-runs

Example body:

{
  "matrix_id": "tm_abc123",
  "detected": [
    {
      "entity_type": "employee",
      "entity_id": "emp_ghost_001",
      "finding": "ghost employee"
    }
  ]
}

Fictix compares submitted findings with the matrix and scenario truth.

List test runs

GET /api/v1/demo-jobs/{job_ref}/test-runs

Get test run

GET /api/v1/demo-jobs/{job_ref}/test-runs/{run_id}

Evaluation pattern

1. Create a demo job.
2. Inject or generate scenarios.
3. Fetch the scenario manifest.
4. Load the company into the target application.
5. Run the target application or agent.
6. Convert the target application's findings into Fictix detected findings.
7. Submit a test run.
8. Store the score and verdict.

Detected finding shape

Detected findings should be specific enough to match planted truth:

{
  "entity_type": "vendor",
  "entity_id": "ven_123",
  "finding": "duplicate vendor"
}

Avoid vague findings such as:

{
  "finding": "something is wrong"
}

Specific findings produce better scoring and better user-facing verdicts.