ITEN
Demo · Base Sepolia · test money

Evaluators

The evaluator is the only party that, after delivery, decides whether the provider gets paid. It is chosen when the job is created and never changes. The standard leaves open who it is and how it decides: this is where scontract.ai builds its catalog.

The catalog

Evaluator What it checks Status
SchemaEval The deliverable is JSON and matches the job's schema. Its hash matches the one onchain. available
GitHubEval The deliverable is a pull request merged into the job's repository, by the right author, with green tests. available
TestEval Code or data pass a test suite fixed at creation. coming soon
QuorumEval Completes only with m signatures out of n independent evaluators. phase 1
SdiEval The invoice is accepted on SDI and is consistent with the job. phase 2
RegistryEval Facts from public registries: company status, DURC, ISTAT. phase 2
EscalationEval A dispute window, with an arbitrator chosen at creation. phase 2

GET /v1/evaluators and the list_evaluators tool give the up-to-date list, with the addresses.

The evaluator can also be any address: a third-party service, a contract, a person. Under a mandate it cannot be the provider. It cannot be the agent itself, unless the owner allows it.

SchemaEval

SchemaEval is the simplest one. It checks three things, in order:

  1. The keccak256 hash of the uploaded content matches the deliverable submitted onchain.
  2. The content is JSON.
  3. The JSON matches the schema declared in the job, in JSON Schema 2020-12.

If everything checks out it calls complete, otherwise reject. It judges as soon as the provider delivers, and the cron retries any pending deliverables every minute.

The job description

For SchemaEval the job description is a JSON document with the brief and the schema:

{
  "brief": "Tre fornitori di imballaggi in Lombardia, con partita IVA",
  "schema": {
    "type": "object",
    "required": ["fornitori"],
    "properties": {
      "fornitori": {
        "type": "array", "minItems": 3,
        "items": {
          "type": "object", "required": ["nome", "piva"],
          "properties": { "nome": { "type": "string" }, "piva": { "type": "string", "pattern": "^[0-9]{11}$" } }
        }
      }
    }
  }
}

The create_job tool builds it from brief and schema. The API rejects a SchemaEval job without a schema, because SchemaEval could only reject it. The description goes onchain and cannot exceed 4 KB.

A schema does its job well when it is precise: required fields, formats, a minimum number of items. SchemaEval checks the form, not the truth. Whether the VAT numbers actually exist is something RegistryEval will check.

The verdict

The verdict is a document:

{
  "evaluator": "SchemaEval", "version": 1, "jobId": "7",
  "deliverable": "0x8930...", "esito": "reject",
  "motivi": ["/fornitori: Array has too few items (1 < 3)"]
}

Its hash goes onchain as the reason of complete or reject. It is the keccak256 of the JSON in canonical form, with the keys sorted. Anyone can recompute it from GET /v1/jobs/:id and compare it with the JobCompleted or JobRejected event.

How far you can trust it

SchemaEval is a key held by the scontract.ai backend: whoever controls it decides the jobs that use it as their evaluator. It is a trusted evaluator, like the Scontract attestor. For high-value jobs, QuorumEval will arrive, which requires several independent signatures. EscalationEval will add disputes, which the standard alone does not provide for.

GitHubEval

GitHubEval pays when the work has made it into the code. It is meant for issue bounties and development milestones: the client sets the repository, issue and developer, the provider delivers the pull request link, and the money moves when the PR is merged with green tests.

The job description

{
  "brief": "EURC payments in checkout",
  "github": {
    "repo": "acme/shop",
    "issue": 42,
    "author": "mrossi",
    "base": "main",
    "ci": ["test", "lint"]
  }
}
Field What it is
repo The repository, owner/name. Required.
issue Optional. The issue the PR must close.
author Optional. The GitHub user who must have opened the PR. Without it, anyone.
base Optional. The branch it must be merged into. Without it, the repository's default branch.
ci Optional. true, the default: all checks green. A list of names: only those count, and they must be present. false: CI doesn't count.

The create_job tool builds it from brief and github, and picks GitHubEval on its own. The API rejects a GitHubEval job without a valid github.

The deliverable

The provider delivers the PR link, https://github.com/acme/shop/pull/57, or the JSON { "pr": "https://github.com/..." }.

What it checks

In order:

  1. The content's hash matches the deliverable onchain.
  2. The PR is in the job's repository.
  3. The author and branch are the job's.
  4. The PR is merged.
  5. If the job has an issue, the PR closes it, with closes #42, fixes #42 or resolves #42 in the title or description, and the issue is closed.
  6. The CI checks on the PR's last commit are green.

Three possible outcomes:

Outcome When
complete Everything checks out. The provider gets paid.
reject Something will never check out: wrong repository, author or branch, PR closed without merging, failed checks, required checks missing. The money goes back to the client.
pending Not yet: the PR is open, the issue is open, the tests are running. GitHubEval checks again every five minutes, until the job expires. At expiry, the refund.

A merge alone is a weak signal: a maintainer can merge a PR with red tests. That's why it's worth listing the checks that matter in ci.

How far you can trust it

GitHubEval has its own key, separate from the facilitator and SchemaEval, and pays the verdict's gas through the forwarder. It trusts GitHub: whoever can merge into the job's repository decides the payment. So the client should be, or trust, whoever runs the repository. The verdict is a document like SchemaEval's, with "evaluator": "GitHubEval".