Query Evaluators

List all evaluators in your organization

Endpoint

method · string · default: "POST"

POST

endpoint · string

/v1/evaluator/query

Authentication

This endpoint requires API key authentication. Include your API key in the request headers:

Authorization: Bearer YOUR_API_KEY

Request Body

No specific parameters required. Send an empty object.

Response

data · array

Array of evaluator objects

id · string

Unique identifier for the evaluator

name · string

Evaluator name

scoring_type · string

Type of scoring: “LLM-BOOLEAN”, “LLM-CHOICE”, “LLM-RANGE”, “PYTHON”, or “LASTMILE”

llm_template · object

LLM configuration for LLM-based evaluators

code_template · object

Python code for code-based evaluators

last_mile_config · object

Configuration for LastMile evaluators

organization_id · string

Organization that owns this evaluator

created_at · string

Creation timestamp

updated_at · string

Last update timestamp

error · string | null

Error message if the request failed, null otherwise

Example Request

curl -X POST https://api.helicone.ai/v1/evaluator/query \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{}'

Example Response

{
  "data": [
    {
      "id": "eval_abc123",
      "name": "Hallucination Detector",
      "scoring_type": "LLM-BOOLEAN",
      "llm_template": {
        "model": "gpt-4",
        "prompt": "Does this response contain hallucinations?"
      },
      "code_template": null,
      "last_mile_config": null,
      "organization_id": "org_xyz789",
      "created_at": "2024-01-10T09:00:00Z",
      "updated_at": "2024-01-15T14:30:00Z"
    },
    {
      "id": "eval_def456",
      "name": "Relevance Score",
      "scoring_type": "LLM-RANGE",
      "llm_template": {
        "model": "gpt-4",
        "prompt": "Rate the relevance from 1-10",
        "min": 1,
        "max": 10
      },
      "code_template": null,
      "last_mile_config": null,
      "organization_id": "org_xyz789",
      "created_at": "2024-01-12T11:00:00Z",
      "updated_at": "2024-01-12T11:00:00Z"
    }
  ],
  "error": null
}

Evaluator Types

LLM-BOOLEAN

Uses an LLM to return a true/false score

LLM-CHOICE

Uses an LLM to select from predefined choices

LLM-RANGE

Uses an LLM to score within a numeric range

PYTHON

Executes custom Python code for evaluation

LASTMILE

Integration with LastMile evaluation framework