fix(api): Update evaluators execution apis #165
Open
nina-kollman wants to merge 3 commits intomainfrom
Open
Conversation
Contributor
📝 WalkthroughWalkthroughUpdated OpenAPI paths in evaluator documentation and the API spec: moved many evaluator execute routes from Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Poem
🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Workflows to automatically generate PRs for you. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 4
♻️ Duplicate comments (1)
api-reference/evaluators/execute-toxicity-detector-evaluator.mdx (1)
2-2:⚠️ Potential issue | 🔴 CriticalSame critical path mismatch as execute-topic-adherence-evaluator.mdx.
The openapi.json schema still defines this endpoint at
/v2/evaluators/execute/toxicity-detector(see openapi.json:2096-2152), but the documentation is being updated to/v2/evaluators/toxicity-detector/execute. Please see the detailed comment on execute-topic-adherence-evaluator.mdx for the full analysis and verification steps.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@api-reference/evaluators/execute-toxicity-detector-evaluator.mdx` at line 2, The documentation path for the toxicity detector endpoint doesn't match the OpenAPI schema: the MDX shows "post /v2/evaluators/toxicity-detector/execute" but openapi.json defines "/v2/evaluators/execute/toxicity-detector"; update the MDX to use the schema canonical path (replace "post /v2/evaluators/toxicity-detector/execute" with "post /v2/evaluators/execute/toxicity-detector") or, if the API surface intentionally changed, update openapi.json accordingly and run the same reconciliation you applied for execute-topic-adherence-evaluator.mdx so both the documentation (execute-toxicity-detector-evaluator.mdx) and the OpenAPI schema consistently use the same endpoint string.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@api-reference/evaluators/create-a-custom-llm-as-a-judge-evaluator.mdx`:
- Line 2: The frontmatter OpenAPI route value "openapi: post /v2/evaluators" is
incorrect; change it to the documented operation path "openapi: post
/v2/evaluators/custom-evaluator" so it matches the OpenAPI spec entry "post
/v2/evaluators/custom-evaluator" and the other page
"create-custom-evaluator.mdx"; update the string in the frontmatter where
"openapi: post /v2/evaluators" appears to exactly "openapi: post
/v2/evaluators/custom-evaluator".
In `@api-reference/evaluators/execute-agent-tool-trajectory-evaluator.mdx`:
- Line 2: The evaluator route in this MDX uses the wrong ordering; change the
OpenAPI path from "/v2/evaluators/{evaluator}/execute" (currently shown as "post
/v2/evaluators/agent-tool-trajectory/execute") to the spec-backed
"/v2/evaluators/execute/{evaluator}" (e.g., "post
/v2/evaluators/execute/agent-tool-trajectory"); apply the same swap to all
evaluator MDX files in api-reference/evaluators/ so routes match the whitelist
pattern "/v2/evaluators/execute/*".
In `@api-reference/evaluators/execute-answer-completeness-evaluator.mdx`:
- Around line 1-3: OpenAPI specs still reference the old path pattern
"/v2/evaluators/execute/{evaluator-name}"; update all occurrences (e.g., the ~35
references in openapi.json and entries in openapi-whitelist.yaml) to the new
path structure "/v2/evaluators/{evaluator-name}/execute". Search for the exact
string "/v2/evaluators/execute/{evaluator-name}" (and any variants like escaped
JSON paths) and replace them with "/v2/evaluators/{evaluator-name}/execute",
ensuring both the operation object keys and any whitelist/path lists are updated
so endpoints like the evaluator "answer-completeness" match the new format used
in execute-answer-completeness-evaluator.mdx.
In `@api-reference/evaluators/execute-topic-adherence-evaluator.mdx`:
- Line 2: Documentation now references the new endpoint pattern POST
/v2/evaluators/topic-adherence/execute but the OpenAPI schema still exposes
/v2/evaluators/execute/topic-adherence; update the OpenAPI spec (openapi.json)
so all evaluator paths follow the new pattern
(/v2/evaluators/<evaluator-name>/execute) — e.g., change
/v2/evaluators/execute/topic-adherence to /v2/evaluators/topic-adherence/execute
(and replicate for the other 34 evaluator endpoints like topic-adherence), or if
backend cannot be changed now, revert the docs in
api-reference/evaluators/execute-topic-adherence-evaluator.mdx to the existing
path until the API is updated; coordinate with the backend team to ensure
whichever source of truth (spec or docs) is aligned before merging.
---
Duplicate comments:
In `@api-reference/evaluators/execute-toxicity-detector-evaluator.mdx`:
- Line 2: The documentation path for the toxicity detector endpoint doesn't
match the OpenAPI schema: the MDX shows "post
/v2/evaluators/toxicity-detector/execute" but openapi.json defines
"/v2/evaluators/execute/toxicity-detector"; update the MDX to use the schema
canonical path (replace "post /v2/evaluators/toxicity-detector/execute" with
"post /v2/evaluators/execute/toxicity-detector") or, if the API surface
intentionally changed, update openapi.json accordingly and run the same
reconciliation you applied for execute-topic-adherence-evaluator.mdx so both the
documentation (execute-toxicity-detector-evaluator.mdx) and the OpenAPI schema
consistently use the same endpoint string.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 7f30d597-a21d-4f38-b5b8-2d5e2e2501f1
📒 Files selected for processing (36)
api-reference/evaluators/create-a-custom-llm-as-a-judge-evaluator.mdxapi-reference/evaluators/execute-agent-efficiency-evaluator.mdxapi-reference/evaluators/execute-agent-flow-quality-evaluator.mdxapi-reference/evaluators/execute-agent-goal-accuracy-evaluator.mdxapi-reference/evaluators/execute-agent-goal-completeness-evaluator.mdxapi-reference/evaluators/execute-agent-tool-error-detector-evaluator.mdxapi-reference/evaluators/execute-agent-tool-trajectory-evaluator.mdxapi-reference/evaluators/execute-answer-completeness-evaluator.mdxapi-reference/evaluators/execute-answer-correctness-evaluator.mdxapi-reference/evaluators/execute-answer-relevancy-evaluator.mdxapi-reference/evaluators/execute-char-count-evaluator.mdxapi-reference/evaluators/execute-char-count-ratio-evaluator.mdxapi-reference/evaluators/execute-context-relevance-evaluator.mdxapi-reference/evaluators/execute-conversation-quality-evaluator.mdxapi-reference/evaluators/execute-faithfulness-evaluator.mdxapi-reference/evaluators/execute-html-comparison-evaluator.mdxapi-reference/evaluators/execute-instruction-adherence-evaluator.mdxapi-reference/evaluators/execute-intent-change-evaluator.mdxapi-reference/evaluators/execute-json-validator-evaluator.mdxapi-reference/evaluators/execute-perplexity-evaluator.mdxapi-reference/evaluators/execute-pii-detector-evaluator.mdxapi-reference/evaluators/execute-placeholder-regex-evaluator.mdxapi-reference/evaluators/execute-profanity-detector-evaluator.mdxapi-reference/evaluators/execute-prompt-injection-evaluator.mdxapi-reference/evaluators/execute-prompt-perplexity-evaluator.mdxapi-reference/evaluators/execute-regex-validator-evaluator.mdxapi-reference/evaluators/execute-secrets-detector-evaluator.mdxapi-reference/evaluators/execute-semantic-similarity-evaluator.mdxapi-reference/evaluators/execute-sexism-detector-evaluator.mdxapi-reference/evaluators/execute-sql-validator-evaluator.mdxapi-reference/evaluators/execute-tone-detection-evaluator.mdxapi-reference/evaluators/execute-topic-adherence-evaluator.mdxapi-reference/evaluators/execute-toxicity-detector-evaluator.mdxapi-reference/evaluators/execute-uncertainty-detector-evaluator.mdxapi-reference/evaluators/execute-word-count-evaluator.mdxapi-reference/evaluators/execute-word-count-ratio-evaluator.mdx
Contributor
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
openapi.json (2)
277-328:⚠️ Potential issue | 🟠 MajorUpdate the whitelist before this spec is regenerated.
These route moves only exist in
openapi.jsonright now.scripts/filter-openapi.jsrebuilds this file from the backend spec usingopenapi-whitelist.yaml, and that whitelist still includes/v2/evaluators/execute/*plus/v2/evaluators/custom-evaluator. On the next regeneration, the new/v2/evaluators/{slug}/executeendpoints andPOST /v2/evaluatorswill be filtered back out.🔧 Suggested follow-up in
openapi-whitelist.yamlinclude_paths: - /v2/metrics_hwm - /v2/organizations - - /v2/evaluators/execute/* - - /v2/evaluators/custom-evaluator + - /v2/evaluators + - /v2/evaluators/*/execute - /v2/metrics - /v2/metrics_hwm - /v2/auto-monitor-setups - /v2/auto-monitor-setups/*Also applies to: 329-2323
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@openapi.json` around lines 277 - 328, The openapi whitelist is out of sync with the new evaluator routes: update openapi-whitelist.yaml to allow the new POST /v2/evaluators and the new /v2/evaluators/{slug}/execute pattern (and any related new evaluator paths) and remove the old entries (/v2/evaluators/execute/* and /v2/evaluators/custom-evaluator) so scripts/filter-openapi.js won’t strip them on regeneration; after editing the whitelist, run the filter-openapi.js regeneration to verify the new /v2/evaluators and /v2/evaluators/{slug}/execute endpoints remain in openapi.json.
277-328:⚠️ Potential issue | 🟠 MajorFix the create-custom-evaluator doc page to use the new path.
Line 277 changes the spec to
POST /v2/evaluators, butapi-reference/evaluators/create-custom-evaluator.mdx:1-5still points atpost /v2/evaluators/custom-evaluator. That page now targets an operation that no longer exists in the published spec.🔧 Suggested follow-up in
api-reference/evaluators/create-custom-evaluator.mdx--- -openapi: post /v2/evaluators/custom-evaluator +openapi: post /v2/evaluators ---🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@openapi.json` around lines 277 - 328, The OpenAPI path for creating custom evaluators was changed to "POST /v2/evaluators" but the docs page still references the old "post /v2/evaluators/custom-evaluator"; update api-reference/evaluators/create-custom-evaluator.mdx to use the new path and operation name by replacing any occurrences of "/v2/evaluators/custom-evaluator" or the old operationId with "/v2/evaluators" (and the corresponding POST operation), and adjust any example curl/JS/HTTP snippets, payloads, and response schema references to match the new operation defined in the spec (ensure examples reference response.CreateEvaluatorResponse and request.CreateCustomEvaluatorInput as in the OpenAPI change).
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Outside diff comments:
In `@openapi.json`:
- Around line 277-328: The openapi whitelist is out of sync with the new
evaluator routes: update openapi-whitelist.yaml to allow the new POST
/v2/evaluators and the new /v2/evaluators/{slug}/execute pattern (and any
related new evaluator paths) and remove the old entries
(/v2/evaluators/execute/* and /v2/evaluators/custom-evaluator) so
scripts/filter-openapi.js won’t strip them on regeneration; after editing the
whitelist, run the filter-openapi.js regeneration to verify the new
/v2/evaluators and /v2/evaluators/{slug}/execute endpoints remain in
openapi.json.
- Around line 277-328: The OpenAPI path for creating custom evaluators was
changed to "POST /v2/evaluators" but the docs page still references the old
"post /v2/evaluators/custom-evaluator"; update
api-reference/evaluators/create-custom-evaluator.mdx to use the new path and
operation name by replacing any occurrences of "/v2/evaluators/custom-evaluator"
or the old operationId with "/v2/evaluators" (and the corresponding POST
operation), and adjust any example curl/JS/HTTP snippets, payloads, and response
schema references to match the new operation defined in the spec (ensure
examples reference response.CreateEvaluatorResponse and
request.CreateCustomEvaluatorInput as in the OpenAPI change).
doronkopit5
approved these changes
Apr 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes: TLP-2068
Summary by CodeRabbit