LLM Eval Harness MCP Server

by razilzulio.github.razilzul/llm-eval-harnessv1.0.0

Run a golden set and gate LLM model swaps on accuracy, with latency and cost per request.

0Dockerstdioofficial registry

context tax

pending

security

pending

cold start

pending

freshness

pending

Our reading of LLM Eval Harness, from the data

LLM Eval Harness is a ai & llm tools MCP server from razilzul, listed in the official MCP Registry.

We haven't measured its tool list yet, so its token cost is unknown.

With 0 GitHub stars it ranks #1,305 of 3,169 ai & llm tools servers we index.

It runs locally on your machine over stdio and needs 2 secrets (ANTHROPIC_API_KEY, GEMINI_API_KEY); use a scoped, least-privilege key.

Install LLM Eval Harness MCP server

claude mcp add llm-eval-harness -e ANTHROPIC_API_KEY='<anthropic-api-key>' -e GEMINI_API_KEY='<gemini-api-key>' -- docker run -i --rm -e ANTHROPIC_API_KEY -e GEMINI_API_KEY ghcr.io/razilzul/llm-eval-harness:1.0.0

{
  "mcpServers": {
    "llm-eval-harness": {
      "command": "docker",
      "args": [
        "run",
        "-i",
        "--rm",
        "-e",
        "ANTHROPIC_API_KEY",
        "-e",
        "GEMINI_API_KEY",
        "ghcr.io/razilzul/llm-eval-harness:1.0.0"
      ],
      "env": {
        "ANTHROPIC_API_KEY": "<anthropic-api-key>",
        "GEMINI_API_KEY": "<gemini-api-key>"
      }
    }
  }
}

Add to ~/.cursor/mcp.json (global) or .cursor/mcp.json (project).

{
  "mcpServers": {
    "llm-eval-harness": {
      "command": "docker",
      "args": [
        "run",
        "-i",
        "--rm",
        "-e",
        "ANTHROPIC_API_KEY",
        "-e",
        "GEMINI_API_KEY",
        "ghcr.io/razilzul/llm-eval-harness:1.0.0"
      ],
      "env": {
        "ANTHROPIC_API_KEY": "<anthropic-api-key>",
        "GEMINI_API_KEY": "<gemini-api-key>"
      }
    }
  }
}

Settings → Developer → Edit Config (claude_desktop_config.json), then restart.

{
  "servers": {
    "llm-eval-harness": {
      "type": "stdio",
      "command": "docker",
      "args": [
        "run",
        "-i",
        "--rm",
        "-e",
        "ANTHROPIC_API_KEY",
        "-e",
        "GEMINI_API_KEY",
        "ghcr.io/razilzul/llm-eval-harness:1.0.0"
      ],
      "env": {
        "ANTHROPIC_API_KEY": "<anthropic-api-key>",
        "GEMINI_API_KEY": "<gemini-api-key>"
      }
    }
  }
}

Add to .vscode/mcp.json in your workspace.

{
  "mcpServers": {
    "llm-eval-harness": {
      "command": "docker",
      "args": [
        "run",
        "-i",
        "--rm",
        "-e",
        "ANTHROPIC_API_KEY",
        "-e",
        "GEMINI_API_KEY",
        "ghcr.io/razilzul/llm-eval-harness:1.0.0"
      ],
      "env": {
        "ANTHROPIC_API_KEY": "<anthropic-api-key>",
        "GEMINI_API_KEY": "<gemini-api-key>"
      }
    }
  }
}

Add to ~/.codeium/windsurf/mcp_config.json.

Configuration

Variable	Required	Secret	Description
ANTHROPIC_API_KEY	—	yes	Only needed to evaluate anthropic:* models. Not needed for mock models.
GEMINI_API_KEY	—	yes	Only needed to evaluate gemini:* models. Not needed for mock models.

Freshness

Not enough public signals yet to judge maintenance.

Last commit (default branch)
2026-10-18 · today · GitHub
Latest release
no data · GitHub
Package published
no data · npm/PyPI
Registry entry updated
2026-10-18 · today · official registry · v1.0.0

How it compares in AI & LLM Tools

Against the most-starred ai & llm tools servers we index. Token cost is per model turn, measured from each server's live tool list.

Server	Stars	Tools	Tokens / turn	Freshness	Runs
LLM Eval Harness (this)	0	—	not measured	—	local
Memory	91,356	—	not measured	Active	local
Claude Flow	76,638	—	not measured	Active	local
Next AI Draw.io	36,318	—	not measured	—	local
Serena Agent	30,371	—	not measured	Active	local
Trycua	30,160	—	not measured	Active	—

All ai & llm tools MCP servers

FAQ

›How do I install the LLM Eval Harness MCP server in Claude Code?

Run: claude mcp add llm-eval-harness -e ANTHROPIC_API_KEY='<anthropic-api-key>' -e GEMINI_API_KEY='<gemini-api-key>' -- docker run -i --rm -e ANTHROPIC_API_KEY -e GEMINI_API_KEY ghcr.io/razilzul/llm-eval-harness:1.0.0. For Cursor, VS Code, Claude Desktop and Windsurf, use the install tabs above.

›Does LLM Eval Harness require an API key?

Yes. It expects ANTHROPIC_API_KEY, GEMINI_API_KEY, of which 2 are secrets.

›Can I use LLM Eval Harness as a remote (hosted) MCP server?

No hosted endpoint is published; it runs locally over stdio.

›Is LLM Eval Harness in the official MCP registry?

Yes, as io.github.razilzul/llm-eval-harness.

Sources and method

Official MCP Registry entry: updated 2026-10-18
GitHub repository: fetched 2026-10-19
Package registry (npm / PyPI): listed
Last checked against all sources: 2026-10-19

The description, packages and links come from the maintainer's own published metadata. Token cost, security signals, freshness and the reading above are MCPSkillHub's own measurements; see the methodology. Spot an error? Request a correction or claim the listing.