{"_id":"mcp-evals","_rev":"4229384","name":"mcp-evals","description":"GitHub Action for evaluating MCP server tool calls using LLM-based scoring","dist-tags":{"latest":"2.0.1"},"maintainers":[{"name":"mclenhard","email":""}],"time":{"modified":"2026-03-23T00:32:01.000Z","created":"2025-04-23T12:47:29.978Z","2.0.1":"2025-06-28T16:02:02.022Z","2.0.0":"2025-06-23T17:14:22.528Z","1.0.18":"2025-04-30T11:59:51.826Z","1.0.17":"2025-04-24T14:00:57.233Z","1.0.16":"2025-04-24T12:53:59.807Z","1.0.15":"2025-04-24T12:37:13.010Z","1.0.14":"2025-04-24T12:14:59.148Z","1.0.13":"2025-04-24T11:48:49.834Z","1.0.12":"2025-04-24T11:46:56.872Z","1.0.11":"2025-04-24T10:57:15.312Z","1.0.10":"2025-04-24T10:54:43.400Z","1.0.9":"2025-04-23T22:17:09.772Z","1.0.8":"2025-04-23T22:16:15.166Z","1.0.7":"2025-04-23T22:12:20.001Z","1.0.6":"2025-04-23T22:07:31.323Z","1.0.5":"2025-04-23T22:05:47.753Z","1.0.4":"2025-04-23T22:03:57.542Z","1.0.3":"2025-04-23T21:57:33.195Z","1.0.2":"2025-04-23T21:42:29.186Z","1.0.1":"2025-04-23T19:52:12.912Z","1.0.0":"2025-04-23T12:47:29.978Z"},"users":{},"author":{"name":"Matthew Lenhard"},"repository":{"type":"git","url":"git+https://github.com/mclenhard/mcp-evals.git"},"versions":{"2.0.1":{"name":"mcp-evals","version":"2.0.1","description":"GitHub Action for evaluating MCP server tool calls using LLM-based scoring","main":"dist/index.js","types":"dist/index.d.ts","type":"module","exports":{".":{"import":"./dist/index.js","types":"./dist/index.d.ts"}},"bin":{"mcp-eval":"dist/cli.js"},"scripts":{"clean":"rm -rf dist tsconfig.tsbuildinfo","build":"npm run clean && tsc && chmod +x dist/cli.js","dev":"tsc -w","test":"jest --config jest.config.cjs","test:unit":"jest --config jest.config.cjs --testPathPattern=unit","test:integration":"jest --config jest.config.cjs --testPathPattern=integration","test:watch":"jest --config jest.config.cjs --watch","test:coverage":"jest --config jest.config.cjs --coverage","eval":"tsx src/cli.ts","lint":"eslint src/**/*.ts"},"repository":{"type":"git","url":"git+https://github.com/mclenhard/mcp-evals.git"},"keywords":["mcp","evaluation","github-actions","llm"],"author":{"name":"Matthew Lenhard"},"license":"MIT","bugs":{"url":"https://github.com/mclenhard/mcp-evals/issues"},"homepage":"https://www.mcpevals.io/","dependencies":{"@actions/core":"^1.10.0","@ai-sdk/anthropic":"^1.2.12","@ai-sdk/openai":"^1.3.17","@anthropic-ai/sdk":"^0.8.0","@modelcontextprotocol/sdk":"^1.11.0","@opentelemetry/api":"^1.9.0","@opentelemetry/exporter-trace-otlp-http":"^0.200.0","@opentelemetry/resources":"^2.0.0","@opentelemetry/sdk-node":"^0.200.0","@opentelemetry/sdk-trace-base":"^2.0.0","@opentelemetry/semantic-conventions":"^1.33.0","@types/js-yaml":"^4.0.9","ai":"^4.3.9","chalk":"^4.1.2","dotenv":"^16.3.1","express":"^5.1.0","js-yaml":"^4.1.0","openai":"^4.24.1","prom-client":"^15.1.3","tsx":"^4.19.3"},"devDependencies":{"@types/express":"^5.0.1","@types/jest":"^30.0.0","@types/node":"^20.10.0","@typescript-eslint/eslint-plugin":"^6.13.0","@typescript-eslint/parser":"^6.13.0","eslint":"^8.54.0","jest":"^29.7.0","react":"^19.1.0","ts-jest":"^29.1.1","ts-node":"^10.9.1","typescript":"^5.3.2"},"peerDependencies":{"react":"^19.1.0"},"_id":"mcp-evals@2.0.1","gitHead":"3c435bebf429cb06d6ae4e09661f18c07a63e5a5","_nodeVersion":"22.12.0","_npmVersion":"10.9.0","dist":{"shasum":"2af4e7a00871a75dfdf6597f1f3e13289804cdd1","size":13434,"noattachment":false,"key":"/mcp-evals/-/mcp-evals-2.0.1.tgz","tarball":"http://registry.cnpm.dingdandao.com/mcp-evals/download/mcp-evals-2.0.1.tgz"},"_npmUser":{"name":"mclenhard","email":"mclenhard@gmail.com","actor":{"name":"mclenhard","email":"mclenhard@gmail.com","type":"user"}},"directories":{},"maintainers":[{"name":"mclenhard","email":""}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/mcp-evals_2.0.1_1751126521827_0.5217191424822174"},"_hasShrinkwrap":false,"_cnpmcore_publish_time":"2025-06-28T16:02:02.022Z","publish_time":1751126522022,"_source_registry_name":"default","_cnpm_publish_time":1751126522022},"2.0.0":{"name":"mcp-evals","version":"2.0.0","description":"GitHub Action for evaluating MCP server tool calls using LLM-based scoring","main":"dist/index.js","types":"dist/index.d.ts","type":"module","exports":{".":{"import":"./dist/index.js","types":"./dist/index.d.ts"}},"bin":{"mcp-eval":"dist/cli.js"},"scripts":{"clean":"rm -rf dist tsconfig.tsbuildinfo","build":"npm run clean && tsc && chmod +x dist/cli.js","dev":"tsc -w","test":"jest --config jest.config.cjs","test:unit":"jest --config jest.config.cjs --testPathPattern=unit","test:integration":"jest --config jest.config.cjs --testPathPattern=integration","test:watch":"jest --config jest.config.cjs --watch","test:coverage":"jest --config jest.config.cjs --coverage","eval":"tsx src/cli.ts","lint":"eslint src/**/*.ts"},"repository":{"type":"git","url":"git+https://github.com/mclenhard/mcp-evals.git"},"keywords":["mcp","evaluation","github-actions","llm"],"author":{"name":"Matthew Lenhard"},"license":"MIT","bugs":{"url":"https://github.com/mclenhard/mcp-evals/issues"},"homepage":"https://www.mcpevals.io/","dependencies":{"@actions/core":"^1.10.0","@ai-sdk/anthropic":"^1.2.12","@ai-sdk/openai":"^1.3.17","@anthropic-ai/sdk":"^0.8.0","@modelcontextprotocol/sdk":"^1.11.0","@opentelemetry/api":"^1.9.0","@opentelemetry/exporter-trace-otlp-http":"^0.200.0","@opentelemetry/resources":"^2.0.0","@opentelemetry/sdk-node":"^0.200.0","@opentelemetry/sdk-trace-base":"^2.0.0","@opentelemetry/semantic-conventions":"^1.33.0","@types/js-yaml":"^4.0.9","ai":"^4.3.9","chalk":"^4.1.2","dotenv":"^16.3.1","express":"^5.1.0","js-yaml":"^4.1.0","openai":"^4.24.1","prom-client":"^15.1.3","tsx":"^4.19.3"},"devDependencies":{"@types/express":"^5.0.1","@types/jest":"^30.0.0","@types/node":"^20.10.0","@typescript-eslint/eslint-plugin":"^6.13.0","@typescript-eslint/parser":"^6.13.0","eslint":"^8.54.0","jest":"^29.7.0","react":"^19.1.0","ts-jest":"^29.1.1","ts-node":"^10.9.1","typescript":"^5.3.2"},"peerDependencies":{"react":"^19.1.0"},"_id":"mcp-evals@2.0.0","gitHead":"3c435bebf429cb06d6ae4e09661f18c07a63e5a5","_nodeVersion":"22.12.0","_npmVersion":"10.9.0","dist":{"shasum":"7f165389c2d4f17a2bb2b13f97bdaa097cf4c31a","size":13430,"noattachment":false,"key":"/mcp-evals/-/mcp-evals-2.0.0.tgz","tarball":"http://registry.cnpm.dingdandao.com/mcp-evals/download/mcp-evals-2.0.0.tgz"},"_npmUser":{"name":"mclenhard","email":"mclenhard@gmail.com","actor":{"name":"mclenhard","email":"mclenhard@gmail.com","type":"user"}},"directories":{},"maintainers":[{"name":"mclenhard","email":""}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/mcp-evals_2.0.0_1750698862318_0.7348023212114174"},"_hasShrinkwrap":false,"_cnpmcore_publish_time":"2025-06-23T17:14:22.528Z","publish_time":1750698862528,"_source_registry_name":"default","_cnpm_publish_time":1750698862528},"1.0.18":{"name":"mcp-evals","version":"1.0.18","description":"GitHub Action for evaluating MCP server tool calls using LLM-based scoring","main":"dist/index.js","types":"dist/index.d.ts","type":"module","exports":{".":{"import":"./dist/index.js","types":"./dist/index.d.ts"}},"bin":{"mcp-eval":"dist/cli.js"},"scripts":{"clean":"rm -rf dist tsconfig.tsbuildinfo","build":"npm run clean && tsc && chmod +x dist/cli.js","dev":"tsc -w","test":"bin/mcp-eval","eval":"tsx src/cli.ts","lint":"eslint src/**/*.ts"},"repository":{"type":"git","url":"git+https://github.com/mclenhard/mcp-evals.git"},"keywords":["mcp","evaluation","github-actions","llm"],"author":{"name":"Matthew Lenhard"},"license":"MIT","bugs":{"url":"https://github.com/mclenhard/mcp-evals/issues"},"homepage":"https://www.mcpevals.io/","dependencies":{"@actions/core":"^1.10.0","@ai-sdk/openai":"^1.3.17","@anthropic-ai/sdk":"^0.8.0","@modelcontextprotocol/sdk":"^1.10.2","ai":"^4.3.9","chalk":"^4.1.2","dotenv":"^16.3.1","openai":"^4.24.1","tsx":"^4.19.3"},"devDependencies":{"@types/node":"^20.10.0","@typescript-eslint/eslint-plugin":"^6.13.0","@typescript-eslint/parser":"^6.13.0","eslint":"^8.54.0","jest":"^29.7.0","react":"^19.1.0","ts-jest":"^29.1.1","ts-node":"^10.9.1","typescript":"^5.3.2"},"peerDependencies":{"react":"^19.1.0"},"_id":"mcp-evals@1.0.18","gitHead":"e608a5fa7aa72960aa13a1998f76e1e8611c5492","_nodeVersion":"22.12.0","_npmVersion":"10.9.0","dist":{"shasum":"67ae90a5ce88398499919f260793fca34c18550d","size":6687,"noattachment":false,"key":"/mcp-evals/-/mcp-evals-1.0.18.tgz","tarball":"http://registry.cnpm.dingdandao.com/mcp-evals/download/mcp-evals-1.0.18.tgz"},"_npmUser":{"name":"mclenhard","email":"mclenhard@gmail.com"},"directories":{},"maintainers":[{"name":"mclenhard","email":""}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/mcp-evals_1.0.18_1746014391587_0.6696432428800245"},"_hasShrinkwrap":false,"_cnpmcore_publish_time":"2025-04-30T11:59:51.826Z","publish_time":1746014391826,"_source_registry_name":"default","_cnpm_publish_time":1746014391826},"1.0.17":{"name":"mcp-evals","version":"1.0.17","description":"GitHub Action for evaluating MCP server tool calls using LLM-based scoring","main":"dist/index.js","types":"dist/index.d.ts","type":"module","exports":{".":{"import":"./dist/index.js","types":"./dist/index.d.ts"}},"bin":{"mcp-eval":"dist/cli.js"},"scripts":{"clean":"rm -rf dist tsconfig.tsbuildinfo","build":"npm run clean && tsc && chmod +x dist/cli.js","dev":"tsc -w","test":"bin/mcp-eval","eval":"tsx src/cli.ts","lint":"eslint src/**/*.ts"},"repository":{"type":"git","url":"git+https://github.com/mclenhard/mcp-evals.git"},"keywords":["mcp","evaluation","github-actions","llm"],"author":{"name":"Matthew Lenhard"},"license":"MIT","bugs":{"url":"https://github.com/mclenhard/mcp-evals/issues"},"homepage":"https://github.com/mclenhard/mcp-evals#readme","dependencies":{"@actions/core":"^1.10.0","@ai-sdk/openai":"^1.3.17","@anthropic-ai/sdk":"^0.8.0","@modelcontextprotocol/sdk":"^1.10.2","ai":"^4.3.9","chalk":"^4.1.2","dotenv":"^16.3.1","openai":"^4.24.1","tsx":"^4.19.3"},"devDependencies":{"@types/node":"^20.10.0","@typescript-eslint/eslint-plugin":"^6.13.0","@typescript-eslint/parser":"^6.13.0","eslint":"^8.54.0","jest":"^29.7.0","react":"^19.1.0","ts-jest":"^29.1.1","ts-node":"^10.9.1","typescript":"^5.3.2"},"peerDependencies":{"react":"^19.1.0"},"_id":"mcp-evals@1.0.17","gitHead":"e608a5fa7aa72960aa13a1998f76e1e8611c5492","_nodeVersion":"22.12.0","_npmVersion":"10.9.0","dist":{"shasum":"725cd7637799ca7f24b5e1f0de61037c9aa5375d","size":6594,"noattachment":false,"key":"/mcp-evals/-/mcp-evals-1.0.17.tgz","tarball":"http://registry.cnpm.dingdandao.com/mcp-evals/download/mcp-evals-1.0.17.tgz"},"_npmUser":{"name":"mclenhard","email":"mclenhard@gmail.com"},"directories":{},"maintainers":[{"name":"mclenhard","email":""}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/mcp-evals_1.0.17_1745503257079_0.8054333100987525"},"_hasShrinkwrap":false,"_cnpmcore_publish_time":"2025-04-24T14:00:57.233Z","publish_time":1745503257233,"_source_registry_name":"default","_cnpm_publish_time":1745503257233},"1.0.16":{"name":"mcp-evals","version":"1.0.16","description":"GitHub Action for evaluating MCP server tool calls using LLM-based scoring","main":"dist/index.js","types":"dist/index.d.ts","type":"module","exports":{".":{"import":"./dist/index.js","types":"./dist/index.d.ts"}},"bin":{"mcp-eval":"dist/cli.js"},"scripts":{"clean":"rm -rf dist tsconfig.tsbuildinfo","build":"npm run clean && tsc && chmod +x dist/cli.js","dev":"tsc -w","test":"bin/mcp-eval","eval":"tsx src/cli.ts","lint":"eslint src/**/*.ts"},"repository":{"type":"git","url":"git+https://github.com/mclenhard/mcp-evals.git"},"keywords":["mcp","evaluation","github-actions","llm"],"author":{"name":"Matthew Lenhard"},"license":"MIT","bugs":{"url":"https://github.com/mclenhard/mcp-evals/issues"},"homepage":"https://github.com/mclenhard/mcp-evals#readme","dependencies":{"@actions/core":"^1.10.0","@ai-sdk/openai":"^1.3.17","@anthropic-ai/sdk":"^0.8.0","@modelcontextprotocol/sdk":"^1.10.2","ai":"^4.3.9","chalk":"^4.1.2","dotenv":"^16.3.1","openai":"^4.24.1","tsx":"^4.19.3"},"devDependencies":{"@types/node":"^20.10.0","@typescript-eslint/eslint-plugin":"^6.13.0","@typescript-eslint/parser":"^6.13.0","eslint":"^8.54.0","jest":"^29.7.0","react":"^19.1.0","ts-jest":"^29.1.1","ts-node":"^10.9.1","typescript":"^5.3.2"},"peerDependencies":{"react":"^19.1.0"},"_id":"mcp-evals@1.0.16","gitHead":"e683bdcd6dec1ba3a4202ea170441d73a298eacf","_nodeVersion":"22.12.0","_npmVersion":"10.9.0","dist":{"shasum":"cd5805fd8db49b44122bb4c0d21e7fcc18772df4","size":6567,"noattachment":false,"key":"/mcp-evals/-/mcp-evals-1.0.16.tgz","tarball":"http://registry.cnpm.dingdandao.com/mcp-evals/download/mcp-evals-1.0.16.tgz"},"_npmUser":{"name":"mclenhard","email":"mclenhard@gmail.com"},"directories":{},"maintainers":[{"name":"mclenhard","email":""}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/mcp-evals_1.0.16_1745499239640_0.16080520357449601"},"_hasShrinkwrap":false,"_cnpmcore_publish_time":"2025-04-24T12:53:59.807Z","publish_time":1745499239807,"_source_registry_name":"default","_cnpm_publish_time":1745499239807},"1.0.15":{"name":"mcp-evals","version":"1.0.15","description":"GitHub Action for evaluating MCP server tool calls using LLM-based scoring","main":"dist/index.js","types":"dist/index.d.ts","type":"module","exports":{".":{"import":"./dist/index.js","types":"./dist/index.d.ts"}},"bin":{"mcp-eval":"dist/cli.js"},"scripts":{"clean":"rm -rf dist tsconfig.tsbuildinfo","build":"npm run clean && tsc && chmod +x dist/cli.js","dev":"tsc -w","test":"bin/mcp-eval","eval":"tsx src/cli.ts","lint":"eslint src/**/*.ts"},"repository":{"type":"git","url":"git+https://github.com/mclenhard/mcp-evals.git"},"keywords":["mcp","evaluation","github-actions","llm"],"author":{"name":"Matthew Lenhard"},"license":"MIT","bugs":{"url":"https://github.com/mclenhard/mcp-evals/issues"},"homepage":"https://github.com/mclenhard/mcp-evals#readme","dependencies":{"@actions/core":"^1.10.0","@ai-sdk/openai":"^1.3.17","@anthropic-ai/sdk":"^0.8.0","@modelcontextprotocol/sdk":"^1.10.2","ai":"^4.3.9","chalk":"^4.1.2","openai":"^4.24.1","tsx":"^4.19.3"},"devDependencies":{"@types/node":"^20.10.0","@typescript-eslint/eslint-plugin":"^6.13.0","@typescript-eslint/parser":"^6.13.0","dotenv":"^16.3.1","eslint":"^8.54.0","jest":"^29.7.0","react":"^19.1.0","ts-jest":"^29.1.1","ts-node":"^10.9.1","typescript":"^5.3.2"},"peerDependencies":{"react":"^19.1.0"},"_id":"mcp-evals@1.0.15","gitHead":"9ed73c6c7970914af6345cffcdced2cf5bdfb2d6","_nodeVersion":"22.12.0","_npmVersion":"10.9.0","dist":{"shasum":"a4bf0583d5006307b9265b38fa390b02d69980fd","size":6570,"noattachment":false,"key":"/mcp-evals/-/mcp-evals-1.0.15.tgz","tarball":"http://registry.cnpm.dingdandao.com/mcp-evals/download/mcp-evals-1.0.15.tgz"},"_npmUser":{"name":"mclenhard","email":"mclenhard@gmail.com"},"directories":{},"maintainers":[{"name":"mclenhard","email":""}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/mcp-evals_1.0.15_1745498232811_0.6590994947791193"},"_hasShrinkwrap":false,"_cnpmcore_publish_time":"2025-04-24T12:37:13.010Z","publish_time":1745498233010,"_source_registry_name":"default","_cnpm_publish_time":1745498233010},"1.0.14":{"name":"mcp-evals","version":"1.0.14","description":"GitHub Action for evaluating MCP server tool calls using LLM-based scoring","main":"dist/index.js","types":"dist/index.d.ts","type":"module","exports":{".":{"import":"./dist/index.js","types":"./dist/index.d.ts"}},"bin":{"mcp-eval":"dist/cli.js"},"scripts":{"clean":"rm -rf dist tsconfig.tsbuildinfo","build":"npm run clean && tsc && chmod +x dist/cli.js","dev":"tsc -w","test":"bin/mcp-eval","eval":"tsx src/cli.ts","lint":"eslint src/**/*.ts"},"repository":{"type":"git","url":"git+https://github.com/mclenhard/mcp-evals.git"},"keywords":["mcp","evaluation","github-actions","llm"],"author":{"name":"Matthew Lenhard"},"license":"MIT","bugs":{"url":"https://github.com/mclenhard/mcp-evals/issues"},"homepage":"https://github.com/mclenhard/mcp-evals#readme","dependencies":{"@actions/core":"^1.10.0","@ai-sdk/openai":"^1.3.17","@anthropic-ai/sdk":"^0.8.0","@modelcontextprotocol/sdk":"^1.10.2","ai":"^4.3.9","chalk":"^4.1.2","openai":"^4.24.1","tsx":"^4.19.3"},"devDependencies":{"@types/node":"^20.10.0","@typescript-eslint/eslint-plugin":"^6.13.0","@typescript-eslint/parser":"^6.13.0","dotenv":"^16.3.1","eslint":"^8.54.0","jest":"^29.7.0","ts-jest":"^29.1.1","ts-node":"^10.9.1","typescript":"^5.3.2"},"peerDependencies":{"react":"^19.1.0"},"_id":"mcp-evals@1.0.14","gitHead":"9ed73c6c7970914af6345cffcdced2cf5bdfb2d6","_nodeVersion":"22.12.0","_npmVersion":"10.9.0","dist":{"shasum":"bf87a894572d7b286b31d2aeb994f5e61a0ff48f","size":6569,"noattachment":false,"key":"/mcp-evals/-/mcp-evals-1.0.14.tgz","tarball":"http://registry.cnpm.dingdandao.com/mcp-evals/download/mcp-evals-1.0.14.tgz"},"_npmUser":{"name":"mclenhard","email":"mclenhard@gmail.com"},"directories":{},"maintainers":[{"name":"mclenhard","email":""}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/mcp-evals_1.0.14_1745496898982_0.8870049923776979"},"_hasShrinkwrap":false,"_cnpmcore_publish_time":"2025-04-24T12:14:59.148Z","publish_time":1745496899148,"_source_registry_name":"default","_cnpm_publish_time":1745496899148},"1.0.13":{"name":"mcp-evals","version":"1.0.13","description":"GitHub Action for evaluating MCP server tool calls using LLM-based scoring","main":"dist/index.js","types":"dist/index.d.ts","type":"module","exports":{".":{"import":"./dist/index.js","types":"./dist/index.d.ts"}},"bin":{"mcp-eval":"dist/cli.js"},"scripts":{"clean":"rm -rf dist tsconfig.tsbuildinfo","build":"npm run clean && tsc && chmod +x dist/cli.js","dev":"tsc -w","test":"bin/mcp-eval","eval":"tsx src/cli.ts","lint":"eslint src/**/*.ts"},"repository":{"type":"git","url":"git+https://github.com/mclenhard/mcp-evals.git"},"keywords":["mcp","evaluation","github-actions","llm"],"author":{"name":"Matthew Lenhard"},"license":"MIT","bugs":{"url":"https://github.com/mclenhard/mcp-evals/issues"},"homepage":"https://github.com/mclenhard/mcp-evals#readme","dependencies":{"@actions/core":"^1.10.0","@ai-sdk/openai":"^1.3.17","@anthropic-ai/sdk":"^0.8.0","@modelcontextprotocol/sdk":"^1.10.2","ai":"^4.3.9","chalk":"^4.1.2","openai":"^4.24.1","tsx":"^4.19.3"},"devDependencies":{"@types/node":"^20.10.0","@typescript-eslint/eslint-plugin":"^6.13.0","@typescript-eslint/parser":"^6.13.0","dotenv":"^16.3.1","eslint":"^8.54.0","jest":"^29.7.0","ts-jest":"^29.1.1","ts-node":"^10.9.1","typescript":"^5.3.2"},"_id":"mcp-evals@1.0.13","gitHead":"c48c32e8ea05d472f1ffcb98f8aa4d967a665983","_nodeVersion":"22.12.0","_npmVersion":"10.9.0","dist":{"shasum":"505eb887ad7c8cba5262e4d39ce039b506ca654a","size":6535,"noattachment":false,"key":"/mcp-evals/-/mcp-evals-1.0.13.tgz","tarball":"http://registry.cnpm.dingdandao.com/mcp-evals/download/mcp-evals-1.0.13.tgz"},"_npmUser":{"name":"mclenhard","email":"mclenhard@gmail.com"},"directories":{},"maintainers":[{"name":"mclenhard","email":""}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/mcp-evals_1.0.13_1745495329657_0.7513944180626424"},"_hasShrinkwrap":false,"_cnpmcore_publish_time":"2025-04-24T11:48:49.834Z","publish_time":1745495329834,"_source_registry_name":"default","_cnpm_publish_time":1745495329834},"1.0.12":{"name":"mcp-evals","version":"1.0.12","description":"GitHub Action for evaluating MCP server tool calls using LLM-based scoring","main":"dist/index.js","types":"dist/index.d.ts","type":"module","exports":{".":{"import":"./dist/index.js","types":"./dist/index.d.ts"}},"bin":{"mcp-eval":"dist/cli.js"},"scripts":{"clean":"rm -rf dist tsconfig.tsbuildinfo","build":"npm run clean && tsc && chmod +x dist/cli.js","dev":"tsc -w","test":"bin/mcp-eval","eval":"tsx src/cli.ts","lint":"eslint src/**/*.ts"},"repository":{"type":"git","url":"git+https://github.com/mclenhard/mcp-evals.git"},"keywords":["mcp","evaluation","github-actions","llm"],"author":{"name":"Matthew Lenhard"},"license":"MIT","bugs":{"url":"https://github.com/mclenhard/mcp-evals/issues"},"homepage":"https://github.com/mclenhard/mcp-evals#readme","dependencies":{"@actions/core":"^1.10.0","@ai-sdk/openai":"^1.3.17","@anthropic-ai/sdk":"^0.8.0","@modelcontextprotocol/sdk":"^1.10.2","ai":"^4.3.9","chalk":"^4.1.2","openai":"^4.24.1","tsx":"^4.19.3"},"devDependencies":{"@types/node":"^20.10.0","@typescript-eslint/eslint-plugin":"^6.13.0","@typescript-eslint/parser":"^6.13.0","dotenv":"^16.3.1","eslint":"^8.54.0","jest":"^29.7.0","ts-jest":"^29.1.1","ts-node":"^10.9.1","typescript":"^5.3.2"},"_id":"mcp-evals@1.0.12","gitHead":"c48c32e8ea05d472f1ffcb98f8aa4d967a665983","_nodeVersion":"22.12.0","_npmVersion":"10.9.0","dist":{"shasum":"7bdc5ec8a443958ef03e202cec47d85b805c7efe","size":6531,"noattachment":false,"key":"/mcp-evals/-/mcp-evals-1.0.12.tgz","tarball":"http://registry.cnpm.dingdandao.com/mcp-evals/download/mcp-evals-1.0.12.tgz"},"_npmUser":{"name":"mclenhard","email":"mclenhard@gmail.com"},"directories":{},"maintainers":[{"name":"mclenhard","email":""}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/mcp-evals_1.0.12_1745495216669_0.3632997891984431"},"_hasShrinkwrap":false,"_cnpmcore_publish_time":"2025-04-24T11:46:56.872Z","publish_time":1745495216872,"_source_registry_name":"default","_cnpm_publish_time":1745495216872},"1.0.11":{"name":"mcp-evals","version":"1.0.11","description":"GitHub Action for evaluating MCP server tool calls using LLM-based scoring","main":"dist/index.js","types":"dist/index.d.ts","type":"module","exports":{".":{"import":"./dist/index.js","types":"./dist/index.d.ts"}},"bin":{"mcp-eval":"dist/cli.js"},"scripts":{"clean":"rm -rf dist tsconfig.tsbuildinfo","build":"npm run clean && tsc && chmod +x dist/cli.js","dev":"tsc -w","test":"bin/mcp-eval","eval":"tsx src/cli.ts","lint":"eslint src/**/*.ts"},"repository":{"type":"git","url":"git+https://github.com/mclenhard/mcp-evals.git"},"keywords":["mcp","evaluation","github-actions","llm"],"author":{"name":"Matthew Lenhard"},"license":"MIT","bugs":{"url":"https://github.com/mclenhard/mcp-evals/issues"},"homepage":"https://github.com/mclenhard/mcp-evals#readme","dependencies":{"@actions/core":"^1.10.0","@ai-sdk/openai":"^1.3.17","@anthropic-ai/sdk":"^0.8.0","@modelcontextprotocol/sdk":"^1.10.2","ai":"^4.3.9","chalk":"^4.1.2","openai":"^4.24.1","tsx":"^4.19.3"},"devDependencies":{"@types/node":"^20.10.0","@typescript-eslint/eslint-plugin":"^6.13.0","@typescript-eslint/parser":"^6.13.0","dotenv":"^16.3.1","eslint":"^8.54.0","jest":"^29.7.0","ts-jest":"^29.1.1","ts-node":"^10.9.1","typescript":"^5.3.2"},"_id":"mcp-evals@1.0.11","gitHead":"c48c32e8ea05d472f1ffcb98f8aa4d967a665983","_nodeVersion":"22.12.0","_npmVersion":"10.9.0","dist":{"shasum":"71bb7bb3399023e51b3fd09d966c2dc2e98cc422","size":6376,"noattachment":false,"key":"/mcp-evals/-/mcp-evals-1.0.11.tgz","tarball":"http://registry.cnpm.dingdandao.com/mcp-evals/download/mcp-evals-1.0.11.tgz"},"_npmUser":{"name":"mclenhard","email":"mclenhard@gmail.com"},"directories":{},"maintainers":[{"name":"mclenhard","email":""}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/mcp-evals_1.0.11_1745492235125_0.7347316091980429"},"_hasShrinkwrap":false,"_cnpmcore_publish_time":"2025-04-24T10:57:15.312Z","publish_time":1745492235312,"_source_registry_name":"default","_cnpm_publish_time":1745492235312},"1.0.10":{"name":"mcp-evals","version":"1.0.10","description":"GitHub Action for evaluating MCP server tool calls using LLM-based scoring","main":"dist/index.js","types":"dist/index.d.ts","type":"module","exports":{".":{"import":"./dist/index.js","types":"./dist/index.d.ts"}},"bin":{"mcp-eval":"dist/cli.js"},"scripts":{"clean":"rm -rf dist tsconfig.tsbuildinfo","build":"npm run clean && tsc && chmod +x dist/cli.js","dev":"tsc -w","test":"bin/mcp-eval","eval":"tsx src/cli.ts","lint":"eslint src/**/*.ts"},"repository":{"type":"git","url":"git+https://github.com/mclenhard/mcp-evals.git"},"keywords":["mcp","evaluation","github-actions","llm"],"author":{"name":"Matthew Lenhard"},"license":"MIT","bugs":{"url":"https://github.com/mclenhard/mcp-evals/issues"},"homepage":"https://github.com/mclenhard/mcp-evals#readme","dependencies":{"@actions/core":"^1.10.0","@ai-sdk/openai":"^1.3.17","@anthropic-ai/sdk":"^0.8.0","@modelcontextprotocol/sdk":"^1.10.2","ai":"^4.3.9","chalk":"^4.1.2","openai":"^4.24.1","tsx":"^4.19.3"},"devDependencies":{"@types/node":"^20.10.0","@typescript-eslint/eslint-plugin":"^6.13.0","@typescript-eslint/parser":"^6.13.0","dotenv":"^16.3.1","eslint":"^8.54.0","jest":"^29.7.0","ts-jest":"^29.1.1","ts-node":"^10.9.1","typescript":"^5.3.2"},"_id":"mcp-evals@1.0.10","gitHead":"c48c32e8ea05d472f1ffcb98f8aa4d967a665983","_nodeVersion":"22.12.0","_npmVersion":"10.9.0","dist":{"shasum":"3c73240897c6e26f3eaa782e97e4395dd27dff8f","size":6376,"noattachment":false,"key":"/mcp-evals/-/mcp-evals-1.0.10.tgz","tarball":"http://registry.cnpm.dingdandao.com/mcp-evals/download/mcp-evals-1.0.10.tgz"},"_npmUser":{"name":"mclenhard","email":"mclenhard@gmail.com"},"directories":{},"maintainers":[{"name":"mclenhard","email":""}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/mcp-evals_1.0.10_1745492083228_0.8208468334813837"},"_hasShrinkwrap":false,"_cnpmcore_publish_time":"2025-04-24T10:54:43.400Z","publish_time":1745492083400,"_source_registry_name":"default","_cnpm_publish_time":1745492083400},"1.0.9":{"name":"mcp-evals","version":"1.0.9","description":"GitHub Action for evaluating MCP server tool calls using LLM-based scoring","main":"dist/index.js","types":"dist/index.d.ts","type":"module","exports":{".":{"import":"./dist/index.js","types":"./dist/index.d.ts"}},"bin":{"mcp-eval":"dist/cli.js"},"scripts":{"clean":"rm -rf dist tsconfig.tsbuildinfo","build":"npm run clean && tsc","dev":"tsc -w","eval":"tsx src/cli.ts","lint":"eslint src/**/*.ts"},"repository":{"type":"git","url":"git+https://github.com/mclenhard/mcp-evals.git"},"keywords":["mcp","evaluation","github-actions","llm"],"author":{"name":"Matthew Lenhard"},"license":"MIT","bugs":{"url":"https://github.com/mclenhard/mcp-evals/issues"},"homepage":"https://github.com/mclenhard/mcp-evals#readme","dependencies":{"@actions/core":"^1.10.0","@ai-sdk/openai":"^1.3.17","@anthropic-ai/sdk":"^0.8.0","@modelcontextprotocol/sdk":"^1.10.2","ai":"^4.3.9","chalk":"^4.1.2","openai":"^4.24.1","tsx":"^4.19.3"},"devDependencies":{"@types/node":"^20.10.0","@typescript-eslint/eslint-plugin":"^6.13.0","@typescript-eslint/parser":"^6.13.0","dotenv":"^16.3.1","eslint":"^8.54.0","jest":"^29.7.0","ts-jest":"^29.1.1","ts-node":"^10.9.1","typescript":"^5.3.2"},"_id":"mcp-evals@1.0.9","gitHead":"c48c32e8ea05d472f1ffcb98f8aa4d967a665983","_nodeVersion":"22.12.0","_npmVersion":"10.9.0","dist":{"shasum":"666f8a99e3feceac6a276717130e75cb3de09831","size":6575,"noattachment":false,"key":"/mcp-evals/-/mcp-evals-1.0.9.tgz","tarball":"http://registry.cnpm.dingdandao.com/mcp-evals/download/mcp-evals-1.0.9.tgz"},"_npmUser":{"name":"mclenhard","email":"mclenhard@gmail.com"},"directories":{},"maintainers":[{"name":"mclenhard","email":""}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/mcp-evals_1.0.9_1745446629510_0.05879453874365148"},"_hasShrinkwrap":false,"_cnpmcore_publish_time":"2025-04-23T22:17:09.772Z","publish_time":1745446629772,"_source_registry_name":"default","_cnpm_publish_time":1745446629772},"1.0.8":{"name":"mcp-evals","version":"1.0.8","description":"GitHub Action for evaluating MCP server tool calls using LLM-based scoring","main":"dist/index.js","types":"dist/index.d.ts","type":"module","exports":{".":{"import":"./dist/index.js","types":"./dist/index.d.ts"}},"bin":{"mcp-eval":"src/cli.ts"},"scripts":{"clean":"rm -rf dist tsconfig.tsbuildinfo","build":"npm run clean && tsc","dev":"tsc -w","eval":"tsx src/cli.ts","lint":"eslint src/**/*.ts"},"repository":{"type":"git","url":"git+https://github.com/mclenhard/mcp-evals.git"},"keywords":["mcp","evaluation","github-actions","llm"],"author":{"name":"Matthew Lenhard"},"license":"MIT","bugs":{"url":"https://github.com/mclenhard/mcp-evals/issues"},"homepage":"https://github.com/mclenhard/mcp-evals#readme","dependencies":{"@actions/core":"^1.10.0","@ai-sdk/openai":"^1.3.17","@anthropic-ai/sdk":"^0.8.0","@modelcontextprotocol/sdk":"^1.10.2","ai":"^4.3.9","chalk":"^4.1.2","openai":"^4.24.1","tsx":"^4.19.3"},"devDependencies":{"@types/node":"^20.10.0","@typescript-eslint/eslint-plugin":"^6.13.0","@typescript-eslint/parser":"^6.13.0","dotenv":"^16.3.1","eslint":"^8.54.0","jest":"^29.7.0","ts-jest":"^29.1.1","ts-node":"^10.9.1","typescript":"^5.3.2"},"_id":"mcp-evals@1.0.8","gitHead":"c48c32e8ea05d472f1ffcb98f8aa4d967a665983","_nodeVersion":"22.12.0","_npmVersion":"10.9.0","dist":{"shasum":"16a4b3bc57e52a8fae0600582ae3902e2a64bf79","size":6803,"noattachment":false,"key":"/mcp-evals/-/mcp-evals-1.0.8.tgz","tarball":"http://registry.cnpm.dingdandao.com/mcp-evals/download/mcp-evals-1.0.8.tgz"},"_npmUser":{"name":"mclenhard","email":"mclenhard@gmail.com"},"directories":{},"maintainers":[{"name":"mclenhard","email":""}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/mcp-evals_1.0.8_1745446574969_0.29202152517608515"},"_hasShrinkwrap":false,"_cnpmcore_publish_time":"2025-04-23T22:16:15.166Z","publish_time":1745446575166,"_source_registry_name":"default","_cnpm_publish_time":1745446575166},"1.0.7":{"name":"mcp-evals","version":"1.0.7","description":"GitHub Action for evaluating MCP server tool calls using LLM-based scoring","main":"dist/index.js","types":"dist/index.d.ts","type":"module","exports":{".":{"import":"./dist/index.js","types":"./dist/index.d.ts"}},"bin":{"mcp-eval":"src/cli.ts"},"scripts":{"clean":"rm -rf dist tsconfig.tsbuildinfo","build":"npm run clean && tsc","dev":"tsc -w","eval":"tsx src/cli.ts","lint":"eslint src/**/*.ts"},"repository":{"type":"git","url":"git+https://github.com/mclenhard/mcp-evals.git"},"keywords":["mcp","evaluation","github-actions","llm"],"author":{"name":"Matthew Lenhard"},"license":"MIT","bugs":{"url":"https://github.com/mclenhard/mcp-evals/issues"},"homepage":"https://github.com/mclenhard/mcp-evals#readme","dependencies":{"@actions/core":"^1.10.0","@ai-sdk/openai":"^1.3.17","@anthropic-ai/sdk":"^0.8.0","@modelcontextprotocol/sdk":"^1.10.2","ai":"^4.3.9","chalk":"^4.1.2","openai":"^4.24.1","tsx":"^4.19.3"},"devDependencies":{"@types/node":"^20.10.0","@typescript-eslint/eslint-plugin":"^6.13.0","@typescript-eslint/parser":"^6.13.0","dotenv":"^16.3.1","eslint":"^8.54.0","jest":"^29.7.0","ts-jest":"^29.1.1","ts-node":"^10.9.1","typescript":"^5.3.2"},"_id":"mcp-evals@1.0.7","gitHead":"c48c32e8ea05d472f1ffcb98f8aa4d967a665983","_nodeVersion":"22.12.0","_npmVersion":"10.9.0","dist":{"shasum":"d9a2772a2678b0ba9e70cc5c30b73c0d605a4162","size":6665,"noattachment":false,"key":"/mcp-evals/-/mcp-evals-1.0.7.tgz","tarball":"http://registry.cnpm.dingdandao.com/mcp-evals/download/mcp-evals-1.0.7.tgz"},"_npmUser":{"name":"mclenhard","email":"mclenhard@gmail.com"},"directories":{},"maintainers":[{"name":"mclenhard","email":""}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/mcp-evals_1.0.7_1745446339813_0.42769065860670374"},"_hasShrinkwrap":false,"_cnpmcore_publish_time":"2025-04-23T22:12:20.001Z","publish_time":1745446340001,"_source_registry_name":"default","_cnpm_publish_time":1745446340001},"1.0.6":{"name":"mcp-evals","version":"1.0.6","description":"GitHub Action for evaluating MCP server tool calls using LLM-based scoring","main":"dist/index.js","types":"dist/index.d.ts","type":"module","exports":{".":{"import":"./dist/index.js","types":"./dist/index.d.ts"}},"bin":{"mcp-eval":"dist/cli.js"},"scripts":{"clean":"rm -rf dist tsconfig.tsbuildinfo","build":"npm run clean && tsc","dev":"tsc -w","eval":"tsx src/cli.ts","lint":"eslint src/**/*.ts"},"repository":{"type":"git","url":"git+https://github.com/mclenhard/mcp-evals.git"},"keywords":["mcp","evaluation","github-actions","llm"],"author":{"name":"Matthew Lenhard"},"license":"MIT","bugs":{"url":"https://github.com/mclenhard/mcp-evals/issues"},"homepage":"https://github.com/mclenhard/mcp-evals#readme","dependencies":{"@actions/core":"^1.10.0","@ai-sdk/openai":"^1.3.17","@anthropic-ai/sdk":"^0.8.0","@modelcontextprotocol/sdk":"^1.10.2","ai":"^4.3.9","chalk":"^4.1.2","openai":"^4.24.1","tsx":"^4.19.3"},"devDependencies":{"@types/node":"^20.10.0","@typescript-eslint/eslint-plugin":"^6.13.0","@typescript-eslint/parser":"^6.13.0","dotenv":"^16.3.1","eslint":"^8.54.0","jest":"^29.7.0","ts-jest":"^29.1.1","ts-node":"^10.9.1","typescript":"^5.3.2"},"_id":"mcp-evals@1.0.6","gitHead":"c48c32e8ea05d472f1ffcb98f8aa4d967a665983","_nodeVersion":"22.12.0","_npmVersion":"10.9.0","dist":{"shasum":"50b015261c029233fd61b8878a5fd68fe64c5924","size":6244,"noattachment":false,"key":"/mcp-evals/-/mcp-evals-1.0.6.tgz","tarball":"http://registry.cnpm.dingdandao.com/mcp-evals/download/mcp-evals-1.0.6.tgz"},"_npmUser":{"name":"mclenhard","email":"mclenhard@gmail.com"},"directories":{},"maintainers":[{"name":"mclenhard","email":""}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/mcp-evals_1.0.6_1745446051133_0.6028233056360313"},"_hasShrinkwrap":false,"_cnpmcore_publish_time":"2025-04-23T22:07:31.323Z","publish_time":1745446051323,"_source_registry_name":"default","_cnpm_publish_time":1745446051323},"1.0.5":{"name":"mcp-evals","version":"1.0.5","description":"GitHub Action for evaluating MCP server tool calls using LLM-based scoring","main":"dist/index.js","types":"dist/index.d.ts","type":"module","exports":{".":{"import":"./dist/index.js","types":"./dist/index.d.ts"}},"bin":{"mcp-eval":"dist/cli.js"},"scripts":{"clean":"rm -rf dist tsconfig.tsbuildinfo","build":"npm run clean && tsc","dev":"tsc -w","eval":"tsx src/cli.ts","lint":"eslint src/**/*.ts"},"repository":{"type":"git","url":"git+https://github.com/mclenhard/mcp-evals.git"},"keywords":["mcp","evaluation","github-actions","llm"],"author":{"name":"Matthew Lenhard"},"license":"MIT","bugs":{"url":"https://github.com/mclenhard/mcp-evals/issues"},"homepage":"https://github.com/mclenhard/mcp-evals#readme","dependencies":{"@actions/core":"^1.10.0","@ai-sdk/openai":"^1.3.17","@anthropic-ai/sdk":"^0.8.0","@modelcontextprotocol/sdk":"^1.10.2","ai":"^4.3.9","chalk":"^4.1.2","openai":"^4.24.1","tsx":"^4.19.3"},"devDependencies":{"@types/node":"^20.10.0","@typescript-eslint/eslint-plugin":"^6.13.0","@typescript-eslint/parser":"^6.13.0","dotenv":"^16.3.1","eslint":"^8.54.0","jest":"^29.7.0","ts-jest":"^29.1.1","ts-node":"^10.9.1","typescript":"^5.3.2"},"_id":"mcp-evals@1.0.5","gitHead":"c48c32e8ea05d472f1ffcb98f8aa4d967a665983","_nodeVersion":"22.12.0","_npmVersion":"10.9.0","dist":{"shasum":"8ab262db511e629f85a32d8fe729d85da5ce1f75","size":6244,"noattachment":false,"key":"/mcp-evals/-/mcp-evals-1.0.5.tgz","tarball":"http://registry.cnpm.dingdandao.com/mcp-evals/download/mcp-evals-1.0.5.tgz"},"_npmUser":{"name":"mclenhard","email":"mclenhard@gmail.com"},"directories":{},"maintainers":[{"name":"mclenhard","email":""}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/mcp-evals_1.0.5_1745445947531_0.5066030408950066"},"_hasShrinkwrap":false,"_cnpmcore_publish_time":"2025-04-23T22:05:47.753Z","publish_time":1745445947753,"_source_registry_name":"default","_cnpm_publish_time":1745445947753},"1.0.4":{"name":"mcp-evals","version":"1.0.4","description":"GitHub Action for evaluating MCP server tool calls using LLM-based scoring","main":"dist/index.js","type":"module","bin":{"mcp-eval":"dist/cli.js"},"scripts":{"clean":"rm -rf dist tsconfig.tsbuildinfo","build":"npm run clean && tsc","dev":"tsc -w","eval":"tsx src/cli.ts","lint":"eslint src/**/*.ts"},"repository":{"type":"git","url":"git+https://github.com/mclenhard/mcp-evals.git"},"keywords":["mcp","evaluation","github-actions","llm"],"author":{"name":"Matthew Lenhard"},"license":"MIT","bugs":{"url":"https://github.com/mclenhard/mcp-evals/issues"},"homepage":"https://github.com/mclenhard/mcp-evals#readme","dependencies":{"@actions/core":"^1.10.0","@ai-sdk/openai":"^1.3.17","@anthropic-ai/sdk":"^0.8.0","@modelcontextprotocol/sdk":"^1.10.2","ai":"^4.3.9","chalk":"^4.1.2","openai":"^4.24.1","tsx":"^4.19.3"},"devDependencies":{"@types/node":"^20.10.0","@typescript-eslint/eslint-plugin":"^6.13.0","@typescript-eslint/parser":"^6.13.0","dotenv":"^16.3.1","eslint":"^8.54.0","jest":"^29.7.0","ts-jest":"^29.1.1","ts-node":"^10.9.1","typescript":"^5.3.2"},"_id":"mcp-evals@1.0.4","gitHead":"c48c32e8ea05d472f1ffcb98f8aa4d967a665983","types":"./dist/index.d.ts","_nodeVersion":"22.12.0","_npmVersion":"10.9.0","dist":{"shasum":"d7382f58a14aa20d97a7aecb9639b81f052a9d3a","size":6202,"noattachment":false,"key":"/mcp-evals/-/mcp-evals-1.0.4.tgz","tarball":"http://registry.cnpm.dingdandao.com/mcp-evals/download/mcp-evals-1.0.4.tgz"},"_npmUser":{"name":"mclenhard","email":"mclenhard@gmail.com"},"directories":{},"maintainers":[{"name":"mclenhard","email":""}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/mcp-evals_1.0.4_1745445837324_0.03070904775985661"},"_hasShrinkwrap":false,"_cnpmcore_publish_time":"2025-04-23T22:03:57.542Z","publish_time":1745445837542,"_source_registry_name":"default","_cnpm_publish_time":1745445837542},"1.0.3":{"name":"mcp-evals","version":"1.0.3","description":"GitHub Action for evaluating MCP server tool calls using LLM-based scoring","main":"dist/index.js","type":"module","bin":{"mcp-eval":"dist/cli.js"},"scripts":{"clean":"rm -rf dist tsconfig.tsbuildinfo","build":"npm run clean && tsc","dev":"tsc -w","eval":"tsx src/cli.ts","lint":"eslint src/**/*.ts"},"repository":{"type":"git","url":"git+https://github.com/mclenhard/mcp-evals.git"},"keywords":["mcp","evaluation","github-actions","llm"],"author":{"name":"Matthew Lenhard"},"license":"MIT","bugs":{"url":"https://github.com/mclenhard/mcp-evals/issues"},"homepage":"https://github.com/mclenhard/mcp-evals#readme","dependencies":{"@actions/core":"^1.10.0","@ai-sdk/openai":"^1.3.17","@anthropic-ai/sdk":"^0.8.0","@modelcontextprotocol/sdk":"^1.10.2","ai":"^4.3.9","chalk":"^4.1.2","openai":"^4.24.1","tsx":"^4.19.3"},"devDependencies":{"@types/node":"^20.10.0","@typescript-eslint/eslint-plugin":"^6.13.0","@typescript-eslint/parser":"^6.13.0","dotenv":"^16.3.1","eslint":"^8.54.0","jest":"^29.7.0","ts-jest":"^29.1.1","ts-node":"^10.9.1","typescript":"^5.3.2"},"_id":"mcp-evals@1.0.3","gitHead":"c48c32e8ea05d472f1ffcb98f8aa4d967a665983","types":"./dist/index.d.ts","_nodeVersion":"22.12.0","_npmVersion":"10.9.0","dist":{"shasum":"d228ba1c3610c4f3d8c0d61e3b100a525c0147a1","size":6203,"noattachment":false,"key":"/mcp-evals/-/mcp-evals-1.0.3.tgz","tarball":"http://registry.cnpm.dingdandao.com/mcp-evals/download/mcp-evals-1.0.3.tgz"},"_npmUser":{"name":"mclenhard","email":"mclenhard@gmail.com"},"directories":{},"maintainers":[{"name":"mclenhard","email":""}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/mcp-evals_1.0.3_1745445453006_0.7655100108404687"},"_hasShrinkwrap":false,"_cnpmcore_publish_time":"2025-04-23T21:57:33.195Z","publish_time":1745445453195,"_source_registry_name":"default","_cnpm_publish_time":1745445453195},"1.0.2":{"name":"mcp-evals","version":"1.0.2","description":"GitHub Action for evaluating MCP server tool calls using LLM-based scoring","main":"dist/index.js","type":"module","bin":{"mcp-eval":"dist/cli.js"},"scripts":{"build":"tsc -b","dev":"tsc -w","eval":"tsx src/cli.ts","lint":"eslint src/**/*.ts"},"repository":{"type":"git","url":"git+https://github.com/mclenhard/mcp-evals.git"},"keywords":["mcp","evaluation","github-actions","llm"],"author":{"name":"Matthew Lenhard"},"license":"MIT","bugs":{"url":"https://github.com/mclenhard/mcp-evals/issues"},"homepage":"https://github.com/mclenhard/mcp-evals#readme","dependencies":{"@actions/core":"^1.10.0","@ai-sdk/openai":"^1.3.17","@anthropic-ai/sdk":"^0.8.0","@modelcontextprotocol/sdk":"^1.10.2","ai":"^4.3.9","chalk":"^4.1.2","openai":"^4.24.1","tsx":"^4.19.3"},"devDependencies":{"@types/node":"^20.10.0","@typescript-eslint/eslint-plugin":"^6.13.0","@typescript-eslint/parser":"^6.13.0","dotenv":"^16.3.1","eslint":"^8.54.0","jest":"^29.7.0","ts-jest":"^29.1.1","ts-node":"^10.9.1","typescript":"^5.3.2"},"_id":"mcp-evals@1.0.2","gitHead":"c48c32e8ea05d472f1ffcb98f8aa4d967a665983","types":"./dist/index.d.ts","_nodeVersion":"22.12.0","_npmVersion":"10.9.0","dist":{"shasum":"d5fba47d04a7baae4184cb582b8036e4d1e61d00","size":5842,"noattachment":false,"key":"/mcp-evals/-/mcp-evals-1.0.2.tgz","tarball":"http://registry.cnpm.dingdandao.com/mcp-evals/download/mcp-evals-1.0.2.tgz"},"_npmUser":{"name":"mclenhard","email":"mclenhard@gmail.com"},"directories":{},"maintainers":[{"name":"mclenhard","email":""}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/mcp-evals_1.0.2_1745444549001_0.3967137312628708"},"_hasShrinkwrap":false,"_cnpmcore_publish_time":"2025-04-23T21:42:29.186Z","publish_time":1745444549186,"_source_registry_name":"default","_cnpm_publish_time":1745444549186},"1.0.1":{"name":"mcp-evals","version":"1.0.1","description":"GitHub Action for evaluating MCP server tool calls using LLM-based scoring","main":"dist/index.js","type":"module","bin":{"mcp-eval":"dist/cli.js"},"scripts":{"build":"tsc -b","dev":"tsc -w","eval":"tsx src/cli.ts","lint":"eslint src/**/*.ts"},"repository":{"type":"git","url":"git+https://github.com/mclenhard/mcp-evals.git"},"keywords":["mcp","evaluation","github-actions","llm"],"author":{"name":"Matthew Lenhard"},"license":"MIT","bugs":{"url":"https://github.com/mclenhard/mcp-evals/issues"},"homepage":"https://github.com/mclenhard/mcp-evals#readme","dependencies":{"@actions/core":"^1.10.0","@ai-sdk/openai":"^1.3.17","@anthropic-ai/sdk":"^0.8.0","@modelcontextprotocol/sdk":"^1.10.2","ai":"^4.3.9","chalk":"^4.1.2","openai":"^4.24.1","tsx":"^4.19.3"},"devDependencies":{"@types/node":"^20.10.0","@typescript-eslint/eslint-plugin":"^6.13.0","@typescript-eslint/parser":"^6.13.0","dotenv":"^16.3.1","eslint":"^8.54.0","jest":"^29.7.0","ts-jest":"^29.1.1","ts-node":"^10.9.1","typescript":"^5.3.2"},"_id":"mcp-evals@1.0.1","gitHead":"c48c32e8ea05d472f1ffcb98f8aa4d967a665983","types":"./dist/index.d.ts","_nodeVersion":"22.12.0","_npmVersion":"10.9.0","dist":{"shasum":"f31b483e82457912563b871255713d2bf11860c1","size":25087,"noattachment":false,"key":"/mcp-evals/-/mcp-evals-1.0.1.tgz","tarball":"http://registry.cnpm.dingdandao.com/mcp-evals/download/mcp-evals-1.0.1.tgz"},"_npmUser":{"name":"mclenhard","email":"mclenhard@gmail.com"},"directories":{},"maintainers":[{"name":"mclenhard","email":""}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/mcp-evals_1.0.1_1745437932640_0.5459626458797531"},"_hasShrinkwrap":false,"_cnpmcore_publish_time":"2025-04-23T19:52:12.912Z","publish_time":1745437932912,"_source_registry_name":"default","_cnpm_publish_time":1745437932912},"1.0.0":{"name":"mcp-evals","version":"1.0.0","description":"GitHub Action for evaluating MCP server tool calls using LLM-based scoring","main":"dist/index.js","type":"module","bin":{"mcp-eval":"dist/cli.js"},"scripts":{"build":"tsc -b","dev":"tsc -w","eval":"tsx src/cli.ts","lint":"eslint src/**/*.ts"},"repository":{"type":"git","url":"git+https://github.com/mclenhard/mcp-evals.git"},"keywords":["mcp","evaluation","github-actions","llm"],"author":{"name":"Matthew Lenhard"},"license":"MIT","bugs":{"url":"https://github.com/mclenhard/mcp-evals/issues"},"homepage":"https://github.com/mclenhard/mcp-evals#readme","dependencies":{"@actions/core":"^1.10.0","@ai-sdk/openai":"^1.3.17","@anthropic-ai/sdk":"^0.8.0","@modelcontextprotocol/sdk":"^1.10.2","ai":"^4.3.9","chalk":"^4.1.2","openai":"^4.24.1","tsx":"^4.19.3"},"devDependencies":{"@types/node":"^20.10.0","@typescript-eslint/eslint-plugin":"^6.13.0","@typescript-eslint/parser":"^6.13.0","dotenv":"^16.3.1","eslint":"^8.54.0","jest":"^29.7.0","ts-jest":"^29.1.1","ts-node":"^10.9.1","typescript":"^5.3.2"},"_id":"mcp-evals@1.0.0","types":"./dist/index.d.ts","_nodeVersion":"22.12.0","_npmVersion":"10.9.0","dist":{"shasum":"92b3f43b30d64340e2ef72eb5fff797e281b7eff","size":25298,"noattachment":false,"key":"/mcp-evals/-/mcp-evals-1.0.0.tgz","tarball":"http://registry.cnpm.dingdandao.com/mcp-evals/download/mcp-evals-1.0.0.tgz"},"_npmUser":{"name":"mclenhard","email":"mclenhard@gmail.com"},"directories":{},"maintainers":[{"name":"mclenhard","email":""}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/mcp-evals_1.0.0_1745412449775_0.7031772201743416"},"_hasShrinkwrap":false,"_cnpmcore_publish_time":"2025-04-23T12:47:29.978Z","publish_time":1745412449978,"_source_registry_name":"default","_cnpm_publish_time":1745412449978}},"readme":"# MCP Evals\n\nA Node.js package and GitHub Action for evaluating MCP (Model Context Protocol) tool implementations using LLM-based scoring, **with built-in observability support**. This helps ensure your MCP server's tools are working correctly, performing well, and are fully observable with integrated monitoring and metrics.\n\n## Installation\n\n### As a Node.js Package\n\n```bash\nnpm install mcp-evals\n```\n\n### As a GitHub Action\n\nAdd the following to your workflow file:\n\n```yaml\nname: Run MCP Evaluations\non:\n  pull_request:\n    types: [opened, synchronize, reopened]\njobs:\n  evaluate:\n    runs-on: ubuntu-latest\n    permissions:\n      contents: read\n      pull-requests: write\n    steps:\n      - uses: actions/checkout@v4\n      \n      - name: Setup Node.js\n        uses: actions/setup-node@v4\n        with:\n          node-version: '20'\n          \n      - name: Install dependencies\n        run: npm install\n        \n      - name: Run MCP Evaluations\n        uses: mclenhard/mcp-evals@v1.0.9\n        with:\n          evals_path: 'src/evals/evals.ts'    # Can also use .yaml files\n          server_path: 'src/index.ts'\n          openai_api_key: ${{ secrets.OPENAI_API_KEY }}\n          model: 'gpt-4'  # Optional, defaults to gpt-4\n```\n\n## Usage -- Evals\n\n### 1. Create Your Evaluation File\n\nYou can create evaluation configurations in either TypeScript or YAML format.\n\n#### Option A: TypeScript Configuration\n\nCreate a file (e.g., `evals.ts`) that exports your evaluation configuration:\n\n```typescript\nimport { EvalConfig } from 'mcp-evals';\nimport { openai } from \"@ai-sdk/openai\";\nimport { grade, EvalFunction} from \"mcp-evals\";\n\nconst weatherEval: EvalFunction = {\n    name: 'Weather Tool Evaluation',\n    description: 'Evaluates the accuracy and completeness of weather information retrieval',\n    run: async () => {\n      const result = await grade(openai(\"gpt-4\"), \"What is the weather in New York?\");\n      return JSON.parse(result);\n    }\n};\nconst config: EvalConfig = {\n    model: openai(\"gpt-4\"),\n    evals: [weatherEval]\n  };\n  \n  export default config;\n  \n  export const evals = [\n    weatherEval,\n    // add other evals here\n]; \n```\n\n#### Option B: YAML Configuration\n\nFor simpler configuration, you can use YAML format (e.g., `evals.yaml`):\n\n```yaml\n# Model configuration\nmodel:\n  provider: openai     # 'openai' or 'anthropic'\n  name: gpt-4o        # Model name\n  # api_key: sk-...   # Optional, uses OPENAI_API_KEY env var by default\n\n# List of evaluations to run\nevals:\n  - name: weather_query_basic\n    description: Test basic weather information retrieval\n    prompt: \"What is the current weather in San Francisco?\"\n    expected_result: \"Should return current weather data for San Francisco including temperature, conditions, etc.\"\n\n  - name: weather_forecast\n    description: Test weather forecast functionality\n    prompt: \"Can you give me the 3-day weather forecast for Seattle?\"\n    expected_result: \"Should return a multi-day forecast for Seattle\"\n\n  - name: invalid_location\n    description: Test handling of invalid location requests\n    prompt: \"What's the weather in Atlantis?\"\n    expected_result: \"Should handle invalid location gracefully with appropriate error message\"\n```\n\n### 2. Run the Evaluations\n\n#### As a Node.js Package\n\nYou can run the evaluations using the CLI with either TypeScript or YAML files:\n\n```bash\n# Using TypeScript configuration\nnpx mcp-eval path/to/your/evals.ts path/to/your/server.ts\n\n# Using YAML configuration\nnpx mcp-eval path/to/your/evals.yaml path/to/your/server.ts\n```\n\n#### As a GitHub Action\n\nThe action will automatically:\n1. Run your evaluations\n2. Post the results as a comment on the PR\n3. Update the comment if the PR is updated\n\n## Evaluation Results\n\nEach evaluation returns an object with the following structure:\n\n```typescript\ninterface EvalResult {\n  accuracy: number;        // Score from 1-5\n  completeness: number;    // Score from 1-5\n  relevance: number;       // Score from 1-5\n  clarity: number;         // Score from 1-5\n  reasoning: number;       // Score from 1-5\n  overall_comments: string; // Summary of strengths and weaknesses\n}\n```\n\n## Configuration\n\n### Environment Variables\n\n- `OPENAI_API_KEY`: Your OpenAI API key (required for OpenAI models)\n- `ANTHROPIC_API_KEY`: Your Anthropic API key (required for Anthropic models)\n\n> [!NOTE]\n> If you're using this GitHub Action with open source software, enable data sharing in the OpenAI billing dashboard to claim 2.5 million free GPT-4o mini tokens per day, making this Action effectively free to use.\n\n### Evaluation Configuration\n\n#### TypeScript Configuration\n\nThe `EvalConfig` interface requires:\n\n- `model`: The language model to use for evaluation (e.g., GPT-4)\n- `evals`: Array of evaluation functions to run\n\nEach evaluation function must implement:\n\n- `name`: Name of the evaluation\n- `description`: Description of what the evaluation tests\n- `run`: Async function that takes a model and returns an `EvalResult`\n\n#### YAML Configuration\n\nYAML configuration files support:\n\n**Model Configuration:**\n- `provider`: Either 'openai' or 'anthropic'\n- `name`: Model name (e.g., 'gpt-4o', 'claude-3-opus-20240229')\n- `api_key`: Optional API key (uses environment variables by default)\n\n**Evaluation Configuration:**\n- `name`: Name of the evaluation (required)\n- `description`: Description of what the evaluation tests (required)\n- `prompt`: The prompt to send to your MCP server (required)\n- `expected_result`: Optional description of expected behavior\n\n**Supported File Extensions:** `.yaml`, `.yml`\n\n## Usage -- Monitoring\n\n> **Note:** The metrics functionality is still in alpha. Features and APIs may change, and breaking changes are possible.\n\n1. Add the following to your application before you initilize the MCP server. \n\n```typescript \nimport { metrics } from 'mcp-evals';\nmetrics.initialize(9090, { enableTracing: true, otelEndpoint: 'http://localhost:4318/v1/traces' });\n```\n\n2. Start the monitoring stack:\n\n```bash\ndocker-compose up -d\n```\n\n3. Run your MCP server and it will automatically connect to the monitoring stack.\n\n### Accessing the Dashboards\n\n- **Prometheus**: http://localhost:9090\n- **Grafana**: http://localhost:3000 (username: admin, password: admin)\n- **Jaeger UI**: http://localhost:16686\n\n## Metrics Available\n\n- **Tool Calls**: Number of tool calls by tool name\n- **Tool Errors**: Number of errors by tool name\n- **Tool Latency**: Distribution of latency times by tool name\n\n\n## License\n\nMIT \n","_attachments":{},"homepage":"https://www.mcpevals.io/","bugs":{"url":"https://github.com/mclenhard/mcp-evals/issues"},"license":"MIT"}