OpenJudge
OpenJudge: A Unified Framework for Holistic Evaluation and Quality Rewards
Install
pip install py-openjudgeInstall command taken from the project's README or registry entry. Always review code before running it.
README excerpt
Read full README on GitHub →Holistic Evaluation, Quality Rewards: Driving Application Excellence
🌟 If you find OpenJudge helpful, please give us a Star! 🌟
🌐 Website | 🚀 Try Online | 📖 Documentation | 🤝 Contributing | 🐾 PawBench | 中文
OpenJudge is an open-source evaluation framework for AI applications (e.g., AI agents or chatbots) designed to evaluate quality and drive continuous application optimization.
In practice, application excellence depends on a trustworthy evaluation workflow: Collect test data → Define graders → Run evaluation at scale → Analyze weaknesses → Iterate quickly.
OpenJudge provides ready-to-use graders and supports generating scenario-specific rubrics (as graders), making this workflow simpler, more professional, and easy to integrate into your workflow. It can also convert grading results into reward signals to help you fine-tune and optimize your application.
🚀 Try it now! Visit openjudge.me/app to use graders online — no installation required. Test built-in graders, build custom rubrics, and explore evaluation results directly in your browser.
📑 Table of Contents
- Key Features
- News
- Online Playground
- Installation
- Quickstart
- Integrations
- Ecosystem
- Contributing
- Community
- Citation
News
2026-06-17 - 🐾 PawBench v1.0 - A Model × Harness co-evaluation benchmark for agentic AI: 150 tasks · 9 models · 3 harnesses, with public prompts, graders, task labels, submissions, and leaderboard slices. 👉 GitHub | Leaderboard
2026-04-07 - 🔒 Skill Graders - 5 new LLM-based graders for evaluating AI Agent Skill packages: threat analysis (AITech taxonomy), declaration alignment, completeness, relevance, and design quality. 👉 Documentation | Cookbook
2026-03-10 - 🛠️ New Skills - Claude authenticity verification, find skills combo, and more. 👉 Browse Skills
2026-02-12 - 📚 Reference Hallucination Arena - Benchmark for evaluating LLM academic reference hallucination. 👉 Documentation | 📊 Leaderboard
2026-01-27 - 🆕 Paper Review - Automatically review academic papers using LLM-powered evaluation. 👉 Documentation
2026-01-27 - 🖥️ OpenJudge UI - A Streamlit-based visual interface for grader testing and Auto Arena. 👉 Try Online | Run locally:
streamlit run ui/app.py
✨ Key Features
📦 Systematic & Quality-Assured Grader Library
Access 50+ production-ready graders featuring a comprehensive taxonomy, rigorously validated for reliable performance.
🎯 General
Similar plugs
Makes your AI agent think like the laziest senior dev in the room. The best code is the code you never wrote.
Turn any codebase, with its docs, SQL schemas, configs, and PDFs, into a queryable knowledge graph. A /graphify skill for Claude Code, Cursor, Codex, and Gemini CLI: local…
Why use many token when few token do trick. Viral skill + proxy for coding agents that cuts 65% of tokens by talking like a caveman.
Production-grade engineering skills for AI coding agents.
The 1st Collaborative AI Design Agent Workspace. Local-first and open source. Create prototypes, websites, apps, dashboards, presentations, images, and videos with your team…