eval

Overview of eval skill

What eval does for AgentHub sessions

eval is an AgentHub skill for ranking completed agent results. It is designed for the /hub:eval command, where several agents have already worked on the same task and you need to score, compare, or choose a winner. The skill supports metric-based evaluation when an eval command is configured, LLM judge evaluation when no metric is available, and a hybrid decision style when both objective scores and code judgment matter.

Best-fit users and jobs

The eval skill is best for developers using an AgentHub-style multi-agent workflow: one task, multiple agent branches or worktrees, then a final comparison step. It helps when you want a repeatable answer to questions like “which agent produced the fastest implementation?”, “which patch is safest to merge?”, or “which result best satisfies the original task?” It is less useful as a standalone benchmarking framework because it assumes AgentHub session structure, agent result posts, branches, and diffs.

What makes eval different from a generic prompt

A generic prompt can ask an LLM to compare outputs, but eval provides a concrete evaluation path: run a metric command per agent when available, or inspect each agent diff and result post when using judge mode. The important differentiator is that the skill orients the assistant around completed AgentHub artifacts instead of free-form opinions. That reduces guesswork and makes the ranking easier to audit.

Main adoption considerations

Before installing or relying on eval, confirm that your workflow stores agent results in the expected AgentHub locations and that agent branches or worktrees are still available. Metric mode also depends on a working evaluation command, metric name, and direction such as lower-is-better latency or higher-is-better score. The repository path for this skill contains only SKILL.md, so most behavior is defined by the command instructions rather than extra helper files inside the skill directory.

How to Use eval skill

eval install and repository check

Install from the GitHub skill repository using your normal skill installer, for example:

npx skills add alirezarezvani/claude-skills --skill eval

After install, read engineering/agenthub/skills/eval/SKILL.md first. There are no local rules/, resources/, references/, or scripts/ folders in this skill directory, so SKILL.md is the source of truth. Note that the skill text references scripts/result_ranker.py for metric mode; verify whether your broader AgentHub setup provides that script or equivalent evaluation runner before depending on metric-based ranking.

Basic eval usage commands

Use the command after agents have completed a session:

/hub:eval
/hub:eval 20260317-143022
/hub:eval --judge

/hub:eval evaluates the latest session. Passing a session ID targets a specific run. --judge forces LLM judge mode, which is useful when your metric command is missing, unreliable, or too narrow to capture correctness.

Inputs that make eval work well

For metric mode, provide or configure: session ID, eval command, metric label, and direction. A strong request is specific:

“Run /hub:eval 20260317-143022 using the configured benchmark. Rank by latency_ms, lower is better, and call out any agent whose result fails tests.”

For LLM judge mode, make sure the assistant can access the base branch, agent branches, git diffs, and result posts such as .agenthub/board/results/agent-1-result.md. A stronger prompt includes the task goal and priority order:

“Use /hub:eval --judge for the latest session. Prioritize correctness first, then minimal risk, then simplicity. Treat changed public APIs as risky unless justified in the result post.”

Practical workflow for reliable rankings

Run eval only after all agents have posted results and their branches are clean enough to diff. Start with metric mode when the task has an objective score such as runtime, test count, accuracy, size, or benchmark output. Use LLM judge mode for design, refactoring, bug fixes, or tasks where a metric can be gamed. For important merges, ask eval to report not just the winner but also the top risks, evidence from diffs, and any tie-breaking assumptions.

eval skill FAQ

Is eval for Model Evaluation or agent result ranking?

This eval skill is primarily for AgentHub agent result ranking, not a general-purpose Model Evaluation suite. It can evaluate model-generated work, but the unit of comparison is an agent’s completed session result: its branch, diff, result note, and optionally a metric command run in its worktree.

When should I use metric mode instead of judge mode?

Use metric mode when success can be measured consistently: tests passed, benchmark score, latency, memory, accuracy, bundle size, or another numeric output. Use judge mode when the real question is whether the patch is correct, maintainable, and safe. If the metric captures only part of the goal, ask for a hybrid read: rank by metric, then flag correctness or regression concerns from diffs.

Can beginners use the eval skill?

Beginners can use eval if they already understand the AgentHub session concept. The command surface is small, but the evaluation quality depends on repository state: branches, worktrees, result posts, and configured eval commands. If those artifacts are missing, a beginner may see confusing or incomplete rankings.

When is eval the wrong tool?

Do not use eval before agents have finished, when there is only one result, or when the assistant cannot access diffs and result files. It is also a poor fit for broad model benchmarking, prompt leaderboard creation, or offline dataset evaluation unless you adapt the surrounding workflow. For those cases, a dedicated evaluation harness is more appropriate.

How to Improve eval skill

Improve eval results with clearer criteria

The biggest quality boost is a precise ranking policy. Tell eval what matters most: correctness, passing tests, performance, simplicity, security, compatibility, or minimal code churn. Avoid vague requests like “pick the best.” Prefer: “Rank agents by correctness first; if tied, prefer fewer changed files and no new dependencies; mention any untested assumptions.”

Prevent common eval failure modes

Common failures include ranking by a misleading metric, ignoring a failing edge case, comparing stale branches, or overvaluing a smaller diff that does not solve the task. Prevent this by confirming the base branch, session ID, metric direction, and task objective before evaluation. For judge mode, ask for evidence-backed rankings tied to specific diffs and result posts.

Iterate after the first ranking

Treat the first eval output as a decision draft. If the winner is surprising, ask for a second pass focused on the disputed criterion: “Re-evaluate only the top two agents for regression risk,” or “Explain whether agent-2’s faster metric comes from skipping required behavior.” This keeps the eval workflow practical without rerunning the entire agent session.

Strengthen the skill for your own workflow

If you maintain an AgentHub setup, improve eval by standardizing result post format, naming metrics consistently, and making the eval command deterministic. Add project-specific guidance for what counts as correctness, what tests must pass, and which risks block a merge. The skill is compact, so local conventions around sessions, branches, and metrics will determine how trustworthy eval feels in daily use.