self-eval

Overview of self-eval skill

What self-eval is for

The self-eval skill is a prompt-only Claude Code skill for honest post-work evaluation. It helps an AI agent assess its own output after a task, code review, implementation session, debugging pass, or planning exercise without defaulting to vague praise or inflated “4/5” scores.

Instead of asking “How good was this?” on one subjective scale, self-eval separates evaluation into two axes: task ambition and execution quality. That makes it useful when you need a calibrated answer to: “Was this genuinely strong work, or just adequate work on an easy task?”

Best-fit users and tasks

Use the self-eval skill if you want an agent to review completed work before you accept it, merge it, or continue building on it. It is especially relevant for engineering teams using AI in code generation, refactoring, issue triage, test writing, or architecture analysis.

It is not a replacement for test suites, human review, security review, or production validation. Its value is in structured self-critique: catching overconfidence, surfacing weaknesses, and creating a score trail across work sessions.

What makes this different from a normal prompt

A normal “evaluate your work” prompt often produces generic positives, mild caveats, and an optimistic score. self-eval adds constraints that make inflation harder: mandatory devil’s advocate reasoning, a fixed scoring matrix, and score persistence to .self-eval-scores.jsonl.

That persistence matters. If recent evaluations cluster too tightly, the skill can flag score inflation patterns instead of treating each review as isolated.

How to Use self-eval skill

self-eval install and repository check

Install from the GitHub skill source using your skill manager, for example:

npx skills add alirezarezvani/claude-skills --skill self-eval

The relevant repository path is:

engineering/skills/self-eval/SKILL.md

There are no external dependencies, helper scripts, bundled rules, or reference files in the current structure. For a fast pre-install review, read SKILL.md first; it contains the scoring model, workflow, and output expectations. Because this is a prompt-only skill, adoption risk is low, but output quality depends heavily on giving it enough task context.

Inputs the skill needs

For useful self-eval usage, do not simply say “evaluate this.” Provide the agent with the completed work and the standard it should be judged against.

Strong inputs include:

• The original user request or issue description
• The final answer, patch, plan, or review produced by the agent
• Relevant constraints, such as time limits, style rules, test requirements, or forbidden approaches
• Evidence of validation, such as tests run, files changed, commands executed, or known gaps
• The intended audience: maintainer, reviewer, product owner, beginner user, or production team

A weak prompt asks for a score. A strong prompt gives the skill enough evidence to distinguish “low-ambition but clean” from “high-ambition but incomplete.”

Prompt pattern for better results

Use self-eval after the work is done, not before. A practical prompt looks like:

Use the self-eval skill to evaluate the work below. Original task: [goal]. Output produced: [answer or diff summary]. Constraints: [tests, style, repo rules]. Validation performed: [commands or none]. Known concerns: [risks]. Give the two-axis evaluation, devil's advocate reasoning, final matrix score, and concrete follow-up actions.

This structure improves the result because it prevents the model from evaluating only tone or effort. It forces the review to consider ambition, execution, evidence, and unresolved risk.

Suggested workflow

Run self-eval at checkpoints where a misleadingly positive assessment would be costly:

1. Complete the task or draft the solution.
2. Ask for self-eval using the original goal and evidence of work.
3. Read the devil’s advocate section before the final score.
4. Convert the weaknesses into a short fix list.
5. Re-run only after meaningful changes, not after cosmetic edits.

If .self-eval-scores.jsonl is created in your working directory, treat it as local evaluation history. Decide whether your team wants that file committed, ignored, or periodically reviewed.

self-eval skill FAQ

Is self-eval for Model Evaluation or code quality?

It is useful for both, but with a narrow meaning. self-eval for Model Evaluation is best understood as task-level evaluation of an AI agent’s own work, not benchmark-grade model measurement. It can help compare sessions, detect inflated self-ratings, and improve review discipline, but it does not replace formal eval harnesses, golden datasets, or human-labeled scoring.

When should I not use self-eval?

Do not use it as the only gate for high-risk work: security-sensitive changes, legal or medical content, production migrations, or anything requiring verified correctness. Also avoid it when there is no concrete artifact to evaluate. The skill needs a task, an output, and evaluation criteria; otherwise it will still produce structure but with weak evidence.

Is this beginner-friendly?

Yes, because it is prompt-only and has no tooling dependencies. Beginners should still read SKILL.md before using it, especially the scoring logic. The main learning curve is not installation; it is providing enough context so the evaluation is not based on vibes.

How is it different from asking for a critique?

A critique can list problems without calibrated scoring. self-eval uses a two-axis model and a matrix-locked final score, so the agent has less freedom to justify whatever rating “feels” right. The devil’s advocate step also requires arguing both for a higher and lower score before resolving the final judgment.

How to Improve self-eval skill

Give self-eval stronger evidence

The best way to improve self-eval results is to make the evidence concrete. Include changed files, failing or passing tests, important omissions, and acceptance criteria. If no tests were run, say so. If the answer intentionally skipped a requirement, include that too.

Better input does not mean longer input; it means evaluable input. “Refactored auth code” is weak. “Refactored auth/session.ts to remove duplicated token parsing; ran npm test -- auth; did not test OAuth callback manually” gives the skill something real to score.

Watch for common failure modes

The most common failure is letting the model reward effort instead of outcome. Another is treating a hard task with partial execution as automatically strong. self-eval is designed to resist that, but only if the user includes the original ambition and the actual deliverable.

Also watch for score-history noise. If .self-eval-scores.jsonl mixes unrelated task types, clustering signals may be less meaningful. A documentation polish pass and a complex migration should not be interpreted as equivalent just because they share a numeric score.

Iterate from score to action

Do not stop at the final score. The useful output is the gap between ambition and execution. Turn that gap into a repair prompt:

Based on the self-eval weaknesses, revise the work to address the top three execution gaps. Do not expand scope. Preserve the original constraints and report what changed.

This keeps the next iteration focused. It also prevents the agent from “improving” the work by adding unrelated features to chase a higher score.

Customize cautiously

If you adapt the self-eval skill, preserve the parts that create calibration: separated axes, devil’s advocate reasoning, fixed score mapping, and score-history awareness. Customizing labels, output format, or team-specific acceptance criteria is usually safe. Removing the constraints that make the model argue against itself will make the skill behave more like an ordinary review prompt.