diff options
Diffstat (limited to '.agents/skills/skill-quality-reviewer/references')
4 files changed, 1523 insertions, 0 deletions
diff --git a/.agents/skills/skill-quality-reviewer/references/batch-review-template.md b/.agents/skills/skill-quality-reviewer/references/batch-review-template.md new file mode 100644 index 0000000..8c1f6ae --- /dev/null +++ b/.agents/skills/skill-quality-reviewer/references/batch-review-template.md @@ -0,0 +1,14 @@ +# Batch Review Template + +## Portfolio Summary +- Skills reviewed: +- Highest-risk skills: +- Common issue clusters: + +## Priority Backlog +### P0 +- +### P1 +- +### P2 +- diff --git a/.agents/skills/skill-quality-reviewer/references/examples-bad.md b/.agents/skills/skill-quality-reviewer/references/examples-bad.md new file mode 100644 index 0000000..ba1671a --- /dev/null +++ b/.agents/skills/skill-quality-reviewer/references/examples-bad.md @@ -0,0 +1,555 @@ +# Common Skill Anti-Patterns + +Collection of common mistakes and anti-patterns to avoid when creating Claude Skills. Learn from these examples to improve your skills. + +## Anti-Pattern 1: Vague Description (F Grade) + +### The Problem + +```yaml +--- +name: helper-skill +description: Use this skill to help with various tasks and provide useful +assistance for users. +version: 1.0.0 +--- +``` + +**Why this fails:** +- ❌ No specific trigger phrases +- ❌ Second person ("Use this skill") +- ❌ Vague and generic +- ❌ No clue what it actually does +- ❌ Users will never discover it + +### Score Impact + +| Dimension | Score | Why | +|-----------|-------|-----| +| Description Quality | 0/100 | No triggers, second person, vague | +| Overall | F | Cannot even get started | + +### The Fix + +```yaml +--- +name: api-contract-manager +description: This skill should be used when the user asks to "validate API +contract", "check API compatibility", "version API schema", or "detect +breaking changes". Manages API contracts through validation, versioning, +and compatibility checking for OpenAPI 3.0 specifications. +version: 1.0.0 +--- +``` + +**Improvement:** 0/100 → 90/100 in description quality + +--- + +## Anti-Pattern 2: Second Person Throughout (D Grade) + +### The Problem + +```markdown +# My Skill + +## Overview +You should use this skill when you need to process data files. + +## How to Use +First, you need to select your input file. You can choose from CSV, +JSON, or XML formats. Then you should configure the processing options. + +## Configuration +You can set the following options: +- You must specify the output format +- You should choose a delimiter +- You need to define the schema +``` + +**Why this fails:** +- ❌ "You should", "you need", "you can" throughout +- ❌ Not imperative form +- ❌ Conversational rather than instructional +- ❌ Unprofessional tone + +### Score Impact + +| Dimension | Score | Why | +|-----------|-------|-----| +| Writing Style | 5/100 | Over 20 instances of "you" | +| Overall | D | Major style issue | + +### The Fix + +```markdown +# Data File Processor + +## Overview +Process data files in CSV, JSON, or XML formats with configurable +schemas and output options. + +## Usage +Select the input file. Configure processing options. Specify output +format and delimiter. + +## Configuration +Set the following options: +- Output format (required) +- Field delimiter +- Data schema +``` + +**Improvement:** 5/100 → 85/100 in writing style + +--- + +## Anti-Pattern 3: Everything in One File (C Grade) + +### The Problem + +``` +my-skill/ +└── SKILL.md (12,000 words) +``` + +**SKILL.md contains:** +- Overview +- Detailed API documentation (3,000 words) +- 20 complete code examples (5,000 words) +- Troubleshooting guide (2,000 words) +- FAQ (1,000 words) +- Changelog (1,000 words) + +**Why this is problematic:** +- ❌ SKILL.md is 12,000 words (should be <5,000) +- ❌ Detailed content always loaded +- ❌ Wastes tokens on rarely-used content +- ❌ Difficult to navigate +- ❌ Violates progressive disclosure + +### Score Impact + +| Dimension | Score | Why | +|-----------|-------|-----| +| Content Organization | 10/100 | No progressive disclosure | +| Overall | C | Major organizational issue | + +### The Fix + +``` +my-skill/ +├── SKILL.md (1,800 words) +├── references/ +│ ├── api-reference.md (3,000 words) +│ ├── troubleshooting.md (2,000 words) +│ └── faq.md (1,000 words) +└── examples/ + ├── basic-usage.md (1,000 words) + ├── advanced-usage.md (2,000 words) + └── edge-cases.md (2,000 words) +``` + +**SKILL.md now contains:** +- Overview +- Quick start +- Basic operations +- Links to detailed references + +**Improvement:** 10/100 → 85/100 in content organization + +--- + +## Anti-Pattern 4: Missing Required Fields (F Grade) + +### The Problem + +```yaml +--- +name: my-skill +# Missing description! +version: 1.0.0 +author: Someone +tags: [utility, helper] +--- +``` + +**Why this fails:** +- ❌ Missing required `description` field +- ❌ Skill will never trigger +- ❌ Has unnecessary fields (author, tags) +- ❌ Cannot be discovered + +### Score Impact + +| Dimension | Score | Why | +|-----------|-------|-----| +| Structural Integrity | 0/100 | Missing required field | +| Description Quality | 0/100 | No description to evaluate | +| Overall | F | Fundamentally broken | + +### The Fix + +```yaml +--- +name: data-validator +description: This skill should be used when the user asks to "validate data", +"check data format", "verify data schema", or "sanitize input". Validate and +sanitize data files against JSON schemas with support for custom validation +rules. +version: 1.0.0 +--- +``` + +**Improvement:** F → B- (80) overall + +--- + +## Anti-Pattern 5: Broken References (D Grade) + +### The Problem + +**SKILL.md says:** +```markdown +## Additional Resources + +See `references/api-guide.md` for complete API documentation. +Check `examples/complete-example.py` for a working example. +``` + +**Actual directory structure:** +``` +my-skill/ +├── SKILL.md +├── references/ +│ └── setup-guide.md # No api-guide.md! +└── examples/ + └── basic-example.py # No complete-example.py! +``` + +**Why this fails:** +- ❌ Referenced files don't exist +- ❌ Users get errors when trying to follow references +- ❌ Incomplete skill structure + +### Score Impact + +| Dimension | Score | Why | +|-----------|-------|-----| +| Structural Integrity | 15/100 | Multiple broken references | +| Overall | D | Structural issues | + +### The Fix + +Option 1: Create the referenced files +``` +my-skill/ +├── SKILL.md +├── references/ +│ ├── api-guide.md # Created! +│ └── setup-guide.md +└── examples/ + ├── complete-example.py # Created! + └── basic-example.py +``` + +Option 2: Update SKILL.md to match actual files +```markdown +## Additional Resources + +See `references/setup-guide.md` for setup instructions. +Check `examples/basic-example.py` for a working example. +``` + +**Improvement:** 15/100 → 90/100 in structural integrity + +--- + +## Anti-Pattern 6: Inconsistent Style (C Grade) + +### The Problem + +```markdown +## Workflow + +### Step 1: Prepare +First you should create the directory structure. + +### Step 2: Configure +Configure the settings file. You need to specify the endpoint. + +### Step 3: Deploy +Deploy the application. Check the logs to verify. +``` + +**Why this fails:** +- ❌ Mixes imperative ("Configure", "Deploy") with second person ("you should") +- ❌ Inconsistent throughout +- ❌ Confusing to read + +### Score Impact + +| Dimension | Score | Why | +|-----------|-------|-----| +| Writing Style | 30/100 | Inconsistent style | +| Overall | C | Style issues | + +### The Fix + +```markdown +## Workflow + +### Step 1: Prepare +Create the directory structure. + +### Step 2: Configure +Specify the endpoint in the settings file. + +### Step 3: Deploy +Deploy the application. Verify by checking the logs. +``` + +**Improvement:** 30/100 → 85/100 in writing style + +--- + +## Anti-Pattern 7: Overly Long Description (B Grade) + +### The Problem + +```yaml +--- +name: file-processor +description: This skill should be used when the user asks to "process files", +"convert file formats", "validate file structure", "check file integrity", +"transform file data", "merge multiple files", "split large files", "compress +file size", "encrypt file contents", "decrypt file contents", "archive files", +"extract archives", "generate file checksums", "verify file signatures", or +"optimize file storage". This comprehensive file processing utility supports +over 50 different file formats including PDF, DOCX, XLSX, CSV, JSON, XML, +HTML, TXT, and many more. It provides advanced features like batch processing, +parallel execution, error recovery, automatic backup creation, detailed logging, +progress tracking, and notification support. Ideal for data migration tasks, +content management workflows, archival operations, and file system maintenance. +version: 1.0.0 +--- +``` + +**Why this is problematic:** +- ❌ 900+ characters (way too long) +- ❌ Lists too many trigger phrases (dilutes effectiveness) +- ❌ Becomes unreadable +- ❌ Loses focus + +### Score Impact + +| Dimension | Score | Why | +|-----------|-------|-----| +| Description Quality | 15/100 | Too long, unfocused | +| Overall | B | Description issue | + +### The Fix + +```yaml +--- +name: file-processor +description: This skill should be used when the user asks to "process files", +"convert file formats", "validate file structure", "merge files", or +"compress files". Supports common formats (PDF, DOCX, CSV, JSON) with +batch processing and error recovery. +version: 1.0.0 +--- +``` + +**Improvement:** 15/100 → 85/100 in description quality + +--- + +## Anti-Pattern 8: No Examples (C Grade) + +### The Problem + +``` +my-skill/ +├── SKILL.md (2,000 words of theory) +└── references/ + └── detailed-guide.md (3,000 words more theory) +``` + +**SKILL.md describes concepts but never shows:** +- How to actually use the skill +- What input looks like +- What output looks like +- Concrete usage patterns + +**Why this fails:** +- ❌ Users can't visualize usage +- ❌ No working code to copy +- ❌ Theory without practice + +### Score Impact + +| Dimension | Score | Why | +|-----------|-------|-----| +| Structural Integrity | 50/100 | No examples despite theory | +| Overall | C | Missing practical component | + +### The Fix + +``` +my-skill/ +├── SKILL.md (2,000 words) +├── references/ +│ └── detailed-guide.md +└── examples/ + ├── basic-usage.sh (working example) + ├── advanced-usage.sh (working example) + └── input-output-examples.md (before/after) +``` + +**Improvement:** 50/100 → 85/100 in structural integrity + +--- + +## Anti-Pattern 9: Subjective Language (D Grade) + +### The Problem + +```markdown +## Overview + +This is a really great skill that I think you'll find super useful. +It's awesome for handling difficult tasks. The best part is how +easy it is to use - you'll love it! + +## Features + +- Amazing performance +- Incredible flexibility +- Fantastic user experience +``` + +**Why this fails:** +- ❌ Highly subjective ("really great", "super useful") +- ❌ Marketing language, not instructional +- ❌ First person ("I think") +- ❌ Unprofessional tone +- ❌ No concrete information + +### Score Impact + +| Dimension | Score | Why | +|-----------|-------|-----| +| Writing Style | 20/100 | Subjective, unprofessional | +| Overall | D | Style issues | + +### The Fix + +```markdown +## Overview + +Handle complex data processing tasks with configurable validation +and error recovery. Supports CSV, JSON, and XML formats with +batch processing capabilities. + +## Features + +- Processes 10,000+ records per second +- Configurable validation rules +- Automatic error recovery +- Detailed logging +``` + +**Improvement:** 20/100 → 80/100 in writing style + +--- + +## Anti-Pattern 10: Invalid YAML (F Grade) + +### The Problem + +```yaml +--- +name: my-skill +description: This skill should be used when the user asks to "validate data" +or "check schemas". It's designed for: data validation, schema checking, and +format verification. +version: 1.0.0 +--- +``` + +**Error:** Colon after "for" creates invalid YAML syntax + +**Why this fails:** +- ❌ YAML cannot be parsed +- ❌ Skill fails to load +- ❌ Completely non-functional + +### Score Impact + +| Dimension | Score | Why | +|-----------|-------|-----| +| Structural Integrity | 0/100 | Invalid YAML | +| Overall | F | Cannot load | + +### The Fix + +```yaml +--- +name: data-validator +description: This skill should be used when the user asks to "validate data", +"check schemas", or "verify formats". Performs data validation, schema +checking, and format verification. +version: 1.0.0 +--- +``` + +**Improvement:** F → B (83) overall + +--- + +## Quick Anti-Pattern Checklist + +### Description Quality +- [ ] Not vague like "helps with tasks" +- [ ] Has 3-5 specific trigger phrases +- [ ] Uses third person, not second person +- [ ] Is 100-300 characters long +- [ ] Mentions concrete use cases + +### Content Organization +- [ ] SKILL.md under 5,000 words +- [ ] Detailed content in references/ +- [ ] Examples in examples/ directory +- [ ] Progressive disclosure followed + +### Writing Style +- [ ] No "you", "your", "you're" in body +- [ ] Uses imperative verbs (Create, Check, Run) +- [ ] Objective and factual +- [ ] No marketing language + +### Structural Integrity +- [ ] YAML has name and description +- [ ] All referenced files exist +- [ ] Examples are complete +- [ ] Scripts are executable + +--- + +## Learn From Mistakes + +Each anti-pattern above represents real issues found in actual skills. Avoid these common mistakes: + +1. **Vague descriptions** → Be specific with trigger phrases +2. **Second person** → Use imperative form +3. **No progressive disclosure** → Move details to references/ +4. **Missing fields** → Include name and description +5. **Broken references** → Verify all files exist +6. **Inconsistent style** → Stick to imperative form +7. **Too long description** → Keep under 300 characters +8. **No examples** → Include working code +9. **Subjective language** → Be factual and objective +10. **Invalid YAML** → Validate YAML syntax + +Use `examples-good.md` as a reference for what to do right. diff --git a/.agents/skills/skill-quality-reviewer/references/examples-good.md b/.agents/skills/skill-quality-reviewer/references/examples-good.md new file mode 100644 index 0000000..ed82f28 --- /dev/null +++ b/.agents/skills/skill-quality-reviewer/references/examples-good.md @@ -0,0 +1,385 @@ +# Exemplary Skill Examples + +Collection of high-quality skill examples demonstrating best practices across all quality dimensions. Study these when creating or improving skills. + +## Example 1: hook-development (A+ Grade) + +An exemplary skill demonstrating excellent description quality, progressive disclosure, and writing style. + +### Frontmatter (Description Quality) + +```yaml +--- +name: hook-development +description: This skill should be used when the user asks to "create a hook", +"add a PreToolUse hook", "validate tool use", "implement prompt-based hooks", +"use ${CLAUDE_PLUGIN_ROOT}", "set up event-driven automation", "block dangerous +commands", or mentions hook events (PreToolUse, PostToolUse, Stop, SubagentStop, +SessionStart, SessionEnd, UserPromptSubmit, PreCompact, Notification). Provides +comprehensive guidance for creating and implementing Claude Code plugin hooks +with focus on advanced prompt-based hooks API. +version: 0.1.0 +--- +``` + +**Why this is excellent:** +- ✅ 8+ specific trigger phrases covering diverse scenarios +- ✅ Third-person format throughout +- ✅ Includes specific hook events for precision +- ✅ Mentions key concepts (prompt-based hooks API) +- ✅ Length is appropriate (~400 characters, acceptable for complex skill) + +### Progressive Disclosure (Content Organization) + +**SKILL.md structure:** +- Overview (50 words) +- Core concepts (200 words) +- Hook events reference (300 words) +- Development workflow (500 words) +- Best practices (400 words) +- **Total: ~1,650 words** - excellent length + +**References/ directory:** +``` +references/ +├── prompt-based-hooks.md (800 words) - Detailed guide +├── hook-patterns.md (1,200 words) - Common patterns +└── migration-guide.md (900 words) - Migration info +``` + +**Why this is excellent:** +- ✅ SKILL.md is concise and focused +- ✅ Detailed content moved to 3 reference files +- ✅ Clear progressive disclosure +- ✅ Each reference file has specific purpose + +### Writing Style (Imperative Form) + +**Example from SKILL.md:** +```markdown +## Hook Development Workflow + +### Step 1: Define the Hook Event +Identify which event the hook should trigger on. Review available events +in the Hook Events Reference section. + +### Step 2: Create hooks.json +Create a hooks.json file in the plugin's .claude-plugin/ directory. +Specify the event type and command to execute. + +### Step 3: Implement the Hook Script +Write the hook script. Ensure it is executable. Follow the prompt-based +hooks API for complex hooks. + +### Step 4: Test the Hook +Test the hook locally. Verify it triggers on the correct event. +Check output and error handling. +``` + +**Why this is excellent:** +- ✅ Every instruction uses imperative form +- ✅ No "you", "your", "should" +- ✅ Clear, actionable steps +- ✅ Consistent style throughout + +--- + +## Example 2: agent-development (A Grade) + +Strong skill with good description and organization. + +### Frontmatter + +```yaml +--- +name: agent-development +description: This skill should be used when the user asks to "create an agent", +"add an agent", "write a subagent", "agent frontmatter", "when to use description", +"agent examples", "agent tools", "agent colors", "autonomous agent", or needs +guidance on agent structure, system prompts, triggering conditions, or agent +development best practices for Claude Code plugins. +version: 0.1.0 +--- +``` + +**Strengths:** +- ✅ Multiple specific trigger phrases +- ✅ Covers agent creation scenarios +- ✅ Mentions specific agent attributes + +### Directory Structure + +``` +agent-development/ +├── SKILL.md (1,438 words) +├── references/ +│ ├── agent-generation-prompt.md (500 words) +│ └── agent-anatomy.md (400 words) +└── examples/ + ├── task-agent.md (complete example) + ├── review-agent.md (complete example) + └── interactive-agent.md (complete example) +``` + +**Strengths:** +- ✅ Lean SKILL.md +- ✅ Focused reference files +- ✅ Complete working examples +- ✅ Clear organization + +--- + +## Example 3: mcp-integration (A- Grade) + +Comprehensive skill with excellent progressive disclosure. + +### Frontmatter + +```yaml +--- +name: mcp-integration +description: This skill should be used when the user asks to "add MCP server", +"integrate MCP", "configure MCP in plugin", "use .mcp.json", "set up Model +Context Protocol", "connect external service", mentions "${CLAUDE_PLUGIN_ROOT} +with MCP", or discusses MCP server types (SSE, stdio, HTTP, WebSocket). Provides +comprehensive guidance for integrating Model Context Protocol servers into Claude +Code plugins for external tool and service integration. +version: 0.1.0 +--- +``` + +**Strengths:** +- ✅ Specific MCP-related trigger phrases +- ✅ Lists server types for precision +- ✅ Mentions key configuration elements + +### Progressive Disclosure + +**SKILL.md**: ~1,800 words (core concepts, basic setup) + +**References/**: +- `mcp-server-types.md` - Detailed server type comparisons +- `mcp-configuration.md` - Configuration examples +- `mcp-best-practices.md` - Integration patterns +- `mcp-troubleshooting.md` - Common issues + +**Why this works:** +- ✅ Main SKILL.md covers essentials +- ✅ Each reference is 1,000-2,000 words +- ✅ Loaded only when needed +- ✅ Clear topic separation + +--- + +## Excellent Description Examples + +### Example: API-Contract Skill (Ideal Description) + +```yaml +--- +name: api-contract-manager +description: This skill should be used when the user asks to "validate API contract", +"check API compatibility", "version API schema", "detect breaking changes", or +"ensure API contract compliance". Manages API contracts through validation, +versioning, and compatibility checking. Use when designing new endpoints or +ensuring backward compatibility. +version: 0.1.0 +--- +``` + +**Analysis:** +- ✅ 5 specific trigger phrases +- ✅ Third person throughout +- ✅ 230 characters (ideal length) +- ✅ Clear use cases +- ✅ Mentions when to use (designing, compatibility) + +### Example: PDF-Editor Skill (Ideal Description) + +```yaml +--- +name: pdf-editor +description: This skill should be used when the user asks to "rotate PDF", "merge +PDF files", "split PDF pages", "compress PDF", or "crop PDF pages". Perform common +PDF manipulation operations including rotation, merging, splitting, compression, +and cropping. Works with local PDF files. +version: 0.1.0 +--- +``` + +**Analysis:** +- ✅ 5 specific operations as triggers +- ✅ Lists all supported operations +- ✅ Specifies working context (local files) +- ✅ 180 characters (excellent length) + +--- + +## Excellent Progressive Disclosure Examples + +### Example: Frontend-Builder Skill + +**SKILL.md** (1,800 words): +- Overview +- Quick Start (5 steps) +- Basic Component Creation +- Common Operations +- Reference to detailed guides + +**References/**: +- `react-patterns.md` (2,500 words) - React specific patterns +- `vue-patterns.md` (2,000 words) - Vue specific patterns +- `styling-guide.md` (1,800 words) - CSS/styling approaches +- `deployment.md` (1,500 words) - Build and deploy + +**Examples/**: +- `hello-world-react/` - Complete React example +- `hello-world-vue/` - Complete Vue example +- `component-library/` - Reusable components + +**Why this works:** +1. SKILL.md gets users started quickly +2. Framework-specific details in references/ +3. Complete examples for copy-paste +4. Progressive: overview → basics → details → examples + +--- + +## Excellent Writing Style Examples + +### Example: From Test-Driven Development Skill + +```markdown +## TDD Workflow + +### Red: Write a Failing Test +Start by writing a test that fails. This test defines the desired behavior. +Run the test to confirm it fails. + +### Green: Make the Test Pass +Write the minimum code to make the test pass. Focus on making it work, +not making it perfect. Run the test to confirm it passes. + +### Refactor: Improve the Code +Refactor the code while keeping tests green. Improve structure, readability, +and maintainability. Run tests after each change. + +### Repeat +Continue the cycle for each feature. Write test, make it pass, refactor. +``` + +**Analysis:** +- ✅ Imperative form throughout +- ✅ Clear, actionable instructions +- ✅ No second person +- ✅ Consistent style +- ✅ Easy to follow + +--- + +## Excellent Reference File Organization + +### Example: Git-Workflow Skill References + +``` +references/ +├── commit-conventions.md (1,500 words) +│ ├── Commit message format +│ ├── Type categories +│ ├── Scope values +│ └── Examples by type +├── branch-strategy.md (1,800 words) +│ ├── Branch types +│ ├── Merging policies +│ ├── Release workflow +│ └── Hotfix procedures +└── troubleshooting.md (1,200 words) + ├── Common issues + ├── Recovery procedures + └── Best practices +``` + +**Each reference file:** +- ✅ Focused on single topic +- ✅ Self-contained +- ✅ Cross-references others when needed +- ✅ Loaded only when relevant + +--- + +## Key Takeaways from Excellent Skills + +### 1. Description Quality + +**Good descriptions:** +- Have 5+ specific trigger phrases +- Use third person consistently +- Are 100-300 characters long +- Include concrete use cases + +**Example template:** +```yaml +description: This skill should be used when the user asks to "action 1", +"action 2", "action 3", or "action 4". Brief description of what the skill +does. When to use it. +``` + +### 2. Progressive Disclosure + +**Good organization:** +- SKILL.md: 1,500-2,000 words (core only) +- references/: 1,500-2,500 words per file (details) +- examples/: Complete working code +- scripts/: Utility tools + +**Rule of thumb:** If a section exceeds 500 words and covers details, +consider moving it to references/. + +### 3. Writing Style + +**Good style:** +- Use imperative verbs: Create, Validate, Check, Run +- Avoid: You should, You can, You need to +- Be objective and factual +- Keep instructions actionable + +### 4. Structural Integrity + +**Good structure:** +``` +skill-name/ +├── SKILL.md (required) +├── references/ (optional, for details) +├── examples/ (optional, for working code) +└── scripts/ (optional, for utilities) +``` + +**Validation checklist:** +- [ ] YAML frontmatter has name and description +- [ ] All referenced files exist +- [ ] Examples are complete +- [ ] Scripts are executable + +--- + +## Study These Skills + +For hands-on learning, study these high-quality skills in your environment: + +```bash +# Explore hook-development structure +ls -la ~/.claude/plugins/cache/*/skills/hook-development/ + +# Read agent-development SKILL.md +cat ~/.claude/skills/agent-development/SKILL.md + +# Review mcp-integration references +ls -la ~/.claude/skills/mcp-integration/references/ +``` + +Each demonstrates different strengths: +- **hook-development**: Progressive disclosure, utilities +- **agent-development**: Clean examples, focused content +- **mcp-integration**: Comprehensive references, clear organization + +Use these as templates when creating or improving your own skills. diff --git a/.agents/skills/skill-quality-reviewer/references/scoring-criteria.md b/.agents/skills/skill-quality-reviewer/references/scoring-criteria.md new file mode 100644 index 0000000..621b18a --- /dev/null +++ b/.agents/skills/skill-quality-reviewer/references/scoring-criteria.md @@ -0,0 +1,569 @@ +# Skill Quality Scoring Criteria + +Detailed evaluation rubrics for each quality dimension assessed by the skill-quality-reviewer. + +## 1. Description Quality (25% Weight) + +Assesses the effectiveness of the frontmatter description in triggering the skill appropriately. + +### Scoring Breakdown + +#### Trigger Phrases Clarity (0-25 points) + +**25 points (Excellent):** +- 5+ specific, concrete trigger phrases +- Phrases cover diverse use cases +- Each phrase is something a user would naturally say +- Phrases include both specific tasks and general scenarios + +**Example:** +```yaml +description: This skill should be used when the user asks to "create a hook", +"add a PreToolUse hook", "validate tool use", "implement prompt-based hooks", +"set up event-driven automation", or mentions hook events (PreToolUse, PostToolUse, +Stop, SubagentStop). +``` + +**20 points (Good):** +- 3-4 specific trigger phrases +- Good variety of scenarios +- Most phrases are natural user language + +**15 points (Acceptable):** +- 2-3 trigger phrases +- Some variety but may miss key scenarios +- Phrases are reasonably specific + +**10 points (Needs Work):** +- 1-2 trigger phrases +- Limited variety +- Some phrases too generic + +**5 points (Poor):** +- Single vague trigger phrase +- Missing common use cases + +**0 points (Fail):** +- No trigger phrases +- Trigger phrases are meaningless + +--- + +#### Third-Person Format (0-25 points) + +**25 points (Excellent):** +- Consistently uses "This skill should be used when..." +- Never uses second person ("you", "your") +- Professional, objective tone + +**20 points (Good):** +- Generally uses third person +- Minor inconsistencies but overall correct + +**10 points (Needs Work):** +- Mix of third and second person +- Inconsistent formatting + +**0 points (Fail):** +- Uses second person ("Use this when you want...") +- First person references + +--- + +#### Description Length (0-25 points) + +**25 points (Excellent):** +- 150-300 characters +- Concise yet comprehensive +- No unnecessary words + +**20 points (Good):** +- 100-150 or 300-400 characters +- Slightly short or long but acceptable + +**15 points (Acceptable):** +- 80-100 or 400-500 characters +- Beginning to be too concise or verbose + +**10 points (Needs Work):** +- 50-80 or 500-700 characters +- Too brief or too verbose + +**5 points (Poor):** +- Under 50 or over 700 characters + +**0 points (Fail):** +- Effectively empty or extremely long + +--- + +#### Specific Scenarios (0-25 points) + +**25 points (Excellent):** +- Multiple concrete scenarios mentioned +- Clear scope boundaries (when NOT to use) +- Specific domains or use cases identified + +**Example:** +```yaml +description: ... Use when designing new API endpoints, validating existing +contracts, or checking breaking changes. NOT for general API testing or +documentation generation. +``` + +**20 points (Good):** +- Good scenario coverage +- Clear use cases + +**15 points (Acceptable):** +- Basic scenario descriptions +- Could be more specific + +**10 points (Needs Work):** +- Vague scenarios +- Unclear when to use + +**0 points (Fail):** +- No scenarios mentioned +- Purely generic description + +--- + +## 2. Content Organization (30% Weight) + +Evaluates adherence to progressive disclosure principles and effective content organization. + +### Scoring Breakdown + +#### Progressive Disclosure (0-30 points) + +**30 points (Excellent):** +- SKILL.md under 2,000 words +- All detailed content in references/ +- Clear separation of overview and details +- References/ properly organized with multiple files + +**25 points (Good):** +- SKILL.md 2,000-3,000 words +- Most detailed content moved to references/ +- Good separation of concerns + +**20 points (Acceptable):** +- SKILL.md 3,000-5,000 words +- Some detailed content still in SKILL.md +- References/ used but could be better + +**15 points (Needs Work):** +- SKILL.md 5,000-7,000 words +- Significant content should be moved +- Limited use of references/ + +**10 points (Poor):** +- SKILL.md over 7,000 words +- Minimal or no references/ usage + +**0 points (Fail):** +- Everything in one large file +- No progressive disclosure + +--- + +#### SKILL.md Length Control (0-25 points) + +**25 points (Excellent):** +- 1,500-2,000 words (optimal) +- Every word serves a purpose +- No redundancy + +**20 points (Good):** +- 1,000-1,500 or 2,000-3,000 words +- Generally concise + +**15 points (Acceptable):** +- 800-1,000 or 3,000-4,000 words +- Some unnecessary content + +**10 points (Needs Work):** +- 500-800 or 4,000-5,000 words +- Could be significantly trimmed + +**5 points (Poor):** +- Under 500 or over 5,000 words +- Major length issues + +--- + +#### References/ Usage (0-25 points) + +**25 points (Excellent):** +- 3+ well-organized reference files +- Clear topics (patterns, advanced, examples, etc.) +- Properly referenced in SKILL.md +- Each file has clear purpose + +**20 points (Good):** +- 2-3 reference files +- Good organization +- Referenced in SKILL.md + +**15 points (Acceptable):** +- 1-2 reference files +- Basic organization + +**10 points (Needs Work):** +- Single reference file or poorly organized +- Not well referenced + +**5 points (Poor):** +- References/ exists but minimal content +- Not referenced from SKILL.md + +**0 points (Fail):** +- No references/ directory when clearly needed + +--- + +#### Logical Organization (0-20 points) + +**20 points (Excellent):** +- Clear section hierarchy +- Logical flow from overview to details +- Easy to navigate +- Consistent structure + +**15 points (Good):** +- Generally well organized +- Minor flow issues + +**10 points (Acceptable):** +- Basic organization present +- Some structural confusion + +**5 points (Needs Work):** +- Poor organization +- Difficult to follow + +**0 points (Fail):** +- No clear structure +- Content randomly arranged + +--- + +## 3. Writing Style (20% Weight) + +Evaluates adherence to skill writing conventions and style consistency. + +### Scoring Breakdown + +#### Imperative Form Usage (0-40 points) + +**40 points (Excellent):** +- 95%+ of instructions use imperative form +- Consistent throughout +- Natural, readable instructions + +**Good examples:** +``` +Create the skill directory structure. +Validate the YAML frontmatter. +Check for required fields. +Read the SKILL.md file. +Generate the quality report. +``` + +**30 points (Good):** +- 80-95% imperative form +- Generally consistent +- Minor exceptions + +**20 points (Acceptable):** +- 60-80% imperative form +- Inconsistent usage + +**10 points (Needs Work):** +- 40-60% imperative form +- Significant inconsistency + +**0 points (Fail):** +- Less than 40% imperative form +- Mostly descriptive or second-person + +--- + +#### No Second Person (0-30 points) + +**30 points (Excellent):** +- Zero instances of "you", "your", "you're" +- Clean imperative style throughout + +**20 points (Good):** +- 1-3 minor instances +- Generally avoids second person + +**10 points (Acceptable):** +- 4-6 instances +- Some second person creeping in + +**5 points (Needs Work):** +- 7-10 instances +- Frequent second person + +**0 points (Fail):** +- 10+ instances +- Heavily uses second person + +--- + +#### Objective Language (0-30 points) + +**30 points (Excellent):** +- Factual, instructional tone +- No subjective opinions +- Professional language +- Clear and direct + +**20 points (Good):** +- Generally objective +- Minor subjectivity + +**15 points (Acceptable):** +- Mostly objective with some opinion +- Generally professional + +**10 points (Needs Work):** +- Subjective language present +- Conversational tone + +**0 points (Fail):** +- Highly subjective +- Unprofessional or casual tone + +--- + +## 4. Structural Integrity (25% Weight) + +Evaluates the physical structure, completeness, and correctness of the skill. + +### Scoring Breakdown + +#### YAML Frontmatter (0-30 points) + +**30 points (Excellent):** +- All required fields present (name, description) +- Optional fields used appropriately (version) +- Valid YAML syntax +- No prohibited fields +- Well-formatted + +**Example of excellent frontmatter:** +```yaml +--- +name: skill-quality-reviewer +description: This skill should be used when the user asks to "analyze skill quality", +"evaluate this skill", "review skill quality", or mentions quality review of Claude Skills. +version: 0.1.0 +--- +``` + +**25 points (Good):** +- All required fields present +- Valid YAML +- Minor formatting issues + +**20 points (Acceptable):** +- Required fields present +- Valid YAML +- Some missing recommended fields + +**10 points (Needs Work):** +- Missing some required fields +- YAML issues present + +**0 points (Fail):** +- Missing required fields +- Invalid YAML syntax +- Cannot parse frontmatter + +--- + +#### Directory Structure (0-30 points) + +**30 points (Excellent):** +- Proper skill structure: + ``` + skill-name/ + ├── SKILL.md + ├── references/ + ├── examples/ + └── scripts/ + ``` +- Each subdirectory has clear purpose +- No unnecessary files +- Clean naming conventions + +**25 points (Good):** +- Core structure correct +- Minor organization issues + +**20 points (Acceptable):** +- Basic structure present +- Some organizational issues + +**10 points (Needs Work):** +- Missing key directories +- Poor organization + +**0 points (Fail):** +- No clear structure +- Chaotic organization + +--- + +#### Resource References (0-40 points) + +**40 points (Excellent):** +- All referenced files exist +- References are accurate and specific +- Examples are complete and working +- Scripts are executable +- No broken links or references + +**Check method:** +- Read SKILL.md +- Extract all references to files (e.g., `references/patterns.md`) +- Verify each file exists +- Verify examples/ and scripts/ contents + +**30 points (Good):** +- All references exist +- Minor issues with completeness + +**20 points (Acceptable):** +- Most references valid +- Some broken or incomplete references + +**10 points (Needs Work):** +- Multiple broken references +- Incomplete examples + +**0 points (Fail):** +- Many broken references +- Examples don't work +- Scripts not executable + +--- + +## Grade Calculation + +### Weighted Score Formula + +``` +Overall Score = (Description Quality × 0.25) + + (Content Organization × 0.30) + + (Writing Style × 0.20) + + (Structural Integrity × 0.25) +``` + +### Letter Grade Mapping + +| Score Range | Grade | Quality Level | Certification | +|-------------|-------|---------------|--------------| +| 97-100 | A+ | Exemplary | Certified | +| 93-96 | A | Excellent | Certified | +| 90-92 | A- | Very Good | Certified | +| 87-89 | B+ | Good | Certified | +| 83-86 | B | Above Average | Certified | +| 80-82 | B- | Solid | Certified | +| 77-79 | C+ | Acceptable | Review | +| 73-76 | C | Satisfactory | Review | +| 70-72 | C- | Minimal | Review | +| 67-69 | D+ | Below Standard | Reject | +| 63-66 | D | Poor | Reject | +| 60-62 | D- | Very Poor | Reject | +| 0-59 | F | Fail | Reject | + +### Certification Thresholds + +For "Certified" status (recommended for sharing): +- Overall score: ≥80/100 +- Description Quality: ≥75/100 +- Content Organization: ≥75/100 +- Writing Style: ≥70/100 +- Structural Integrity: ≥80/100 + +--- + +## Quick Reference Scoring Guide + +### Description Quality Quick Checks + +- [ ] Description has 3-5 specific trigger phrases +- [ ] Uses third person ("This skill should be used when...") +- [ ] Length is 100-300 characters +- [ ] Mentions concrete use cases + +### Content Organization Quick Checks + +- [ ] SKILL.md under 5,000 words (ideally 1,500-2,000) +- [ ] Detailed content in references/ +- [ ] Examples in examples/ directory +- [ ] Scripts in scripts/ directory + +### Writing Style Quick Checks + +- [ ] Instructions use verbs (Create, Validate, Check) +- [ ] No "you", "your", "you're" in body +- [ ] Objective, instructional tone +- [ ] Consistent style throughout + +### Structural Integrity Quick Checks + +- [ ] YAML has name and description +- [ ] All referenced files exist +- [ ] Examples are complete +- [ ] Scripts are executable + +--- + +## Common Scenarios + +### Scenario 1: New Skill Submission + +**Context:** Developer submits a new skill for review. + +**Evaluation Focus:** +- Description quality (is it clear when to use?) +- Progressive disclosure (is content organized?) +- Writing style (does it follow conventions?) + +**Passing Criteria:** B- (80) or higher overall + +--- + +### Scenario 2: Skill Quality Audit + +**Context:** Periodic review of existing skills. + +**Evaluation Focus:** +- All dimensions equally +- Identify improvement opportunities +- Track quality over time + +**Passing Criteria:** C (73) or higher overall + +--- + +### Scenario 3: Pre-Publication Check + +**Context:** Final check before sharing skill publicly. + +**Evaluation Focus:** +- Structural integrity (must be complete) +- Description quality (must be discoverable) +- No critical issues + +**Passing Criteria:** B (83) or higher overall + +--- + +This scoring criteria document provides the detailed rubrics used by skill-quality-reviewer to evaluate Claude Skills. Use this as a reference when interpreting quality reports or planning improvements. |