diff options
Diffstat (limited to '')
| -rw-r--r-- | .agents/skills/command-development/references/testing-strategies.md | 702 |
1 files changed, 702 insertions, 0 deletions
diff --git a/.agents/skills/command-development/references/testing-strategies.md b/.agents/skills/command-development/references/testing-strategies.md new file mode 100644 index 0000000..7b482fb --- /dev/null +++ b/.agents/skills/command-development/references/testing-strategies.md @@ -0,0 +1,702 @@ +# Command Testing Strategies + +Comprehensive strategies for testing slash commands before deployment and distribution. + +## Overview + +Testing commands ensures they work correctly, handle edge cases, and provide good user experience. A systematic testing approach catches issues early and builds confidence in command reliability. + +## Testing Levels + +### Level 1: Syntax and Structure Validation + +**What to test:** +- YAML frontmatter syntax +- Markdown format +- File location and naming + +**How to test:** + +```bash +# Validate YAML frontmatter +head -n 20 .claude/commands/my-command.md | grep -A 10 "^---" + +# Check for closing frontmatter marker +head -n 20 .claude/commands/my-command.md | grep -c "^---" # Should be 2 + +# Verify file has .md extension +ls .claude/commands/*.md + +# Check file is in correct location +test -f .claude/commands/my-command.md && echo "Found" || echo "Missing" +``` + +**Automated validation script:** + +```bash +#!/bin/bash +# validate-command.sh + +COMMAND_FILE="$1" + +if [ ! -f "$COMMAND_FILE" ]; then + echo "ERROR: File not found: $COMMAND_FILE" + exit 1 +fi + +# Check .md extension +if [[ ! "$COMMAND_FILE" =~ \.md$ ]]; then + echo "ERROR: File must have .md extension" + exit 1 +fi + +# Validate YAML frontmatter if present +if head -n 1 "$COMMAND_FILE" | grep -q "^---"; then + # Count frontmatter markers + MARKERS=$(head -n 50 "$COMMAND_FILE" | grep -c "^---") + if [ "$MARKERS" -ne 2 ]; then + echo "ERROR: Invalid YAML frontmatter (need exactly 2 '---' markers)" + exit 1 + fi + echo "✓ YAML frontmatter syntax valid" +fi + +# Check for empty file +if [ ! -s "$COMMAND_FILE" ]; then + echo "ERROR: File is empty" + exit 1 +fi + +echo "✓ Command file structure valid" +``` + +### Level 2: Frontmatter Field Validation + +**What to test:** +- Field types correct +- Values in valid ranges +- Required fields present (if any) + +**Validation script:** + +```bash +#!/bin/bash +# validate-frontmatter.sh + +COMMAND_FILE="$1" + +# Extract YAML frontmatter +FRONTMATTER=$(sed -n '/^---$/,/^---$/p' "$COMMAND_FILE" | sed '1d;$d') + +if [ -z "$FRONTMATTER" ]; then + echo "No frontmatter to validate" + exit 0 +fi + +# Check 'model' field if present +if echo "$FRONTMATTER" | grep -q "^model:"; then + MODEL=$(echo "$FRONTMATTER" | grep "^model:" | cut -d: -f2 | tr -d ' ') + if ! echo "sonnet opus haiku" | grep -qw "$MODEL"; then + echo "ERROR: Invalid model '$MODEL' (must be sonnet, opus, or haiku)" + exit 1 + fi + echo "✓ Model field valid: $MODEL" +fi + +# Check 'allowed-tools' field format +if echo "$FRONTMATTER" | grep -q "^allowed-tools:"; then + echo "✓ allowed-tools field present" + # Could add more sophisticated validation here +fi + +# Check 'description' length +if echo "$FRONTMATTER" | grep -q "^description:"; then + DESC=$(echo "$FRONTMATTER" | grep "^description:" | cut -d: -f2-) + LENGTH=${#DESC} + if [ "$LENGTH" -gt 80 ]; then + echo "WARNING: Description length $LENGTH (recommend < 60 chars)" + else + echo "✓ Description length acceptable: $LENGTH chars" + fi +fi + +echo "✓ Frontmatter fields valid" +``` + +### Level 3: Manual Command Invocation + +**What to test:** +- Command appears in `/help` +- Command executes without errors +- Output is as expected + +**Test procedure:** + +```bash +# 1. Start Claude Code +claude --debug + +# 2. Check command appears in help +> /help +# Look for your command in the list + +# 3. Invoke command without arguments +> /my-command +# Check for reasonable error or behavior + +# 4. Invoke with valid arguments +> /my-command arg1 arg2 +# Verify expected behavior + +# 5. Check debug logs +tail -f ~/.claude/debug-logs/latest +# Look for errors or warnings +``` + +### Level 4: Argument Testing + +**What to test:** +- Positional arguments work ($1, $2, etc.) +- $ARGUMENTS captures all arguments +- Missing arguments handled gracefully +- Invalid arguments detected + +**Test matrix:** + +| Test Case | Command | Expected Result | +|-----------|---------|-----------------| +| No args | `/cmd` | Graceful handling or useful message | +| One arg | `/cmd arg1` | $1 substituted correctly | +| Two args | `/cmd arg1 arg2` | $1 and $2 substituted | +| Extra args | `/cmd a b c d` | All captured or extras ignored appropriately | +| Special chars | `/cmd "arg with spaces"` | Quotes handled correctly | +| Empty arg | `/cmd ""` | Empty string handled | + +**Test script:** + +```bash +#!/bin/bash +# test-command-arguments.sh + +COMMAND="$1" + +echo "Testing argument handling for /$COMMAND" +echo + +echo "Test 1: No arguments" +echo " Command: /$COMMAND" +echo " Expected: [describe expected behavior]" +echo " Manual test required" +echo + +echo "Test 2: Single argument" +echo " Command: /$COMMAND test-value" +echo " Expected: 'test-value' appears in output" +echo " Manual test required" +echo + +echo "Test 3: Multiple arguments" +echo " Command: /$COMMAND arg1 arg2 arg3" +echo " Expected: All arguments used appropriately" +echo " Manual test required" +echo + +echo "Test 4: Special characters" +echo " Command: /$COMMAND \"value with spaces\"" +echo " Expected: Entire phrase captured" +echo " Manual test required" +``` + +### Level 5: File Reference Testing + +**What to test:** +- @ syntax loads file contents +- Non-existent files handled +- Large files handled appropriately +- Multiple file references work + +**Test procedure:** + +```bash +# Create test files +echo "Test content" > /tmp/test-file.txt +echo "Second file" > /tmp/test-file-2.txt + +# Test single file reference +> /my-command /tmp/test-file.txt +# Verify file content is read + +# Test non-existent file +> /my-command /tmp/nonexistent.txt +# Verify graceful error handling + +# Test multiple files +> /my-command /tmp/test-file.txt /tmp/test-file-2.txt +# Verify both files processed + +# Test large file +dd if=/dev/zero of=/tmp/large-file.bin bs=1M count=100 +> /my-command /tmp/large-file.bin +# Verify reasonable behavior (may truncate or warn) + +# Cleanup +rm /tmp/test-file*.txt /tmp/large-file.bin +``` + +### Level 6: Bash Execution Testing + +**What to test:** +- !` commands execute correctly +- Command output included in prompt +- Command failures handled +- Security: only allowed commands run + +**Test procedure:** + +```bash +# Create test command with bash execution +cat > .claude/commands/test-bash.md << 'EOF' +--- +description: Test bash execution +allowed-tools: Bash(echo:*), Bash(date:*) +--- + +Current date: !`date` +Test output: !`echo "Hello from bash"` + +Analysis of output above... +EOF + +# Test in Claude Code +> /test-bash +# Verify: +# 1. Date appears correctly +# 2. Echo output appears +# 3. No errors in debug logs + +# Test with disallowed command (should fail or be blocked) +cat > .claude/commands/test-forbidden.md << 'EOF' +--- +description: Test forbidden command +allowed-tools: Bash(echo:*) +--- + +Trying forbidden: !`ls -la /` +EOF + +> /test-forbidden +# Verify: Permission denied or appropriate error +``` + +### Level 7: Integration Testing + +**What to test:** +- Commands work with other plugin components +- Commands interact correctly with each other +- State management works across invocations +- Workflow commands execute in sequence + +**Test scenarios:** + +**Scenario 1: Command + Hook Integration** + +```bash +# Setup: Command that triggers a hook +# Test: Invoke command, verify hook executes + +# Command: .claude/commands/risky-operation.md +# Hook: PreToolUse that validates the operation + +> /risky-operation +# Verify: Hook executes and validates before command completes +``` + +**Scenario 2: Command Sequence** + +```bash +# Setup: Multi-command workflow +> /workflow-init +# Verify: State file created + +> /workflow-step2 +# Verify: State file read, step 2 executes + +> /workflow-complete +# Verify: State file cleaned up +``` + +**Scenario 3: Command + MCP Integration** + +```bash +# Setup: Command uses MCP tools +# Test: Verify MCP server accessible + +> /mcp-command +# Verify: +# 1. MCP server starts (if stdio) +# 2. Tool calls succeed +# 3. Results included in output +``` + +## Automated Testing Approaches + +### Command Test Suite + +Create a test suite script: + +```bash +#!/bin/bash +# test-commands.sh - Command test suite + +TEST_DIR=".claude/commands" +FAILED_TESTS=0 + +echo "Command Test Suite" +echo "==================" +echo + +for cmd_file in "$TEST_DIR"/*.md; do + cmd_name=$(basename "$cmd_file" .md) + echo "Testing: $cmd_name" + + # Validate structure + if ./validate-command.sh "$cmd_file"; then + echo " ✓ Structure valid" + else + echo " ✗ Structure invalid" + ((FAILED_TESTS++)) + fi + + # Validate frontmatter + if ./validate-frontmatter.sh "$cmd_file"; then + echo " ✓ Frontmatter valid" + else + echo " ✗ Frontmatter invalid" + ((FAILED_TESTS++)) + fi + + echo +done + +echo "==================" +echo "Tests complete" +echo "Failed: $FAILED_TESTS" + +exit $FAILED_TESTS +``` + +### Pre-Commit Hook + +Validate commands before committing: + +```bash +#!/bin/bash +# .git/hooks/pre-commit + +echo "Validating commands..." + +COMMANDS_CHANGED=$(git diff --cached --name-only | grep "\.claude/commands/.*\.md") + +if [ -z "$COMMANDS_CHANGED" ]; then + echo "No commands changed" + exit 0 +fi + +for cmd in $COMMANDS_CHANGED; do + echo "Checking: $cmd" + + if ! ./scripts/validate-command.sh "$cmd"; then + echo "ERROR: Command validation failed: $cmd" + exit 1 + fi +done + +echo "✓ All commands valid" +``` + +### Continuous Testing + +Test commands in CI/CD: + +```yaml +# .github/workflows/test-commands.yml +name: Test Commands + +on: [push, pull_request] + +jobs: + test: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v2 + + - name: Validate command structure + run: | + for cmd in .claude/commands/*.md; do + echo "Testing: $cmd" + ./scripts/validate-command.sh "$cmd" + done + + - name: Validate frontmatter + run: | + for cmd in .claude/commands/*.md; do + ./scripts/validate-frontmatter.sh "$cmd" + done + + - name: Check for TODOs + run: | + if grep -r "TODO" .claude/commands/; then + echo "ERROR: TODOs found in commands" + exit 1 + fi +``` + +## Edge Case Testing + +### Test Edge Cases + +**Empty arguments:** +```bash +> /cmd "" +> /cmd '' '' +``` + +**Special characters:** +```bash +> /cmd "arg with spaces" +> /cmd arg-with-dashes +> /cmd arg_with_underscores +> /cmd arg/with/slashes +> /cmd 'arg with "quotes"' +``` + +**Long arguments:** +```bash +> /cmd $(python -c "print('a' * 10000)") +``` + +**Unusual file paths:** +```bash +> /cmd ./file +> /cmd ../file +> /cmd ~/file +> /cmd "/path with spaces/file" +``` + +**Bash command edge cases:** +```markdown +# Commands that might fail +!`exit 1` +!`false` +!`command-that-does-not-exist` + +# Commands with special output +!`echo ""` +!`cat /dev/null` +!`yes | head -n 1000000` +``` + +## Performance Testing + +### Response Time Testing + +```bash +#!/bin/bash +# test-command-performance.sh + +COMMAND="$1" + +echo "Testing performance of /$COMMAND" +echo + +for i in {1..5}; do + echo "Run $i:" + START=$(date +%s%N) + + # Invoke command (manual step - record time) + echo " Invoke: /$COMMAND" + echo " Start time: $START" + echo " (Record end time manually)" + echo +done + +echo "Analyze results:" +echo " - Average response time" +echo " - Variance" +echo " - Acceptable threshold: < 3 seconds for fast commands" +``` + +### Resource Usage Testing + +```bash +# Monitor Claude Code during command execution +# In terminal 1: +claude --debug + +# In terminal 2: +watch -n 1 'ps aux | grep claude' + +# Execute command and observe: +# - Memory usage +# - CPU usage +# - Process count +``` + +## User Experience Testing + +### Usability Checklist + +- [ ] Command name is intuitive +- [ ] Description is clear in `/help` +- [ ] Arguments are well-documented +- [ ] Error messages are helpful +- [ ] Output is formatted readably +- [ ] Long-running commands show progress +- [ ] Results are actionable +- [ ] Edge cases have good UX + +### User Acceptance Testing + +Recruit testers: + +```markdown +# Testing Guide for Beta Testers + +## Command: /my-new-command + +### Test Scenarios + +1. **Basic usage:** + - Run: `/my-new-command` + - Expected: [describe] + - Rate clarity: 1-5 + +2. **With arguments:** + - Run: `/my-new-command arg1 arg2` + - Expected: [describe] + - Rate usefulness: 1-5 + +3. **Error case:** + - Run: `/my-new-command invalid-input` + - Expected: Helpful error message + - Rate error message: 1-5 + +### Feedback Questions + +1. Was the command easy to understand? +2. Did the output meet your expectations? +3. What would you change? +4. Would you use this command regularly? +``` + +## Testing Checklist + +Before releasing a command: + +### Structure +- [ ] File in correct location +- [ ] Correct .md extension +- [ ] Valid YAML frontmatter (if present) +- [ ] Markdown syntax correct + +### Functionality +- [ ] Command appears in `/help` +- [ ] Description is clear +- [ ] Command executes without errors +- [ ] Arguments work as expected +- [ ] File references work +- [ ] Bash execution works (if used) + +### Edge Cases +- [ ] Missing arguments handled +- [ ] Invalid arguments detected +- [ ] Non-existent files handled +- [ ] Special characters work +- [ ] Long inputs handled + +### Integration +- [ ] Works with other commands +- [ ] Works with hooks (if applicable) +- [ ] Works with MCP (if applicable) +- [ ] State management works + +### Quality +- [ ] Performance acceptable +- [ ] No security issues +- [ ] Error messages helpful +- [ ] Output formatted well +- [ ] Documentation complete + +### Distribution +- [ ] Tested by others +- [ ] Feedback incorporated +- [ ] README updated +- [ ] Examples provided + +## Debugging Failed Tests + +### Common Issues and Solutions + +**Issue: Command not appearing in /help** + +```bash +# Check file location +ls -la .claude/commands/my-command.md + +# Check permissions +chmod 644 .claude/commands/my-command.md + +# Check syntax +head -n 20 .claude/commands/my-command.md + +# Restart Claude Code +claude --debug +``` + +**Issue: Arguments not substituting** + +```bash +# Verify syntax +grep '\$1' .claude/commands/my-command.md +grep '\$ARGUMENTS' .claude/commands/my-command.md + +# Test with simple command first +echo "Test: \$1 and \$2" > .claude/commands/test-args.md +``` + +**Issue: Bash commands not executing** + +```bash +# Check allowed-tools +grep "allowed-tools" .claude/commands/my-command.md + +# Verify command syntax +grep '!\`' .claude/commands/my-command.md + +# Test command manually +date +echo "test" +``` + +**Issue: File references not working** + +```bash +# Check @ syntax +grep '@' .claude/commands/my-command.md + +# Verify file exists +ls -la /path/to/referenced/file + +# Check permissions +chmod 644 /path/to/referenced/file +``` + +## Best Practices + +1. **Test early, test often**: Validate as you develop +2. **Automate validation**: Use scripts for repeatable checks +3. **Test edge cases**: Don't just test the happy path +4. **Get feedback**: Have others test before wide release +5. **Document tests**: Keep test scenarios for regression testing +6. **Monitor in production**: Watch for issues after release +7. **Iterate**: Improve based on real usage data |
