diff options
Diffstat (limited to '')
| -rw-r--r-- | .agents/skills/ml-paper-writing/references/literature-research/arxiv-search-guide.md | 199 | ||||
| -rw-r--r-- | .agents/skills/ml-paper-writing/references/literature-research/paper-quality-criteria.md | 303 |
2 files changed, 502 insertions, 0 deletions
diff --git a/.agents/skills/ml-paper-writing/references/literature-research/arxiv-search-guide.md b/.agents/skills/ml-paper-writing/references/literature-research/arxiv-search-guide.md new file mode 100644 index 0000000..c37c868 --- /dev/null +++ b/.agents/skills/ml-paper-writing/references/literature-research/arxiv-search-guide.md @@ -0,0 +1,199 @@ +# arXiv Literature Search Guide + +## Overview + +This guide provides workflows for discovering and evaluating recent ML research papers on arXiv. Use this when conducting literature reviews, finding related work, or staying updated on recent publications. + +--- + +## Search Strategies + +### 1. Keyword-Based Search + +**arXiv Search URL Pattern:** +``` +https://arxiv.org/search/?searchtype=all&query=KEYWORDS&abstracts=show&order=-announced_date_first +``` + +**Common ML Search Keywords:** +- **General ML**: `machine learning`, `deep learning`, `neural networks` +- **Specific Areas**: `reinforcement learning`, `transformer`, `attention mechanism`, `graph neural networks` +- **Applications**: `computer vision`, `natural language processing`, `reinforcement learning` +- **Methods**: `self-supervised learning`, `contrastive learning`, `foundation models` + +**Tips:** +- Combine keywords with `+` for AND operation +- Use `|` for OR operation +- Put phrases in quotes for exact matches: `"attention is all you need"` + +### 2. Category-Based Search + +**Recommended arXiv Categories for ML:** +- `cs.LG` (Machine Learning) +- `cs.AI` (Artificial Intelligence) +- `cs.CV` (Computer Vision and Pattern Recognition) +- `cs.CL` (Computation and Language) +- `cs.NE` (Neural and Evolutionary Computing) +- `stat.ML` (Machine Learning - Statistics) + +**Category Filter URL:** +``` +https://arxiv.org/search/?cat:cs.LG+OR+cat:cs.AI+AND+all:transformer&abstracts=show&order=-announced_date_first +``` + +### 3. Time-Based Filtering + +**Recent Papers (Last 3 Months):** +- Use `order=-announced_date_first` for newest first +- Manually filter by submission date +- Check paper metadata for submission date + +--- + +## Using Chrome MCP for arXiv Search + +When available, use Chrome MCP tools for automated arXiv searching: + +1. **Navigate to arXiv search** using Chrome MCP navigation +2. **Extract paper information** from search results: + - Paper title + - Authors + - arXiv ID + - Abstract preview + - Publication date + +3. **Navigate to individual papers** for detailed review + +--- + +## Paper Quality Evaluation + +Evaluate papers using the 5-dimension criteria below: + +| Dimension | Weight | Key Points | +|-----------|--------|------------| +| **Innovation** | 30% | Novelty of contribution | +| **Method Completeness** | 25% | Clarity and reproducibility | +| **Experimental Thoroughness** | 25% | Validation depth | +| **Writing Quality** | 10% | Clarity of expression | +| **Relevance & Impact** | 10% | Domain importance | + +### Scoring Guidelines (1-5 scale) + +**Innovation (30%):** +- 5: Breakthrough contribution, major impact +- 4: Significant improvement, new insights +- 3: Methodological innovation +- 2: Incremental improvement +- 1: Minor improvements + +**Method Completeness (25%):** +- 5: Complete and rigorous, easily reproducible +- 4: Very detailed, mostly reproducible +- 3: Core method clear, basically reproducible +- 2: Lacks key details +- 1: Unclear description + +**Experimental Thoroughness (25%):** +- 5: Comprehensive multi-dataset, ablation studies +- 4: Multiple datasets, reasonable ablations +- 3: Main experiments complete +- 2: Limited experiments +- 1: Minimal validation + +**Writing Quality (10%):** +- 5: Excellent clarity and rigor +- 4: Clear and well-structured +- 3: Understandable +- 2: Some ambiguity +- 1: Confusing + +**Relevance & Impact (10%):** +- 5: Solves important problem, wide impact +- 4: Important domain problem +- 3: Meaningful contribution +- 2: Niche problem +- 1: Limited impact + +### Selection Process + +1. **Screen by title/abstract** for relevance +2. **Navigate to full paper** for detailed review +3. **Score each dimension** (1-5) +4. **Calculate weighted total** +5. **Rank and select** top papers + +--- + +## Extracting Paper Metadata + +**From arXiv Abstract Page (`https://arxiv.org/abs/ARXIV_ID`):** + +- Title (from `<h1>` tag) +- Authors (from `.authors` element) +- Abstract (from `blockquote.abstract`) +- Submission date (from `.dateline`) +- arXiv ID (from URL) +- Categories (from `.subjects`) +- Comments (if present) +- Code repository (check abstract for GitHub links) + +--- + +## Integration with Citation Workflow + +After finding relevant papers: + +1. **Verify citations** using Semantic Scholar API (see `../citation-workflow.md`) +2. **Fetch BibTeX** programmatically via DOI +3. **Store in bibliography** with verification status + +--- + +## Common Use Cases + +### Finding Related Work + +When writing a paper, use arXiv search to: +1. Find recent papers on your topic +2. Identify state-of-the-art methods +3. Discover competing approaches +4. Find baseline comparisons + +### Staying Updated + +Set up regular searches for: +- Your specific research area +- Competing labs/researchers +- New methods in your domain +- Conference proceedings (preprints) + +### Literature Reviews + +For comprehensive reviews: +1. Start with broad keyword searches +2. Filter by recent publications (last 1-3 years) +3. Use citation chaining (forward and backward) +4. Evaluate and select high-quality papers +5. Organize by theme/contribution + +--- + +## Tips for Effective Searching + +1. **Use specific keywords** rather than broad terms +2. **Combine techniques** (keywords + categories + time filters) +3. **Check code availability** (many arXiv papers link to GitHub) +4. **Look for citations** to understand impact +5. **Read abstracts carefully** before full papers +6. **Use paper metrics** (citation count, code stars) as indicators + +--- + +## External Resources + +- **arXiv**: https://arxiv.org/ +- **Semantic Scholar**: https://www.semanticscholar.org/ +- **Papers With Code**: https://paperswithcode.com/ +- **Connected Papers**: https://www.connectedpapers.com/ +- **arXiv API**: http://export.arxiv.org/api_help/ diff --git a/.agents/skills/ml-paper-writing/references/literature-research/paper-quality-criteria.md b/.agents/skills/ml-paper-writing/references/literature-research/paper-quality-criteria.md new file mode 100644 index 0000000..0e07a8a --- /dev/null +++ b/.agents/skills/ml-paper-writing/references/literature-research/paper-quality-criteria.md @@ -0,0 +1,303 @@ +# ML Paper Quality Evaluation Criteria + +## Overview + +Use these criteria to evaluate ML research papers found during literature search or when selecting papers for detailed review. The 5-dimension framework provides structured assessment for paper selection and comparison. + +--- + +## Evaluation Dimensions + +| Dimension | Weight | Description | +|-----------|--------|-------------| +| **Innovation** | 30% | Novelty and originality of contribution | +| **Method Completeness** | 25% | Clarity, rigor, and reproducibility | +| **Experimental Thoroughness** | 25% | Validation depth and analysis quality | +| **Writing Quality** | 10% | Clarity and presentation | +| **Relevance & Impact** | 10% | Domain importance and potential impact | + +--- + +## Detailed Scoring Rubrics + +### 1. Innovation (30%) + +**Score 5 - Breakthrough:** +- Proposes entirely new paradigm or framework +- Solves long-standing open problem +- Major impact expected on the field +- Challenges fundamental assumptions + +**Score 4 - Significant Innovation:** +- Substantial improvement over existing methods +- New insights or perspectives +- Novel combination of techniques +- Clear advancement over state-of-the-art + +**Score 3 - Methodological Innovation:** +- New method or architecture proposed +- Some novelty but incremental +- Reasonable contribution +- Standard type of innovation + +**Score 2 - Incremental Improvement:** +- Minor improvements to existing methods +- Limited novelty +- Small advancement +- Mostly derivative + +**Score 1 - Trivial:** +- Minimal contribution +- Obvious extension +- No real innovation +- Known results + +**Evaluation Questions:** +- Does this paper propose something genuinely new? +- Does it advance the state-of-the-art? +- Will this influence future work? +- Is the contribution significant or marginal? + +--- + +### 2. Method Completeness (25%) + +**Score 5 - Complete and Rigorous:** +- Full mathematical derivation +- All hyperparameters specified +- Complete algorithmic details +- Easily reproducible +- Code available + +**Score 4 - Very Complete:** +- Detailed method description +- Most important details included +- Mostly reproducible +- Minor gaps in documentation + +**Score 3 - Reproducible:** +- Core method clearly described +- Key details present +- Can be reproduced with effort +- Some ambiguity in details + +**Score 2 - Lacks Details:** +- Key details missing +- Difficult to reproduce +- Incomplete description +- Ambiguous in important areas + +**Score 1 - Unclear:** +- Method description unclear +- Missing critical information +- Cannot determine validity +- Poorly explained + +**Evaluation Questions:** +- Can another researcher reproduce this work? +- Are all important details specified? +- Is mathematical derivation sound? +- Is code available and documented? + +--- + +### 3. Experimental Thoroughness (25%) + +**Score 5 - Comprehensive:** +- Multiple diverse datasets +- Extensive ablation studies +- Statistical significance testing +- Thorough analysis and discussion +- Comparison with strong baselines + +**Score 4 - Very Thorough:** +- Multiple datasets +- Reasonable ablation studies +- Proper baseline comparisons +- Good analysis + +**Score 3 - Adequate:** +- Main experiments complete +- Standard datasets +- Basic baselines +- Results are credible + +**Score 2 - Limited:** +- Limited experiments +- Few datasets +- Weak baselines +- Minimal analysis + +**Score 1 - Insufficient:** +- Minimal validation +- Toy examples only +- No meaningful comparisons +- Results not convincing + +**Evaluation Questions:** +- Are experiments comprehensive? +- Are baselines strong and appropriate? +- Are statistical tests used? +- Is there ablation analysis? +- Are results on standard datasets? + +--- + +### 4. Writing Quality (10%) + +**Score 5 - Excellent:** +- Clear, precise, well-structured +- Logical flow throughout +- Professional presentation +- High-quality figures +- No ambiguity + +**Score 4 - Very Good:** +- Clear and well-written +- Mostly logical structure +- Good presentation +- Minor issues + +**Score 3 - Understandable:** +- Basically clear +- Some organizational issues +- Acceptable presentation +- Understandable with effort + +**Score 2 - Fair:** +- Some confusing sections +- Organization problems +- Presentation issues +- Hard to follow at times + +**Score 1 - Poor:** +- Unclear or confusing +- Poor organization +- Difficult to understand +- Major presentation problems + +**Evaluation Questions:** +- Is the paper easy to understand? +- Is the structure logical? +- Are figures/tables clear? +- Is the writing professional? + +--- + +### 5. Relevance & Impact (10%) + +**Score 5 - High Impact:** +- Solves important problem +- Broad applicability +- Expected wide influence +- Addresses fundamental challenge + +**Score 4 - Domain Important:** +- Important problem in field +- Significant potential impact +- Relevant to many researchers + +**Score 3 - Meaningful:** +- Meaningful contribution +- Moderate impact expected +- Relevant to subset of field + +**Score 2 - Niche:** +- Specialized problem +- Limited applicability +- Narrow impact + +**Score 1 - Limited:** +- Very narrow problem +- Minimal impact expected +- Limited relevance + +**Evaluation Questions:** +- Is this an important problem? +- Will this influence future work? +- Is it relevant to current research needs? +- Does it address a significant challenge? + +--- + +## Scoring Calculation + +**Weighted Total:** +``` +Total = (Innovation × 0.30) + (Method × 0.25) + (Experiments × 0.25) + (Writing × 0.10) + (Impact × 0.10) +``` + +**Example Calculation:** +- Innovation: 4/5 +- Method: 3/5 +- Experiments: 4/5 +- Writing: 3/5 +- Impact: 4/5 + +``` +Total = (4 × 0.30) + (3 × 0.25) + (4 × 0.25) + (3 × 0.10) + (4 × 0.10) + = 1.20 + 0.75 + 1.00 + 0.30 + 0.40 + = 3.65 / 5.0 +``` + +--- + +## Selection Process + +### For Literature Reviews + +1. **Screen papers** by title/abstract for relevance +2. **Full review** of potentially relevant papers +3. **Score each paper** using all 5 dimensions +4. **Rank by total score** +5. **Select top papers** for detailed review + +### Quality Thresholds + +- **Excellent**: 4.0+ (include definitely) +- **Good**: 3.5-3.9 (include if relevant) +- **Fair**: 3.0-3.4 (include if highly relevant) +- **Poor**: <3.0 (exclude unless essential) + +--- + +## Quick Screening Indicators + +Before detailed review, check: + +**Positive Indicators:** +- Published at top venue (NeurIPS, ICML, ICLR) +- Citations in top papers +- Code available with stars +- Authors from top labs +- Clear novelty in abstract + +**Negative Indicators:** +- Vague abstract +- Limited experiments mentioned +- No baselines mentioned +- Poor writing in abstract +- incremental claims only + +--- + +## Integration with Paper Discovery + +When using arXiv search (`arxiv-search-guide.md`): + +1. **Search** for relevant papers +2. **Extract metadata** from arXiv pages +3. **Quick screen** by abstract/relevance +4. **Detailed review** of promising papers +5. **Score using** these criteria +6. **Rank and select** top candidates + +--- + +## Notes + +- These criteria are designed for ML papers specifically +- Adjust weights based on your specific needs +- Use scores as relative comparisons, not absolute judgments +- Consider venue reputation as additional signal +- Code availability is increasingly important for reproducibility |