From 898b52edcb47bcb3e9d6106e74ca73e74ea01e70 Mon Sep 17 00:00:00 2001 From: sillylaird Date: Thu, 3 Sep 2026 00:33:59 +0000 Subject: import live www.sillylaird.ca webroot --- .../references/knowledge/README.md | 58 ++ .../design-simplification-papers-kaiming-he.md | 605 ++++++++++++ .../knowledge/kaiming_he_injection_record.json | 153 +++ .../knowledge/paper-miner-writing-memory.md | 37 + .../knowledge/rethinking-papers-kaiming-he.md | 1023 ++++++++++++++++++++ .../references/knowledge/review-response.md | 429 ++++++++ .../references/knowledge/structure.md | 349 +++++++ .../references/knowledge/submission-guides.md | 326 +++++++ .../knowledge/theory-driven-papers-kaiming-he.md | 561 +++++++++++ .../references/knowledge/writing-techniques.md | 637 ++++++++++++ 10 files changed, 4178 insertions(+) create mode 100644 .agents/skills/ml-paper-writing/references/knowledge/README.md create mode 100644 .agents/skills/ml-paper-writing/references/knowledge/design-simplification-papers-kaiming-he.md create mode 100644 .agents/skills/ml-paper-writing/references/knowledge/kaiming_he_injection_record.json create mode 100644 .agents/skills/ml-paper-writing/references/knowledge/paper-miner-writing-memory.md create mode 100644 .agents/skills/ml-paper-writing/references/knowledge/rethinking-papers-kaiming-he.md create mode 100644 .agents/skills/ml-paper-writing/references/knowledge/review-response.md create mode 100644 .agents/skills/ml-paper-writing/references/knowledge/structure.md create mode 100644 .agents/skills/ml-paper-writing/references/knowledge/submission-guides.md create mode 100644 .agents/skills/ml-paper-writing/references/knowledge/theory-driven-papers-kaiming-he.md create mode 100644 .agents/skills/ml-paper-writing/references/knowledge/writing-techniques.md (limited to '.agents/skills/ml-paper-writing/references/knowledge') diff --git a/.agents/skills/ml-paper-writing/references/knowledge/README.md b/.agents/skills/ml-paper-writing/references/knowledge/README.md new file mode 100644 index 0000000..42abdf6 --- /dev/null +++ b/.agents/skills/ml-paper-writing/references/knowledge/README.md @@ -0,0 +1,58 @@ +# Academic Writing Knowledge Base + +This knowledge base contains reusable academic writing knowledge mined from papers. + +## Canonical maintained memory + +The canonical paper-miner memory is: + +- `paper-miner-writing-memory.md` + +This is the **only maintained paper-miner writing memory**. + +It stores: +- writing patterns mined, +- structure signals, +- reusable phrasing, +- venue-specific signals, +- how those signals help future writing, +- and a source index. + +## Maintenance rule + +`paper-miner` always writes mined writing knowledge into `paper-miner-writing-memory.md`. + +This memory is: +- **global**, +- **cross-project**, +- **not project-specific**. + +If `paper-miner` is invoked inside a project, it may use project context to understand relevance, but it still writes only to the global memory. + +## Legacy files + +Older files such as: +- `structure.md` +- `writing-techniques.md` +- `submission-guides.md` +- `review-response.md` + +may still exist as historical material, but new paper-miner updates should treat `paper-miner-writing-memory.md` as the canonical maintained memory. + +## Usage + +Use this knowledge base when: +- drafting papers, +- improving section structure, +- borrowing reusable phrasing patterns, +- preparing rebuttals, +- studying venue-facing writing signals. + +## Contributing + +When `paper-miner` analyzes a new paper: +1. extract actionable writing knowledge, +2. merge it into `paper-miner-writing-memory.md`, +3. preserve source attribution, +4. avoid duplicate patterns, +5. keep the memory compact and reusable. diff --git a/.agents/skills/ml-paper-writing/references/knowledge/design-simplification-papers-kaiming-he.md b/.agents/skills/ml-paper-writing/references/knowledge/design-simplification-papers-kaiming-he.md new file mode 100644 index 0000000..eea5087 --- /dev/null +++ b/.agents/skills/ml-paper-writing/references/knowledge/design-simplification-papers-kaiming-he.md @@ -0,0 +1,605 @@ +# Design Simplification Papers: Less Is More + +**Source**: Kaiming He et al., "Exploring Plain Vision Transformer Backbones for Object Detection" (ViTDet, 2022) + +**Paper Type**: Design simplification / Minimal adaptations paper + +**Core Pattern**: Challenge design assumptions → Minimize changes → Surprising effectiveness → Fair comparison + +--- + +## 1. Abstract Structure: The "Surprisingly" Framework + +### Pattern: Conventional Practice → Simple Alternative → Unexpected Results + +**Template**: +```markdown +Abstract: +1. [Context] Standard practice in [domain] is [conventional design] +2. [Challenge] With [new technology], this faces [challenges] +3. [Common Solution] Most work addresses this by [abandoning philosophy / adding complexity] +4. [Our Direction] We explore [different direction]: [minimal approach] +5. [Surprisingly 1] Surprisingly, we observe: (i) [simple finding 1] + and (ii) [simple finding 2] +6. [Surprisingly 2] More surprisingly, [stronger claim under conditions] +7. [Implications] This enables [benefit] without [traditional requirement] +``` + +### ViTDet Abstract Example (annotated): + +```latex +Modern object detectors consist of hierarchical backbone feature extractors +and detection-specific necks/heads (e.g., FPN, RPN). + +With Vision Transformers (ViT) emerging as powerful backbones, their plain, +non-hierarchical nature poses challenges: How to address multi-scale objects? + +One solution abandons the plain ViT philosophy, re-introducing hierarchical +designs (e.g., Swin). + +We pursue a different direction: plain ViT backbones with minimal adaptations. + +Surprisingly, we observe: (i) A simple feature pyramid from a single-scale map +is sufficient (without FPN), and (ii) Window attention without shifting is +sufficient (with a few propagation blocks). + +More surprisingly, under some circumstances, our ViTDet can compete with +leading hierarchical detectors like Swin. + +With MAE pre-training, ViTDet outperforms hierarchical counterparts, +especially for larger models. + +This decouples pre-training from fine-tuning, maintaining independence of +upstream vs downstream tasks. +``` + +### Key Techniques: + +1. **"Modern...consist of..."** - Establish conventional practice +2. **"With...emerging as..."** - New technology, new challenge +3. **"abandons the...philosophy"** - Critique common solutions +4. **"We pursue a different direction"** - Clear positioning +5. **"Surprisingly, we observe: (i)... and (ii)..."** - First surprise +6. **"More surprisingly..."** - Second, deeper surprise +7. **"under some circumstances"** - Measured claim +8. **"sufficient"** - Scientific (not "optimal") +9. **"without [common practice]"** - Negative emphasis + +--- + +## 2. Introduction: The "Challenge Assumptions" Framework + +### Pattern: Tradition → New Challenge → Common Compromise → Your Alternative → Philosophy + +**Structure**: +```markdown +1. [Traditional Practice] Established design in [field] +2. [Evolution] How this emerged historically ("For a long while...") +3. [New Challenge] [New technology] with [different characteristics] +4. [Philosophy Clash] Original [tech] has "'minimalist' pursuit" + - Questions: "How can we...?" "Is [X] too inefficient?" +5. [Common Solution] One solution: [abandon philosophy] → [revert to old design] + - Acknowledge: "has shown successful results" +6. [Your Direction] "we pursue a different direction" + - Motivation: "If successful, enables [benefit]" +7. [Philosophy] "in part follows the [philosophy] of '[concept]'" +8. [Surprising Findings] "Surprisingly, we observe..." +9. [Implications] "More surprisingly..." → Competitive results +``` + +### ViTDet Introduction Flow: + +#### Traditional Practice (Establish Context) +```latex +Modern object detectors in general consist of a backbone feature extractor +that is agnostic to the detection task and a set of necks and heads that +incorporate detection-specific prior knowledge. + +Common components in the necks/heads may include Region-of-Interest (RoI) +operations, Region Proposal Networks (RPN) or anchors, Feature Pyramid +Networks (FPN), etc. +``` + +**Technique**: +- **"in general consist of"** - Standard architecture +- **"agnostic to"** vs **"detection-specific"** - Clear division +- **"may include"** - Examples, not exhaustive + +#### Historical Evolution +```latex +For a long while, these backbones have been multi-scale, hierarchical +architectures due to the de facto design of convolutional networks (ConvNet), +which has heavily influenced the neck/head design for detecting objects at +multiple scales (e.g., FPN). +``` + +**Technique**: +- **"For a long while"** - Historical dimension +- **"due to...which has heavily influenced"** - Causal chain +- **"de facto design"** - Established convention + +#### New Technology Challenge +```latex +Over the past year, Vision Transformers (ViT) have been established as a +powerful backbone for visual recognition. + +Unlike typical ConvNets, the original ViT is a plain, non-hierarchical +architecture that maintains a single-scale feature map throughout. + +Its 'minimalist' pursuit is met with challenges when applied to object +detection—e.g., How can we address multi-scale objects in a downstream task +with a plain backbone from upstream pre-training? Is a plain ViT too +inefficient to use with high-resolution detection images? +``` + +**Technique**: +- **"Over the past year...have been established as"** - Timeframe +- **"Unlike typical ConvNets"** - Direct contrast +- **"plain, non-hierarchical"**, **"single-scale"** - Key characteristics +- **"'minimalist' pursuit"** - Philosophy (in quotes) +- **Two questions**: Challenge reader to think + +#### Common Solution (Acknowledge then Pivot) +```latex +One solution, which abandons this pursuit, is to re-introduce hierarchical +designs into the backbone. + +This solution, e.g., Swin Transformers and related works, can inherit the +ConvNet-based detector design and has shown successful results. +``` + +**Technique**: +- **"which abandons this pursuit"** - Critique (respectful) +- **"can inherit"** - Acknowledge advantage +- **"has shown successful results"** - Don't deny effectiveness + +#### Your Different Direction +```latex +In this work, we pursue a different direction: we explore object detectors +that use only plain, non-hierarchical backbones. + +If this direction is successful, it will enable the use of original ViT +backbones for object detection; this will decouple the pre-training design +from the fine-tuning demands, maintaining the independence of upstream vs. +downstream tasks, as has been the case for ConvNet-based research. +``` + +**Technique**: +- **"we pursue a different direction"** - Clear positioning +- **"If this direction is successful, it will enable..."** - Motivation +- **"decouple"**, **"independence"** - Philosophy keywords +- **"as has been the case for..."** - Historical precedent + +#### Philosophy Elevation +```latex +This direction also in part follows the ViT philosophy of 'fewer inductive +biases' in the pursuit of universal features. + +As the non-local self-attention computation can learn translation-equivariant +features, they may also learn scale-equivariant features from certain forms +of supervised or self-supervised pre-training. +``` + +**Technique**: +- **"in part follows the...philosophy of"** - Theoretical connection +- **"fewer inductive biases"** - Core concept +- **Analogy**: translation-equivariant → scale-equivariant +- **"may also learn"** - Speculation (honest) + +#### Surprising Findings +```latex +Surprisingly, we observe: (i) it is sufficient to build a simple feature +pyramid from a single-scale feature map (without the common FPN design) and +(ii) it is sufficient to use window attention (without shifting) aided with +very few cross-window propagation blocks. +``` + +**Technique**: +- **"Surprisingly, we observe:"** - Marker +- **(i)** and **(ii)** - Structured list +- **"sufficient to"** - Not "optimal", scientific phrasing +- **"without the common [X]"** - Negative differentiation + +#### Deeper Surprise +```latex +More surprisingly, under some circumstances, our plain-backbone detector, +named ViTDet, can compete with the leading hierarchical-backbone detectors +(e.g., Swin, MViT). + +With Masked Autoencoder (MAE) pre-training, our plain-backbone detector can +outperform the hierarchical counterparts that are pre-trained on ImageNet-1K/21K +with supervision (Figure 3). + +The gains are more prominent for larger model sizes. +``` + +**Technique**: +- **"More surprisingly"** - Progressive emphasis +- **"under some circumstances"** - Measured claim +- **"named ViTDet"** - Brand at results +- **Specific comparison**: MAE vs ImageNet supervised +- **"The gains are more prominent for..."** - Pattern observation + +--- + +## 3. Methods Section: The "Minimal Adaptations" Narrative + +### Pattern: We Don't Aim to Invent, We Minimize + +**Structure**: +```markdown +1. [Declaration] We do NOT aim to develop new components +2. [Philosophy] Instead, make minimal adaptations sufficient to overcome challenges +3. [Specific] In particular, [what we actually do] +4. [Abandonment] This abandons [traditional component] +5. [Decoupling] Adaptations only during fine-tuning, do not alter pre-training +6. [Contrast] This is in contrast to [recent methods] that [what they do] +7. [Benefit] Our scenario enables [benefit], without [cost] +``` + +### ViTDet Methods Narrative: + +```latex +In our study, we do not aim to develop new components; instead, we make +minimal adaptations that are sufficient to overcome the aforementioned +challenges. + +In particular, our detector builds a simple feature pyramid from only the +last feature map of a plain ViT backbone (Figure 1). + +This abandons the FPN design and waives the requirement of a hierarchical +backbone. + +These adaptations are made only during fine-tuning and do not alter pre-training. + +This is in contrast to the recent methods that modify the attention computation +directly with backbone pre-training (e.g., Swin, MViT). + +Our scenario enables us to use the original ViT backbone for detection, without +redesigning pre-training architectures. +``` + +**Techniques**: +- **"we do not aim to develop new components"** - Clear scope +- **"minimal adaptations"** - Philosophy +- **"sufficient to"** - Not maximal, necessary +- **"This abandons..."** - What you give up +- **"only during fine-tuning"** - Temporal boundary +- **"do not alter pre-training"** - Upstream independence +- **"This is in contrast to"** - Competitor positioning +- **"enables us to use"** - Practical benefit + +--- + +## 4. Fair Comparison: The "Equal Effort" Declaration + +### Pattern: Admit Complexity → Claim Effort → Demonstrate Fairness + +**Structure**: +```markdown +1. [Admit] Modern systems involve [complexity] +2. [Claim] To compare as fairly as possible, we [effort] +3. [Specific 1] Use same [implementation] for all +4. [Specific 2] Different backbones get [appropriate treatment] +5. [Validation] Our results for [competitor] are [better/equal] to original +6. [Implication] Since we reproduce others well, comparisons are fair +``` + +### ViTDet Fair Comparison Statement: + +```latex +Modern detection systems involve many implementation details and subtleties. + +To focus on comparing backbones under as fair conditions as possible, we +incorporate the Swin and MViTv2 backbones into our implementation. + +We use the same implementation of Mask R-CNN and Cascade Mask R-CNN for all +ViT, Swin, and MViTv2 backbones. + +We use FPN for the hierarchical backbones of Swin/MViTv2. + +We search for optimal hyper-parameters separately for each backbone. +``` + +**Techniques**: +- **"involve many implementation details and subtleties"** - Admit difficulty +- **"under as fair conditions as possible"** - Effort disclaimer +- **"incorporate...into our implementation"** - What we did +- **"use the same...for all"** - Unified framework +- **"search for optimal...separately"** - Equal effort + +#### Self-Validation +```latex +Our Swin results are better than their counterparts in the original paper; +our MViTv2 results are better than or on par with those reported in the +original paper. +``` + +**Technique**: +- Report self-results → Show competence → Imply fairness + +--- + +## 5. Results: Multi-Factor Analysis + +### Pattern: Factors, Trends, Wall-Clock Time + +**Structure**: +```markdown +1. [Acknowledge Complexity] Comparisons involve [factors] +2. [Identify Trend] Our method presents better [trend behavior] +3. [Qualify] When [condition], our method [advantage] +4. [Expand] Moreover, [second dimension advantage] +5. [Explain] as [reason related to simplicity] +``` + +### ViTDet Results Narrative: + +```latex +Figure 3 plots the trade-offs. + +The comparisons here involve two factors: the backbone and the pre-training +strategy. + +Our plain-backbone detector, combined with MAE pre-training, presents better +scaling behavior. + +When the models are large, our method outperforms the hierarchical +counterparts of Swin/MViTv2, including those using IN-21K supervised +pre-training. + +Moreover, the plain ViT has a better wall-clock performance (Figure 3 right), +as the simpler blocks are more hardware-friendly. +``` + +**Techniques**: +- **"involve two factors"** - Analysis depth +- **"presents better scaling behavior"** - Trend, not just points +- **"When the models are large"** - Qualify claim +- **"Moreover"** - Second dimension +- **"better wall-clock performance"** - Practical metric +- **"simpler blocks are more hardware-friendly"** - Explain why + +--- + +## 6. "Surprisingly" Usage: Multi-Level Pattern + +### Level 1: Basic Surprise (Abstract) + +```latex +Surprisingly, we observe: (i) [simple sufficient without common practice] +and (ii) [simple sufficient without common practice] +``` + +**Characteristics**: +- Two findings (i) and (ii) +- "sufficient" not "optimal" +- "without [common practice]" +- Structured presentation + +### Level 2: Competitive Surprise (Introduction) + +```latex +More surprisingly, under some circumstances, our [method] can compete +with the leading [competitors]. +``` + +**Characteristics**: +- "More surprisingly" - Progressive +- "under some circumstances" - Measured +- "compete with" - Not "beat", competitive +- Name competitors specifically + +### Level 3: Superiority Surprise (Introduction) + +```latex +With [specific condition], our [method] can outperform the [competitors] +that use [stronger condition]. + +The gains are more prominent for [specific condition]. +``` + +**Characteristics**: +- Specific conditions compared +- "outperform" - Stronger claim here +- Pattern observation: "more prominent for" +- Shows understanding of when/where + +--- + +## 7. Ablation Study: Incremental + Destructive + +### Pattern: Baseline → Incremental Additions → Sufficient + +**Table Design**: +```markdown +Table X: [Component] Ablation +┌──────────────────────────────────────────┐ +│ Baseline | Metric | Δ │ +├──────────────────────────────────────────┤ +│ no [component] | 47.8 | - │ +│ (a) [common] | 50.3 | +2.5 │ +│ (b) [variant] | 50.9 | +3.1 │ +│ (c) ours: simple | 51.2 | +3.4 ✓ │ +├──────────────────────────────────────────┤ +│ Conclusion: Our simple [X] is sufficient │ +└──────────────────────────────────────────┘ +``` + +### ViTDet Table 1 Example: + +```latex +pyramid design APbox APmask +───────────────────────────────────────── +no feature pyramid 47.8 42.5 +(a) FPN, 4-stage 50.3 44.9 +(b) FPN, last-map 50.9 45.3 +(c) simple feature pyramid 51.2 45.5 +``` + +**Techniques**: +- **Baseline**: "no [X]" shows it's needed +- **(a), (b), (c)**: Progressive variations +- **Δ标注**: (+2.5) - Show incremental gains +- **Conclusion text**: "our simple pyramid is sufficient" + +--- + +## 8. "Interestingly" Usage: Pattern + Explanation + +### Pattern: Observation → Literature Support → Explanation + +**Structure**: +```markdown +Interestingly, [observation]. + +This is in line with the observation in [paper] that [their finding]. + +[Additional explanation or hypothesis]. +``` + +### ViTDet Example: + +```latex +Interestingly, performing propagation in the last 4 blocks is nearly as +good as even placement. + +This is in line with the observation in ViT [14] that ViT has longer +attention distance in later blocks and is more localized in earlier ones. +``` + +**Techniques**: +- **"Interestingly"** - Marker for unexpected +- **Observation**: Specific finding +- **"in line with the observation in"** - Literature support +- **Explanation**: Why it makes sense + +--- + +## 9. Minimalism Keywords: Design Simplification Vocabulary + +**Philosophy Keywords**: +- "minimal" - "minimal adaptations" +- "sufficient" - "is sufficient to" (not "optimal") +- "simple" - "simple feature pyramid" +- "plain" - "plain backbone" +- "decouple" - "decouple pre-training from fine-tuning" +- "independence" - "independence of upstream vs downstream" + +**Direction Keywords**: +- "pursue a different direction" - Positioning +- "in contrast to" - Differentiation +- "abandons" - What you give up +- "enables" - What your approach allows + +**Measured Claim Keywords**: +- "under some circumstances" - Not always +- "can compete with" - Competitive, not dominant +- "more prominent for" - When effect is stronger +- "is sufficient" - Necessary, not maximal + +**Surprise Markers** (use in order): +1. "Surprisingly" - First finding +2. "More surprisingly" - Deeper finding +3. "Interestingly" - Pattern observation +4. "Notably" - Important detail +5. "It is worth noting that" - Caveat/clarification + +--- + +## 10. Common Mistakes in Design Simplification Papers + +### ❌ Don't: +- Claim your method is "optimal" - You're simplifying, not optimizing +- Attack common practices - Acknowledge their value first +- Overgeneralize - "under some circumstances" is honest +- Forget to show fair comparison - Prove you tried hard with baselines +- Hide complexity - Admit what you don't address + +### ✅ Do: +- Use "sufficient" instead of "optimal" +- Say what you DON'T do ("do not aim to develop new components") +- Emphasize minimal changes ("minimal adaptations") +- Report when your method wins and when it doesn't +- Show "surprisingly" findings with proper qualification +- Demonstrate fair comparison effort +- Adapt only where necessary (fine-tuning, not pre-training) + +--- + +## 11. Revision Checklist for Design Simplification Papers + +**Before Submission, Verify:** + +- [ ] Abstract has "Surprisingly, we observe: (i)... and (ii)..." +- [ ] Introduction establishes conventional practice first +- [ ] Common solution is acknowledged ("has shown successful results") +- [ ] "We pursue a different direction" is stated clearly +- [ ] Philosophy is elevated ("fewer inductive biases") +- [ ] "More surprisingly" used for deeper claim +- [ ] Methods section says "we do not aim to develop new components" +- [ ] "minimal adaptations" philosophy stated +- [ ] "only during fine-tuning" boundary specified +- [ ] Fair comparison effort described explicitly +- [ ] Self-validation shown (our reproduction of others is good) +- [ ] Multi-factor analysis in results (scaling, wall-clock) +- [ ] Ablations show incremental progression +- [ ] "sufficient" used, not "optimal" +- [ ] Under what conditions is stated ("under some circumstances") + +--- + +## 12. Example: Applying This Pattern + +### Original Idea (Not Design Simplification): +"We propose a new feature pyramid that improves detection AP by 3%." + +### Design Simplification Version: +"Modern detectors use hierarchical backbones with FPN. With plain ViT +emerging as powerful backbones, a common solution re-introduces hierarchy +(abandoning the plain philosophy). We pursue a different direction: plain +backbones with minimal adaptations. Surprisingly, we observe a simple feature +pyramid from a single-scale map is sufficient (without FPN). More +surprisingly, with MAE pre-training, ViTDet competes with hierarchical +detectors, especially for larger models. This decouples pre-training from +fine-tuning, maintaining upstream/downstream independence." + +**The Design Simplification Frame**: +- Conventional: Hierarchy + FPN +- Challenge: Plain ViT is...plain +- Common: Swin (abandons philosophy) +- Ours: Minimal adaptations +- Surprise: Simple is sufficient +- Philosophy: Decoupling, independence + +--- + +## Paper Metadata + +**Title**: Exploring Plain Vision Transformer Backbones for Object Detection (ViTDet) + +**Authors**: Yanghao Li, Hanzi Mao, Kaiming He + +**Venue**: ECCV 2022 + +**arXiv**: 2203.16527 + +**Key Concepts**: +- Plain ViT for detection (no hierarchy needed) +- Simple feature pyramid (no FPN needed) +- Minimal adaptations philosophy +- Decoupling pre-training from fine-tuning +- MAE pre-training synergy + +--- + +## Extracted by + +**Date**: 2026-01-26 + +**Source**: Analysis of ViTDet paper (21 pages) + +**Extraction Focus**: Design simplification paper writing patterns, "surprisingly" +findings reporting, minimal adaptations philosophy, fair comparison strategies + +**For Integration**: ml-paper-writing skill knowledge base diff --git a/.agents/skills/ml-paper-writing/references/knowledge/kaiming_he_injection_record.json b/.agents/skills/ml-paper-writing/references/knowledge/kaiming_he_injection_record.json new file mode 100644 index 0000000..7ffd675 --- /dev/null +++ b/.agents/skills/ml-paper-writing/references/knowledge/kaiming_he_injection_record.json @@ -0,0 +1,153 @@ +{ + "metadata": { + "source": "Kaiming He Papers Analysis", + "date": "2026-01-26", + "papers_analyzed": 11, + "analysis_method": "Text extraction and pattern mining", + "latest_addition": { + "papers": ["Mean Flows", "ViTDet", "MoCo v2", "Deconstructing Denoising Diffusion Models", "Autoregressive Image Generation (MAR)"], + "extraction_date": "2026-01-26", + "new_knowledge_files": [ + "theory-driven-papers-kaiming-he.md", + "design-simplification-papers-kaiming-he.md" + ] + } + }, + "knowledge_files": { + "structure.md": { + "status": "updated", + "last_update": "2026-01-26", + "contains": "Basic structure patterns from 19 Kaiming He papers" + }, + "writing-techniques.md": { + "status": "needs_update", + "last_update": "2026-01-26", + "contains": "Basic writing techniques from 19 Kaiming He papers" + }, + "rethinking-papers-kaiming-he.md": { + "status": "complete", + "focus": "Rethinking papers, challenging conventional wisdom", + "source_paper": "Autoregressive Image Generation without Vector Quantization (NeurIPS 2024 Spotlight)" + }, + "theory-driven-papers-kaiming-he.md": { + "status": "new", + "focus": "Theory-driven papers, first principles, MeanFlow Identity", + "source_paper": "Mean Flows for One-step Generative Modeling (2025)" + }, + "design-simplification-papers-kaiming-he.md": { + "status": "new", + "focus": "Design simplification, minimal adaptations, 'Surprisingly' findings", + "source_paper": "Exploring Plain Vision Transformer Backbones for Object Detection (ViTDet, ECCV 2022)" + } + }, + "patterns_extracted": { + "introduction_frameworks": { + "principle_introduction": { + "source": "MeanFlows", + "pattern": "Background → Problem → Critique (Despite...) → Core Concept → Theory → Advantage → Results", + "keywords": ["principled", "intrinsic", "well-defined", "naturally", "first principles"] + }, + "challenge_assumptions": { + "source": "ViTDet", + "pattern": "Traditional → New Challenge → Common Solution → Our Direction → Philosophy → Surprisingly → Implications", + "keywords": ["minimal adaptations", "sufficient", "decouple", "independence", "surprisingly"] + }, + "rethinking_conventional_wisdom": { + "source": "MAR", + "pattern": "Conventional wisdom → Question → Analysis → Alternative → Results → Vision", + "keywords": ["Conventional wisdom holds that", "Is it necessary", "not a necessity"] + } + }, + "surprisingly_findings": { + "level_1": { + "pattern": "Surprisingly, we observe: (i)... and (ii)...", + "usage": "First-level surprise - basic findings", + "example": "ViTDet Abstract" + }, + "level_2": { + "pattern": "More surprisingly, under some circumstances...", + "usage": "Second-level surprise - competitive results", + "example": "ViTDet Introduction" + }, + "level_3": { + "pattern": "With [condition], outperforms... gains more prominent for...", + "usage": "Third-level surprise - superiority under conditions", + "example": "ViTDet Introduction" + }, + "variants": { + "interestingly": "Observation + literature support + explanation", + "notably": "Important detail or counter-intuitive result", + "it_is_worth_noting": "Technical caveat or clarification" + } + }, + "ablation_techniques": { + "incremental_tables": { + "pattern": "Baseline → (a) → (b) → (c) with Δ标注", + "source": "ViTDet Table 1" + }, + "destructive_comparison": { + "pattern": "Intentionally wrong values to prove necessity", + "source": "MeanFlows Table 1b" + }, + "narrative_structure": { + "observation_then_explain": "Observe pattern → Provide explanation (literature/hypothesis/theory)" + } + }, + "theoretical_derivation": { + "naming_identity": { + "pattern": "Define → Derive → Name ('X Identity')", + "source": "MeanFlows MeanFlow Identity" + }, + "step_by_step": { + "pattern": "Motivation → Derivation with 'Now we...' → Justification with 'where...'", + "source": "MeanFlows Section 2" + } + }, + "comparison_techniques": { + "principled_vs_heuristic": { + "pattern": "At the core...does not depend on...In contrast, typically rely on...", + "source": "MeanFlows" + }, + "fair_comparison_declaration": { + "pattern": "Admit complexity → Claim effort → Demonstrate fairness", + "source": "ViTDet" + }, + "multi_factor_analysis": { + "pattern": "Factors identified → Trend behavior → Wall-clock time", + "source": "ViTDet Results" + } + }, + "keyword_strategies": { + "theory_paper": ["principled", "intrinsic", "well-defined", "naturally", "self-contained", "solely originated from"], + "design_paper": ["minimal", "sufficient", "decouple", "independence", "surprisingly", "abandons"], + "rethinking_paper": ["Conventional wisdom holds that", "not a necessity", "orthogonal to", "uncharted realm"] + } + }, + "papers_analyzed_list": [ + "Non-local Neural Networks", + "SlowFast Networks", + "Rethinking ImageNet Pre-training", + "Faster R-CNN", + "Delving Deep into Rectifiers (PReLU)", + "Spatial Pyramid Pooling (SPP-net)", + "Deconstructing Denoising Diffusion Models", + "Autoregressive Image Generation without Vector Quantization (MAR)", + "Mean Flows for One-step Generative Modeling", + "Exploring Plain Vision Transformer Backbones for Object Detection (ViTDet)", + "MoCo v2: Improved Baselines with Momentum Contrastive Learning" + ], + "integration_summary": { + "total_papers": 11, + "knowledge_files": 5, + "patterns_extracted": 25, + "paper_types_identified": [ + "Theory-driven (MeanFlows)", + "Design simplification (ViTDet)", + "Rethinking (MAR)", + "Deconstruction (DDM)", + "Milestone (PReLU)", + "Multi-task (SPP-net)", + "Technical note (MoCo v2)" + ] + } +} diff --git a/.agents/skills/ml-paper-writing/references/knowledge/paper-miner-writing-memory.md b/.agents/skills/ml-paper-writing/references/knowledge/paper-miner-writing-memory.md new file mode 100644 index 0000000..7320060 --- /dev/null +++ b/.agents/skills/ml-paper-writing/references/knowledge/paper-miner-writing-memory.md @@ -0,0 +1,37 @@ +# Paper-Miner Writing Memory + +This is the **canonical global writing memory** maintained by `paper-miner`. + +It stores reusable academic writing knowledge mined from papers across venues and projects. + +## Rules + +- This is the **only maintained paper-miner writing memory**. +- `paper-miner` writes here even when invoked inside a specific project. +- Do **not** create project-specific paper-miner writing memory. +- Do **not** split new mined knowledge across multiple maintained category files. +- Keep source attribution explicit and avoid duplicate entries. + +## Writing patterns mined + + + +## Structure signals + + + +## Reusable phrasing + + + +## Venue-specific signals + + + +## How this helps our writing + + + +## Source index + + diff --git a/.agents/skills/ml-paper-writing/references/knowledge/rethinking-papers-kaiming-he.md b/.agents/skills/ml-paper-writing/references/knowledge/rethinking-papers-kaiming-he.md new file mode 100644 index 0000000..26a7c86 --- /dev/null +++ b/.agents/skills/ml-paper-writing/references/knowledge/rethinking-papers-kaiming-he.md @@ -0,0 +1,1023 @@ +# "Rethinking" Papers: Challenging Conventional Wisdom + +**Source**: Kaiming He et al., "Autoregressive Image Generation without Vector Quantization" (NeurIPS 2024 Spotlight) + +**Paper Type**: Paradigm-challenging / "Rethinking" paper + +**Core Pattern**: Question deeply-held assumptions → Propose alternative → Demonstrate effectiveness + +--- + +## 1. Abstract Structure: Challenging Conventional Wisdom + +### Pattern: The "Conventional Wisdom" Opening + +**Template**: +```markdown +Abstract: +1. [Hook] Conventional wisdom holds that [widely-believed assumption]. +2. [Observation] We observe that [counter-point or nuance]. +3. [Question] Is it necessary for [core assumption]? +4. [Proposal] In this work, we propose [alternative approach]. +5. [Solution] Rather than [traditional method], we [novel method]. +6. [Results] This approach [eliminates/enables] X, achieving [strong results]. +7. [Vision] We hope this work will [broader impact statement]. +``` + +### MAR Abstract Example (annotated): + +```latex +Conventional wisdom holds that [autoregressive models for image generation +are typically accompanied by vector-quantized tokens]. + +We observe that while [discrete-valued space can facilitate representing +a categorical distribution], it is [not a necessity for autoregressive modeling]. + +In this work, we propose to [model the per-token probability distribution +using a diffusion procedure], which allows us to [apply autoregressive models +in a continuous-valued space]. + +Rather than using [categorical cross-entropy loss], we define a +[Diffusion Loss function] to model the per-token probability. + +This approach [eliminates the need for discrete-valued tokenizers]. + +We evaluate its effectiveness across [a wide range of cases], including +[standard autoregressive models and generalized masked autoregressive (MAR) variants]. + +By removing vector quantization, our image generator achieves [strong results] +while enjoying [the speed advantage of sequence modeling]. + +We hope this work will motivate [the use of autoregressive generation in +other continuous-valued domains and applications]. +``` + +### Key Techniques: + +1. **"Conventional wisdom holds that..."** - Strong, respectful opening that acknowledges prevailing beliefs +2. **"We observe that..."** - Signals empirical insight rather than mere speculation +3. **"it is not a necessity for..."** - Gentle but direct challenge to the assumption +4. **"Rather than..."** - Clear alternative positioning +5. **"This approach eliminates the need for..."** - Practical benefit statement +6. **"We hope this work will..."** - Forward-looking vision + +--- + +## 2. Introduction: The Question-Driven Framework + +### Pattern: Start with the Question + +**Traditional Introduction** (less effective): +- "Image generation is an important problem..." +- "Existing methods use VQ tokenizers..." +- "We propose a new method..." + +**"Rethinking" Introduction** (Kaiming He style): +```markdown +1. [Establish Context] Domain context + prevailing approach +2. [Identify Assumption] The widely-held belief +3. [Formulate Question] "Is it necessary for..." +4. [Analyze Requirements] What is truly essential? +5. [Propose Alternative] If alternative model exists... +6. [Present Solution] Our specific proposal +7. [Broader Implications] What this enables +``` + +### MAR Introduction Structure (lines 20-79): + +#### Part 1: Context and Prevailing Belief (lines 20-29) +```latex +Autoregressive models are currently the de facto solution to generative +models in natural language processing [38, 39, 3]. These models predict +the next word or token in a sequence based on the previous words as input. + +Given the discrete nature of languages, the inputs and outputs of these +models are in a categorical, discrete-valued space. + +This prevailing approach has led to a widespread belief that autoregressive +models are inherently linked to discrete representations. +``` + +**Technique**: Start from uncontroversial facts (NLP success) → Show how they led to a belief (AR ↔ discrete) + +#### Part 2: Consequences of the Belief (lines 25-29) +```latex +As a result, research on generalizing autoregressive models to continuous- +valued domains—most notably, image generation—has intensely focused on +discretizing the data [6, 13, 40]. + +A commonly adopted strategy is to train a discrete-valued tokenizer on +images, which involves a finite vocabulary obtained by vector quantization +(VQ) [51, 41, 41]. +``` + +**Technique**: Show concrete consequences of the belief (VQ tokenizers everywhere) + +#### Part 3: The Research Question (lines 30-33) +```latex +In this work, we aim to address the following question: + +Is it necessary for autoregressive models to be coupled with vector- +quantized representations? +``` + +**Technique**: Explicit question format. Bold, direct, centered. + +#### Part 4: Analyzing What's Essential (lines 31-37) +```latex +We note that the autoregressive nature, i.e., "predicting next tokens based +on previous ones", is independent of whether the values are discrete or +continuous. + +What is needed is to model the per-token probability distribution, which +can be measured by a loss function and used to draw samples from. + +Discrete-valued representations can be conveniently modeled by a categorical +distribution, but it is not conceptually necessary. +``` + +**Technique**: Distinguish mechanism (autoregression) from implementation (discrete tokens) + +#### Part 5: The "If..." Condition (lines 36-37) +```latex +If alternative models for per-token probability distributions are presented, +autoregressive models can be approached without vector quantization. +``` + +**Technique**: Logical bridge - "If X, then Y" sets up your contribution + +#### Part 6: Your Solution (lines 38-58) +```latex +With this observation, we propose to model the per-token probability +distribution by a diffusion procedure operating on continuous-valued domains. + +[Technical description of Diffusion Loss...] +``` + +#### Part 7: Benefits and Implications (lines 59-79) +```latex +Our approach eliminates the need for discrete-valued tokenizers. Vector- +quantized tokenizers are difficult to train and are sensitive to gradient +approximation strategies [51, 41, 40, 27]. Their reconstruction quality +often falls short compared to continuous-valued counterparts [42]. + +Our approach allows autoregressive models to enjoy the benefits of higher- +quality, non-quantized tokenizers. + +[...] + +The effectiveness of our method reveals a largely uncharted realm of image +generation: modeling the interdependence of tokens by autoregression, +jointly with the per-token distribution by diffusion. +``` + +**Technique**: Multiple benefit levels: +- **Eliminates pain points**: VQ is hard to train +- **Enables benefits**: Higher-quality tokenizers +- **Reveals new territory**: "uncharted realm" + +--- + +## 3. Rethinking Section: Dissecting the Status Quo + +### Pattern: "Rethinking X" Chapter + +When challenging conventional wisdom, dedicate a section to re-examining the assumptions. + +### MAR Section 3.1: "Rethinking Discrete-Valued Tokens" + +**Structure**: +```markdown +1. [Setup] Define the conventional approach mathematically +2. [Analysis] Identify the essential properties (not the implementation) +3. [Insight] Show that the implementation is not necessary +4. [Conclusion] State what's actually needed +``` + +### MAR Example (lines 110-126): + +#### Step 1: Define the Conventional Approach +```latex +To begin with, we revisit the roles of discrete-valued tokens in +autoregressive generation models. + +Denote as x the ground-truth token to be predicted at the next position. +With a discrete tokenizer, x can be represented as an integer: 0 ≤ x < K, +with a vocabulary size K. + +The autoregressive model produces a continuous-valued D-dim vector z ∈ R^D, +which is then projected by a K-way classifier matrix W ∈ R^(K×D). + +Conceptually, this formulation models a categorical probability distribution +in the form of p(x|z) = softmax(Wz). +``` + +**Technique**: Mathematically precise setup. Shows you deeply understand the status quo. + +#### Step 2: Identify Essential Properties +```latex +In the context of generative modeling, this probability distribution must +exhibit two essential properties. + +(i) A loss function that can measure the difference between the estimated +and true distributions. + +(ii) A sampler that can draw samples from the distribution x ~ p(x|z) at +inference time. +``` + +**Technique**: Abstract away from implementation to functional requirements + +#### Step 3: Show Convention is Not Necessary +```latex +In the case of categorical distribution, this can be simply done by the +cross-entropy loss. + +[...] + +This analysis suggests that discrete-valued tokens are not necessary for +autoregressive models. +``` + +**Technique**: "This analysis suggests" - evidence-based conclusion + +#### Step 4: What's Actually Needed +```latex +Instead, it is the requirement of modeling a distribution that is essential. + +A discrete-valued token space implies a categorical distribution, whose loss +function and sampler are simple to define. + +What we actually need are a loss function and its corresponding sampler for +distribution modeling. +``` + +**Technique**: Reposition the problem - it's about distribution modeling, not discrete tokens + +--- + +## 4. Unifying Different Methods: The "Generalized Framework" Pattern + +### Pattern: Show Two Seemingly Different Methods Are Special Cases + +**MAR Section 3.4**: "Unifying Autoregressive and Masked Generative Models" + +**Structure**: +```markdown +1. [Observation] These methods seem different but share a core principle +2. [Insight] The core principle is X, not the surface differences +3. [Demonstration] Show how Method A fits the principle +4. [Demonstration] Show how Method B fits the principle +5. [Unification] Both are special cases of a general framework +6. [Benefit] This understanding enables new combinations/insights +``` + +### MAR Example (lines 181-259): + +#### Opening Observation +```latex +We show that masked generative models, e.g., MaskGIT [4] and MAGE [29], +can be generalized under the broad concept of autoregression, i.e., +next token prediction. +``` + +#### The Insight +```latex +Bidirectional attention can perform autoregression. + +The concept of autoregression is orthogonal to network architectures: +autoregression can be done by RNNs [50], CNNs [49, 7], and +Transformers [38, 36, 6]. + +When using Transformers, although autoregressive models are popularly +implemented by causal attention, we show that they can also be done by +bidirectional attention. +``` + +**Technique**: "Orthogonal to" - shows independence of concepts + +#### Visual Demonstration (Figure 2) +```latex +Note that the goal of autoregression is to predict the next token given +the previous tokens; it does not constrain how the previous tokens +communicate with the next token. + +We can adopt the bidirectional attention implementation as done in Masked +Autoencoder (MAE) [21]. See Figure 2(b). +``` + +**Technique**: Use visualization to show "unexpected" equivalence + +#### Generalization (Figure 3) +```latex +Autoregressive models in random orders... + +Masked autoregressive models: In masked generative modeling [4, 29], +the models predict a random subset of tokens based on known/predicted +tokens. + +This can be formulated as permuting the token sequence by a random order, +and then predicting multiple tokens based on previous tokens. + +Conceptually, this is an autoregressive procedure... +``` + +**Technique**: Mathematical unification - same formula, different instantiations + +#### Naming the Generalization +```latex +We refer to this variant as Masked Autoregressive (MAR) models. + +MAR is a random-order autoregressive model that can predict multiple +tokens simultaneously. +``` + +**Technique**: Define new terminology to capture the unified concept + +--- + +## 5. Flexibility Arguments: Demonstrating Generality + +### Pattern: Show Your Method Works in Many Configurations + +**MAR Section 5.1**: "Flexibility of Diffusion Loss" + +**Structure**: +```markdown +1. [Claim] One significant advantage is flexibility with various X +2. [Demonstration 1] Works with configuration A +3. [Demonstration 2] Works with configuration B (even surprising case) +4. [Demonstration 3] Works with configuration C +5. [Insight] This flexibility enables Y +``` + +### MAR Example (lines 349-366, Table 2): + +#### Opening Claim +```latex +One significant advantage of Diffusion Loss is its flexibility with +various tokenizers. +``` + +#### Surprising Case 1: VQ Tokenizers +```latex +Diffusion Loss can be easily used even given a VQ tokenizer. + +We simply treat the continuous-valued latent before the VQ layer as the +tokens. + +This variant gives us 7.82 FID (w/o CFG), compared favorably with 8.79 +FID (Table 1) of cross-entropy loss using the same VQ tokenizer. +``` + +**Technique**: Show it works even with the "wrong" tokenizer type (VQ) + +#### Surprising Case 2: Mismatched Stride +```latex +Interestingly, Diffusion Loss also enables us to use tokenizers with +mismatched strides. + +In Table 2, we study a KL-8 tokenizer whose stride is 8 and output +sequence length is 32×32. Without increasing the sequence length of the +generator, we group 2×2 tokens into a new token. + +Despite the mismatch, we are able to obtain decent results... +``` + +**Technique**: "Interestingly" + "Despite the mismatch" - emphasizes robustness + +#### Surprising Case 3: Different Architecture +```latex +Further, this property allows us to investigate other tokenizers, e.g., +Consistency Decoder [35], a non-VQ tokenizer of a different +architecture/stride designed for different goals. +``` + +**Technique**: Show applicability beyond original design space + +#### Comprehiveness Check +```latex +For comprehensiveness, we also train a KL-16 tokenizer on ImageNet using +the code of [42]... +``` + +**Technique**: "For comprehensiveness" - signals thoroughness + +--- + +## 6. Speed/Accuracy Trade-offs: Plotting the Pareto Frontier + +### Pattern: Show Your Method Dominates or Expands the Frontier + +**MAR Section 5.2**: "Speed/accuracy Trade-off" (Figure 6) + +**Structure**: +```markdown +1. [Setup] We enjoy flexibility of X +2. [Visualization] Figure/plot showing trade-off curves +3. [Comparison 1] Compare within your variants +4. [Comparison 2] Compare to other methods +5. [Highlight] Notable point on the curve +``` + +### MAR Example (lines 425-474): + +#### Setup +```latex +Following MaskGIT [4], our MAR enjoys the flexibility of predicting +multiple tokens at a time. + +This is controlled by the number of autoregressive steps at inference time. +``` + +#### Comparison Within Family +```latex +Figure 6 plots the speed/accuracy trade-off. + +MAR has a better trade-off than its AR counterpart, noting that AR is +with the efficient kv-cache. +``` + +**Technique**: "noting that..." - acknowledge AR's advantage, still win + +#### Comparison to Other Methods +```latex +With Diffusion Loss, MAR also shows a favorable trade-off in comparison +with the recently popular Diffusion Transformer (DiT) [37]. + +As a latent diffusion model, DiT models the interdependence of all tokens +by the diffusion process. The speed/accuracy trade-off of DiT is mainly +controlled by its diffusion steps. + +Unlike our diffusion process on a small MLP, the diffusion process of DiT +involves the entire Transformer architecture. + +Our method is more accurate and faster. +``` + +**Technique**: Explain *why* you win (architectural difference) + +#### Highlight Specific Point +```latex +Notably, our method can generate at a rate of <0.3 second per image with +a strong FID of <2.0. +``` + +**Technique**: "Notably" + impressive numbers - memorable takeaway + +--- + +## 7. System-Level Comparison Tables: Organizing by Categories + +### Pattern: Group Related Work, Highlight Your Position + +**MAR Table 4**: "System-level comparison on ImageNet 256×256" + +**Structure**: +```markdown +Table 4: +┌─────────────────────────────────┐ +│ Category 1: pixel-based │ +│ - Method A │ +│ - Method B │ +├─────────────────────────────────┤ +│ Category 2: vector-quantized │ +│ - Method C │ +│ - Method D │ +├─────────────────────────────────┤ +│ Category 3: continuous-valued │ +│ - Method E │ +│ - Method F │ +├─────────────────────────────────┤ +│ **Your Methods** (highlighted) │ +│ - Your Method B │ +│ - Your Method L │ +│ - Your Method H │ +└─────────────────────────────────┘ +``` + +### MAR Example (lines 448-470): + +```latex +Table 4: System-level comparison on ImageNet 256×256 conditional +generation. Diffusion Loss enables Masked Autoregression to achieve +leading results in comparison with previous systems. + +†: LDM operates on continuous-valued tokens, though this result uses +a quantized tokenizer. + +w/o CFG w/ CFG +#params FID↓ IS↑ Pre.↑ Rec.↑ FID↓ IS↑ Pre.↑ Rec.↑ +──────────────────────────────────────────────────────────── +pixel-based +ADM [10] 554M 10.94 101.0 0.69 0.63 4.59 186.7 0.82 0.52 +VDM++ [26] 2B 2.40 225.3 - - 2.12 267.7 - - + +vector-quantized tokens +Autoreg. w/ VQGAN [13] 1.4B 15.78 78.3 - - - - - - +MaskGIT [4] 227M 6.18 182.1 0.80 0.51 - - - - +MAGE [29] 230M 6.93 195.8 - - - - - - +MAGVIT-v2 [55] 307M 3.65 200.5 - - 1.78 319.4 - - + +continuous-valued tokens +LDM-4† [42] 400M 10.56 103.5 0.71 0.62 3.60 247.7 0.87 0.48 +U-ViT-H/2-G [2] 501M - - - - 2.29 263.9 0.82 0.57 +DiT-XL/2 [37] 675M 9.62 121.5 0.67 0.67 2.27 278.2 0.83 0.57 +[... other methods ...] + +MAR-B, Diff Loss 208M 3.48 192.4 0.78 0.58 2.31 281.7 0.82 0.57 +MAR-L, Diff Loss 479M 2.60 221.4 0.79 0.60 1.78 296.0 0.81 0.60 +MAR-H, Diff Loss 943M 2.35 227.8 0.79 0.62 1.55 303.7 0.81 0.62 +``` + +**Key Techniques**: +1. **Group by paradigm**: pixel-based → VQ → continuous-valued +2. **Show progression**: Earlier methods → later methods → yours +3. **Multiple metrics**: FID, IS, Precision, Recall (comprehensive) +4. **Multiple configurations**: w/o CFG and w/ CFG (fair comparison) +5. **Model sizes**: Show efficiency (your 479M vs their 675M) +6. **Dagger (†)**: Clarify edge cases (LDM uses quantized despite continuous) +7. **Bold your methods**: Visual separation (here shown as last group) + +--- + +## 8. Admitting Limitations: Honest and Specific + +### Pattern: Acknowledge Weaknesses Proactively + +**MAR Appendix A**: "Limitations and Broader Impacts" + +**Structure**: +```markdown +Limitations: +1. [Specific artifact] Show example (Figure 8) +2. [Dependency] Acknowledge reliance on external components +3. [Scope] Be honest about what you haven't tested + +Broader Impacts: +1. [Positive] How this advances the field +2. [Positive] Practical applications +3. [Negative] Potential misuse considerations +``` + +### MAR Limitations Example (lines 629-642): + +#### Limitation 1: Visual Artifacts +```latex +First of all, our image generation system can produce images with +noticeable artifacts (Figure 8). + +This limitation is commonly observed in existing methods, especially when +trained on controlled, academic data (e.g., ImageNet). + +Research-driven models trained on ImageNet still have a noticeable gap +in visual quality in comparison with commercial models trained on +massive data. +``` + +**Techniques**: +- Show example (Figure 8) - transparency +- "commonly observed" - you're not uniquely bad +- Contextualize: academic vs. commercial data + +#### Limitation 2: External Dependency +```latex +Second, our image generation system relies on existing pre-trained +tokenizers. + +The quality of our system can be limited by the quality of these tokenizers. + +Pre-training better tokenizers is beyond the scope of this paper. +``` + +**Techniques**: +- "relies on" - direct acknowledgment +- "can be limited by" - honest constraint +- "beyond the scope" - boundary setting + +#### Limitation 3: Unexplored Territory +```latex +Last, we note that given the limited computational resources, we have +primarily tested our method on the ImageNet benchmark. + +Further validation is needed to assess the scalability and robustness +of our approach in more diverse and real-world scenarios. +``` + +**Techniques**: +- "given limited computational resources" - context, not excuse +- "primarily tested" - honesty about scope +- "Further validation is needed" - explicit call for future work + +--- + +## 9. Visionary Conclusion: The "We Hope" Pattern + +### Pattern: End with Forward-Looking Impact Statement + +**MAR Conclusion** (lines 482-488): + +```latex +The effectiveness of Diffusion Loss on various autoregressive models +suggests new opportunities: modeling the interdependence of tokens by +autoregression, jointly with the per-token distribution by diffusion. + +This is unlike the common usage of diffusion that models the joint +distribution of all tokens. + +Our strong results on image generation suggest that autoregressive models +or their extensions are powerful tools beyond language modeling. + +These models do not need to be constrained by vector-quantized +representations. + +We hope our work will motivate the research community to explore sequence +models with continuous-valued representations in other domains. +``` + +### Structure: +```markdown +1. [Synthesis] What your results suggest (new opportunities) +2. [Contrast] How this differs from common usage +3. [Generalization] Broader lesson (powerful tools beyond X) +4. [Freedom] Removing constraints (not limited by Y) +5. [Call to action] "We hope" - inspire future work +``` + +### Key Phrases: +- "suggests new opportunities" +- "unlike the common usage" +- "powerful tools beyond [original domain]" +- "do not need to be constrained by" +- "We hope our work will motivate..." + +--- + +## 10. Writing Style: Kaiming He's Signature Techniques + +### Tone Characteristics: + +1. **Respectful but Firm Challenge** + - "Conventional wisdom holds that..." (respect) + - "...is not a necessity" (firm) + - Never dismissive of prior work + +2. **Empirically Grounded** + - "We observe that..." (not "We believe that...") + - "This analysis suggests..." (evidence-based) + - Show, don't just tell + +3. **Logical Precision** + - Distinguish mechanism from implementation + - "If X, then Y" formulations + - Mathematical clarity in conceptual arguments + +4. **Measured Claims** + - "can be" not "is always" + - "suggests" not "proves" + - "enables" not "guarantees" + +5. **Generous Citation** + - Cite the methods you're challenging + - Credit relevant prior work + - "following [X]" for techniques you adopt + +### Sentence Patterns: + +**Challenge Patterns:** +- "Conventional wisdom holds that [X]. We observe that [Y]." +- "This prevailing approach has led to a widespread belief that [X]." +- "[X] is not a necessity for [Y]." +- "What is needed is [X], not [Y]." + +**Insight Patterns:** +- "The concept of [X] is orthogonal to [Y]." +- "This is unlike the common usage of [X]." +- "These models do not need to be constrained by [X]." +- "This reveals a largely uncharted realm of [X]." + +**Demonstration Patterns:** +- "Interestingly, [X] also enables us to..." +- "Despite [challenge], we are able to..." +- "For comprehensiveness, we also..." +- "Notably, [impressive result]." + +**Vision Patterns:** +- "We hope this work will motivate..." +- "This suggests new opportunities for..." +- "We believe this will be beneficial to..." +- "Further validation is needed to assess..." + +--- + +## 11. Figures: Making the Invisible Visible + +### Pattern: Use Visualization to Show Conceptual Equivalence + +**MAR Figure 2**: "Bidirectional attention can do autoregression" + +**Purpose**: Challenge the assumption that autoregression requires causal attention + +**Technique**: Side-by-side comparison +- (a) Causal attention (conventional) +- (b) Bidirectional attention (surprising but equivalent) + +**Annotation Key Points**: +- "next token prediction" (highlight both do this) +- "loss on [all/unknown] tokens" (show the difference) +- Arrows showing attention flow + +**MAR Figure 3**: "Generalized Autoregressive Models" + +**Purpose**: Show AR, random-order AR, and MAR are special cases + +**Technique**: Progressive generalization +- (a) AR, raster order (standard) +- (b) AR, random order (small variation) +- (c) Masked AR (generalization) + +**Visual Language**: +- "unknown" vs "known/predicted" tokens +- Color coding for what's predicted at each step +- Consistent notation across panels + +**Key**: Figures should make conceptual contributions visually obvious + +--- + +## 12. Related Work: Positioning Within Traditions + +### Pattern: Acknowledge Lineage, Show Departure + +**MAR Section 2**: "Related Work" (lines 80-103) + +**Structure**: +```markdown +1. [Main tradition] Sequence Models for Image Generation + - Pioneering work [citations] + - Your position: "Related to our work, [X] also focuses on..." + - Contrast: "In [X], [limitation]. In contrast, our method..." + +2. [Adjacent tradition] Diffusion for Representation Learning + - Prior work [citations] + - Clarify difference: "These efforts have been focused on [X], + rather than [Y]" + - "In their scenarios, [goal] is not a goal" + +3. [Conceptual connection] Diffusion for Policy Learning + - "Our work is conceptually related to [X]" + - Analogy: "In image generation, we can think of [X] as [Y]" + - Difference: "Despite this conceptual connection, [key difference]" +``` + +### Key Techniques: + +1. **"Related to our work..."** - Acknowledge contemporaries generously +2. **"In contrast..."** - Sharp differentiation +3. **"These efforts have been focused on..."** - Respectful boundary setting +4. **"In their scenarios..."** - Contextualize differences +5. **"Conceptually related to..."** - Find connections across domains +6. **"We can think of X as Y"** - Analogies for insight +7. **"Despite this conceptual connection..."** - Honest differences + +--- + +## 13. Experiments: Ablation as Storytelling + +### Pattern: Experiments Should Validate the "Rethinking" + +**MAR Section 5**: Experiments organized to prove the rethinking + +**Flow**: +```markdown +1. Properties of Diffusion Loss (prove the alternative works) + - vs Cross-entropy (Table 1) + - Flexibility (Table 2) + - Ablations (Table 3, Figures 4-5) + +2. Properties of Generalized AR (prove the unification) + - From AR to MAR (Table 1) + - Speed/accuracy trade-off (Figure 6) + +3. Benchmarking (prove the results are strong) + - System-level comparison (Table 4) + - Qualitative results (Figure 7) +``` + +**Key**: Every experiment should map to a conceptual claim from the introduction + +### Table Design Patterns: + +**Comparison Tables** (Table 1): +- Group by variants (AR → MAR) +- Show progression (Cross-ent → Diff Loss) +- Multiple configurations (w/ and w/o CFG) +- Clear metrics (FID↓, IS↑) + +**Flexibility Tables** (Table 2): +- "Flexibility of X" in caption +- Diverse configurations +- Architecture variations +- Stride mismatches +- rFID column (reconstruction quality) + +**Ablation Tables** (Table 3): +- One dimension varied (MLP width) +- Show params and inference time +- Demonstrate efficiency +- Default row marked (implicit) + +--- + +## 14. Summary: The "Rethinking" Paper Template + +### When to Use This Pattern: + +**Your paper is a "Rethinking" paper if:** +- You challenge a widely-held assumption in your field +- You show two seemingly different methods are equivalent +- You propose removing a "necessary" component +- You unify multiple approaches under one framework + +**Not a "Rethinking" paper if:** +- You propose an incremental improvement +- You combine existing techniques without conceptual insight +- You're optimizing within an established paradigm + +### Abstract Template: +``` +Conventional wisdom holds that [assumption]. We observe that [nuance]. +Is it necessary for [assumption]? In this work, we propose [alternative]. +Rather than [traditional method], we [novel method]. This approach [benefits]. +We hope this work will [vision]. +``` + +### Introduction Template: +``` +[Context] Domain X uses method Y widely. +[Belief] This has led to belief that Z. +[Question] Is Z necessary? +[Analysis] What's actually needed is W. +[Proposal] We propose alternative that achieves W. +[Results] Strong performance + [specific benefit]. +[Vision] This opens new direction. +``` + +### Section Template ("Rethinking X"): +``` +1. Define the conventional approach mathematically +2. Identify essential properties (2-3 items) +3. Show conventional is just one way to achieve these +4. State what's actually needed (abstract) +5. Position your method as satisfying these needs +``` + +### Unification Template: +``` +[Observation] Methods A and B seem different. +[Insight] But they share core principle X. +[Demonstration] Show A = X + implementation Y +[Demonstration] Show B = X + implementation Z +[Generalization] Both are special cases of framework F +[Benefit] This understanding enables [new possibilities] +``` + +### Conclusion Template: +``` +Our results suggest [new opportunity]. +Unlike common usage that [old approach], we show [new insight]. +This reveals [uncharted realm]. +We hope this will motivate [future direction]. +``` + +--- + +## 15. Common Pitfalls to Avoid + +### ❌ Don't: +- Dismiss prior work as "wrong" - it led to the belief for good reasons +- Overstate your case - "completely eliminates" vs "reduces the need for" +- Ignore implementation realities - VQ was solving real problems +- Be vague about alternatives - show concrete alternatives +- Forget to cite the methods you're challenging + +### ✅ Do: +- Acknowledge why the conventional wisdom emerged +- Show limitations respectfully +- Provide rigorous comparisons +- Admit when your method doesn't help +- Credit relevant work extensively +- Use precise language ("can be" not "is always") + +--- + +## 16. Citation Integration Examples + +### How MAR Cites Challenges: + +**Citing the method you're improving:** +```latex +Discrete tokenizers are difficult to train [51, 41, 40, 27]. +``` +- Multiple citations show it's a known problem +- Recent citations ([27]) show it's still active + +**Citing contemporaries:** +```latex +Related to our work, the recent work on GIVT [48] also focuses on +continuous-valued tokens in sequence models. + +GIVT and our work both reveal the significance and potential of this +direction. + +In GIVT, [limitation]. In contrast, our method [advantage]. +``` +- "Related to our work" - generous positioning +- "both reveal" - shared progress +- "In contrast" - clear differentiation + +**Citing the technique you adapt:** +```latex +We can adopt the bidirectional attention implementation as done in +Masked Autoencoder (MAE) [21]. + +See Figure 2(b). +``` +- "as done in" - giving credit +- Reference to figure for transparency + +--- + +## 17. Revision Checklist for "Rethinking" Papers + +**Before Submission, Verify:** + +- [ ] Abstract clearly states the challenged assumption +- [ ] Introduction asks an explicit question ("Is it necessary...") +- [ ] "Rethinking" section dissects the status quo rigorously +- [ ] Unification section shows mathematical/conceptual equivalence +- [ ] Experiments directly test the rethinking claims +- [ ] Flexibility/robustness experiments included +- [ ] Limitations acknowledge remaining challenges +- [ ] Related work cites challenged methods generously +- [ ] Conclusion frames work as opening new territory +- [ ] Figures make conceptual contributions visible +- [ ] Tone is respectful but firm +- [ ] Claims are measured ("can be", "suggests", not "proves") +- [ ] Vision statement inspires without overpromising + +--- + +## 18. Example: Applying This Pattern + +### Original Idea (Not "Rethinking"): +"We propose a new tokenizer that improves image quality by 10%." + +### Rethinking Version: +"Conventional wisdom holds that autoregressive models require +vector-quantized tokenizers. We observe that VQ introduces artifacts and +training difficulties. Is quantization necessary for autoregressive +image generation? We propose Diffusion Loss that enables continuous-valued +tokens, eliminating VQ while achieving SOTA results. We hope this will +motivate continuous-valued modeling in other domains." + +**The Rethinking Frame:** +- Assumption: AR ↔ VQ (challenged) +- Insight: What's needed is distribution modeling, not discreteness +- Alternative: Diffusion Loss for continuous tokens +- Benefit: Eliminates VQ artifacts + enables better tokenizers +- Vision: Continuous-valued AR beyond images + +--- + +## Paper Metadata + +**Title**: Autoregressive Image Generation without Vector Quantization + +**Authors**: Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, Kaiming He + +**Venue**: NeurIPS 2024 (Spotlight) + +**arXiv**: 2406.11838 + +**Code**: https://github.com/LTH14/mar + +**Key Citations**: +- VQ-VAE [51]: Original VQ paper +- MaskGIT [4], MAGE [29]: Masked generative models +- MAE [21]: Bidirectional attention implementation +- DiT [37]: Speed/accuracy comparison +- LDM [42]: Tokenizer source + +--- + +## Extracted by + +**Date**: 2025-01-26 + +**Source**: Full PDF analysis of arXiv:2406.11838v3 + +**Extraction Focus**: Writing patterns for challenging conventional wisdom, +unifying methods, and structuring "rethinking" papers + +**For Integration**: ml-paper-writing skill knowledge base diff --git a/.agents/skills/ml-paper-writing/references/knowledge/review-response.md b/.agents/skills/ml-paper-writing/references/knowledge/review-response.md new file mode 100644 index 0000000..e6e5748 --- /dev/null +++ b/.agents/skills/ml-paper-writing/references/knowledge/review-response.md @@ -0,0 +1,429 @@ +# Review Response and Rebuttal Strategies + +This file contains effective strategies for responding to reviewer comments and addressing reviewer concerns, extracted from successful ML conference paper rebuttals. + +--- + +## General Rebuttal Principles + +### Core Philosophy +**Source:** Analysis of successful NeurIPS/ICML rebuttals + +**Key Principles:** +1. **Respectful tone**: Thank reviewers for their time +2. **Direct addressing**: Respond point-by-point to each concern +3. **Evidence-based**: Support claims with data, experiments, or citations +4. **Concise communication**: Be clear but brief +5. **No over-committing**: Only promise what can be done + +### Response Structure + +**Template:** +```markdown +# Response to Reviewer [Number] + +Thank you for this insightful comment. We [address the concern]. + +[Specific response to concern]. + +[Additional evidence/experiments if needed]. + +We have revised the manuscript to clarify this point (see changes marked in blue). +``` + +--- + +## Addressing Specific Concerns + +### Concern: Clarity Issues + +**Strategy:** +- Acknowledge the confusion +- Clarify with revised text +- Add examples if helpful + +**Template:** +```markdown +# Response to Clarity Concern + +We apologize for the confusion. The original text was: + +[Original unclear text] + +We have revised this to: +"Revised text with clearer explanation" + +We also added an example (Figure X) to illustrate this concept. +``` + +**Real Example:** +- **Concern:** "The algorithm description is unclear." +- **Response:** "We've rewritten Algorithm 1 with more detailed steps and added pseudocode. We also included a concrete example in Appendix B to illustrate the algorithm's execution." + +### Concern: Missing Experiments + +**Strategy:** +- Assess whether experiment is feasible +- If yes: add experiment and report results +- If not: explain why experiment is not essential +- Offer alternative evidence if possible + +**Template:** +```markdown +# Response to Missing Experiment Request + +We agree that [experiment] would strengthen the evaluation. We have: +[Option 1: Added experiment and results] +OR +[Option 2: Explained why not essential with alternative evidence] + +We believe this addresses the concern while maintaining focus on our core contribution. +``` + +**Real Example:** +- **Concern:** "Add comparison with Method X on dataset Y." +- **Response:** "We've added results on dataset Y (Table 3). Our method outperforms Method X by 5%. We also include ablation showing our improvement comes from [feature], not just better optimization." + +### Concern: Statistical Significance + +**Strategy:** +- Add statistical tests if appropriate +- Report confidence intervals +- Discuss practical significance vs statistical significance +- Note sample size limitations + +**Template:** +```markdown +# Response to Statistical Significance + +We agree statistical testing is important. We have: +- Added paired t-test results showing significance (p<0.01) +- Included 95% confidence intervals in Figure 3 +- Reported standard deviations across 5 runs +- Noted that while some differences are not statistically significant due to sample size, they are practically meaningful for [application] + +We have updated Section 4.2 with these statistical details. +``` + +### Concern: Insufficient Baselines + +**Strategy:** +- Add missing baselines if available +- Explain why certain baselines are inappropriate +- Cite reasons for exclusions with references + +**Template:** +```markdown +# Response to Baseline Concern + +We have added comparisons with: +- [Method A]: Added in Table 2 +- [Method B]: Excluded because [reason with citation] + +For Method B, while it seems related, it [specific reason why not comparable], making direct comparison inappropriate. +``` + +### Concern: Writing Quality + +**Strategy:** +- Revise problematic text +- Fix grammatical issues +- Improve flow and clarity +- Add signposting + +**Template:** +```markdown +# Response to Writing Concern + +We've revised the writing to address your concerns: +- Restructured Section 3 for better flow +- Fixed typos and grammar +- Added transition sentences between paragraphs +- Clarified technical terminology + +The revised manuscript has been proofread and edited for clarity. +``` + +### Concern: Overclaiming + +**Strategy:** +- Tone down absolute statements +- Add qualifications where appropriate +- Acknowledge limitations more explicitly +- Reframe claims to match evidence + +**Template:** +```markdown +# Response to Overclaiming Concern + +We accept that our original claim was too strong. We have revised the text: + +Original: "Our method achieves state-of-the-art on all tasks." +Revised: "Our method achieves state-of-the-art on [specific tasks] and competitive performance on [other tasks]." + +We also added a Limitations section acknowledging that our method may not generalize to [condition]. +``` + +--- + +## Tone and Phrasing Patterns + +### Opening Statements + +**Thanking:** +- "Thank you for this insightful comment." +- "We appreciate the reviewer's suggestion to..." +- "We thank the reviewer for pointing this out." + +**Acknowledging Valid Points:** +- "The reviewer is right that..." +- "We agree this is a limitation." +- "This is an excellent suggestion." + +### Addressing Disagreements + +**Respectful Disagreement:** +- "We respectfully disagree with this assessment based on..." +- "While we understand the concern, our results suggest..." +- "We believe our approach is justified because..." + +**Providing Evidence:** +- "Our experimental results (Table 3) show..." +- "As shown in Figure 4, the difference is..." +- "This is supported by prior work [Citation]." + +### Making Commitments + +**Full Commitments:** +- "We will add this experiment in the revised version." +- "We have added additional ablation studies in Section 5." + +**Partial Commitments:** +- "We have added clarification in the appendix due to space constraints." +- "We've expanded discussion of this point in the revision." + +**Declining Requests:** +- "Unfortunately, due to [constraint], we cannot add this experiment." +- "This would require substantial additional resources beyond our current scope." +- "We believe this is beyond the scope of the current paper but note it as future work." + +--- + +## Common Rebuttal Strategies + +### Strategy: Organized Response + +**Structure:** +```markdown +# Summary of Changes + +We thank the reviewers for their constructive feedback. In this response, we: +- [Major change 1] +- [Major change 2] +- [Improvement 3] + +We believe these changes have significantly strengthened the paper. + +# Response to Reviewer 1 + +[Point-by-point responses] + +# Response to Reviewer 2 + +[Point-by-point responses] +``` + +### Strategy: Evidence-Based Arguments + +**Template:** +```markdown +# Response to Technical Concern + +Our approach is valid because: +1. [Reason 1 with reference/evidence] +2. [Reason 2 with data/figure] +3. [Reason 3 with theoretical justification] + +This is supported by [Citation], which demonstrates that [fact]. +``` + +### Strategy: Highlighting Improvements + +**Template:** +```markdown +# Major Revisions + +1. **New Experiments**: Added comparison with [method] on [dataset] +2. **New Analysis**: Included ablation study in Table 4 +3. **Clarified Writing**: Rewrote Section 3 for clarity +4. **Added Limitations**: New section 5.2 acknowledging constraints + +These additions strengthen our core claims about [contribution]. +``` + +--- + +## Venue-Specific Considerations + +### NeurIPS + +**Emphasis:** +- Novelty and conceptual contribution +- Broader impact (lay summary) +- Reproducibility checklist + +**Rebuttal Focus:** +- How work advances understanding +- Significance of contribution +- Ethical considerations + +### ICML + +**Emphasis:** +- Methodological rigor +- Theoretical contributions +- Broader impact statement + +**Rebuttal Focus:** +- Soundness of methods +- Theoretical guarantees +- Practical implications + +### ICLR + +**Emphasis:** +- Experimental thoroughness +- Limitations acknowledgment +- LLM usage disclosure + +**Rebuttal Focus:** +- Comprehensive evaluation +- Honest limitation discussion +- Transparency about methods + +### ACL + +**Emphasis:** +- Linguistic appropriateness +- Ethical considerations +- Clear limitations + +**Rebuttal Focus:** +- Language quality and appropriateness +- Data provenance and ethics +- Practical utility + +--- + +## Tips for Successful Rebuttals + +### Before Writing + +1. **Understand the concerns**: Read carefully, identify key issues +2. **Prioritize**: Address major concerns first +3. **Be realistic**: Only promise what can deliver +4. **Gather evidence**: Collect data, results, citations +5. **Coordinate**: Discuss with co-authors if applicable + +### While Writing + +1. **Be specific**: Reference exact sections, figures, tables +2. **Be concise**: Keep responses focused and brief +3. **Be respectful**: Thank reviewers, acknowledge good points +4. **Be confident**: Defend your work appropriately +5. **Be honest**: Acknowledge limitations, don't overpromise + +### Common Mistakes to Avoid + +- **Defensive tone**: Don't argue excessively +- **Vague responses**: Be specific about changes +- **Ignoring concerns**: Address every point +- **Over-promising**: Only commit to feasible additions +- **Disorganized:** +- **Poor formatting:** Use clear sections and structure +- **Rude language:** Maintain professional tone + +--- + +## Rebuttal Examples + +### Example 1: Clarity Concern + +**Reviewer:** "The method description in Section 3 is unclear and hard to follow." + +**Response:** +```markdown +We apologize for the confusion. We have rewritten Section 3.2 to clarify our algorithm: + +**Original:** "We process the data using our method and get results." + +**Revised:** "Our method consists of three stages: (1) We first normalize the input +features using [technique]. (2) We then apply our core algorithm, which iteratively [process]. +(3) Finally, we post-process the outputs using [method]." + +We also added Algorithm 1 with detailed steps and included a concrete example in +Appendix A. We believe this revision makes the method reproducible and clear. +``` + +### Example 2: Missing Baseline + +**Reviewer:** "You should compare with Method X (Smith et al., 2022)." + +**Response:** +```markdown +Thank you for this suggestion. We have added comparisons with Method X in our +revised manuscript: + +**Results in Table 3:** Our method achieves 92% accuracy compared to Method X's +85% on dataset Y. This 7% improvement demonstrates the value of our [key innovation]. + +**Ablation Study:** We show in Table 4 that our improvement comes specifically from +[feature], not just better optimization. + +We chose not to include Method Z because [reason with citation]. +``` + +### Example 3: Overclaiming + +**Reviewer:** "The abstract claims 'state-of-the-art' too broadly." + +**Response:** +```markdown +We accept this critique. Our original claim was too broad. We have revised the +abstract: + +**Original:** "Our method achieves state-of-the-art performance across all tasks." + +**Revised:** "Our method achieves state-of-the-art on [specific tasks A and B] (Table 1) +and competitive performance on [other tasks C and D] (Table 2)." + +We also added a Limitations section (Section 5) noting that performance may vary +across domains and tasks. +``` + +--- + +## Final Checklist + +Before submitting rebuttal: + +- [ ] All reviewer concerns addressed +- [ ] Responses are clear and specific +- [ ] Tone is respectful and professional +- [ ] Changes are marked in manuscript +- [ ] Evidence provided for claims +- [ ] Feasible commitments made +- [ ] Co-authors agree with responses +- [ ] Proofread for errors +- [ ] Check formatting requirements + +--- + +## Notes + +- **Learn from successful rebuttals**: Read well-received papers' reviewer exchanges +- **Practice humility**: Acknowledge mistakes, show willingness to improve +- **Focus on core contribution**: Defend your main contribution without overclaiming +- **Keep it concise**: Reviewers are busy; be respectful of their time + +**Updates:** This file is periodically updated with new strategies and examples from successful rebuttals. diff --git a/.agents/skills/ml-paper-writing/references/knowledge/structure.md b/.agents/skills/ml-paper-writing/references/knowledge/structure.md new file mode 100644 index 0000000..9c77d88 --- /dev/null +++ b/.agents/skills/ml-paper-writing/references/knowledge/structure.md @@ -0,0 +1,349 @@ +# Paper Structure Patterns + +This file contains actionable patterns for organizing ML conference papers, extracted from successful publications. + +--- + +## Introduction Patterns + +### Pattern: Contribution Statement Structure +**Source:** "Attention Is All You Need", NeurIPS (2017) +**Context:** Introducing the main contribution + +**Pattern:** +1. Start with broader context or problem +2. Narrow down to specific limitation +3. Present your approach as solution +4. State clear contribution upfront + +**Example Template:** +```markdown +[Context/Problem]: Existing approaches struggle with [limitation] due to [reason]. + +[Our Approach]: We propose [method name], which [key innovation]. + +[Contribution]: This achieves [result] and enables [capability]. +``` + +**Application:** Use this pattern when introducing your main contribution in the first or second paragraph of the introduction. + +--- + +### Pattern: Bulleted Contribution List +**Source:** "BERT: Pre-training of Deep Bidirectional Transformers", NAACL (2019) +**Context:** Summarizing contributions for clarity + +**Pattern:** +- Place near end of Introduction (after Related Work) +- Use 2-4 bullets +- Each bullet: 1-2 lines max (in two-column format) +- Start with strong verbs ("We propose", "We demonstrate", "We show") + +**Example Template:** +```markdown +Our contributions are three-fold: +- We propose [method], which achieves [result]. +- We demonstrate that [technique] improves [metric]. +- We show that [approach] enables [new capability]. +``` + +**Application:** Use this when you need to clearly delineate multiple contributions for reviewers. + +--- + +### Pattern: Related Work Organization +**Source:** "Attention Is All You Need", NeurIPS (2017) +**Context:** Structuring literature review + +**Pattern:** +- Organize methodologically, not chronologically +- Group papers by approach/assumption +- Contrast your approach with each group +- Use "One line of work uses X whereas we use Y because..." + +**Example Template:** +```markdown +[Approach Category]: Several approaches use [assumption A] [refs]. +[Contrast]: We adopt [assumption B] because it allows [benefit]. + +[Alternative Category]: Other methods focus on [aspect C] [refs]. +[Positioning]: We build on this by adding [our innovation]. +``` + +**Application:** Use this to position your work relative to existing literature without paper-by-paper reviews. + +--- + +## Methods Section Patterns + +### Pattern: Algorithm Presentation +**Source:** "Adam: A Method for Stochastic Optimization", ICLR (2015) +**Context:** Describing algorithms clearly + +**Pattern:** +1. High-level overview first +2. Mathematical formulation +3. Algorithm pseudocode (if complex) +4. Implementation details + +**Example Template:** +```markdown +[Overview]: We formulate [problem] as optimization. Let [objective] be our goal. + +[Method]: Our approach optimizes [objective] using [technique]. +Specifically, we [algorithm description]. + +[Algorithm]: The full procedure is shown in Algorithm 1. + +[Implementation]: In practice, we [practical details]. +``` + +**Application:** Use this when presenting novel algorithms or optimization methods. + +--- + +### Pattern: Component Breakdown +**Source:** "BERT: Pre-training of Deep Bidirectional Transformers", NAACL (2019) +**Context:** Describing multi-component systems + +**Pattern:** +- Present model architecture first +- Break down into key components +- Explain each component's role +- Show how components interact + +**Example Template:** +```markdown +[Architecture]: Our model consists of [N components]: [list]. + +[Component 1]: The [component] module [function]. +[Component 2]: The [component] layer [operation]. + +[Integration]: These components are stacked sequentially, with [connection pattern]. +``` + +**Application:** Use this when describing complex architectures with multiple interacting parts. + +--- + +## Results Section Patterns + +### Pattern: Quantitative Opening +**Source:** "BERT: Pre-training of Deep Bidirectional Transformers", NAACL (2019) +**Context:** Presenting main findings + +**Pattern:** +- Start with strongest quantitative result +- Use exact numbers and metrics +- Include comparison to baselines +- State statistical significance + +**Example Template:** +```markdown +[Main Result]: Our method achieves [score] on [dataset], improving +over the previous best of [baseline] by [margin] (p<0.001). + +[Comparison]: Compared to baselines: +- [Method A]: [score] +- [Method B]: [score] +- Ours: [score] + +[Significance]: Results are averaged over N runs; standard deviations shown in parentheses. +``` + +**Application:** Use this to open your Results section with your strongest finding. + +--- + +### Pattern: Table Integration +**Source:** "Attention Is All All You Need", NeurIPS (2017) +**Context:** Presenting results in tables + +**Pattern:** +- Bold best results in each column +- Include direction indicators (↑↓) +- Provide table caption that stands alone +- Reference table in text before presenting + +**Example Template:** +```markdown +Table 1 shows our method's performance. Our model (bold) outperforms +all baselines across datasets. + +[Table content] + +As shown in Table 1, we achieve state-of-the-art on [datasets]. +``` + +**Application:** Use this when presenting comparative results in table format. + +--- + +## Discussion Section Patterns + +### Pattern: Limitations First +**Source:** "Attention Is All You Need", NeurIPS (2017) +**Context:** Acknowledging limitations proactively + +**Pattern:** +- State limitations clearly in first paragraph +- Explain why limitations don't undermine core claims +- Distinguish between limitations and future work + +**Example Template:** +```markdown +[Limitation Statement]: Our approach has [limitation]. Specifically, +[constraint]. + +[Mitigation]: Despite this, our core findings about [main contribution] remain +valid because [reason]. + +[Future Work]: Addressing this limitation is an important direction for +future research. +``` + +**Application:** Use this to acknowledge limitations honestly while maintaining paper strength. + +--- + +### Pattern: Broader Impact Framing +**Source:** "Language Models are Few-Shot Learners", GPT-3 Paper (2020) +**Context:** Discussing wider implications + +**Pattern:** +- Start with direct implications +- Expand to related domains +- Consider societal impact (if appropriate) +- End with forward-looking statement + +**Example Template:** +```markdown +[Direct Impact]: Our findings suggest that [implication for domain]. + +[Broader Implications]: Beyond [specific domain], this approach could +enable [application in other areas]. + +[Future Outlook]: As [trend] continues, methods like ours will become +increasingly important for [reason]. +``` + +**Application:** Use this when writing the final paragraphs of Discussion or Conclusion. + +--- + +## Transition Patterns + +### Pattern: Section Transitions +**Source:** "Attention Is All You Need", NeurIPS (2017) +**Context:** Moving between sections + +**Pattern:** +- Introduction → Methods: "We now describe our approach." +- Methods → Results: "We evaluate our method on [tasks]." +- Results → Discussion: "These results suggest that [insight]." + +**Example Template:** +```markdown +[Transition to Methods]: Having established [motivation], we present +our method. + +[Transition to Results]: To validate our approach, we conduct experiments +on [datasets]. + +[Transition to Discussion]: The experimental results reveal several insights +about [phenomenon], which we discuss next. +``` + +**Application:** Use these to create smooth transitions between major sections. + +--- + +## Notes + +- **Consistency**: Maintain consistent terminology throughout the paper +- **Flow**: Each section should logically lead to the next +- **Clarity**: Make structure explicit with signposting +- **Audience**: Write for tired reviewers - make their job easy + + + +## 何凯明（Kaiming He）的论文结构模式 + +> 来源: 分析了何凯明的 19 篇代表性论文 +> 添加时间: {datetime.now().strftime('%Y-%m-%d')} + +### 摘要结构模式 + +何凯明在摘要中常用的开场模式： + +**模式 1: 直接陈述贡献** +``` +We introduce [method name], a [key feature] framework for [task]. +We show that [method] achieves [result] on [dataset]. +``` + +**模式 2: 问题-解决方案** +``` +[Problem] is difficult for [task]. We present [solution] +that addresses this by [key mechanism]. +``` + +**示例** (来自 ResNet): +``` +Deeper neural networks are more difficult to train. We present a +residual learning framework to ease the training of networks that +are substantially deeper than those used previously. +``` + +### 引言结构模式 + +**三段式引言:** +1. **问题陈述** (2-3段) - 描述挑战和现有方法 +2. **方法概述** (1-2段) - 简洁介绍解决方案 +3. **主要贡献** (1段) - 列表形式，每条 1-2 行 + +**贡献列表模式:** +```markdown +- 我们提出了 [方法]，解决了 [问题] +- 我们展示了 [方法] 在 [数据集] 上的 [性能提升] +- 我们证明了 [原理] 是有效的 +``` + +### 方法部分结构 + +何凯明的方法部分通常包含： + +1. **符号定义** - 清晰定义所有变量和符号 +2. **问题形式化** - 数学公式表达 +3. **方法描述** - 逐步算法解释 +4. **实现细节** - 网络架构、训练设置 + +**常用句式:** +- "Let us consider [变量] as [定义]" +- "Formally, we define [公式]" +- "We hypothesize that [假设]" +- "To the extreme, [极端情况]" + +### 实验部分结构 + +1. **实验设置** - 数据集、评价指标、实现细节 +2. **主要结果** - 核心性能对比 +3. **消融实验** - 组件分析 +4. **可视化分析** - 图表展示 + +**结果描述模式:** +- "Table X shows that [结果]" +- "Fig. Y illustrates that [观察]" +- "Our method achieves [指标] on [任务]" +- "This represents a [X]% improvement over baseline" + +### 相关工作部分组织 + +何凯明倾向于**主题式组织**而非时间顺序： + +**好的组织方式:** +- "One line of work uses [方法A] [引用], whereas we use [方法B]" +- "[方法A] [引用] assumes [假设], but we show [反驳]" + +**避免:** +- "X et al. introduced [方法]. Y et al. improved [方法]" diff --git a/.agents/skills/ml-paper-writing/references/knowledge/submission-guides.md b/.agents/skills/ml-paper-writing/references/knowledge/submission-guides.md new file mode 100644 index 0000000..993df52 --- /dev/null +++ b/.agents/skills/ml-paper-writing/references/knowledge/submission-guides.md @@ -0,0 +1,326 @@ +# Submission Guides and Venue Requirements + +This file contains venue-specific submission requirements, formatting guidelines, and checklist items extracted from ML conference and journal publications. + +--- + +## NeurIPS Requirements + +### Format Requirements +**Source:** NeurIPS 2025 Conference Guidelines + +**Page Limits:** +- Main paper: 9 pages (excluding references) +- References: Unlimited (don't count toward page limit) +- Appendices: Allowed but reviewers not required to read + +**Required Sections:** +- Abstract: 150-250 words +- Introduction: Must clearly state contribution +- Methods: Sufficient for reproduction +- Experiments: Comprehensive evaluation +- Discussion: Interpret results and limitations +- References: Complete citations +- **Checklist**: Mandatory submission checklist (16 items) +- **Lay Summary**: Required for accepted papers (1 page, non-technical) + +**Formatting:** +- Double-blind review (anonymize submissions) +- LaTeX template required +- 9pt font, two-column format +- Margins: 1 inch on all sides + +**Submission Checklist Items:** +1. Do the main claims of the paper match the checklist? +2. Have you checked the checklist for missing items? +3. Is the paper anonymized? +4. Are all figures/tables clear and readable? +5. Have you included code and data availability statements? +6. Are all equations properly formatted? +7. Is the abstract within word limit? +8. Are all citations complete and correct? +9. Have you addressed potential ethical concerns? +10. Are experimental settings clearly described? +11. Is statistical significance properly reported? +12. Have you included limitations? +13. Is the broader impact discussed (if required)? +14. Are all figures referenced in text? +15. Is the supplementary material well-organized? +16. Have you proofread for typos and grammar? + +--- + +## ICML Requirements + +### Format Requirements +**Source:** ICML 2026 Conference Guidelines + +**Page Limits:** +- Main paper: 8 pages +- Camera-ready: +1 page (9 total) +- References: Unlimited (don't count toward page limit) + +**Required Sections:** +- Abstract: Clear summary of contribution +- Introduction: Problem and contribution +- Methods: Complete description +- Experiments: Thorough evaluation +- **Broader Impact Statement**: Required (after conclusion) +- References: Complete citations + +**Formatting:** +- Double-blind review (anonymize submissions) +- LaTeX template required +- Two-column format +- Margins as per template + +**Broader Impact Statement:** +- Discuss positive and negative societal impacts +- Consider biases, fairness, environmental impact +- 1 page maximum +- Required for all submissions + +--- + +## ICLR Requirements + +### Format Requirements +**Source:** ICLR 2026 Conference Guidelines + +**Page Limits:** +- Main paper: 9 pages +- Camera-ready: +1 page (10 total) +- References: Unlimited (don't count toward page limit) + +**Required Sections:** +- Abstract: Summary of contribution +- Introduction: Clear problem statement +- Methods: Reproducible description +- Experiments: Comprehensive evaluation +- **Limitations Section**: MANDATORY +- **LLM Disclosure**: Required if using LLMs +- References: Complete citations + +**Formatting:** +- Double-blind review (anonymize submissions) +- LaTeX template required +- Two-column format + +**LLM Disclosure Requirements:** +- Describe LLM use in the paper +- Include model details (architecture, training data, compute) +- Acknowledge LLM limitations +- Note any automated text generation + +--- + +## ACL Requirements + +### Format Requirements +**Source:** ACL 2025 Conference Guidelines + +**Page Limits:** +- Main paper: 8 pages (long papers) +- Short papers: 4 pages +- References: Unlimited (don't count toward page limit) + +**Required Sections:** +- Abstract: 150-200 words +- Introduction: Background and contribution +- Methods: Technical description +- Experiments: Evaluation +- **Limitations Section**: MANDATORY +- **Ethics Statement**: Required if applicable +- References: Complete citations + +**Formatting:** +- Double-blind review (anonymize submissions) +- LaTeX template required (ACL style files) +- Two-column format + +**Ethics Considerations:** +- Human subjects research: IRB approval required +- Data privacy: Anonymization and consent +- Environmental impact: Compute resource usage + +--- + +## AAAI Requirements + +### Format Requirements +**Source:** AAAI 2026 Conference Guidelines + +**Page Limits:** +- Main paper: 7 pages +- Camera-ready: +1 page (8 total) +- References: Unlimited (don't count toward page limit) + +**Required Sections:** +- Abstract: 150-250 words +- Introduction: Problem and contribution +- Methods: Technical description +- Experiments: Evaluation +- References: Complete citations + +**Formatting:** +- Double-blind review (anonymize submissions) +- **Strict style file adherence**: Must use official template +- Two-column format +- No modifications to style files + +**Strict Requirements:** +- Follow AAAI template exactly +- No custom formatting beyond template +- Font sizes and margins as specified +- Page limits strictly enforced + +--- + +## COLM Requirements + +### Format Requirements +**Source:** COLM 2025 Conference Guidelines + +**Page Limits:** +- Main paper: 9 pages +- Camera-ready: +1 page (10 total) +- References: Unlimited (don't count toward page limit) + +**Required Sections:** +- Abstract: Summary of contribution +- Introduction: Problem and contribution +- Methods: Technical description +- Experiments: Evaluation +- **Focus**: Language models specifically +- References: Complete citations + +**Formatting:** +- Double-blind review (anonymize submissions) +- LaTeX template required +- Two-column format + +**Language Model Focus:** +- Papers should address language model challenges +- Method contributions applicable to LM community +- Experimental setup relevant to language tasks + +--- + +## Common Submission Requirements + +### Double-Blind Review +**Applies to:** NeurIPS, ICML, ICLR, ACL, AAAI, COLM + +**Requirements:** +- Remove author names and affiliations +- Anonymize citations to own work (use [Anonymous, 2024]) +- Remove acknowledgments that reveal identity +- Avoid distinctive phrases that identify authors +- Supplementary materials must also be anonymized + +**Common Mistakes:** +- Forgetting to anonymize GitHub links +- Including author names in file paths +- Thanking specific colleagues in acknowledgments +- Citing own work with author names + +### Code and Data Availability + +**Increasingly Required:** + +**NeurIPS:** Encourages code/data availability statements +**ICML:** Encourages reproducibility +**ICLR:** Recommends code sharing + +**Best Practices:** +- Include code repository link (anonymized if under review) +- Provide data access instructions +- Describe hyperparameters and settings +- Note any proprietary constraints + +### Supplementary Materials + +**General Guidelines:** +- Appendices allowed but not required reading +- Use for additional experiments, proofs, tables +- Keep main paper self-contained +- Reference supplementary in main text + +**Formatting:** +- Same style as main paper +- Clear section numbering (S1, S2, etc.) +- Include in submission PDF or as separate file + +--- + +## Citation Styles + +### Common Styles in ML + +**IEEE Style (Numbered):** +``` +[1] J. Doe, "Paper title," Conference Name, Year. +[2] A. Smith, "Another paper," Journal Name, vol. 10, pp. 1-15, 2020. +``` + +**ACM Style (Numbered):** +``` +[J. Doe and A. Smith, "Paper title," Conference Name, Year. +[A. Smith and B. Jones, "Another paper," Journal Name, 2020. +``` + +**Author-Year (APA-like):** +``` +Doe (2020) J. Doe. Paper title. Conference Name. +Smith (2019) A. Smith. Another paper. Journal Name. +``` + +### Reference Management + +**Best Practices:** +- Use consistent style throughout +- Include DOIs when available +- Provide arXiv links for preprints +- Check for broken links +- Verify all citations before submission + +--- + +## Checklists and Templates + +### Pre-Submission Checklist + +**Content:** +- [ ] Novel contribution clearly stated +- [ ] Related work comprehensive +- [ ] Methods reproducible +- [ ] Results support all claims +- [ ] Limitations acknowledged +- [ ] Broader impact addressed (if required) +- [ ] Ethics compliance verified + +**Formatting:** +- [ ] Page limits respected +- [ ] Style file followed exactly +- [ ] References complete and consistent +- [ ] Figures/tables clear and readable +- [ ] Equations numbered and referenced +- [ ] Supplementary material organized + +**Anonymity:** +- [ ] Author names removed +- [ ] Acknowledgements anonymized +- [ ] Self-citations anonymized +- [ ] GitHub links anonymized +- [ ] Identifying information removed + +--- + +## Notes + +- **Always verify**: Requirements change between years - always check current conference guidelines +- **Plan ahead**: Some venues have strict formatting - start early +- **Read examples**: Look at well-received papers from previous years +- **Ask for help**: If unsure about a requirement, consult program chairs or experienced colleagues + +**Updates:** This file is regularly updated as new conference requirements become available. diff --git a/.agents/skills/ml-paper-writing/references/knowledge/theory-driven-papers-kaiming-he.md b/.agents/skills/ml-paper-writing/references/knowledge/theory-driven-papers-kaiming-he.md new file mode 100644 index 0000000..8c0a0e3 --- /dev/null +++ b/.agents/skills/ml-paper-writing/references/knowledge/theory-driven-papers-kaiming-he.md @@ -0,0 +1,561 @@ +# Theory-Driven Papers: From First Principles + +**Source**: Kaiming He et al., "Mean Flows for One-step Generative Modeling" (2025) + +**Paper Type**: Theory-driven / First-principles paper + +**Core Pattern**: Start from first principles → Derive theory → Build method → Demonstrate superiority + +--- + +## 1. Abstract Structure: The "Principle Introduction" Framework + +### Pattern: From Theory to Results + +**Template**: +```markdown +Abstract: +1. [Background] Established framework provides [foundation] +2. [Problem] Recent research focuses on [challenge], but existing methods have [limitation] +3. [Critique] Despite encouraging results, [specific problem with prior approaches] +4. [Core Concept] We introduce [new concept], in contrast to [old concept] +5. [Theory] Solely from definition, we derive [theoretical foundation] +6. [Advantage] This provides [principled basis] vs [heuristic approaches] +7. [Results] Achieves [strong result] - [relative improvement] over SOTA +8. [Significance] Self-contained, [independence from external components] +``` + +### MeanFlows Abstract Example (annotated): + +```latex +Flow Matching provides an intuitive and conceptually simple framework for +constructing flow paths that transport one distribution to another. + +Recent research has paid significant attention to few-step—and in +particular, one-step, feedforward—generative models. + +Despite encouraging results, the consistency constraint is imposed as a +property of the network's behavior, while the properties of the underlying +ground-truth field that should guide learning remain unknown. + +The core idea is to introduce a new ground-truth field representing the +average velocity, in contrast to the instantaneous velocity typically +modeled in Flow Matching. + +Solely originated from this definition, we derive a well-defined, intrinsic +relation between the average and instantaneous velocities, which naturally +serves as a principled basis for guiding network training. + +Our method achieves an FID of 3.43 using 1-NFE generation, significantly +outperforming previous state-of-the-art methods by a relative margin of 50% +to 70%. + +It is trained entirely from scratch, without any pre-training, distillation, +or curriculum learning. +``` + +### Key Techniques: + +1. **"Provides an intuitive and conceptually simple framework"** - Light touch introduction +2. **"Recent research has paid significant attention to..."** - Establish context +3. **"Despite encouraging results..."** - The critique pattern (acknowledge then problem) +4. **"The core idea is to introduce..."** - Clear concept statement +5. **"in contrast to"** - Conceptual differentiation +6. **"Solely originated from this definition"** - First-principles emphasis +7. **"well-defined, intrinsic relation"** - Theory keywords +8. **"naturally serves as a principled basis"** - Naturalness emphasis +9. **Relative improvement (50-70%)** - More impactful than absolute numbers +10. **Negative list** - What you DON'T need (pre-training, distillation, curriculum) + +--- + +## 2. Introduction: The "Critique-First" Framework + +### Pattern: Build Up → Identify Flaw → Propose Alternative + +**Structure**: +```markdown +1. [Background] Established field with [characteristic] +2. [Problem Shift] Research focus has moved to [new direction] +3. [Specific Problem] Existing approaches address this by [method] +4. [The Critique] Despite [acknowledgment], [fundamental problem] + - "imposed as a property of [X]" + - "[Y] remains unknown" +5. [Consequences] Consequently, [practical problems] +6. [Your Concept] We propose [alternative] with [differentiation] +7. [Theory] From [first principles], we derive [result] +8. [Advantage] This is [principled/natural/intrinsic] vs [heuristic/artificial] +9. [Results] [Quantitative result] with [qualitative advantage] +``` + +### MeanFlows Introduction Flow: + +#### Background (Light Touch) +```latex +Flow Matching provides an intuitive and conceptually simple framework +for constructing flow paths that transport one distribution to another. +``` + +**Technique**: +- "intuitive and conceptually simple" - Modest, not revolutionary +- Focus on what it IS, not how important it is + +#### Problem Shift +```latex +Closely related to diffusion models, Flow Matching focuses on the velocity +fields that guide model training. + +Both Flow Matching and diffusion models perform iterative sampling during +generation. Recent research has paid significant attention to few-step +—and in particular, one-step, feedforward—generative models. +``` + +**Technique**: +- "Closely related to" - Establish connection +- "Recent research has paid significant attention to" - Research trend +- "few-step—and in particular, one-step" - Progressive emphasis + +#### The Critique (Key Pattern) +```latex +Consistency Models [46, 43, 15, 31] achieve few-step generation by enforcing +a consistency constraint on the velocity field. + +Despite encouraging results, the consistency constraint is imposed as a +property of the network's behavior, while the properties of the underlying +ground-truth field that should guide learning remain unknown. + +Consequently, training can be unstable and requires a carefully designed +'discretization curriculum' to progressively constrain the time domain. +``` + +**Technique**: +- **"Despite encouraging results"** - Always acknowledge first +- **"imposed as a property of the network's behavior"** - It's artificial +- **"underlying ground-truth field...remain unknown"** - Missing theory +- **"Consequently"** - Show practical consequences +- **Specific problems**: "training can be unstable", "requires...curriculum" + +#### Your Concept +```latex +The core idea is to introduce a new ground-truth field representing the +average velocity, in contrast to the instantaneous velocity typically +modeled in Flow Matching. +``` + +**Technique**: +- **"The core idea is to introduce"** - Direct statement +- **"in contrast to"** - Conceptual differentiation +- **Old vs New**: "average velocity" vs "instantaneous velocity" + +#### Theory First +```latex +Average velocity is defined as the ratio of displacement to a time interval, +with displacement given by the time integral of the instantaneous velocity. + +Solely originated from this definition, we derive a well-defined, intrinsic +relation between the average and instantaneous velocities, which naturally +serves as a principled basis for guiding network training. +``` + +**Technique**: +- **"Solely originated from this definition"** - Pure derivation +- **"well-defined, intrinsic relation"** - Theory keywords +- **"naturally serves as"** - Not forced +- **"principled basis"** - Foundation + +--- + +## 3. Methods Section: The "Named Identity" Pattern + +### Pattern: Define → Derive → Name + +**Structure**: +```markdown +1. [Concept Name] Define with formal notation +2. [Motivation] Explain why we need this +3. [Derivation] Step-by-step with justifications +4. [Naming] Give it a memorable name +5. [Comparison] Contrast with prior approaches +``` + +### MeanFlows Example: + +#### Step 1: Concept Naming +```latex +Average Velocity. We define average velocity as the displacement between +two time steps t and r (obtained by integration) divided by the time interval. + +Formally, the average velocity u is: + +u(zt, r, t) ≜ 1/(t−r) ∫_r^t v(zτ, τ)dτ. (3) +``` + +**Techniques**: +- **Bold heading**: "Average Velocity." - Makes it memorable +- **Text description first**: Explain before formula +- **"Formally,"**: Signals math coming +- **≜ symbol**: "defined as" (clearer than =) + +#### Step 2: Derivation with Motivation +```latex +To have a formulation amenable to training, we rewrite Eq. (3) as: + +(t−r)u(zt, r, t) = ∫_r^t v(zτ, τ)dτ. (4) + +Now we differentiate both sides with respect to t, treating r as independent +of t. This leads to: + +d/dt(t−r)u = d/dt∫_r^t v(zτ, τ)dτ + +⇒ u + (t−r)d/dt u = v(zt, t), (5) + +where the manipulation of the left hand side employs the product rule and +the right hand side uses the fundamental theorem of calculus. +``` + +**Techniques**: +- **"To have a formulation amenable to training"** - Explain why +- **"Now we differentiate..."** - Guide reader +- **Step-by-step**: Don't skip +- **"where..."**: Explain each manipulation +- **"⇒" symbol**: Clear direction + +#### Step 3: Naming the Identity +```latex +Rearranging terms, we obtain the identity: + +u(zt, r, t) = v(zt, t) − (t−r)d/dt u(zt, r, t) (6) + +We refer to this equation as the "MeanFlow Identity", which describes the +relation between v and u. +``` + +**Techniques**: +- **"Rearranging terms, we obtain..."** - What you did +- **"We refer to this equation as the 'X Identity'"** - Brand it +- **Explain**: "which describes..." - What it does + +--- + +## 4. Comparison: Principled vs Heuristic + +### Pattern: Emphasize Theoretical Independence + +**Structure**: +```markdown +1. [Your Core] At the core of our method is [fundamental principle] +2. [Independence] This [does not depend on / is independent of] [implementation] +3. [Contrast] In contrast, prior works typically rely on [heuristic/artificial constraint] +4. [Qualitative] [Natural/principled/intrinsic] vs [imposed/empirical/heuristic] +``` + +### MeanFlows Example: + +```latex +At the core of our method is the functional relationship between two +underlying fields v and u, which naturally leads to the MeanFlow Identity +that u must satisfy (Eq. (6)). + +This identity does not depend on the introduction of neural networks. + +In contrast, prior works typically rely on extra consistency constraints, +imposed on the behavior of the neural network. +``` + +**Techniques**: +- **"At the core of our method is..."** - What matters +- **"naturally leads to"** - Not forced +- **"does not depend on"** - Independence +- **"In contrast"** - Clear pivot +- **"imposed on"** - Theirs is artificial + +### Specific Method Comparison + +```latex +Consistency Models [46, 43, 15, 31] are focused on paths anchored at the +data side: in our notations, this corresponds to fixing r≡0 for any t. + +As a result, Consistency Models are conditioned on a single time variable, +unlike ours. +``` + +**Techniques**: +- **"focused on X"** - Their scope +- **"in our notations, this corresponds to..."** - Precise mapping +- **"As a result"** - Consequence +- **"unlike ours"** - One-word differentiation + +--- + +## 5. Results: Significant Improvements with Context + +### Pattern: Relative Improvement + Independence + +**Structure**: +```markdown +1. [Absolute] We achieve [metric] on [task] +2. [Relative] This represents [X-Y%] relative improvement over [comparison] +3. [Context] Our method is [self-contained / independent] +4. [Negative List] without [list of things you don't need] +``` + +### MeanFlows Example: + +```latex +Our method achieves an FID of 3.43 using 1-NFE generation. + +This result significantly outperforms previous state-of-the-art methods in +its class by a relative margin of 50% to 70% (Fig. 1). + +In addition, our method stands as a self-contained generative model: it is +trained entirely from scratch, without any pre-training, distillation, or +curriculum learning. +``` + +**Techniques**: +- **Absolute first**: "FID of 3.43" +- **"significantly outperforms"** - Strong but not "dramatically" +- **"by a relative margin of 50% to 70%"** - Range, not single number +- **Reference to figure**: "(Fig. 1)" +- **"In addition"** - Second dimension of value +- **"self-contained"** - Independence keyword +- **"trained entirely from scratch"** - Complete independence +- **Negative list**: "without any pre-training, distillation, or curriculum learning" + +--- + +## 6. Table Design: System-Level Comparison + +### Pattern: Multiple Paradigms, Clear Highlighting + +**Structure**: +```markdown +Table X: +┌────────────────────────────────────┐ +│ Left side: Your direct competitors │ +│ (1-NFE and 2-NFE methods) │ +├────────────────────────────────────┤ +│ Right side: Other paradigms │ +│ (GANs, autoregressive, etc.) │ +├────────────────────────────────────┤ +│ **Your method** (bold, positioned) │ +└────────────────────────────────────┘ +``` + +### MeanFlows Table 2 Organization: + +```latex +Table 2: Comparison on ImageNet 256×256. + +Left: 1-NFE and 2-NFE diffusion/flow models +Right: Other generative models + +Highlighted: MeanFlow (our method) +``` + +**Key Techniques**: +1. **Split paradigm**: Direct competitors on left, others on right +2. **Fair metrics**: params, NFE, FID (same for all) +3. **Bold your method**: Visual emphasis +4. **Position strategically**: Where you look best +5. **Comprehensive**: Include all major paradigms + +--- + +## 7. Figure Design: Visual Storytelling + +### Pattern: Multi-Panel Narrative + +**MeanFlows Figure 1**: "One-step generation on ImageNet 256×256 from scratch" + +**Structure**: +- **Main panel**: Generated images (visual evidence) +- **Caption**: Detailed comparison table +- **Annotations**: FID scores of competing methods +- **Highlight**: "Our MeanFlow (MF) model achieves significantly better..." + +**Techniques**: +1. **Title tells the story**: "from scratch" - key differentiator +2. **Images + numbers**: Both visual and quantitative +3. **Competitor scores in caption**: Reader doesn't need to flip pages +4. **"significantly better"**: In the figure caption itself + +--- + +## 8. Ablation Study: Destructive Testing + +### Pattern: Prove Necessity by Breaking Things + +**Structure**: +```markdown +Table X: +┌──────────────────────────────────┐ +│ (a) Vary one design dimension │ +│ - Show effect of parameter │ +│ - Mark default in gray │ +├──────────────────────────────────┤ +│ (b) Destructive comparison │ +│ - Intentionally use WRONG values │ +│ - Show only correct works │ +└──────────────────────────────────┘ +``` + +### MeanFlows Table 1 Example: + +#### Part (a): Design Sweep +```latex +(a) Ratio of sampling r≠t +% of r≠t FID, 1-NFE +0% (= FM) 328.91 +25% 61.06 +50% 63.14 +100% 67.32 +``` + +**Techniques**: +- **Descriptive caption**: "Ratio of sampling r≠t" +- **Show failure mode**: "0% (= FM) 328.91" - pure FM fails +- **Range**: 0% to 100% of parameter +- **Default marked**: In original (not shown here) + +#### Part (b): Destructive Testing +```latex +(b) JVP computation +jvp tangent FID, 1-NFE +(v, 0, 1) 61.06 +(v, 0, 0) [wrong] 268.06 +(v, 1, 0) [wrong] 329.22 +(v, 1, 1) [wrong] 137.96 +``` + +**Techniques**: +- **"Destructive comparison"** in caption +- **"intentionally performed"** in text +- **Only first row works**: Others are wrong by design +- **Proves necessity**: "meaningful results are achieved only when..." + +--- + +## 9. Writing Style: Theory Keywords + +### Emphasis Words for Theory-Driven Papers + +**Naturalness Keywords** (use these to describe your theory): +- "naturally" - "This naturally leads to..." +- "intrinsic" - "intrinsic relation" +- "well-defined" - "well-defined problem" +- "principled" - "principled basis" +- "first principles" - "from first principles" +- "solely originated from" - "solely from definition" + +**Independence Keywords**: +- "does not depend on" - Theory independence +- "independent of" - Implementation independence +- "self-contained" - System independence +- "from scratch" - No external dependencies +- "without any X" - Negative list + +**Differentiation Keywords**: +- "in contrast to" - Conceptual contrast +- "unlike" - Direct comparison +- "typically" - "typically modeled" (their approach) +- "prior works typically rely on" - Their limitation +- "imposed as" - Artificial constraint (theirs) + +### Avoid These (Too Promotional): +- ❌ "revolutionary" - Too strong +- ❌ "breakthrough" - Let others say it +- ❌ "completely eliminates" - Too absolute +- ✅ "significantly outperforms" - Strong but measured +- ✅ "substantial improvement" - Professional + +--- + +## 10. Common Mistakes in Theory Papers + +### ❌ Don't: +- Derive without explaining motivation - Why are we doing this? +- Skip steps in derivation - Readers aren't you +- Use heuristics without admitting it - Be honest +- Overclaim - "proves optimal" vs "improves over" +- Forget to acknowledge dependencies - If you use X, say it + +### ✅ Do: +- Start from first principles explicitly +- Give each equation/dentity a memorable name +- Show "destructive" ablations to prove necessity +- Report relative improvements (more impactful) +- Use "principled" keywords consistently +- Admit what you DON'T need (negative list) + +--- + +## 11. Revision Checklist for Theory-Driven Papers + +**Before Submission, Verify:** + +- [ ] Abstract starts from established framework (not "X is important") +- [ ] Introduction has "Despite encouraging results..." critique +- [ ] Core concept has a memorable name +- [ ] Derivation is step-by-step with justifications +- [ ] Key equation is named ("X Identity") +- [ ] Theory is contrasted as "principled" vs "heuristic" +- [ ] Results include relative improvement (X-Y%) +- [ ] Self-containment is emphasized (what you don't need) +- [ ] Ablations include destructive tests +- [ ] Tables organize by paradigm, highlight your position +- [ ] Figures tell visual story with captions +- [ ] Theory keywords used consistently (principled, intrinsic, natural) + +--- + +## 12. Example: Applying This Pattern + +### Original Idea (Not Theory-Driven): +"We propose a new training method that improves FID by 20%." + +### Theory-Driven Version: +"Flow Matching provides an intuitive framework for generative modeling, +but recent one-step methods impose consistency constraints heuristically. +Despite encouraging results, the underlying ground-truth field properties +remain unknown. We introduce average velocity (in contrast to instantaneous +velocity), deriving the MeanFlow Identity solely from first principles. +This provides a principled basis for training, achieving 3.43 FID with +50-70% relative improvement. Our method is self-contained, trained from +scratch without pre-training or distillation." + +**The Theory-Driven Frame**: +- Foundation: Flow Matching (established) +- Problem: Heuristic constraints (theory gap) +- Concept: Average velocity (new) +- Theory: MeanFlow Identity (derived) +- Result: Strong + independent (no external deps) + +--- + +## Paper Metadata + +**Title**: Mean Flows for One-step Generative Modeling + +**Authors**: Kaiming He et al. + +**Year**: 2025 + +**Key Concepts**: +- Average velocity vs instantaneous velocity +- MeanFlow Identity +- Principled vs heuristic training +- Self-contained generative models + +--- + +## Extracted by + +**Date**: 2026-01-26 + +**Source**: Analysis of Mean Flows paper (16 pages) + +**Extraction Focus**: Theory-driven paper writing patterns, first-principles +derivations, principled vs heuristic positioning + +**For Integration**: ml-paper-writing skill knowledge base diff --git a/.agents/skills/ml-paper-writing/references/knowledge/writing-techniques.md b/.agents/skills/ml-paper-writing/references/knowledge/writing-techniques.md new file mode 100644 index 0000000..ae91c01 --- /dev/null +++ b/.agents/skills/ml-paper-writing/references/knowledge/writing-techniques.md @@ -0,0 +1,637 @@ +# Writing Techniques and Patterns + +This file contains actionable sentence patterns, transition phrases, and writing techniques extracted from successful ML conference papers. + +--- + +## Transition Phrases + +### Literature Review Transitions +**Source:** Various NeurIPS/ICML papers + +**Introducing Problems:** +- "However, these methods suffer from [limitation]." +- "Despite recent progress, [challenge] remains unsolved." +- "While existing approaches address [aspect], they struggle with [issue]." + +**Presenting Solutions:** +- "To address this, we propose..." +- "We overcome this limitation by..." +- "Our key insight is that..." + +**Connecting to Related Work:** +- "Building on [prior work], we extend..." +- "Unlike approaches that [method], we instead..." +- "Following the success of [paper], we apply..." + +### Methods Section Transitions +**Source:** "BERT: Pre-training of Deep Bidirectional Transformers", NAACL (2019) + +**Describing Components:** +- "Our model consists of two main components: [A] and [B]." +- "We divide our approach into [N] stages: [list]." + +**Explaining Rationale:** +- "We choose this architecture because..." +- "This formulation allows us to..." +- "Motivated by [intuition], we design..." + +### Results Section Transitions +**Source:** "Attention Is All You Need", NeurIPS (2017) + +**Presenting Findings:** +- "Our method achieves [result], outperforming baselines by [margin]." +- "As shown in Table 1, our approach..." +- "Figure 2 demonstrates that..." + +**Analyzing Results:** +- "These results suggest that [insight]." +- "Notably, we observe that..." +- "This improvement indicates that..." + +### Discussion Transitions +**Source:** "Language Models are Few-Shot Learners", GPT-3 (2020) + +**Interpreting Findings:** +- "These findings reveal that..." +- "This performance gap suggests that..." +- "The strong correlation between...indicates..." + +**Connecting to Broader Context:** +- "Beyond the specific task, our results imply..." +- "This has important implications for..." + +**Acknowledging Limitations:** +- "It is important to note that our study is limited to..." +- "While these results are promising, several questions remain..." + +--- + +## Sentence Patterns + +### Claim Presentation +**Source:** "Attention Is All You Need", NeurIPS (2017) + +**Strong Claims:** +- "We show that [approach] achieves [result]." +- "We demonstrate that [method] outperforms..." +- "We prove that [technique] converges to..." + +**Nuanced Claims:** +- "Our results suggest that [factor] contributes to..." +- "We observe that [phenomenon] emerges when..." +- "Experiments indicate that [approach] is particularly effective for..." + +### Technical Description +**Source:** "Adam: A Method for Stochastic Optimization", ICLR (2015) + +**Algorithm Description:** +- "Formally, we optimize [objective] using [method]." +- "The update rule for [parameter] is given by..." +- "We modify the standard [approach] by..." + +**Implementation Details:** +- "In practice, we implement [feature] as..." +- "For computational efficiency, we approximate..." +- "We initialize [parameters] using..." + +### Results Presentation +**Source:** "BERT: Pre-training of Deep Bidirectional Transformers", NAACL (2019) + +**Quantitative Results:** +- "Our model achieves [score] (±[std]), improving over..." +- "On [dataset], we obtain [result], compared to..." +- "We observe a [percentage]% improvement over baselines." + +**Statistical Reporting:** +- "Results are averaged over N runs with different seeds." +- "Standard deviations are shown in parentheses." +- "The improvement is statistically significant (p<0.01)." + +--- + +## Clarity Techniques + +### Active Voice Usage +**Source:** Various well-written papers + +**Passive (avoid):** +- "The model was trained using..." +- "Experiments were conducted on..." + +**Active (prefer):** +- "We trained the model using..." +- "We conducted experiments on..." + +**Guideline:** Use active voice for actions you performed. Use passive for general facts or when the actor is unclear. + +### Specificity Over Generality +**Source:** "Attention Is All You Need", NeurIPS (2017) + +**Vague (avoid):** +- "This approach improves performance." +- "The method learns good representations." + +**Specific (prefer):** +- "This approach improves accuracy by 15%." +- "The method learns representations that transfer to downstream tasks." + +**Guideline:** Be quantitative whenever possible. Use specific numbers and metrics. + +### Signposting +**Source:** "BERT: Pre-training of Deep Bidirectional Transformers", NAACL (2019) + +**Section Openings:** +- "We now describe our model architecture." +- "We evaluate on three tasks: [list]." +- "The results suggest three key insights:" + +**Internal Structure:** +- "First, we [action]. Next, we [action]. Finally, we [action]." +- "Our approach has three stages: [A], [B], and [C]." + +**Guideline:** Use explicit signposting to help tired reviewers follow your paper. + +--- + +## Common Phrase Templates + +### Opening Abstract +**Good Examples:** +- "We introduce [method], a novel approach for [task]." +- "We present [method], which achieves [result] by [mechanism]." +- "We propose [framework] to address [challenge]." + +**Avoid:** +- "In this paper, we study..." (generic) +- "Large language models have..." (overused opening) + +### Introducing Related Work +**Good Examples:** +- "Recent work has shown promise in [area] [refs]." +- "Several approaches have been proposed for [task] [refs]." +- "The standard approach to [problem] is [method] [refs]." + +### Describing Experiments +**Good Examples:** +- "We evaluate on [datasets], comparing against [baselines]." +- "We conduct ablation studies to validate [component]." +- "To verify [claim], we experiment with [variations]." + +### Presenting Results +**Good Examples:** +- "Table 1 shows that our method outperforms all baselines." +- "As shown in Figure 3, performance improves as [factor] increases." +- "Our method achieves state-of-the-art on [task/metric]." + +### Discussing Limitations +**Good Examples:** +- "Our approach has limitations: [constraint]." +- "We note that our method is currently restricted to [condition]." +- "A key limitation is [issue], which we leave for future work." + +--- + +## Writing Principles + +### From Top Papers + +**Clarity First:** +- "Make it easy for reviewers to understand your contribution." +- "Use concrete examples and specific language." +- "Avoid vague or ambiguous statements." + +**Rigorous Presentation:** +- "Provide enough detail for reproduction." +- "Include error bars and statistical tests." +- "Show negative results when relevant." + +**Storytelling:** +- "Your paper tells a story: problem → approach → solution → impact." +- "Make the narrative clear in the introduction." +- "Each section should advance the story." + +**Honesty:** +- "Acknowledge limitations explicitly." +- "Don't overclaim results." +- "Trust reviewers to appreciate honesty." + +--- + +## Notes + +- **Adapt patterns**: These templates can and should be adapted to your specific context +- **Venue matters**: Some venues prefer certain styles (check venue-specific guides) +- **Consistency**: Use consistent terminology throughout +- **Tone**: Maintain professional, objective tone +- **Length**: Keep transitions concise; don't over-explain + +**Attribution:** All patterns extracted from analyzed papers with source citations for traceability. + + +--- + +## "Surprisingly" Findings: Multi-Level Reporting Pattern + +**Source**: Kaiming He et al., "Exploring Plain Vision Transformer Backbones for Object Detection" (ViTDet, ECCV 2022), "Mean Flows" (2025) + +**Paper Type**: Design simplification, unexpected findings + +### The Three-Level "Surprisingly" Pattern + +#### Level 1: Basic Surprise (Abstract/Opening) + +**Pattern**: +```markdown +Surprisingly, we observe: (i) [simple sufficient without common practice] +and (ii) [simple sufficient without common practice] +``` + +**Example (ViTDet Abstract)**: +```latex +Surprisingly, we observe: (i) it is sufficient to build a simple feature +pyramid from a single-scale feature map (without the common FPN design) and +(ii) it is sufficient to use window attention (without shifting) aided with +very few cross-window propagation blocks. +``` + +**Key Techniques**: +- **Structured list**: Use (i) and (ii) to separate findings +- **"sufficient"**: Scientific phrasing (not "optimal") +- **"without [common practice]"**: Negative differentiation + +#### Level 2: Competitive Surprise (Introduction) + +**Pattern**: +```markdown +More surprisingly, under some circumstances, our [method] can compete +with the leading [competitors]. +``` + +**Example (ViTDet Introduction)**: +```latex +More surprisingly, under some circumstances, our plain-backbone detector, +named ViTDet, can compete with the leading hierarchical-backbone detectors +(e.g., Swin, MViT). +``` + +**Key Techniques**: +- **"More surprisingly"**: Progressive emphasis +- **"under some circumstances"**: Measured claim +- **"can compete with"**: Not "beat", competitive +- **Name competitors**: Specific (Swin, MViT) + +#### Level 3: Superiority Surprise (Results) + +**Pattern**: +```markdown +With [specific condition], our [method] can outperform the [competitors] +that use [stronger condition]. The gains are more prominent for [condition]. +``` + +**Example**: +```latex +With Masked Autoencoder (MAE) pre-training, our plain-backbone detector can +outperform the hierarchical counterparts that are pre-trained on ImageNet-1K/21K +with supervision (Figure 3). The gains are more prominent for larger model sizes. +``` + +**Key Techniques**: +- **Specific conditions compared**: MAE vs ImageNet supervised +- **"outperform"**: Stronger claim here (qualified by conditions) +- **"The gains are more prominent for..."**: Pattern observation + +--- + +### "Surprisingly" Variants + +#### "Interestingly" - Pattern Observation + Explanation + +**Pattern**: +```markdown +Interestingly, [observation]. This is in line with the observation in [paper] +that [their finding]. [Additional explanation]. +``` + +**Example (ViTDet)**: +```latex +Interestingly, performing propagation in the last 4 blocks is nearly as +good as even placement. This is in line with the observation in ViT [14] +that ViT has longer attention distance in later blocks and is more localized +in earlier ones. +``` + +**Use when**: You have literature support for your observation + +#### "Notably" - Important Detail + +**Pattern**: +```markdown +Notably, [counter-intuitive result or impressive number]. +``` + +**Examples**: +- "Notably, even embedding only the interval t−r yields reasonable results." +- "Notably, our method is self-contained and trained entirely from scratch." + +**Use when**: Emphasizing importance or counter-intuitive finding + +#### "It is worth noting that" - Caveat/Clarification + +**Pattern**: +```markdown +It is worth noting that [technical caveat or clarification]. +``` + +**Examples**: +- "It is worth noting that even when the conditional flows are designed to be straight ('rectified'), the marginal velocity field typically induces a curved trajectory." +- "It is worth noting that the 3.34× memory (49G) is estimated as if the same training implementation could be used, which is not practical and requires special memory optimization." + +**Use when**: Preventing misunderstanding or clarifying technical details + +--- + +### When to Use "Surprisingly" + +**DO use**: +- When finding genuinely contradicts common practice +- When simple solution works as well as complex one +- When you have explanation (literature, hypothesis, theory) +- With measured claims ("under some circumstances", "can compete") +- With "sufficient" not "optimal" + +**DON'T use**: +- For incremental improvements (use "additionally" instead) +- Without explanation/justification +- Overgeneralizing ("always", "proves") +- For expected results + +--- + +## Ablation Study Writing Techniques + +**Source**: Kaiming He papers (ViTDet, MeanFlows, MoCo v2) + +### Table Design: Incremental Progression + +**Pattern**: +```markdown +Table X: [Component] Ablation +┌──────────────────────────────────────────┐ +│ no [component] | AP | Δ │ +│ (a) [common variant] | AP | +X.X │ +│ (b) [another variant] | AP | +Y.Y │ +│ (c) ours: simple | AP | +Z.Z ✓ │ +└──────────────────────────────────────────┘ +``` + +**Example (ViTDet Table 1)**: +```latex +pyramid design APbox APmask +───────────────────────────────────────── +no feature pyramid 47.8 42.5 +(a) FPN, 4-stage 50.3 44.9 +(b) FPN, last-map 50.9 45.3 +(c) simple feature pyramid 51.2 45.5 +``` + +**Techniques**: +- **Baseline**: "no [X]" shows it's needed +- **(a), (b), (c)**: Progressive variations +- **Δ标注**: (+2.5) - Show incremental gains +- **Correspondence**: "The entries (a-c) correspond to Figure X (a-c)" +- **Conclusion**: "our simple pyramid is sufficient" + +--- + +### Destructive Ablation: Proving Necessity + +**Pattern**: +```markdown +We conduct a destructive comparison in which [wrong choice] is intentionally +performed. Meaningful results are achieved only when [correct choice]. +``` + +**Example (MeanFlows Table 1b)**: +```latex +In Tab. 1b, we conduct a destructive comparison in which incorrect JVP +computation is intentionally performed. + +jvp tangent FID, 1-NFE +(v, 0, 1) [correct] 61.06 +(v, 0, 0) [wrong] 268.06 +(v, 1, 0) [wrong] 329.22 +(v, 1, 1) [wrong] 137.96 + +It shows that meaningful results are achieved only when the JVP computation +is correct. +``` + +**Use when**: You need to prove a design choice is necessary (not just optional) + +--- + +### Ablation Narrative: Observation → Explanation + +**Pattern 1: Observation + Literature Support** +```latex +We observe that [observation]. This is consistent with the observation in +[paper] that [their finding]. +``` + +**Pattern 2: Observation + Hypothesis** +```latex +We hypothesize that this is because [reason 1] and also because [reason 2]. +``` + +**Pattern 3: Observation + Theory** +```latex +[Observation]. This indicates that [theoretical explanation]. +``` + +--- + +## Theory-Driven Paper Keywords + +**Source**: Kaiming He et al., "Mean Flows for One-step Generative Modeling" (2025) + +### Naturalness Keywords (use to describe your theory) + +- **"naturally"** - "This naturally leads to..." +- **"intrinsic"** - "intrinsic relation between..." +- **"well-defined"** - "well-defined problem" +- **"principled"** - "principled basis for..." +- **"first principles"** - "from first principles" +- **"solely originated from"** - "solely from definition" + +### Independence Keywords + +- **"does not depend on"** - Theory independence from implementation +- **"independent of"** - Independent of specific choices +- **"self-contained"** - System independence +- **"from scratch"** - No external dependencies +- **"without any X"** - Negative list (what you don't need) + +### Differentiation Keywords + +- **"in contrast to"** - Conceptual contrast +- **"unlike"** - Direct comparison +- **"typically"** - "typically modeled" (their approach) +- **"prior works typically rely on"** - Their limitation +- **"imposed as"** - Artificial constraint (theirs) + +### Avoid (Too Promotional) + +- ❌ "revolutionary" - Let others say it +- ❌ "breakthrough" - Overused +- ❌ "completely eliminates" - Too absolute +- ✅ "significantly outperforms" - Strong but measured +- ✅ "substantial improvement" - Professional + +--- + +## Design Simplification Paper Keywords + +**Source**: Kaiming He et al., "Exploring Plain Vision Transformer Backbones for Object Detection" (ViTDet, 2022) + +### Philosophy Keywords + +- **"minimal"** - "minimal adaptations" +- **"sufficient"** - "is sufficient to" (not "optimal") +- **"simple"** - "simple feature pyramid" +- **"plain"** - "plain backbone" +- **"decouple"** - "decouple pre-training from fine-tuning" +- **"independence"** - "independence of upstream vs downstream" + +### Direction Keywords + +- **"pursue a different direction"** - Clear positioning +- **"in contrast to"** - Differentiation +- **"abandons"** - What you give up (respectfully) +- **"enables"** - What your approach allows + +### Measured Claim Keywords + +- **"under some circumstances"** - Not always +- **"can compete with"** - Competitive, not dominant +- **"more prominent for"** - When effect is stronger +- **"is sufficient"** - Necessary, not maximal + + +--- + +## Updated: 何凯明的写作技巧 + +> 来源: 分析了何凯明的 11 篇代表性论文（扩展分析，包括 MeanFlows、ViTDet、MAR 等） +> 添加时间: 2026-01-26 + +> 扩展内容包括： +> - "Surprisingly" 发现的多层次报告模式 +> - Ablation Study 的增量式和破坏性实验设计 +> - 理论驱动型论文的关键词策略 +> - 设计简化型论文的关键词策略 + +### 句子结构偏好 + +**主动语态优先** (被动语态仅 9.3%) +何凯明偏好使用主动、直接的陈述： + +**✅ 推荐 (何凯明的风格):** +- "We present a framework for [task]" +- "Our method achieves [result]" +- "This formulation enables [benefit]" + +**❌ 避免:** +- "A framework is presented for [task]" +- "Results are achieved by our method" + +### 贡献表达方式 + +何凯明常用的贡献表达模式： + +**模式 1: 直接陈述** +``` +We propose [method] that [feature]. +We demonstrate [result] on [dataset]. +``` + +**模式 2: 对比强调** +``` +Unlike [previous work], our approach [difference]. +This leads to [improvement] in [metric]. +``` + +**模式 3: 问题-解决方案** +``` +[Challenge] remains difficult. We address this by [solution]. +``` + +### 技术术语使用 + +何凯明论文中的高频术语组合： + +| 术语类别 | 常用术语 | +|---------|---------| +| **网络架构** | deep neural networks, convolutional, residual, activation | +| **训练过程** | training, validation, optimization, convergence | +| **性能评估** | outperforms, achieves, improves, surpasses | +| **方法定位** | state-of-the-art, baseline, framework, algorithm | +| **所有权** | our method, our approach, our framework | + +### 过渡短语 + +何凯明论文中常用的过渡短语（按频率排序）： + +1. **however** - 用于对比不同观点 +2. **in addition/additionally** - 补充信息 +3. **furthermore** - 递进说明 +4. **therefore/thus** - 得出结论 +5. **specifically** - 举例说明 +6. **conversely** - 对比说明 + +### 数值结果呈现 + +何凯明在呈现数值结果时的模式： + +**精确性优先:** +``` +Our method achieves 76.4% accuracy (Table X). +This represents a 28% relative improvement. +``` + +**对比式呈现:** +``` +Compared to baseline (73.2%), our method (76.4%) improves +by 3.2 percentage points. +``` + +**强调意义:** +``` +This result won the 1st place in [competition/task]. +``` + +### 图表引用模式 + +何凯明引用图表的标准格式： + +**图表引入:** +- "Fig. X shows [现象]" +- "Table Y summarizes [结果]" +- "As shown in Fig. Z, [结论]" + +**图表描述:** +- "The solid line denotes [条件 A], the dashed line [条件 B]" +- "The blue curve shows [指标], while the red curve shows [指标]" + +### 网络架构描述 + +何凯明在描述网络架构时的特点： + +1. **表格化呈现** - 使用表格列出层配置 +2. **可视化辅助** - 配合架构图 +3. **简洁符号** - 使用清晰的数学符号 +4. **示例:** +``` +layer name | output size | configuration +conv1 | 112×112 | 7×7, 64, /2 +``` -- cgit v1.2.3