aboutsummaryrefslogtreecommitdiffstats
path: root/.agents/skills/results-report
diff options
context:
space:
mode:
authorsillylaird <sillyfanboy@gmail.com>2026-09-03 00:33:59 +0000
committersillylaird <sillyfanboy@gmail.com>2026-09-03 00:33:59 +0000
commit898b52edcb47bcb3e9d6106e74ca73e74ea01e70 (patch)
tree85c6ee5ad58b860144551184d4cf86b560c62b91 /.agents/skills/results-report
downloadwww-main.tar.gz
www-main.zip
import live www.sillylaird.ca webrootHEADmain
Diffstat (limited to '')
-rw-r--r--.agents/skills/results-report/SKILL.md180
-rw-r--r--.agents/skills/results-report/examples/example-results-report.md83
-rw-r--r--.agents/skills/results-report/references/EVIDENCE-PROPAGATION.md25
-rw-r--r--.agents/skills/results-report/references/decision-oriented-analysis.md17
-rw-r--r--.agents/skills/results-report/references/figure-interpretation.md17
-rw-r--r--.agents/skills/results-report/references/report-naming.md53
-rw-r--r--.agents/skills/results-report/references/report-structure.md63
-rw-r--r--.agents/skills/results-report/references/statistical-completeness.md19
8 files changed, 457 insertions, 0 deletions
diff --git a/.agents/skills/results-report/SKILL.md b/.agents/skills/results-report/SKILL.md
new file mode 100644
index 0000000..d533539
--- /dev/null
+++ b/.agents/skills/results-report/SKILL.md
@@ -0,0 +1,180 @@
+---
+name: results-report
+description: This skill should be used when the user asks to "write an experiment report", "summarize experimental results", "do experiment retrospection", "write a results report", "写实验总结报告", "写实验复盘", or mentions turning completed experiment artifacts into a structured, decision-oriented research report. It assumes strict analysis should come from `results-analysis` first.
+version: 0.1.0
+tags: [Research, Reporting, Experiments, Obsidian]
+---
+
+# Results Report
+
+Write the **complete post-experiment summary report** after analysis artifacts are ready.
+
+This skill is for the stage **after** `results-analysis`.
+
+## Role boundary
+
+### `results-analysis` does
+- strict statistics,
+- real figures,
+- figure interpretation scaffolding,
+- stats appendix.
+
+### `results-report` does
+- complete experiment wrap-up report,
+- decision-oriented narrative,
+- figure-by-figure interpretation inside a coherent structure,
+- limitations, failure cases, and next actions,
+- Obsidian write-back into `Results/Reports/`.
+
+Do not replace strict analysis with confident prose. If the analysis bundle is missing, first identify the blocker and request or produce the missing bundle.
+
+## Default output
+
+The default report is an **internal research report**, not manuscript prose.
+
+It should be named as:
+
+```text
+YYYY-MM-DD--{experiment-line}--r{round}--{purpose}.md
+```
+
+Example:
+- `2026-03-18--freezing--r03--transfer-summary.md`
+- `2026-03-18--contrastive-adversarial--r02--ablation-report.md`
+
+The note title should be:
+
+```text
+{Experiment Line} / Round {N} / {Purpose} / {YYYY-MM-DD}
+```
+
+Read `references/report-naming.md` before finalizing the filename or note title.
+
+## Required frontmatter
+
+```yaml
+---
+type: results-report
+date: 2026-03-18
+experiment_line: freezing
+round: 3
+purpose: transfer-summary
+status: active
+source_artifacts:
+ - analysis-output/analysis-report.md
+ - analysis-output/stats-appendix.md
+linked_experiments:
+ - Experiments/Freezing-Study.md
+linked_results:
+ - Results/Freezing-vs-Adapter.md
+---
+```
+
+## Default report structure
+
+The report must include all sections below.
+
+1. **Executive Summary**
+2. **Experiment Identity and Decision Context**
+3. **Setup and Evaluation Protocol**
+4. **Main Findings**
+5. **Statistical Validation**
+6. **Figure-by-Figure Interpretation**
+7. **Failure Cases / Negative Results / Limitations**
+8. **What Changed Our Belief**
+9. **Next Actions**
+10. **Artifact and Reproducibility Index**
+
+Read `references/report-structure.md` before writing.
+
+## Workflow
+
+### 1. Confirm the report object
+
+Lock these fields first:
+- date,
+- experiment line,
+- round,
+- purpose,
+- linked experiment note,
+- linked durable result note if one already exists.
+
+If round is unknown, do not silently invent a semantic round. Use `r00` only as a temporary placeholder and state that it should be normalized later.
+
+### 2. Read the strict analysis bundle
+
+Minimum required inputs:
+- `analysis-report.md`
+- `stats-appendix.md`
+- `figure-catalog.md`
+- actual figures, if available
+
+If these are missing, either generate them first with `results-analysis` or explicitly state which claims cannot be supported.
+
+### 3. Write the report as a decision object
+
+This report is not a transcript of outputs.
+
+Each section must answer a real question:
+- What did we test?
+- What changed numerically?
+- What is actually supported?
+- What failed or remains uncertain?
+- What should we do next?
+
+Read `references/decision-oriented-analysis.md` for the expected reasoning depth.
+
+### 4. Interpret figures inside the report
+
+Do not only attach figures.
+
+For each main figure:
+- introduce why it is included,
+- state the key observation,
+- explain the supported interpretation,
+- explain the decision implication.
+
+Read `references/figure-interpretation.md` and `references/statistical-completeness.md` as needed.
+
+### 5. Choose the write target explicitly
+
+If the current repo is bound to an Obsidian project knowledge base:
+- create or update `Results/Reports/{report-name}.md`,
+- link back to the relevant `Experiments/` note,
+- update the matching canonical `Results/` note when a durable conclusion is now supported,
+- append a short trace to today's `Daily/` note,
+- update `.claude/project-memory/<project_id>.md`.
+
+If the repo is **not** bound:
+- write the report as a local markdown artifact in the requested output location or next to the analysis bundle,
+- keep the same filename contract,
+- explicitly say that no Obsidian write-back was attempted.
+
+Use `obsidian-project-memory` conventions only for bound repos. Internal experiment reports belong in `Results/Reports/`, not `Writing/`.
+
+### 6. End with explicit next actions
+
+The report must end with operational decisions, for example:
+- stop a weak branch,
+- schedule one missing ablation,
+- promote a stable finding into manuscript-facing writing,
+- update the active plan.
+
+## Required quality bar
+
+- The report must be dateable, searchable, and attributable to one experiment line and one round.
+- The report must cite actual evidence from the analysis bundle.
+- The report must include negative results when they matter.
+- The report must separate stable conclusion from tentative interpretation.
+- The report must say what changed in project belief and what should happen next.
+
+## Reference files
+
+Load only what is needed:
+- `references/report-structure.md`
+- `references/report-naming.md`
+- `references/figure-interpretation.md`
+- `references/statistical-completeness.md`
+- `references/decision-oriented-analysis.md`
+- `references/EVIDENCE-PROPAGATION.md`
+- `examples/example-results-report.md`
diff --git a/.agents/skills/results-report/examples/example-results-report.md b/.agents/skills/results-report/examples/example-results-report.md
new file mode 100644
index 0000000..603bf59
--- /dev/null
+++ b/.agents/skills/results-report/examples/example-results-report.md
@@ -0,0 +1,83 @@
+---
+type: results-report
+date: 2026-03-18
+experiment_line: freezing
+round: 3
+purpose: transfer-summary
+status: active
+source_artifacts:
+ - analysis-output/analysis-report.md
+ - analysis-output/stats-appendix.md
+ - analysis-output/figure-catalog.md
+linked_experiments:
+ - Experiments/Freezing-vs-Adapter.md
+linked_results:
+ - Results/Adapter-Improves-Transfer.md
+---
+
+# Freezing / Round 3 / transfer-summary / 2026-03-18
+
+## Executive Summary
+- Round 3 tested whether a subject adapter recovers the performance lost by freezing most of the encoder.
+- Across 5 seeds per condition, the adapter reduced mean WER by **3.8 absolute points** relative to the frozen encoder baseline.
+- The current evidence supports keeping the adapter branch active, while pure freezing should be deprioritized.
+
+## Experiment Identity and Decision Context
+- Experiment line: freezing
+- Round: 3
+- Purpose: resolve whether the freezing gap is best handled by lightweight adaptation or by abandoning the freezing branch.
+- Decision pressure: choose the next transfer branch before scheduling the next low-resource run block.
+
+## Setup and Evaluation Protocol
+- Same subject pool and split as rounds 1-2.
+- 5 seeds per condition.
+- Primary metric: WER (lower is better).
+- Compared methods: Full fine-tuning, Subject Adapter, Frozen Encoder.
+- Statistical unit: seed-level final WER.
+
+## Main Findings
+- Subject Adapter: **27.6 ± 1.0 WER**, 95% CI **[26.4, 28.8]**.
+- Frozen Encoder: **31.4 ± 1.5 WER**, 95% CI **[29.6, 33.2]**.
+- Full fine-tuning: **25.9 ± 0.8 WER**, 95% CI **[24.9, 26.9]**.
+- Adapter beats Frozen Encoder in all 5 paired seed comparisons.
+
+## Statistical Validation
+- Adapter vs Frozen Encoder: paired Wilcoxon signed-rank test, **p = 0.031**, Holm-corrected **p = 0.047**, matched-rank biserial effect size **r = 0.90**.
+- Full fine-tuning vs Adapter: paired t-test, **p = 0.11**, Cohen's **d = 0.64**.
+- Interpretation: the adapter gain over pure freezing is supported at current `n = 5`; the gap to full fine-tuning is directionally consistent but still underpowered.
+- Unsupported claim boundary: this report does **not** claim generalization beyond the current subject pool or low-resource regime.
+
+## Figure-by-Figure Interpretation
+### Figure 1 — Main comparison
+- Why included: this is the core decision figure.
+- Evidence carried in: mean WER, 95% CI, and paired-seed comparisons.
+- Supported interpretation: lightweight subject adaptation closes most of the freezing gap.
+- Decision implication: future transfer experiments should center on adapter design, not frozen-only variants.
+
+### Figure 2 — Training dynamics
+- Why included: to explain stability differences.
+- Evidence carried in: per-epoch validation traces across seeds.
+- Supported interpretation: the frozen baseline oscillates more after epoch 8, matching its wider uncertainty interval.
+- Decision implication: branch weakness is not only lower final accuracy but also worse optimization stability.
+
+## Failure Cases / Negative Results / Limitations
+- Full fine-tuning still leads in absolute WER.
+- The evidence is limited to one subject pool and 5 seeds.
+- No low-resource stress test or out-of-domain subject split has been run yet.
+- Adapter width was fixed in this round, so capacity trade-offs remain unresolved.
+
+## What Changed Our Belief
+- Before round 3, it was plausible that freezing should be abandoned entirely.
+- After round 3, the better hypothesis is that freezing alone is too rigid, but freezing plus lightweight adaptation remains viable.
+
+## Next Actions
+- Run one low-resource robustness check for the adapter branch.
+- Add a width ablation around the current best adapter size.
+- Update the canonical result note for adapter-improves-transfer.
+
+## Artifact and Reproducibility Index
+- `analysis-output/analysis-report.md`
+- `analysis-output/stats-appendix.md`
+- `analysis-output/figure-catalog.md`
+- `analysis-output/figures/figure-01-main-comparison.pdf`
+- `analysis-output/figures/figure-02-training-dynamics.pdf`
diff --git a/.agents/skills/results-report/references/EVIDENCE-PROPAGATION.md b/.agents/skills/results-report/references/EVIDENCE-PROPAGATION.md
new file mode 100644
index 0000000..bd46910
--- /dev/null
+++ b/.agents/skills/results-report/references/EVIDENCE-PROPAGATION.md
@@ -0,0 +1,25 @@
+# Evidence Propagation
+
+Use this file to keep `results-analysis` outputs aligned with the final report.
+
+## Mapping rule
+
+- `analysis-report.md` -> main findings and narrative summary
+- `stats-appendix.md` -> test choice, uncertainty, effect size, correction rule
+- `figure-catalog.md` -> figure purpose and per-figure interpretation scaffolding
+- figure files -> visual evidence cited in `Figure-by-Figure Interpretation`
+
+## Minimum statistical carry-over
+
+Every strong claim in a results report should preserve:
+- sample size or run/seed count,
+- metric definition,
+- uncertainty summary,
+- test name,
+- effect size when relevant,
+- multiple-comparison handling when relevant.
+
+## Unsupported claim rule
+
+If the analysis bundle does not support a claim strongly enough, keep the claim tentative and say why.
+Do not upgrade a suggestive result into a decisive conclusion during report writing.
diff --git a/.agents/skills/results-report/references/decision-oriented-analysis.md b/.agents/skills/results-report/references/decision-oriented-analysis.md
new file mode 100644
index 0000000..a63a0c3
--- /dev/null
+++ b/.agents/skills/results-report/references/decision-oriented-analysis.md
@@ -0,0 +1,17 @@
+# Decision-Oriented Analysis
+
+The purpose of a post-experiment report is not only to record what happened.
+It should change the project's next decision.
+
+## Required final questions
+- What should stop?
+- What should continue?
+- What should be tested next?
+- What should be promoted into a durable result note?
+- What, if anything, is ready for manuscript-facing writing?
+
+## Good closing pattern
+- “This round supports X.”
+- “It does not yet resolve Y.”
+- “The main blocker is Z.”
+- “Therefore the next concrete action is A.”
diff --git a/.agents/skills/results-report/references/figure-interpretation.md b/.agents/skills/results-report/references/figure-interpretation.md
new file mode 100644
index 0000000..1156d74
--- /dev/null
+++ b/.agents/skills/results-report/references/figure-interpretation.md
@@ -0,0 +1,17 @@
+# Figure Interpretation in Results Reports
+
+A results report should not dump figures.
+
+For each major figure, write four blocks:
+- **Why this figure exists**
+- **What to notice**
+- **What interpretation is supported**
+- **What this changes in the project decision**
+
+## Example micro-structure
+
+### Figure X
+- Purpose: compare adapter vs freezing under the same transfer setting.
+- Observation: adapter improves mean WER and reduces variance.
+- Interpretation: subject-specific adaptation likely resolves part of the transfer mismatch.
+- Decision implication: prioritize adapter ablations before expanding frozen-only variants.
diff --git a/.agents/skills/results-report/references/report-naming.md b/.agents/skills/results-report/references/report-naming.md
new file mode 100644
index 0000000..7132468
--- /dev/null
+++ b/.agents/skills/results-report/references/report-naming.md
@@ -0,0 +1,53 @@
+# Report Naming Standard
+
+## Filename
+
+Use:
+
+```text
+YYYY-MM-DD--{experiment-line}--r{round}--{purpose}.md
+```
+
+Rules:
+- date must be the report date,
+- `experiment-line` should be short and stable,
+- `round` should be zero-padded only if that is already the project convention; otherwise `r3` / `r03` are both acceptable if used consistently,
+- `purpose` should describe why the report exists, not a vague label like `summary` unless that is truly the purpose.
+
+Recommended purpose values:
+- `transfer-summary`
+- `ablation-report`
+- `failure-analysis`
+- `robustness-check`
+- `round-review`
+
+## Title
+
+Use:
+
+```text
+{Experiment Line} / Round {N} / {Purpose} / {YYYY-MM-DD}
+```
+
+## Frontmatter fields
+
+Required:
+- `type: results-report`
+- `date`
+- `experiment_line`
+- `round`
+- `purpose`
+- `status`
+- `source_artifacts`
+- `linked_experiments`
+- `linked_results`
+
+## Placement in Obsidian
+
+Internal reports go to:
+
+```text
+Results/Reports/{filename}
+```
+
+Do not put internal experiment reports in `Writing/` unless they are already manuscript/slides/rebuttal material.
diff --git a/.agents/skills/results-report/references/report-structure.md b/.agents/skills/results-report/references/report-structure.md
new file mode 100644
index 0000000..e0eb247
--- /dev/null
+++ b/.agents/skills/results-report/references/report-structure.md
@@ -0,0 +1,63 @@
+# Results Report Structure
+
+## 1. Executive Summary
+Answer:
+- what was tested,
+- what the highest-confidence conclusion is,
+- what decision this changes.
+
+## 2. Experiment Identity and Decision Context
+Answer:
+- which experiment line this belongs to,
+- why this round was run,
+- what prior uncertainty or decision it was meant to resolve.
+
+## 3. Setup and Evaluation Protocol
+Answer:
+- datasets / subjects / splits,
+- methods compared,
+- primary metrics,
+- repeated-run structure,
+- any deviations from prior protocol.
+
+## 4. Main Findings
+Answer:
+- what changed most,
+- which comparison matters most,
+- where the largest gains or failures appear.
+
+## 5. Statistical Validation
+Answer:
+- what evidence supports the major claims,
+- what tests were used,
+- where the evidence is weak.
+
+## 6. Figure-by-Figure Interpretation
+For each main figure:
+- why it is shown,
+- what to notice,
+- what is supported,
+- what remains uncertain.
+
+## 7. Failure Cases / Negative Results / Limitations
+Answer:
+- what did not work,
+- what instability appeared,
+- what limits the current conclusion.
+
+## 8. What Changed Our Belief
+Answer:
+- which prior hypothesis is strengthened,
+- weakened,
+- or still unresolved.
+
+## 9. Next Actions
+Answer:
+- stop / continue / ablate / scale / write.
+
+## 10. Artifact and Reproducibility Index
+List:
+- source artifacts,
+- figure paths,
+- scripts/logs,
+- linked Obsidian notes.
diff --git a/.agents/skills/results-report/references/statistical-completeness.md b/.agents/skills/results-report/references/statistical-completeness.md
new file mode 100644
index 0000000..4c8756c
--- /dev/null
+++ b/.agents/skills/results-report/references/statistical-completeness.md
@@ -0,0 +1,19 @@
+# Statistical Completeness for Results Reports
+
+A report may summarize statistics, but it must not silently weaken them.
+
+## Always carry forward
+- sample size / seed count
+- metric direction
+- descriptive statistics
+- uncertainty estimate
+- test choice
+- effect size
+- correction rule when relevant
+- evidence boundary
+
+## Never do this in the report
+- upgrade a trend to a conclusion
+- omit the sample size
+- replace effect size with adjectives like “large” without numbers
+- cite a figure without saying what the uncertainty represents