diff options
| author | sillylaird <sillyfanboy@gmail.com> | 2026-09-03 00:33:59 +0000 |
|---|---|---|
| committer | sillylaird <sillyfanboy@gmail.com> | 2026-09-03 00:33:59 +0000 |
| commit | 898b52edcb47bcb3e9d6106e74ca73e74ea01e70 (patch) | |
| tree | 85c6ee5ad58b860144551184d4cf86b560c62b91 /.agents/skills/results-report | |
| download | www-main.tar.gz www-main.zip | |
Diffstat (limited to '.agents/skills/results-report')
8 files changed, 457 insertions, 0 deletions
diff --git a/.agents/skills/results-report/SKILL.md b/.agents/skills/results-report/SKILL.md new file mode 100644 index 0000000..d533539 --- /dev/null +++ b/.agents/skills/results-report/SKILL.md @@ -0,0 +1,180 @@ +--- +name: results-report +description: This skill should be used when the user asks to "write an experiment report", "summarize experimental results", "do experiment retrospection", "write a results report", "写实验总结报告", "写实验复盘", or mentions turning completed experiment artifacts into a structured, decision-oriented research report. It assumes strict analysis should come from `results-analysis` first. +version: 0.1.0 +tags: [Research, Reporting, Experiments, Obsidian] +--- + +# Results Report + +Write the **complete post-experiment summary report** after analysis artifacts are ready. + +This skill is for the stage **after** `results-analysis`. + +## Role boundary + +### `results-analysis` does +- strict statistics, +- real figures, +- figure interpretation scaffolding, +- stats appendix. + +### `results-report` does +- complete experiment wrap-up report, +- decision-oriented narrative, +- figure-by-figure interpretation inside a coherent structure, +- limitations, failure cases, and next actions, +- Obsidian write-back into `Results/Reports/`. + +Do not replace strict analysis with confident prose. If the analysis bundle is missing, first identify the blocker and request or produce the missing bundle. + +## Default output + +The default report is an **internal research report**, not manuscript prose. + +It should be named as: + +```text +YYYY-MM-DD--{experiment-line}--r{round}--{purpose}.md +``` + +Example: +- `2026-03-18--freezing--r03--transfer-summary.md` +- `2026-03-18--contrastive-adversarial--r02--ablation-report.md` + +The note title should be: + +```text +{Experiment Line} / Round {N} / {Purpose} / {YYYY-MM-DD} +``` + +Read `references/report-naming.md` before finalizing the filename or note title. + +## Required frontmatter + +```yaml +--- +type: results-report +date: 2026-03-18 +experiment_line: freezing +round: 3 +purpose: transfer-summary +status: active +source_artifacts: + - analysis-output/analysis-report.md + - analysis-output/stats-appendix.md +linked_experiments: + - Experiments/Freezing-Study.md +linked_results: + - Results/Freezing-vs-Adapter.md +--- +``` + +## Default report structure + +The report must include all sections below. + +1. **Executive Summary** +2. **Experiment Identity and Decision Context** +3. **Setup and Evaluation Protocol** +4. **Main Findings** +5. **Statistical Validation** +6. **Figure-by-Figure Interpretation** +7. **Failure Cases / Negative Results / Limitations** +8. **What Changed Our Belief** +9. **Next Actions** +10. **Artifact and Reproducibility Index** + +Read `references/report-structure.md` before writing. + +## Workflow + +### 1. Confirm the report object + +Lock these fields first: +- date, +- experiment line, +- round, +- purpose, +- linked experiment note, +- linked durable result note if one already exists. + +If round is unknown, do not silently invent a semantic round. Use `r00` only as a temporary placeholder and state that it should be normalized later. + +### 2. Read the strict analysis bundle + +Minimum required inputs: +- `analysis-report.md` +- `stats-appendix.md` +- `figure-catalog.md` +- actual figures, if available + +If these are missing, either generate them first with `results-analysis` or explicitly state which claims cannot be supported. + +### 3. Write the report as a decision object + +This report is not a transcript of outputs. + +Each section must answer a real question: +- What did we test? +- What changed numerically? +- What is actually supported? +- What failed or remains uncertain? +- What should we do next? + +Read `references/decision-oriented-analysis.md` for the expected reasoning depth. + +### 4. Interpret figures inside the report + +Do not only attach figures. + +For each main figure: +- introduce why it is included, +- state the key observation, +- explain the supported interpretation, +- explain the decision implication. + +Read `references/figure-interpretation.md` and `references/statistical-completeness.md` as needed. + +### 5. Choose the write target explicitly + +If the current repo is bound to an Obsidian project knowledge base: +- create or update `Results/Reports/{report-name}.md`, +- link back to the relevant `Experiments/` note, +- update the matching canonical `Results/` note when a durable conclusion is now supported, +- append a short trace to today's `Daily/` note, +- update `.claude/project-memory/<project_id>.md`. + +If the repo is **not** bound: +- write the report as a local markdown artifact in the requested output location or next to the analysis bundle, +- keep the same filename contract, +- explicitly say that no Obsidian write-back was attempted. + +Use `obsidian-project-memory` conventions only for bound repos. Internal experiment reports belong in `Results/Reports/`, not `Writing/`. + +### 6. End with explicit next actions + +The report must end with operational decisions, for example: +- stop a weak branch, +- schedule one missing ablation, +- promote a stable finding into manuscript-facing writing, +- update the active plan. + +## Required quality bar + +- The report must be dateable, searchable, and attributable to one experiment line and one round. +- The report must cite actual evidence from the analysis bundle. +- The report must include negative results when they matter. +- The report must separate stable conclusion from tentative interpretation. +- The report must say what changed in project belief and what should happen next. + +## Reference files + +Load only what is needed: +- `references/report-structure.md` +- `references/report-naming.md` +- `references/figure-interpretation.md` +- `references/statistical-completeness.md` +- `references/decision-oriented-analysis.md` +- `references/EVIDENCE-PROPAGATION.md` +- `examples/example-results-report.md` diff --git a/.agents/skills/results-report/examples/example-results-report.md b/.agents/skills/results-report/examples/example-results-report.md new file mode 100644 index 0000000..603bf59 --- /dev/null +++ b/.agents/skills/results-report/examples/example-results-report.md @@ -0,0 +1,83 @@ +--- +type: results-report +date: 2026-03-18 +experiment_line: freezing +round: 3 +purpose: transfer-summary +status: active +source_artifacts: + - analysis-output/analysis-report.md + - analysis-output/stats-appendix.md + - analysis-output/figure-catalog.md +linked_experiments: + - Experiments/Freezing-vs-Adapter.md +linked_results: + - Results/Adapter-Improves-Transfer.md +--- + +# Freezing / Round 3 / transfer-summary / 2026-03-18 + +## Executive Summary +- Round 3 tested whether a subject adapter recovers the performance lost by freezing most of the encoder. +- Across 5 seeds per condition, the adapter reduced mean WER by **3.8 absolute points** relative to the frozen encoder baseline. +- The current evidence supports keeping the adapter branch active, while pure freezing should be deprioritized. + +## Experiment Identity and Decision Context +- Experiment line: freezing +- Round: 3 +- Purpose: resolve whether the freezing gap is best handled by lightweight adaptation or by abandoning the freezing branch. +- Decision pressure: choose the next transfer branch before scheduling the next low-resource run block. + +## Setup and Evaluation Protocol +- Same subject pool and split as rounds 1-2. +- 5 seeds per condition. +- Primary metric: WER (lower is better). +- Compared methods: Full fine-tuning, Subject Adapter, Frozen Encoder. +- Statistical unit: seed-level final WER. + +## Main Findings +- Subject Adapter: **27.6 ± 1.0 WER**, 95% CI **[26.4, 28.8]**. +- Frozen Encoder: **31.4 ± 1.5 WER**, 95% CI **[29.6, 33.2]**. +- Full fine-tuning: **25.9 ± 0.8 WER**, 95% CI **[24.9, 26.9]**. +- Adapter beats Frozen Encoder in all 5 paired seed comparisons. + +## Statistical Validation +- Adapter vs Frozen Encoder: paired Wilcoxon signed-rank test, **p = 0.031**, Holm-corrected **p = 0.047**, matched-rank biserial effect size **r = 0.90**. +- Full fine-tuning vs Adapter: paired t-test, **p = 0.11**, Cohen's **d = 0.64**. +- Interpretation: the adapter gain over pure freezing is supported at current `n = 5`; the gap to full fine-tuning is directionally consistent but still underpowered. +- Unsupported claim boundary: this report does **not** claim generalization beyond the current subject pool or low-resource regime. + +## Figure-by-Figure Interpretation +### Figure 1 — Main comparison +- Why included: this is the core decision figure. +- Evidence carried in: mean WER, 95% CI, and paired-seed comparisons. +- Supported interpretation: lightweight subject adaptation closes most of the freezing gap. +- Decision implication: future transfer experiments should center on adapter design, not frozen-only variants. + +### Figure 2 — Training dynamics +- Why included: to explain stability differences. +- Evidence carried in: per-epoch validation traces across seeds. +- Supported interpretation: the frozen baseline oscillates more after epoch 8, matching its wider uncertainty interval. +- Decision implication: branch weakness is not only lower final accuracy but also worse optimization stability. + +## Failure Cases / Negative Results / Limitations +- Full fine-tuning still leads in absolute WER. +- The evidence is limited to one subject pool and 5 seeds. +- No low-resource stress test or out-of-domain subject split has been run yet. +- Adapter width was fixed in this round, so capacity trade-offs remain unresolved. + +## What Changed Our Belief +- Before round 3, it was plausible that freezing should be abandoned entirely. +- After round 3, the better hypothesis is that freezing alone is too rigid, but freezing plus lightweight adaptation remains viable. + +## Next Actions +- Run one low-resource robustness check for the adapter branch. +- Add a width ablation around the current best adapter size. +- Update the canonical result note for adapter-improves-transfer. + +## Artifact and Reproducibility Index +- `analysis-output/analysis-report.md` +- `analysis-output/stats-appendix.md` +- `analysis-output/figure-catalog.md` +- `analysis-output/figures/figure-01-main-comparison.pdf` +- `analysis-output/figures/figure-02-training-dynamics.pdf` diff --git a/.agents/skills/results-report/references/EVIDENCE-PROPAGATION.md b/.agents/skills/results-report/references/EVIDENCE-PROPAGATION.md new file mode 100644 index 0000000..bd46910 --- /dev/null +++ b/.agents/skills/results-report/references/EVIDENCE-PROPAGATION.md @@ -0,0 +1,25 @@ +# Evidence Propagation + +Use this file to keep `results-analysis` outputs aligned with the final report. + +## Mapping rule + +- `analysis-report.md` -> main findings and narrative summary +- `stats-appendix.md` -> test choice, uncertainty, effect size, correction rule +- `figure-catalog.md` -> figure purpose and per-figure interpretation scaffolding +- figure files -> visual evidence cited in `Figure-by-Figure Interpretation` + +## Minimum statistical carry-over + +Every strong claim in a results report should preserve: +- sample size or run/seed count, +- metric definition, +- uncertainty summary, +- test name, +- effect size when relevant, +- multiple-comparison handling when relevant. + +## Unsupported claim rule + +If the analysis bundle does not support a claim strongly enough, keep the claim tentative and say why. +Do not upgrade a suggestive result into a decisive conclusion during report writing. diff --git a/.agents/skills/results-report/references/decision-oriented-analysis.md b/.agents/skills/results-report/references/decision-oriented-analysis.md new file mode 100644 index 0000000..a63a0c3 --- /dev/null +++ b/.agents/skills/results-report/references/decision-oriented-analysis.md @@ -0,0 +1,17 @@ +# Decision-Oriented Analysis + +The purpose of a post-experiment report is not only to record what happened. +It should change the project's next decision. + +## Required final questions +- What should stop? +- What should continue? +- What should be tested next? +- What should be promoted into a durable result note? +- What, if anything, is ready for manuscript-facing writing? + +## Good closing pattern +- “This round supports X.” +- “It does not yet resolve Y.” +- “The main blocker is Z.” +- “Therefore the next concrete action is A.” diff --git a/.agents/skills/results-report/references/figure-interpretation.md b/.agents/skills/results-report/references/figure-interpretation.md new file mode 100644 index 0000000..1156d74 --- /dev/null +++ b/.agents/skills/results-report/references/figure-interpretation.md @@ -0,0 +1,17 @@ +# Figure Interpretation in Results Reports + +A results report should not dump figures. + +For each major figure, write four blocks: +- **Why this figure exists** +- **What to notice** +- **What interpretation is supported** +- **What this changes in the project decision** + +## Example micro-structure + +### Figure X +- Purpose: compare adapter vs freezing under the same transfer setting. +- Observation: adapter improves mean WER and reduces variance. +- Interpretation: subject-specific adaptation likely resolves part of the transfer mismatch. +- Decision implication: prioritize adapter ablations before expanding frozen-only variants. diff --git a/.agents/skills/results-report/references/report-naming.md b/.agents/skills/results-report/references/report-naming.md new file mode 100644 index 0000000..7132468 --- /dev/null +++ b/.agents/skills/results-report/references/report-naming.md @@ -0,0 +1,53 @@ +# Report Naming Standard + +## Filename + +Use: + +```text +YYYY-MM-DD--{experiment-line}--r{round}--{purpose}.md +``` + +Rules: +- date must be the report date, +- `experiment-line` should be short and stable, +- `round` should be zero-padded only if that is already the project convention; otherwise `r3` / `r03` are both acceptable if used consistently, +- `purpose` should describe why the report exists, not a vague label like `summary` unless that is truly the purpose. + +Recommended purpose values: +- `transfer-summary` +- `ablation-report` +- `failure-analysis` +- `robustness-check` +- `round-review` + +## Title + +Use: + +```text +{Experiment Line} / Round {N} / {Purpose} / {YYYY-MM-DD} +``` + +## Frontmatter fields + +Required: +- `type: results-report` +- `date` +- `experiment_line` +- `round` +- `purpose` +- `status` +- `source_artifacts` +- `linked_experiments` +- `linked_results` + +## Placement in Obsidian + +Internal reports go to: + +```text +Results/Reports/{filename} +``` + +Do not put internal experiment reports in `Writing/` unless they are already manuscript/slides/rebuttal material. diff --git a/.agents/skills/results-report/references/report-structure.md b/.agents/skills/results-report/references/report-structure.md new file mode 100644 index 0000000..e0eb247 --- /dev/null +++ b/.agents/skills/results-report/references/report-structure.md @@ -0,0 +1,63 @@ +# Results Report Structure + +## 1. Executive Summary +Answer: +- what was tested, +- what the highest-confidence conclusion is, +- what decision this changes. + +## 2. Experiment Identity and Decision Context +Answer: +- which experiment line this belongs to, +- why this round was run, +- what prior uncertainty or decision it was meant to resolve. + +## 3. Setup and Evaluation Protocol +Answer: +- datasets / subjects / splits, +- methods compared, +- primary metrics, +- repeated-run structure, +- any deviations from prior protocol. + +## 4. Main Findings +Answer: +- what changed most, +- which comparison matters most, +- where the largest gains or failures appear. + +## 5. Statistical Validation +Answer: +- what evidence supports the major claims, +- what tests were used, +- where the evidence is weak. + +## 6. Figure-by-Figure Interpretation +For each main figure: +- why it is shown, +- what to notice, +- what is supported, +- what remains uncertain. + +## 7. Failure Cases / Negative Results / Limitations +Answer: +- what did not work, +- what instability appeared, +- what limits the current conclusion. + +## 8. What Changed Our Belief +Answer: +- which prior hypothesis is strengthened, +- weakened, +- or still unresolved. + +## 9. Next Actions +Answer: +- stop / continue / ablate / scale / write. + +## 10. Artifact and Reproducibility Index +List: +- source artifacts, +- figure paths, +- scripts/logs, +- linked Obsidian notes. diff --git a/.agents/skills/results-report/references/statistical-completeness.md b/.agents/skills/results-report/references/statistical-completeness.md new file mode 100644 index 0000000..4c8756c --- /dev/null +++ b/.agents/skills/results-report/references/statistical-completeness.md @@ -0,0 +1,19 @@ +# Statistical Completeness for Results Reports + +A report may summarize statistics, but it must not silently weaken them. + +## Always carry forward +- sample size / seed count +- metric direction +- descriptive statistics +- uncertainty estimate +- test choice +- effect size +- correction rule when relevant +- evidence boundary + +## Never do this in the report +- upgrade a trend to a conclusion +- omit the sample size +- replace effect size with adjectives like “large” without numbers +- cite a figure without saying what the uncertainty represents |
