diff options
Diffstat (limited to '')
3 files changed, 126 insertions, 0 deletions
diff --git a/.agents/skills/results-analysis/examples/example-analysis-report.md b/.agents/skills/results-analysis/examples/example-analysis-report.md new file mode 100644 index 0000000..6aa39cf --- /dev/null +++ b/.agents/skills/results-analysis/examples/example-analysis-report.md @@ -0,0 +1,57 @@ +# Strict Analysis Report + +## Analysis Question +- Compare Full Model, Frozen Encoder, and Subject Adapter on cross-subject decoding. +- Determine whether the adapter closes the transfer gap without sacrificing stability. + +## Data Inventory +- 3 model families +- 5 random seeds per family +- subject-level WER and CER +- training logs for convergence +- ablation outputs for adapter width + +## Executive Summary +- Subject Adapter improves mean WER over Frozen Encoder by 3.8 absolute points. +- The improvement is statistically significant under paired testing after multiple-comparison correction. +- Full fine-tuning remains strongest overall, but Subject Adapter offers the best compute-performance tradeoff. +- Performance variance is lower than Frozen Encoder, suggesting a more stable transfer path. + +## Main Findings + +### 1. Main comparison +- Full fine-tuning: `31.4 ± 1.9` WER +- Subject Adapter: `33.2 ± 1.3` WER +- Frozen Encoder: `37.0 ± 2.1` WER + +Observation: +- Subject Adapter consistently outperforms Frozen Encoder across all 5 seeds. + +Interpretation: +- The transfer bottleneck is not purely feature reuse; a light adaptation head captures subject-specific alignment that freezing alone misses. + +Implication: +- Future work should prioritize adapter variants before investing in heavier full-model tuning. + +### 2. Stability +- Subject Adapter has the smallest seed variance among transfer-friendly methods. +- Convergence curves show less oscillation after epoch 8. + +Interpretation: +- Adapter tuning is not only better on average, but easier to optimize. + +### 3. Ablation +- Reducing adapter width from 256 to 64 hurts WER by 1.6 points. +- Increasing from 256 to 512 yields only marginal improvement. + +Interpretation: +- Most of the benefit is captured at moderate width; scale-up is not the current bottleneck. + +## Caveats +- Only 5 seeds; inference about tails remains limited. +- Subject count is modest, so subject-level heterogeneity may still be under-estimated. +- Training-time comparison depends on identical hardware assumptions. + +## Recommended Next Move +- Promote this finding into a `results-report` note focused on adapter vs freezing. +- Run one robustness pass on held-out low-resource subjects. diff --git a/.agents/skills/results-analysis/examples/example-figure-catalog.md b/.agents/skills/results-analysis/examples/example-figure-catalog.md new file mode 100644 index 0000000..d9222f1 --- /dev/null +++ b/.agents/skills/results-analysis/examples/example-figure-catalog.md @@ -0,0 +1,33 @@ +# Figure Catalog + +## Figure 1 — Main comparison +- **Filename**: `figures/figure-01-main-comparison.pdf` +- **Purpose**: Compare WER across Full fine-tuning, Subject Adapter, and Frozen Encoder. +- **Data source**: `metrics/summary.csv` +- **Plot type**: Bar + scatter overlay of seed-level points. +- **Error bars**: 95% CI. +- **Caption must include**: + - metric direction, + - number of seeds, + - meaning of error bars, + - whether significance markers are corrected. +- **Key observation**: Subject Adapter closes most of the freezing gap. +- **Interpretation checklist**: + - Is the adapter improvement consistent across seeds? + - Is the gap practically meaningful, not just statistically significant? + - Does the figure support a design decision? + +## Figure 2 — Convergence dynamics +- **Filename**: `figures/figure-02-training-dynamics.pdf` +- **Purpose**: Compare optimization stability over epochs. +- **Data source**: `logs/train_curves.csv` +- **Plot type**: Mean line + uncertainty ribbon. +- **Caption must include**: + - smoothing rule if any, + - whether ribbon is std or CI, + - training/eval split. +- **Key observation**: Frozen Encoder shows larger oscillation after epoch 8. +- **Interpretation checklist**: + - Is instability transient or persistent? + - Does curve shape match final metric variance? + - Does this explain why one method is harder to tune? diff --git a/.agents/skills/results-analysis/examples/example-stats-appendix.md b/.agents/skills/results-analysis/examples/example-stats-appendix.md new file mode 100644 index 0000000..79ecbfc --- /dev/null +++ b/.agents/skills/results-analysis/examples/example-stats-appendix.md @@ -0,0 +1,36 @@ +# Statistical Appendix + +## Primary Metric +- Word Error Rate (WER), lower is better. + +## Sample Structure +- Unit of analysis: seed-level run, paired by shared data split and subject pool. +- Number of seeds per condition: 5. + +## Descriptive Statistics + +| Condition | Mean WER | Std | 95% CI | +|---|---:|---:|---:| +| Full fine-tuning | 31.4 | 1.9 | [29.8, 33.0] | +| Subject Adapter | 33.2 | 1.3 | [32.1, 34.3] | +| Frozen Encoder | 37.0 | 2.1 | [35.1, 38.9] | + +## Assumption Checks +- Shapiro-Wilk on paired differences: `p = 0.19` +- No strong evidence against normality for the primary contrast. +- Given small n, interpretation remains conservative. + +## Inferential Tests + +| Contrast | Test | Statistic | p | Effect size | Correction | +|---|---|---|---:|---:|---| +| Subject Adapter vs Frozen Encoder | paired t-test | `t(4) = -4.11` | 0.014 | Cohen's `d = 1.84` | Holm | +| Full fine-tuning vs Subject Adapter | paired t-test | `t(4) = -2.03` | 0.112 | Cohen's `d = 0.91` | Holm | + +## Interpretation Guardrails +- First contrast is supported after correction. +- Second contrast trends in favor of full fine-tuning but is not conclusive at this sample size. + +## Blockers / Limits +- No subject-level bootstrap yet. +- No calibration analysis available. |