blob: 559f29a65f9c606a70b721ff2b83b47080e961e2 (
plain) (
blame)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
|
# Common Pitfalls in Experimental Analysis
## Statistical pitfalls
- Reporting only the best run
- Mixing seed-level and subject-level units
- Running many contrasts without correction
- Reporting significance without effect size
- Using parametric tests after failed assumptions without explanation
## Visualization pitfalls
- No real figure despite readable data
- Plot without uncertainty information
- Overcrowded multi-panel figure with no message hierarchy
- Caption missing n / error-bar meaning
- Figure not referenced or interpreted in text
## Reasoning pitfalls
- Confusing correlation with mechanism
- Treating trend as conclusion
- Ignoring negative results
- Hiding instability behind a mean value
- Turning raw logs into durable conclusions too early
## Reporting pitfalls
- Writing paper prose before evidence is stabilized
- Mixing analysis artifact with final narrative artifact
- Not separating blocker from conclusion
- Forgetting to state what decision the analysis changes
|
