1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
|
# Detailed Directory Structure
This document provides a comprehensive breakdown of the ML project template directory structure.
## Root Level Files
| File | Purpose |
|------|---------|
| `README.md` | Project documentation, installation guide, usage examples |
| `TODO.md` | Task tracking with weekly focus and daily tasks |
| `.gitignore` | Git ignore patterns for Python, Jupyter, IDEs, logs, cache |
| `pyproject.toml` | Project configuration for build system and dependencies |
| `uv.lock` | Locked dependency versions for reproducibility |
## run/ - Execution Layer
### pipeline/
Main workflow scripts organized by stage:
| Directory | Purpose |
|-----------|---------|
| `training/` | Training execution scripts (training.sh, inference.sh) |
| `prepare_data/` | Data preparation and preprocessing pipelines |
| `analysis/` | Evaluation and analysis workflows |
### conf/
Hydra configuration files organized by module:
| Directory | Purpose |
|-----------|---------|
| `training/` | Training hyperparameters, model configs, optimizer settings |
| `dataset/` | Dataset configurations, data paths, preprocessing options |
| `model/` | Model architecture configurations |
| `prepare_data/` | Data preparation parameters |
| `analysis/` | Analysis and evaluation configurations |
| `dir/` | Directory path configurations |
| `analysis/` | Analysis-specific settings |
## src/ - Source Code Layer
### data_module/ - Data Processing Module
```
data_module/
├── __init__.py # Module exports
├── utils.py # Data-specific utility functions
├── dataset/ # Dataset implementations
│ ├── __init__.py # Dataset factory and registry
│ └── simple_dataset.py # Simple dataset example
├── augmentation/ # Data augmentation methods
│ ├── __init__.py
│ ├── mixup.py # Mixup augmentation
│ ├── random_shift.py # Random shifting
│ ├── channel_mask.py # Channel masking
│ ├── time_masking.py # Time masking
│ └── add_noise.py # Noise injection
├── collate_fn/ # Batch collation functions
│ ├── __init__.py
│ └── simple_collate_fn.py
├── compute_metrics/ # Metrics computation
│ ├── __init__.py
│ └── simple_compute_metrics.py
├── prepare_data/ # Data preparation logic
│ ├── __init__.py
│ ├── prepare_data.py
│ └── generate_yaml.py
└── data_func/ # Data utility functions
├── __init__.py
└── simple_data_func.py
```
### model_module/ - Model Module
```
model_module/
├── __init__.py # Module exports
└── model/ # Model implementations
└── [model files]
```
### trainer_module/ - Training Module
Contains training loop logic, validation, and checkpoint management.
### analysis_module/ - Analysis Module
Contains evaluation, visualization, and result analysis code.
### llm/ - LLM Module
LLM-related code and integrations.
### utils/ - Shared Utilities
```
utils/
├── __init__.py
├── helpers.py # Helper functions (import_modules, etc.)
├── logging.py # Logging configuration
├── get_optimizer.py # Optimizer factory
├── get_scheduler.py # Learning rate scheduler factory
├── get_callback.py # Training callbacks
├── get_activation.py # Activation functions
└── get_checkpoint_aggregation.py # Checkpoint handling
```
## data/ - Data Layer
Following the Cookiecutter Data Science standard:
| Directory | Purpose |
|-----------|---------|
| `raw/` | Original, immutable data dump |
| `processed/` | Cleaned, transformed data ready for use |
| `external/` | Data from third-party sources |
## outputs/ - Output Layer
| Directory | Purpose |
|-----------|---------|
| `logs/` | Training logs, tensorboard logs |
| `checkpoints/` | Model checkpoints for resuming training |
| `tables/` | Result tables, CSV outputs |
| `figures/` | Plots, visualizations, figures |
## Module Interaction Flow
```
run/pipeline/ -> src/trainer_module/ -> src/model_module/
src/data_module/ src/utils/
src/utils/
run/conf/ -> Hydra config loader -> All modules
```
## File Naming Conventions
- **Modules**: `simple_dataset.py`, `custom_model.py`
- **Pipelines**: `training.sh`, `inference.sh`
- **Configs**: `config.yaml`, dataset-specific names
- **Utilities**: Descriptive names (`get_optimizer.py`, `helpers.py`)
## Python Package Structure
Each module is a proper Python package:
- Has `__init__.py` with factory/registry logic
- Can be imported as `from src.module import Component`
- Subpackages are automatically discovered via `import_modules()`
|
