DL Hyperparameter Tuning

JOY JHJEONG·2021년 12월 4일

Purpose of Hyperparameter Tuning
- Increase Model Performance
- Reduce True Risk(Generalization Error) of Model
- Reduce True Risk on Validation Set, approximately
Options We Have for Hyperparameter Tuning
- Model Related
  - Number of hidden layer
  - Number of hidden unit
  - Activation Function
- Optimization Related
  - Type of Optimizer
  - Learning rate
  - L2 coef
  - Dropout Rate
  - Batch Size
    Dependent on Data, Model
    - Increase batch size until OOM
    - If overfitting is severe, reduce batch size
  - Epoch
    - Regularly Measure Train Loss & Val Loss
    - Early Stopping
Four Way to Tune Experiment
- Grid Layout
- Random Layout
- Hand Tuning
- Bayesian Optimization

Data Scientist를 향한 공부 기록✏️