Advanced Deep Learning Techniques

Course structure

• Neural network architectures (feed-forward, recurrent, convolutional, generative, autoencoders, Unet, GAN, attention layer)
• Optimizers and their evolution (Steepest Gradient Descent, Stochastic Gradient Descent, Mini-Batch Gradient Descent, Nesterov Accelerated Gradient, Adagrad, AdaDelta, Adam, Learning rate tuning)
• Loss functions and their properties (Mean squared error, Mean absolute error, Negative, Log Likelihood – cross entropy)
• Regularization in Neural Networks (Dropout, Early stopping, Data augmentation, Batch and layer normalization)
• Initialization (Gradient vanishing problem, Zero initialization, He initialization, Xavier initialization)
• Semi-supervised learning (Pseudo Labeling, Mean-Teacher, PI-Model)
• Practical examples of semi-supervised techniques applications
• Confidence estimation (Logit analysis, Confidence networks)
• Practical examples of confidence estimation
• AutoML approaches (Hyper-parameter optimization, grid search, Bayesian optimization, Meta-Learning, Neural network search)
• Practical examples with the AutoKeras
• ML Explainability (Interpretable models, Partial Dependence Plot, Permutation feature importance, Surrogate models, Activation Maximization, Grad CAM)