The "Black" Art & "Elusive" Science of Training Deep Neural Networks

School of Computing, University of Nebraska-Lincoln
Spring 2021 (3-week session): CSCE 496/896

Synopsis: Deep Learning or representational learning is an exciting venture of modern Artificial Intelligence for solving hard problems in computer vision, natural language processing, and speech recognition, to name a few. A Deep Neural Network (DNN) is a gigantic conglomeration of computational units or artificial neurons, structured hierarchically in many successive layers. DNNs discover hidden patterns from the input data by creating layers of increasingly meaningful representations of the data. To unleash the full potential of DNNs, one must have knowledge of effective DNN architectures, learning algorithms, and optimization strategies. Training DNNs is tricky, no less than summoning a “genie”. Many consider it “black art”. This course will attempt to demystify the training process of DNN models with an emphasis on the modern state-of-the-art DNN architectures. It will explore the “black” art and “elusive” science of training DNNs.

Philosophy: I will advocate my mentor Richard Feynman's philosophy on learning: "What I cannot create, I do not understand". Your understanding of the DNN training issues will be incomplete unless you are able to implement DNNs such as CNN or RNN from scratch (without using any high-level ML libraries such as Scikit-Learn, Keras, or PyTorch). Start by implementing a multi-layer perceptron (MLP) using NumPy. Refer to the pseudocode from my GitHub notebook on MLP. Then, try implementing a CNN using NumPy. Refer to the pseudocode from my GitHub notebook on CNN.

Dr. M. R. Hasan
Office Hours
See the course Canvas page

Lecture Time
Monday, Tuesday, Wednesday, Thursday, Friday: 10.00 AM - 11.30 AM via Zoom

GitHub repositories of my tutorials on Machine Learning and Deep Learning

Key repositories:


Week Date Topic & PDF Slides Video Links
1 Jan 4
1 Jan 5
1 Jan 6 Jupyter Notebooks:
1 Jan 7
1 Jan 8 Jupyter Notebooks:
1 Jan 9
2 Jan 11
2 Jan 12
2 Jan 13
2 Jan 14
2 Jan 15
2 Jan 17
3 Jan 18 Martin Luther King Holiday
3 Jan 19 CNN Visualization: Jupyter Notebooks:
3 Jan 20
3 Jan 21 Jupyter Notebooks:

Text Resources
I will refer to the following texts in some lectures for discussing the foundational concepts in Machine Learning
  • Machine Learning: A Probabilistic Perspective by Kevin P. Murphy
  • Pattern Recognition and Machine Learning by Christopher M. Bishop
  • Introduction to Machine learning (3rd ed.) by Ethem Alpaydin
Following books are useful for Python refresher/introduction
  • Data Science from Scratch by Joel Grus (O’Reilly)
  • Python for Data Analysis (2nd Edition) by Wes McKinney (O'Reilley)
  • Python Machine Learning by Sebastian Raschka (Packt Publishing)
Statistics, Linear Algebra & Calculus
  • Advanced Engineering Mathematics (10th Ed.) by Erwin Kreyszig
  • All of Statistics: A Concise Course in Statistical Inference by Larry Wasserman
Interesting & Enlightening Texts

Deep Learning Courses Elsewhere

Collaboration Tool

Google Colab Tutorials


Open Data Repositories

ML Podcasts


Conferences Proceedings