Lecture 13: Fully Connected Neural Networks — Architectures & Learning
From perceptron to MLPs, activations, forward/backprop, and CIFAR-10
Overview
Linear models struggle with non-linear patterns, as seen in classic datasets like two moons. This lecture introduces fully connected neural networks (multi-layer perceptrons) and the crucial role of non-linear activation functions in enabling complex decision boundaries. We build MLPs in PyTorch, trace forward propagation step-by-step, discuss backpropagation intuition and gradient checking, visualize capacity vs. complexity through decision boundaries, and finish by training on the CIFAR-10 dataset.
Learning Objectives
By the end of this lecture, you will be able to:
- Understand why non-linear activation functions are essential for neural networks
- Build multi-layer perceptrons (MLPs) using PyTorch layers
- Trace forward propagation through a network step-by-step
- Explain backpropagation intuitively and implement gradient checking
- Train neural networks on real-world datasets (CIFAR-10)
- Diagnose initialization and compare generalization using both accuracy and cross-entropy
Materials
TipQuick Access
Datasets & Acknowledgments
- CIFAR-10 (Krizhevsky et al.): 60k 32×32 color images across 10 classes, accessed via
torchvision.datasets.CIFAR10 - Synthetic data: Two moons for decision boundaries and width comparisons; random inputs for initialization diagnostics
- Libraries: PyTorch/torchvision, scikit-learn, NumPy, Matplotlib
- Figures/visualizations: Decision boundaries, activation functions, and architecture diagrams generated programmatically in the notebook
Previous: ← Lecture 12: PCA and Dimensionality Reduction | Next: Lecture 14: Understanding Transformers →