Lecture 12: Simplifying Data — PCA and Friends

Dimensionality Reduction for Visualization and Understanding

Overview

PCA compresses data by retaining variance and minimizing squared reconstruction error. We connect the geometry to centered SVD, then test what changes when useful class information has low variance. Nonlinear maps and real essay embeddings show why visible islands, word associations and authorship predictions need careful interpretation.

Learning Objectives

By the end of this lecture, you will:

  • Explain PCA geometrically and understand how it finds directions of maximum variance for data compression
  • Implement centered PCA with SVD and distinguish retained variance from predictive information
  • Explain how t-SNE and UMAP emphasize neighborhoods, and inspect the effects of their settings
  • Interpret DR visualizations critically, understanding what relationships are preserved and what might be artifacts
  • Analyze real essay embeddings while checking prompt, length and recorded-origin associations
  • Optimize DR workflows by using PCA preprocessing to speed up nonlinear methods and choosing appropriate hyperparameters

Materials

TipQuick Access

Read the lecture notes

PCA and Dimensionality Reduction Notebook

Datasets & Acknowledgments

  • DAIGT V3 (thedrcat, Kaggle): Detecting AI Generated Text dataset with 130k texts, used to demonstrate real-world DR applications for distinguishing AI vs. human writing
  • Mammoth 3D (PaCMAP repository): 10k-point digitization of mammoth skeleton for testing global structure preservation
  • Synthetic datasets: Projection clouds, class-signal examples and a Swiss roll
  • Libraries: scikit-learn (PCA, t-SNE), UMAP-learn, sentence-transformers (text embeddings)
  • Methodology inspiration: Yang Lu (uwyang), “Visualizing AI v.s. Human writing embeddings using TSNE for AI detection (Part 1/n)” — our analysis follows similar preprocessing and visualization techniques
  • Key papers: van der Maaten & Hinton (2008) for t-SNE, McInnes et al. (2018) for UMAP

Previous: ← Lecture 11: K-Means Clustering | Next: Lecture 13: Neural Network Architecture →