Contents

Intro

Welcome! My name is Trevor Gordon and I have just completed my MS in Electrical Engineering at Columbia University with a focus in machine learning and reinforcement learning (Dec 2022).

Please see my LinkedIn and resume here for information on my experience:

This jupyter notebook was a place for notes on my courses while at Columbia. This includes theory, formulas, code snippets and links to useful resources. These notes are a work in progress as the semester progresses.

Projects

  • Thesis: Towards real world dynamic spectrum access with deep reinforcement learning. Short blog here

  • Neural Networks & Deep Learning: Piano music generation using LSTM neural network and TensorFlow. Checkout an explanation and results here

  • An Exploration of Socially Driven Multi-Agent Reinforcement Learning Models: Literature review and study on open source multi agent reinforcement learning research. Checkout the final report

  • GPU Programming Project: Developed random forests from scratch using CUDA and NVIDIA GPU. Checkout the final report and code

Courses

Research

  • Reinforcement Learning for wireless communication systems

Advanced Deep Learning

  • Object Detection classical approaches

    • Gradient Vector, HOG, and SS

    • CNN, DPM and Overfeat

    • R-CNN Family

    • Fast Detection Models

      • YOLO: You Only Look Once

      • SSD: Single Shot MultiBox Detector

      • YOLOv2 / YOLO9000

      • YOLOv3

  • GANs

Sparse Models For High Dimensional Data

  • Course Introduction, Motivating Examples, Linear Algebra Review

  • Sparse Solutions, l0, l0 uniqueness, NP-hardness

  • l1 relaxation, l1 recovery under incoherence

  • Recovery under RIP, random matrices

  • Rank minimization: Motivating examples, Nuclear norm relaxation

  • Rank RIP (briefly), Matrix completion

  • Robust PCA, General low-complexity models

  • Optimization I: Review of first order methods, proximal methods, acceleration

  • Optimization II: Constraints, Augmented Lagrangian, ADMM

  • Nonconvex Optimization: dictionaries, deconvolution, neural networks

  • Nonconvex Optimization: geometric theory and intuitions

  • Low-Dimensional Models and Neural Networks I: optimization phenomena

  • Low-Dimensional Models and Neural Networks II: dedicated constructions, invariance

  • Low-Dimensional Models and Neural Networks III: generative models

  • Student course project presentations (final exam slot)

Robot Learning

  • Unsupervised learning: clustering and dimensionality reduction

  • Supervised learning I: linear regression

  • Supervised learning II: classification, decision trees

  • Guest lecture: supervised learning for a robotic hand orthosis

  • Learning for Computer Vision: “traditional” methods

  • 3D Computer Vision

  • Deep Learning: fundamentals, Part I

  • Deep Learning: fundamentals, Part II

  • Deep Learning architectures: CNNs

  • Deep Learning architectures: AEs, RNNs

  • Learning dynamics (forward) models

  • Reinforcement Learning I: fundamentals, Q-Learning

  • Reinforcement Learning II: non-determinism, policy iteration

  • Reinforcement Learning III: Deep Q-Networks

  • Reinforcement Learning IV: Policy Gradient and Actor Critic

  • Open-source tools and libraries for robot learning

  • Domain Randomization and Policies with Memory

Reinforcement Learning

  • Introduction

  • Bandit problems and on-line learning

  • Markov decision processes, returns and value functions

  • RL solutions: dynamic programming methods

  • RL solutions: Monte-Carlo methods

  • RL solutions: temporal difference methods

  • RL algorithms from scratch and intro to OpenAI Gym

  • Eligibility traces.

  • Value function approximation

  • Models and planning

  • Case studies: applications in artificial intelligence

  • Deep reinforcement learning

Neural Networks and Deep Learning

Machine Learning for signals, information, data

  • Introduction, maximum likelihood estimation

  • Linear regression, least squares, geometric view

  • Ridge regression, probabilistic views of linear regression

  • Bias-variance, Bayes rule, maximum a posteriori

  • Bayesian linear regression

  • Sparsity, subset selection for linear regression

  • Nearest neighbor classification, Bayes classifiers

  • Linear classifiers, perceptron

  • Logistic regression, Laplace approximation

  • Kernel methods, Gaussian processes

  • Maximum margin, support vector machines

  • Rrees, random forests

  • Boosting

  • Clustering, k-means

  • EM algorithm, missing data

  • Mixtures of Gaussians

  • Matrix factorization

  • Non-negative matrix factorization

  • Latent factor models, PCA and variations

  • Markov models

  • Hidden Markov models

  • Continuous state-space models

  • Association analysis

Heterogeneous Computing for Signal and Data Processing

  • Introduction

  • GPUs

  • OpenCL

  • CUDA

  • Portability and Scalability in HPC

  • Data Parallelism and Threads

  • Memory Hierarchy

  • Memory Allocation and Data Movement

  • Kernel-Based Parallel Programming

  • Memory Bandwidth and Coalescing

  • Matrix-Matrix Multiplications

  • Thread, Warps and Wavefronts

  • Thread Scheduling

  • Tiled Processing for 1D, 2D

  • Control Divergence

  • Convolution and Tiled Convolution

  • Reduction Kernels

  • Atomic Operations

  • Histogram Kernel

  • Applications: Deep Learning, Imaging, Video, …

  • Profiling and Debugging

next

Music Generation RNN