Reading List

A growing collection of papers, articles, books, literature, and ideas I enjoy & recommend¹

By topic: Deep Learning, Philosophy, Fiction, Film, Words

Updated every month or so.

Deep Learning

See an additional curated list of great papers discussed in my first book as case studies here.

Deep Learning Theory (“Theory” 😉)

• “Beyond Neural Scaling Laws: Beating Power Law Scaling via Data Pruning” by Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, Ari Morcos
• “The Forward-Forward Algorithm: Some Preliminary Investigations” by Geoffrey Hinton
• “Learning in High Dimension Always Amounts to Extrapolation” by Randall Balestriero, Jerome Pesenti, and Yann LeCun
• “The Lottery Ticket Hypothesis” by Jonathan Frankle and Michael Carbin
• “Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask” by Hattie Zhou et al.
• “Deep Double Descent” by Preetum Nakkiran et al.
• “Adversarial Examples Are Not Bugs, They Are Features” by Andrew Ilyas et al.
• “Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets” by Alethea Power et al.
• “What’s Hidden in a Randomly Weighted Neural Network?” by Vivek Ramanjuan and Michell Wortsman
• “The Modern Mathematics of Deep Learning” by Julius Berner et al.
• “Learning to learn by gradient descent by gradient descent” by Marcin Andrychowicz et al.
• “On Exact Computation with an Infinitely Wide Neural Net” by Sanjeev Arora
• “Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning” by Yarin Gal and Zoubin Ghahramani

Bias and Fairness

• “On the (im)possibility of fairness” by Sorelle A. Friedler et al.
• “What Do Compressed Deep Neural Networks Forget?” by Sarah Hooker et al.
• “Characterising Bias in Compressed Models” by Sara Hooker et al.
• “Moving Beyond ‘Algorithmic Bias is a Data Problem’” by Sara Hooker

Computer Vision

• “DeepInsight: A methodology to transform a non-image data to an image for convolution neural network architecture”
• “Explaining and Harnessing Adversarial Examples by Ian J. Goodfellow et al.
• “Robust Physical-World Attacks on Deep Learning Visual Classification” by Kevin Eykholt & Ivan Evtimov et al.
• “EfficientNet: Rethinking Model Scaling for Convoltuional Neural Networks” by Mingxing Tan et al.
• “Natural Language Descriptions of Deep Visual Features” by Evan Hernandez et al.
• “Feature Visualization” by Chris Olah et al.
• “ResMLP: Feedforward networks for image classification with data-efficient training”

Language Modeling

• “Why Can GPT Learn In-Context? Language Models Secretly Perform Gradient Descent as Meta-Optimizers” by Damai Dai et al.
• “Task-Aware Representation of Sentences for Generic Text Classification” by Kishaloy Halder et al.
• “General-Purpose Question-Answering with MACAW” by Oyvind Tafjord et al.
• “Single Headed Attention RNN: Stop Thinking With Your Head” by Stephen Merity
• “GPT-D: Inducing Dementia-related Linguistic Anomalies by Deliberate Degradation of Artificial Neural Language Models” by Changye Li et al.
• “Predictability and Surprise in AI Models” by Deep Ganguli et al.
• “A Mathematical Framework for Transformer Circuits” by Nelson Elhage
• “Language Models as Knowledge Bases?” by Fabio Petroni
• “An Attention Free Transformer” by Shuangfei ZHai et al.
• “Transformers are Graph Neural Networks” by Chaitanya Joshi
• “Generating Words from Embeddings” from Rajat’s Blog

Emergent Language

• “Compositional Observer Communication Learning from Raw Visual Input” by Edward Choi, Angeliki Lazaridou, Nando de Freitas
• “Emergence of Grounded Compositional Language in Multi-Agent Populations” by Igor Mordatch and Pieter Abbeel
• “On the Spontaneous Emergence of Discrete and Compositional Signals” by Nur Geffen Lan, Emmanuel Chemla, and Shane Steinert-Threlkel
• “Learning Emergent Discrete Message Communication for Cooperative Reinforcement Learning” by Sheng Li, Yutai Zhou, Ross Allen, Mykel J. Kochenderfer
• “AI Programmer: Autonomously Creating Software Programs Using Genetic Algorithms” by Kory Becker and Justin Gottschlich

Meta Stuff

• “On the Measure of Intelligence” by Francois Chollet
• “Computing Machinery and Intelligence” by Alan Turing
• “The Bitter Lesson” by Rich Sutton

Shameless Self-Promotion

• Modern Deep Learning Design and Application Development by Andre Ye
• Deep Learning for Tabular Data by Andre Ye and Andy Wang

Philosophy

Currently Reading

• The Poverty of Ethics by Anat Amar
• On the Suffering of the World by Arthur Schopenhauer
• Queer Phenomenology by Sara Ahmed
• Silicon Valley of Dreams by David Pellow and Lisa Sun-Hee Park
• Bad Gays by Huw Lemmey and Ben Miller

Pretending to be Currently Reading

• Capital by Karl Marx
• Of Grammatology by Derrida
• The Sublime Object of Ideology by Slavoj Zizek

Next on the reading List

• Full Surrogacy Now by Sophie Lewis
• The Politics of Friendship by Jacques Derrida

Film

Ultimate Favorites

• Summer of ‘85
• Brokeback Mountain
• La La Land
• Parasite

Pretty Good

• The Menu
• Synecdoche: New York
• 2001: A Space Odyssey
• Interstellar
• A Quiet Place
• Call Me By Your Name
• Ex Machina
• The White Lotus
• Everything Everywhere All At Once
• Triangle of Sadness
• Tár

Fiction

Currently Reading

• None :(

Pretending to be Currently (Re-)Reading

• Tender is the Flesh by Agustina Bazterrica
• The Idiot by Fyodor Dostoevsky
• Infinite Jest by David Foster Wallace

The Good Stuff

• Crime and Punishment by Fyodor Dostoevsky
• Infinite Jest by David Foster Wallace
• Never Let Me Go by Kazuo Ishiguro