06 Perceptrons and Neural Networks

In this block we cover:

• Introduction
  • Neurons
  • Single layer perceptron
  • Learning algorithms
• Deep Neural Networks
• Multi layer perceptron and the feed-forward neural network
  • Learning for deep neural networks
  • Other types of neural networks and their value:
    • Feed-forward
    • Convolutional
    • Recurrent
    • Recursive
    • Auto-encoders

Lectures:

Neural Nets and the Perceptron:

• 06.1 Neural Nets and the Perceptron
• 6.1.P Portfolio Discussion on Neural Nets and the Perceptron

CNNs and Transformers:

• 06.2 CNNs and Transformers
• 6.2.P Portfolio Discussion on CNNs and Transformers

Workshop:

• 06.3 Workshop on Pytorch for Computer Vision (.ipynb)
• Colab Notebook: Workshop 06.3 on Pytorch for Computer Vision

Assessments:

• The main long form portfolio Portfolio 6.
• Block06 on Noteable via Blackboard (2026-27 link)
• Group Sprint 1 on Classification is due this week; see Assessments.
• Group Sprint 2 on Neural Networks is set this week; see Assessments.

References

Neural Networks textbooks

• Chapter 11 of The Elements of Statistical Learning: Data Mining, Inference, and Prediction (Friedman, Hastie and Tibshirani).
• Russell and Norvig Artificial Intelligence: A Modern Approach
  • Chapter 20 Section 5: Neural Networks

Theoretical practicalities

• Bengio 2012 Practical Recommendations for Gradient-Based Training of Deep Architectures (in the book “Neural Networks: Tricks of the Trade”)
• Kull et al 2019 NeurIPS Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration
• Swish: Ramachandran, Zoph and Le Searching for Activation Functions

Important historical papers:

• McCulloch and Pitts (1943) A logical calculus of the ideas immanent in nervous activity
• Minsky and Papert 1969 Perceptrons
• Hecht-Nielsen, Robert. “Theory of the backpropagation neural network.” Neural networks for perception. Academic Press, 1992. 65-93.
• Bishop 1994 Mixture Density Networks

Likelihood and modelling applications of Neural Networks:

• Chilinski and Silva Neural Likelihoods via Cumulative Distribution Functions
• Albawi, Mohammed and Al-Zawi Understanding of a convolutional neural network
• Omi, Ueda and Aihara Fully Neural Network based Model for GeneralTemporal Point Processes

Implementations and Examples

• Neural Network Models in R

Navigation

Previous: Block 05. Next: Block 07.