07 Perceptrons and Neural Networks

In this block we cover:

• Introduction
  • Neurons
  • Single layer perceptron
  • Learning algorithms
• Deep Neural Networks
• Multi layer perceptron and the feed-forward neural network
  • Learning for deep neural networks
  • Other types of neural networks and their value:
    • Feed-forward
    • Convolutional
    • Recurrent
    • Recursive
    • Auto-encoders

Lectures

• Neural Nets and the Perceptron
• Practicalities of Neural Nets

Worksheets:

• Worksheet 7.1 Neural Networks
• Portfolio 07

Assessments:

• Assessment 2 will be set in this week; see Assessments. This is a summative assessment (i.e. does contribute to your grade) and will be due in Week 12.

Workshop:

• Python Notebook: Workshop 07.3 on Tensorflow with Keras
  • Neural Network run as a script.
  • You should understand how to run python as a script.
    • This is run (for me) with python block07-NeuralNetworksScript.py
    • Remember to put the data and python script in the directory you are running in!

References

Neural Networks textbooks

• Chapter 11 of The Elements of Statistical Learning: Data Mining, Inference, and Prediction (Friedman, Hastie and Tibshirani).
• Russell and Norvig Artificial Intelligence: A Modern Approach
  • Chapter 20 Section 5: Neural Networks

Theoretical practicalities

• Bengio 2012 Practical Recommendations for Gradient-Based Training of Deep Architectures (in the book “Neural Networks: Tricks of the Trade”)
• Kull et al 2019 NeurIPS Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration
• Swish: Ramachandran, Zoph and Le Searching for Activation Functions

Important historical papers:

• McCulloch and Pitts (1943) A logical calculus of the ideas immanent in nervous activity
• Minsky and Papert 1969 Perceptrons
• Hecht-Nielsen, Robert. “Theory of the backpropagation neural network.” Neural networks for perception. Academic Press, 1992. 65-93.
• Bishop 1994 Mixture Density Networks

Likelihood and modelling applications of Neural Networks:

• Chilinski and Silva Neural Likelihoods via Cumulative Distribution Functions
• Albawi, Mohammed and Al-Zawi Understanding of a convolutional neural network
• Omi, Ueda and Aihara Fully Neural Network based Model for GeneralTemporal Point Processes

Implementations and Examples

• Neural Network Models in R
• Installing Tensoflow

Previous: Block 06. Next: Block 08.