12 Parallel Infrastructure and Spark

This Block is unassessed except where it overlaps with other blocks. You may find the Parallel Data lecture helpful for Block 10 on parallel algorithms.

In this block we cover:

• Big Data
• Streaming
• Hadoop Distributed file system (HDFS)
• Hadoop MapReduce
• Spark overview
• Resilient Distributed Datasets (RDDs)
• Spark
• Accessing Spark through pyspark

Lectures

• Parallel data with MapReduce
  • 12.1 Parallel Data with MapReduce and Spark (Part 1, Introduction) (38:44)
• Spark
  • 12.2 Parallel Data with MapReduce and Spark (Part 2, Spark) (28:17)

Worksheets:

• Worksheet 12.1 Parallel Infrastructure

Workshop:

• The workshop this week involves considerable setup.
  • You are advised to do this first - it is discussed in the first video.
• All Workshop Content is accessed via the GitHub repository:
  • 12.2 Python Notebook: Workshop on Coding Parallel Algorithms
• Workshop Videos:
  • 12.2.0 Pyspark Install Notes (Workshop Part 0) (6:32)
  • 12.2.1 Hadoop (Workshop Part 1) (20:59)
  • 12.2.2 Pyspark (Workshop Part 2) (30:45)

References

• General parallel algorithms:
  • Streaming and Sketching
  • Parallel algorithms for dense matrix multiplication
• Map Reduce
  • Apache Hadoop
  • Gentle introduction to MapReduce
  • A Q&A
  • Lecture@poznan
  • Basic MapReduce Algorithms Design
  • Tutorialspoint Mapreduce
  • Hadoop for Streaming applications

  References:

• Laws governing data science:
  • Human Rights Act (HRA 1998)
  • EU General Data Protection Regulation (GDPR 2018)
  • Data Protection Act 2018 (DPA 2018)
• Privacy:
  • The Algorithmic Foundations of Differential Privacy by Dwork and Roth (2014).
  • ONS policy on disclosure control.
  • Sweeney 1997. Weaving technology and policy together to maintain confi-dentiality. Journal of Law, Medicines Ethics, 25:98–110.
  • Narayanan and Shmatikov 2008. Robust de-anonymization of largesparse datasets (how to break anonymity of the netflix prize dataset). IEEE Sec. and Priv.
  • Statistical Disclosure Attacks by George Danezis.
• Interpretability and fairness:
  • Book: “Interpretable Machine Learning: A Guide for Making Black Box Models Explainable.” Christoph Molnar 2019.
  • Algorithmic Bias Tutorial by Francesco Bonchi with Slides from KDD 2016
  • Hardt, Price and Srebo Equality of Opportunity in Supervised Learning 2016 explored in https://blog.acolyer.org/2018/05/07/equality-of-opportunity-in-supervised-learning/.
• Spark:
  • Spark RDD guide
  • pyspark
  • Spark RDD Transformations

Previous: Block 11.