12 Parallel Infrastructure and Spark

This Block is unassessed except where it overlaps with other blocks. You may find the Parallel Data lecture helpful for Block 10 on parallel algorithms.

In this block we cover:

• Big Data
• Streaming
• Hadoop Distributed file system (HDFS)
• Hadoop MapReduce
• Spark overview
• Resilient Distributed Datasets (RDDs)
• Spark
• Accessing Spark through pyspark

Lectures

• Parallel data with MapReduce
  • 12.1 Parallel Data with MapReduce and Spark (Part 1, Introduction) (38:44)
• Spark
  • 12.2 Parallel Data with MapReduce and Spark (Part 2, Spark) (28:17)

Workshop:

• The workshop this week involves considerable setup.
  • You are advised to do this first - it is discussed in the first video.
• All Workshop Content is accessed via the GitHub repository:
  • 12.2 Python Notebook: Workshop on Coding Parallel Algorithms
• Workshop Videos:
  • 12.2.0 Pyspark Install Notes (Workshop Part 0) (6:32)
  • 12.2.1 Hadoop (Workshop Part 1) (20:59)
  • 12.2.2 Pyspark (Workshop Part 2) (30:45)

References

• General parallel algorithms:
  • Streaming and Sketching
  • Parallel algorithms for dense matrix multiplication
• Map Reduce
  • Apache Hadoop
  • Gentle introduction to MapReduce
  • A Q&A
  • Lecture@poznan
  • Basic MapReduce Algorithms Design
  • Tutorialspoint Mapreduce
  • Hadoop for Streaming applications

Worksheets (unassessed)

• Worksheet 12.1 Parallel Infrastructure

Navigation:

Previous: Block 11.