11 Parallel Infrastructure and Spark

In this block we cover:

• Big Data
• Streaming
• Hadoop Distributed file system (HDFS)
• Hadoop MapReduce
• Spark overview
• Resilient Distributed Datasets (RDDs)
• Spark
• Accessing Spark through pyspark

Lectures

• Parallel data with MapReduce and Spark
  • 11.1.1 Parallel Data with MapReduce and Spark (Part 1, Introduction) (38:44)
    • Slides
  • 11.1.2 Parallel Data with MapReduce and Spark (Part 2, Spark) (28:17)
    • Slides

Worksheets:

• Worksheet 11.1 Parallel Infrastructure

Workshop:

• The workshop this week involves considerable setup, and the use of BlueCrystal Phase 4 (or setting up your own environment…).
  • You are advised to do this first - it is discussed in the first video.
  • There are two ways to achieve the learning. The first is by getting Spark working on your local machine so that you can access
• All Workshop Content is accessed via the GitHub repository:
  • 11.2 Python Notebook: Workshop on Coding Parallel Algorithms
• Video Lectures:
  • 11.2.0 Pyspark Install Notes (Workshop Part 0) (6:32)
  • 11.2.1 Hadoop (Workshop Part 1) (20:59)
  • 11.2.2 Pyspark (Workshop Part 2) (30:45)

References

• General parallel algorithms:
  • Streaming and Sketching
  • Parallel algorithms for dense matrix multiplication
• Map Reduce
  • Apache Hadoop
  • Gentle introduction to MapReduce
  • A Q&A
  • Lecture@poznan
  • Basic MapReduce Algorithms Design
  • Tutorialspoint Mapreduce
  • Hadoop for Streaming applications
• Spark:
  • Spark RDD guide
  • pyspark
  • Spark RDD Transformations

Previous: Block 10. Next: Block 12.