NYC Open Data: Streaming Python on Hadoop

Our upcoming 12-Week Data Science bootcamp starts on January, 11th, 2016. Apply now to secure a spot in our winter cohort!

In the meantime, come join the NYC Open Data Meetup Group and learn how easily you can use Hadoop for Machine Learning.

If you are hiring Data Scientists, call us at +1-888-752-7585 (USA) or reach us at info@nycdatascience.com to share your openings and set up interviews with our excellent students.

What is an NYC Open Data Meetup event like? Here’s an example:

This summer, Sam Kamin (Vice President of Engineering, NYC Data Science Academy) gave a teaser version of NYC Data Science Academy’s five-week Hadoop course. (The current version on offer is a six-week evening program, “Big Data with Hadoop and Spark.”) See below for a class syllabus from the five-week Hadoop course.

First, though, here are the slides from the Meetup:

(Slides can also be accessed here through SlideShare.)

This five-week course is an intensive, hands-on introduction to the Hadoop ecosystem of Big Data technologies. The emphasis in this course is on learning several of the major components of Apache Hadoop – HDFS, MapReduce, Hive, Pig, Streaming – by doing exercises of increasing complexity. Programming will be done in Python. Students are expected to be familiar with using an operating system from the command line; knowledge of Python is helpful; the material in Learn Python the Hard Way is sufficient background knowledge. The course format is mixed lecture/lab. Students will need to bring their own laptops to connect to our server; instructions will be provided ahead of time as to how to install any required software.

SYLLABUS
Week 1 – Introduction: MapReduce Overview of Big Data and the Hadoop ecosystem the concept of MapReduce
HDFS – Hadoop Distributed File System
MapReduce with Python streaming

Week 2 – More on MapReduce
More on Big Data, the Hadoop ecosystem, and MapReduce.
Mixed case studies and exercises using MR with Python streaming

Week 3 – Hive: A database for Big Data
Hive concepts
HiveQL
User-defined functions in the Hive language
User-defined functions in Python (using streaming)
Advanced topic: Hive queries in Python code

Week 4 – Pig: Simplified MapReduce
Basic concepts
Pig Latin
Pig functions and macros
User-defined functions

Week 5 – Project day

The Hadoop ecosystem
Brief intro to Spark
Brief intro to Mahout
Case studies/project ideas