NYC Open Data: Streaming Python on Hadoop

NYC Open Data: Streaming Python on Hadoop

Posted by Vivian Zhang

Updated: Sep 19, 2015

Our upcoming 12-Week Data Science bootcamp starts on January, 11th, 2016. Apply now to secure a spot in our winter cohort!

In the meantime, come join the NYC Open Data Meetup Group and learn how easily you can use Hadoop for Machine Learning.

If you are hiring Data Scientists, call us at +1-888-752-7585 (USA) or reach us at to share your openings and set up interviews with our excellent students.


What is an NYC Open Data Meetup event like? Here’s an example:

This summer, Sam Kamin (Vice President of Engineering, NYC Data Science Academy) gave a teaser version of NYC Data Science Academy’s five-week Hadoop course. (The current version on offer is a six-week evening program, Big Data with Hadoop and Spark.”) See below for a class syllabus from the five-week Hadoop course.

First, though, here are the slides from the Meetup:

(Slides can also be accessed here through SlideShare.)

Plus video of the event:

  Speaker Bio:
Sam Kamin is currently a Vice President of Engineering at NYC Data Science Academy. He is also Associate Professor Emeritus from the University of Illinois Champaign Urbana where he taught computer science for over 20 years and was head of the undergraduate program. Most recently he was an engineer at Google before joining NYC Data Science Academy.
Now for the five-week Hadoop course: 

This five-week course is an intensive, hands-on introduction to the Hadoop ecosystem of Big Data technologies. The emphasis in this course is on learning several of the major components of Apache Hadoop – HDFS, MapReduce, Hive, Pig, Streaming – by doing exercises of increasing complexity. Programming will be done in Python. Students are expected to be familiar with using an operating system from the command line; knowledge of Python is helpful; the material in Learn Python the Hard Way is sufficient background knowledge. The course format is mixed lecture/lab. Students will need to bring their own laptops to connect to our server; instructions will be provided ahead of time as to how to install any required software.

Week 1 – Introduction: MapReduce Overview of Big Data and the Hadoop ecosystem the concept of MapReduce
HDFS – Hadoop Distributed File System
MapReduce with Python streaming

Week 2 – More on MapReduce
More on Big Data, the Hadoop ecosystem, and MapReduce.
Mixed case studies and exercises using MR with Python streaming

Week 3 – Hive: A database for Big Data
Hive concepts
User-defined functions in the Hive language
User-defined functions in Python (using streaming)
Advanced topic: Hive queries in Python code

Week 4 – Pig: Simplified MapReduce
Basic concepts
Pig Latin
Pig functions and macros
User-defined functions

Week 5 – Project day

The Hadoop ecosystem
Brief intro to Spark
Brief intro to Mahout
Case studies/project ideas

Vivian Zhang

Vivian is the CTO and School Director of NYC Data Science Academy and CTO of SupStat. With her extensive experience working in the data science field, she developed expertise in multiple programming languages, including R, Python, Hadoop, and Spark. In August 2016, Forbes ranked her amongst one of the nine women leading the pack in data analytics. In 2013, she created the NYC Open Data Meetup group, which stands as one of the largest data science communities offering meetups, conferences, and a weekly newsletter. In her spare time, Vivian enjoys meeting people and sharing her motivational stories with our students and other professionals

View all articles

Topics from this blog: Community

Interested in becoming a Data Scientist?

Answer 3 Simple Questions and Get Immediate Course Recommendations.