Video | Strata + Hadoop World NYC 2016 | “The Evolution of Massive Scale Data Processing”

In this video, Tyler Akida presents a whirlwind tour of the evolution of massive-scale data processing at Google, from the original MapReduce paradigm to the high-level pipelines of Flume to the streaming approach of MillWheel to the portable, unified streaming/batch model of Google Cloud Dataflow and Apache Beam (incubating).

Tyler also highlights similarities and differences with related open source systems such as Flink, Spark, Storm, and Gearpump, calling out ways in which they’re converging on and diverging from the Beam model and what that means when running Beam pipelines on their respective runners. Watch Video

VIDEO | NYC Machine Learning Meetup 2016 | Dan Melamed “How To Learn From What Your Users Might Not See”

At the Machine Learning Meetup in NYC, Dan Melamed gave a machine learning talk titled: “How To Learn From What Your Users Might Not See”. This talk will focus on contextual bandits and their applications.

In this tutorial, Dan will show how to learn from such data in a principled, efficient, and unbiased manner. The techniques that he will describe were largely responsible for a click-thru rate gain of over 25% on Watch Video

“10 Myths About Machine Learning” – (Prof. Pedro Domingos)

Machine Learning is at the core of data science and we see it’s applications all over now (i.e. recommender engines, etc.). As Pedro Domingos’s Professor of Computer Science U. Washington writes in the piece, “In reality, the main purpose of machine learning is to predict the future.” It’s important to be aware of the MYTHs associated with Machine Learning. Read More

How AirBnB Scaled from 5 to 70+ Data Scientists in 2 Years (via Kaggle)

In 2013, Airbnb had a small, centralized team of five data scientists serving the data needs of the company. Since then, they have grown to become one of the largest, most innovative startup teams with over 70 data scientists now serving separate business units. In addition to setting a consistently high bar on new hires and focusing on technical mentorship from peers, the structure of the organization has been key to successful growth. Read More

50+ Open Source Tools for Big Data

Open source software tools have become all the rage, especially around big data and that is a GOOD thing. It allows for many players to work off of the same code base to build more add-on tools and it’s cheap and easy for the masses to get set up and use them. Hadoop, R, Cassandra, Mongo DB, Neo4i and HBase are among the most popular, but there are many more.

I have accumulated 3 lists that are very popular. Please let me know if you see things missing and I’ll attempt to create one large master list and post it on the site. Read More…

Blog Publisher / Head of Data Science Search

Founder & Head of Data Science Search at Starbridge Partners, LLC.