Ben Hindman was a PhD student at AMPLab, the famous Berkeley lab for Algorithms, Machines, and People, when the Chief Scientist of Twitter asked him to help orchestrate their workflows. Ben was researching better ways to build distributed systems for parallel computing, a venerable Computer Science tradition going back decades.
AMPLab was a hotbed of applied research that lead to Apache Mesos, the system Ben built, pace Spark, that started as a demo app for Mesos, the computational genomics movement lead by David Patterson, and many other ideas that influenced the direction of Big Data and AI.
Mesos changed the way we view data centers. DC/OS was the Data Center Operating System managing jobs and resources, and from Twitter it went to AirBnB, Uber, Apple, and other web-scale companies. The whole idea of computers as cattle, not pets, and devops was powerfully enabled by Mesos. Kubernetes shares most things Mesos pioneered and the skills are transportable.
I sat down with Ben at D2IQ, the company scaling cloud as an OS, to talk about the history and the future of cloud management.