I met Matei, the creator of Apache Spark, when he was a graduate student at Berkeley, when I ran into problems with running Spark on AWS. Then we started the very first Spark meetup at SF Scala, and Matei lead the revolution in Big Data that made it interactive at scale. Databricks, the company he cofounded, is now a unicorn, and Spark is the choice for ETL and ML/AI workloads the world over.
In this episode, we sat down at the Databricks office, overlooking downtown San Francisco, to talk about the direction that data engineering and data science is taking, and how Spark meets the demands of today and tomorrow at every turn. We also mention Weld and Rust, and other interesting topics.