Deep Learning on Apache Spark™ : Workflows and Best Practices

On-Demand Webinar

The combination of Deep Learning with Apache Spark™ has the potential for tremendous impact in many sectors of the industry. This webinar, based on the experience gained in assisting customers with the Databricks Unified Analytics Platform, will present some best practices for building deep learning pipelines with Spark.


Rather than comparing deep learning systems or specific optimizations, this webinar will focus on issues that are common to deep learning frameworks when running on a Apache Spark™ cluster, including:

  • optimizing cluster setup;
  • configuring the cluster;
  • ingesting data; and
  • monitoring long-running jobs.

We will demonstrate the techniques we cover using Google’s popular TensorFlow library. More specifically, we will cover typical issues users encounter when integrating deep learning libraries with Spark clusters.
Clusters can be configured to avoid task conflicts on GPUs and to allow using multiple GPUs per worker. Setting up pipelines for efficient data ingest improves job throughput, and monitoring facilitates both the work of configuration and the stability of deep learning jobs.


Presenters

  • Tim Hunter

    Software Engineer

    Tim Hunter is a software engineer at Databricks and contributes to the Apache Spark MLlib project. He has been building distributed Machine Learning systems with Spark since version 0.2, before Spark was an Apache Software Foundation project.

  • Jules S. Damji

    Spark Community Evanglist

    Jules S. Damji is an Apache Spark Community Evangelist with Databricks. He is a hands-on developer with over 15 years of experience and has worked at leading companies building large-scale distributed systems.