Apache® Spark™ is being widely adopted as the general processing engine by organizations of all sizes for processing large-scale data workloads. To effectively run mission-critical production workloads on Spark, data teams spend significant effort and time creating and maintaining a complex technology stack that can help deploy secure, highly available, and multi-tenant Spark clusters, simplify data exploration and the development of Spark applications, and push Spark applications into production and monitor their execution.

This whitepaper shows how Databricks enables developers, data engineering, and data science teams to focus on getting value out of their data instead of being bogged down with operational details by providing three sets of critical functionalities out-of-the-box for its users:

• Easily scale up or down fully managed Spark clusters with just a few clicks
• Interactively explore data, write code, and debug applications in a single environment
• Schedule production jobs and get from prototype to production without re-engineering