The idea of DevOps revolutionised software development by making it feasible to develop, integrate and test software in a continuous manner, resulting in faster time-to-value and a lower overall cost of deployment. The need to continually innovate with data in a production setting has led to the inception of two related concepts: “DataOps” and “MLOps”, which seek to bring these same benefits of reduced cycle times and reduced risk to the spheres of data analytics and machine learning, respectively.

In this workshop, we will demystify these two ideas, discuss the basis for their sudden popularity and relevance, and walk you through in a hands-on fashion how they can be implemented in production. This workshop is designed to be a learning opportunity for practitioners like you, so please bring your laptop so you can work through the examples with us, live.

We will open the workshop with a common discussion about data in a production setting:

• Why DataOps?
• Making data a first-class citizen
• Handling evolving schemas elegantly with Delta Lake
• Managing data quality with Delta Lake
• Monitoring data quality with Statistical Process Control

AGENDA AT A GLANCE

 

02.00pm - 02.30pm - Check-in & Refreshments

02.30pm - 02.50pm - Accelerating collaboration between data science & data engineering 

02.50pm - 03.00pm - Success story

03.00pm - 04.20pm - Workshops: DataOps 

04.00pm - 04.15pm - Wrap-up

04.15pm - 05.30pm - Networking drinks & technical questions to the experts

Andy Weaver, Solution Architect, Databricks