Skip to main content
SpringerLink
Log in
Apress
Book cover

The Definitive Guide to Azure Data Engineering

Modern ELT, DevOps, and Analytics on the Azure Cloud Platform

  • Book
  • © 2021

Overview

Authors:
  1. Ron C. L'Esteve

    1. Chicago, USA

    View author publications

    You can also search for this author in PubMed   Google Scholar

  • Provides step-by-step examples of Azure data engineering concepts and solutions
  • Promotes a standard for data engineering excellence through quality patterns and practices
  • Leaves readers with valuable skills in Azure data engineering that lead to high-performing solutions

This is a preview of subscription content, log in via an institution to check access.

Access this book

Log in via an institution
eBook USD 44.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book USD 59.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Other ways to access

Licence this eBook for your library

Institutional subscriptions

Table of contents (24 chapters)

  1. Front Matter

    Pages i-xxiii
    Download chapter PDF

  2. Getting Started

    1. Front Matter

      Pages 1-1
      Download chapter PDF

    2. The Tools and Prerequisites

      • Ron C. L’Esteve
      Pages 3-33

    3. Data Factory vs. SSIS vs. Databricks

      • Ron C. L’Esteve
      Pages 35-39

    4. Design a Data Lake Storage Gen2 Account

      • Ron C. L’Esteve
      Pages 41-54

  3. Azure Data Factory for ELT

    1. Front Matter

      Pages 55-55
      Download chapter PDF

    2. Dynamically Load a SQL Database to Data Lake Storage Gen2

      • Ron C. L’Esteve
      Pages 57-81

    3. Use COPY INTO to Load a Synapse Analytics Dedicated SQL Pool

      • Ron C. L’Esteve
      Pages 83-91

    4. Load Data Lake Storage Gen2 Files into a Synapse Analytics Dedicated SQL Pool

      • Ron C. L’Esteve
      Pages 93-113

    5. Create and Load Synapse Analytics Dedicated SQL Pool Tables Dynamically

      • Ron C. L’Esteve
      Pages 115-131

    6. Build Custom Logs in SQL Database for Pipeline Activity Metrics

      • Ron C. L’Esteve
      Pages 133-146

    7. Capture Pipeline Error Logs in SQL Database

      • Ron C. L’Esteve
      Pages 147-172

    8. Dynamically Load a Snowflake Data Warehouse

      • Ron C. L’Esteve
      Pages 173-214

    9. Mapping Data Flows for Data Warehouse ETL

      • Ron C. L’Esteve
      Pages 215-236

    10. Aggregate and Transform Big Data Using Mapping Data Flows

      • Ron C. L’Esteve
      Pages 237-257

    11. Incrementally Upsert Data

      • Ron C. L’Esteve
      Pages 259-294

    12. Load Excel Sheets into Azure SQL Database Tables

      • Ron C. L’Esteve
      Pages 295-320

    13. Delta Lake

      • Ron C. L’Esteve
      Pages 321-346

  4. Real-Time Analytics in Azure

    1. Front Matter

      Pages 347-347
      Download chapter PDF

    2. Stream Analytics Anomaly Detection

      • Ron C. L’Esteve
      Pages 349-381
Back to top

Keywords

About this book

Build efficient and scalable batch and real-time data ingestion pipelines, DevOps continuous integration and deployment pipelines, and advanced analytics solutions on the Azure Data Platform. This book teaches you to design and implement robust data engineering solutions using Data Factory, Databricks, Synapse Analytics, Snowflake, Azure SQL database, Stream Analytics, Cosmos database, and Data Lake Storage Gen2. You will learn how to engineer your use of these Azure Data Platform components for optimal performance and scalability. You will also learn to design self-service capabilities to maintain and drive the pipelines and your workloads. 



The approach in this book is to guide you through a hands-on, scenario-based learning process that will empower you to promote digital innovation best practices while you work through your organization’s projects, challenges, and needs. The clear examples enable you to use this book as a reference and guide for building data engineering solutions in Azure. After reading this book, you will have a far stronger skill set and confidence level in getting hands on with the Azure Data Platform.





What You Will Learn
  • Build dynamic, parameterized ELT data ingestion orchestration pipelines in Azure Data Factory
  • Create data ingestion pipelines that integrate control tables for self-service ELT
  • Implement a reusable logging framework that can be applied to multiple pipelines
  • Integrate Azure Data Factory pipelines with a variety of Azure data sources and tools
  • Transform data with Mapping Data Flows in Azure Data Factory
  • Apply Azure DevOps continuous integration and deployment practices to your Azure Data Factory pipelines and development SQL databases
  • Design and implement real-time streaming and advanced analytics solutions using Databricks, Stream Analytics, and Synapse Analytics
  • Get started with a variety of Azure data services through hands-on examples




Who This Book Is For


Data engineers and data architects who are interested in learning architectural and engineering best practices around ELT and ETL on the Azure Data Platform, those who are creating complex Azure data engineering projects and are searching for patterns of success, and aspiring cloud and data professionals involved in data engineering, data governance, continuous integration and deployment of DevOps practices, and advanced analytics who want a full understanding of the many different tools and technologies that Azure Data Platform provides




Authors and Affiliations

  • Chicago, USA

    Ron C. L'Esteve

About the author

​Ron L’Esteve is a professional author residing in Chicago, IL, USA. His passion for Azure Data Engineering stems from his deep experience with implementing, leading, and delivering Azure Data projects for numerous clients. He is a trusted architectural leader and digital innovation strategist, responsible for scaling key data architectures, defining the road map and strategy for the future of data and business intelligence (BI) needs, and challenging customers to grow by thoroughly understanding the fluid business opportunities and enabling change by translating them into high quality and sustainable technical solutions that solve the most complex business challenges and promote digital innovation and transformation. Ron has been an advocate for data excellence across industries and consulting practices, while empowering self-service data, BI, and AI through his contributions to the Microsoft technical community.

Bibliographic Information

Publish with us

Policies and ethics

Back to top

Search

Navigation