Skip to content

vplauzon/real-time-lab

Repository files navigation

Real Time Analytics in Azure Lab

Hands on lab for real time analytics in Azure.

This is a one day hands-on lab on Real Time Analytics in Azure.

Azure services used in this lab:

Preparation

Getting familiar with Kusto, Azure Data Explorer query language. We suggest the following resources:

Design Session

Look at the business scenario and requirements of the hands on lab.

Break into teams of 3 to 8 individuals to design a target solution on Azure.

The solution should address each of the business requirements from the previous section. Draw a diagram with the main components.

Each team will then present their solution.

Prefered solution

The instructor will present the prefered solution.

The hands on lab modules are based on that solution.

Hands on Lab Modules

Each module contains a challenge and objectives.

They also contain a suggested solution.

We recommend trying the modules without looking at the suggested solution first. This would maximize the challenges and therefore the benefits.

Module 1 - Know the data

In this first module, we'll explore the data we are going to work with.

Objectives:

  • Provision an Azure Data Explorer (Kusto) cluster
  • Query telemetry samples
  • Prepare ingestion by developing queries to transform the raw data

Go to module 1 instructions.

Module 2 - Setup real-time ingestion

In this module we will setup the real time ingestion of data into our Kusto Cluster.

Objectives:

  • Setup a simulator of IoT data for Azure Event Hub
  • Continuously ingest raw data in Azure Data Explorer

Go to module 2 instructions.

Module 3 - Transform the data at ingestion time

In this module we will take the raw JSON data we setup for continuous ingestion in Module 2 and continuously transform it into strongly-typed data, using the queries we developed in module 1.

Objectives:

  • Get strong-type table populated in near-real time

Go to module 3 instructions.

Module 4 - Querying real time data

Now that we have data ingested in real time, we are going to query it to get insights.

Objectives:

  • Look at ingestion latency
  • Query and chart data
  • Look at Azure Data Explorer metrics (monitoring)

Go to module 4 instructions.

Module 5 - Time series

In this module, we'll query to the next level. We'll look at each drone's telemetry as its own time series and try to find anomalies.

Objectives:

  • Get comfortable with time series
  • Analyse failures in drones using time series tooling

Go to module 5 instructions.

Module 6 - Imperfect telemetry

  • Changing upstream process to reveal "real" telemetry with late arrivals + duplicates
  • Introduce ASA to the rescue

Module 7 - Integrate external data

In this module we are going to integrate external data from Azure SQL DB to enrich the ingested data.

  • Ingest a couple of reference data tables (or reference?)
  • Author update policies to transform the data using reference tables
  • Query some more
  • Setup continuous exporting

Go to module 5 instructions.

Module 8 - Reporting

  • Setup Power BI to query near real time data

About

Hands on lab for real time analytics in Azure

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published