Skip to content

Latest commit

 

History

History
68 lines (49 loc) · 4.8 KB

readme.md

File metadata and controls

68 lines (49 loc) · 4.8 KB
Raw

Trimble2Kusto

Purpose

Ingest real-time data from Trimble via SignalR/WebSocket into Kusto aka. Azure Data Explorer, Synapse Data Explorer dedicated-pool, and Microsoft Fabric KQL Database in Real-Time Intelligence.

Description

This repo contains sample DotNet Console App program in C# that uses WebSocket, SignalR, Kusto SDK. The program will connect to the Trimble Notification Service, provided a valid api-key, via WebSocket and streams messages (service-telemetry, GPS, ETA, OoC, Weather, etc.) into Kusto using DotNet SDK client libraries for real-time analytics.

The sample assets in this repo are not production solutions. They are samples for proof-of-concept or minimal-viable-product at best. The pipelines establish a data workflow for analytics end2end, including dashboard visualizations.

If you're wondering, what is Kusto, check out these videos:

  1. The mechanics of Real-Time Intelligence in Microsoft Fabric
  2. Real-Time Intelligence in Microsoft Fabric
  3. ADX at a glance
  4. How customers are using it?

Summary of Console App

Example of using WebSocket, Trimble Notification Service, Kusto SDK, and SignalR to stream messages into Kusto.

Dependencies

  • .NET 8.0.201
  • NuGet packages:
    • Microsoft.AspNetCore.SignalR.Client
    • Microsoft.Azure.Kusto.Data
    • Microsoft.Azure.Kusto.Ingest

For your local IDE, recommend using VSCode with extensions C# Dev Kit, and Nuget Package Manager. Install the package dependencies using PowerShell terminal in VSCode by entering command: dotnet add package <package-name>

Additionally, StreamingIngestionSample console app is included in this repo to test ingestion & Kusto client libraries using fictitious data.

Additional resources

Remarks

This sample Program.cs assumes:

  1. A valid Trimble API key with the TM add-on.
  2. Your Kusto cluster has streaming enabled. (Streaming is enabled by default for Fabric KQL Database & ADX Free-Personal Clusters)
  3. Your Kusto database has a table with the following schema:
.create table TrimbleNotificationRaw (message:dynamic)

Running Program.cs

  1. Add required DotNet packages.
  2. Enter the required parameters, then build the program.
  3. Deploy the program as an Azure Function when ready.
  4. Sample Azure Function - SignalRToKusto

Conceptual Pipelines

  1. Batching: SQL on-prem > csv file contains +1K "OFD/Active TripIds" (Query & File-transfer) > GET TrimbleAPI TripEvents (using Powershell) > store response in a file per Trip > Load local files to Fabric KQL Database using Get Data > run KQL queries & stored functions to flatten data for analytics > visualize query results in PBI or RTA Dashboard > Apply PBI filters for Dashboard use and parameters for RTA Dashboard.

    in summary...

    SQL > csv file > Trimble API > json files > Fabric KQL DB > KQL queries > PBI + RTA Dashboard.

  2. Streaming: Trimble Notification Service > C# Client App (via Streaming) > Fabric KQL DB > KQL queries > Dashboards (TBD).

DISCLAIMER

This Sample Code is provided for the purpose of illustration only and is not intended to be used in a production environment. THIS SAMPLE CODE AND ANY RELATED INFORMATION ARE PROVIDED "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY AND/OR FITNESS FOR A PARTICULAR PURPOSE. We grant You a nonexclusive, royalty-free right to use and modify the Sample Code for your internal use only, provided that You agree: (i) to not use Our name, logo, or trademarks to market Your software product in which the Sample Code is embedded; (ii) to include a valid copyright notice on Your software product in which the Sample Code is embedded; and (iii) to indemnify, hold harmless, and defend Us and Our suppliers from and against any claims or lawsuits, including attorneys' fees, that arise or result from the use or distribution of the Sample Code.