Nuget framework for massive Microsoft Dataverse data operations.
Xrm.DataManager.Framework provides folling features :
- Batching patterns : Full, Delta, ...
- Performances : Perf tuning, multi threading, ...
- Errors handling and retry : API Limits catching and waits
- Tracing : Supports file output, App Insights and Graylog...
- Configuration management : Simplified conf ...
Xrm.DataManager.Framework is designed to simplify dev work by reducing core development and focus to functional implementation.
This project is open to community contributions in order to enrich this tooling and helps everyone to bring more capabilities for Microsoft Dataverse data operations.
Create a new C# console app projet and add Xrm.DataManager.Framework Nuget package.
You need to specify following parameters in 'app.config':
| Parameter name | Default or available values | Description |
|---|---|---|
| Crm.ConnectionString | Specify connection string to Microsoft Dataverse | |
| Job.Names | ex: DeleteAccountsJobs, DeleteContactsJob | Specify if you want to run multiple jobs (classe names splitted with ',') |
| Process.Duration.MaxHours | Default: 8 | Max job duration, to prevent nigh batch to run in prod during the day |
| Process.Query.RecordLimit | Default: 2500 | Records count to retrieve for each page (max 5000) |
| Process.Thread.Number | Default: 10 | Specify the number of threads to run |
| LogLevel | Verbose = 0, Information = 1, ErrorsAndSuccess = 2, ErrorsOnly = 3 |
Tracing level |
| AppInsights.Instrumentation.Key | Guid | Application Insights telemetry key |
| Graylog.Url | URL | Graylog Gelf endpoint URI |
3 patterns are available:
PickAndProcessDataJob: Query X rows from given QueryExpression and process them then loop as long as data is retrieved.FullProcessDataJob: Query all rows from given QueryExpression, load all data in memory and process.InputFileProcessDataJob: Map a csv or txt file to table/columns and process each line.
Theses patterns provide following overridable methods:
- GetName : Set a display name for job.
- IsEnabled : Specify if job is runnable or not.
- OverrideThreadNumber : Specify thread number for current job (For debug only)
- PreOperation : This method is called before proceeding data, you can use it to put data in cache for example or disable audit or whatever.
- GetQuery : Specify the query expression to retrieve data.
- PrepareData : Usefull with
FullProcessDataJobto apply data aggregation like groupings. - ProcessRecord : Apply custom operation for a single row.
- PostOperation : This method is called after all data have been processed, you can use it to send notification email or reactivate audit or whatever.