From aa92272404f327e8d866c2a845ab1b2a36c7d288 Mon Sep 17 00:00:00 2001 From: Ed Ball Date: Tue, 4 Jun 2024 15:03:14 -0700 Subject: [PATCH] Update docs. --- BulkInsert.md | 51 ++++++++++++++++++ Faithlife.Utility.Dapper.md | 11 ++++ Faithlife.Utility.Dapper/BulkInsertUtility.md | 25 +++++++++ .../BulkInsertUtility/BulkInsert.md | 31 +++++++++++ .../BulkInsertUtility/BulkInsertAsync.md | 32 +++++++++++ .../GetBulkInsertCommands.md | 32 +++++++++++ README.md | 2 +- _layouts/default.html | 10 ++-- assets/github.png | Bin 2541 -> 4044 bytes assets/page.css | 7 +-- index.md | 12 +++-- 11 files changed, 197 insertions(+), 16 deletions(-) create mode 100644 BulkInsert.md create mode 100644 Faithlife.Utility.Dapper.md create mode 100644 Faithlife.Utility.Dapper/BulkInsertUtility.md create mode 100644 Faithlife.Utility.Dapper/BulkInsertUtility/BulkInsert.md create mode 100644 Faithlife.Utility.Dapper/BulkInsertUtility/BulkInsertAsync.md create mode 100644 Faithlife.Utility.Dapper/BulkInsertUtility/GetBulkInsertCommands.md diff --git a/BulkInsert.md b/BulkInsert.md new file mode 100644 index 0000000..447bb99 --- /dev/null +++ b/BulkInsert.md @@ -0,0 +1,51 @@ +# BulkInsert + +The [`BulkInsert`](Faithlife.Utility.Dapper/BulkInsertUtility/BulkInsert.md) (and [`BulkInsertAsync`](Faithlife.Utility.Dapper/BulkInsertUtility/BulkInsertAsync.md) extension methods allow efficient insertion of many rows into a database table with a familiar Dapper-like API. + +## Problem + +Dapper already has a mechanism for "bulk insert". Calling `Execute` with an `IEnumerable` will execute the specified `INSERT` command once for each item in the sequence. Unfortunately, this can be an extremely slow way to insert a large number of rows into a database. + +Each DBMS has its own preferred approaches for efficiently inserting many rows into a database, but the most portable way is to execute an `INSERT` command with multiple rows in the `VALUES` clause, like so: + +``` +INSERT INTO widgets (name, size) VALUES ('foo', 22), ('bar', 14), ('baz', 42) +``` + +Building a SQL statement for a large number of rows is straightforward, but runs the risk of SQL injection problems if the SQL isn't escaped propertly. + +Using command parameters is safer, but building and executing the SQL is more complex. Furthermore, databases often have a limit on the maximum number of command parameters that can be used, so it can be necessary to execute multiple SQL statements, one for each batch of rows to insert. + +## Solution + +`BulkInsert` is a simple Dapper-like extension method that builds the SQL commands for each batch and leverages Dapper to inject the command parameters. + +```csharp +var widgets = new[] { new { name = "foo", size = 22 }, new { name = "bar", size = 14 }, new { name = "baz", size = 42 } }; +connection.BulkInsert("INSERT INTO widgets (name, size) VALUES (@name, @size) ...", widgets); +``` + +The `...` after the `VALUES` clause must be included. It is used by `BulkInsert` to find the end of the `VALUES` clause that will be transformed. The call above will build a SQL statement like so: + +``` +INSERT INTO widgets (name, size) VALUES (@name_0, @size_0), (@name_1, @size_1) +``` + +The actual SQL statement will have as many parameters as needed to insert all of the specified rows. If the total number of command parameters exceeds 999 (the maximum number that [SQLite](https://www.sqlite.org/) supports and an efficient number for [MySql.Data](https://www.nuget.org/packages/MySql.Data/)), it will execute multiple SQL commands until all of the rows are inserted. + +All of the transformed SQL will be executed for each batch, so including additional statements before or after the `INSERT` statement is not recommended. + +Execute the method within a transaction if it is important to avoid inserting only some of the rows if there is an error. + +## Reference + +The `BulkInsert` and `BulkInsertAsync` methods of the `BulkInsertUtility` static class are extension methods on `IDbConnection`. They support these arguments: + +* `sql` – The SQL to execute, which must have a `VALUES` clause that is followed by `...`. +* `commonParam` – The values of any command parameters that are outside the `VALUES` clause, or are common to every row. (Optional.) +* `insertParams` – The values of the command parameters for each row to be inserted. +* `transaction` – The current transaction, if any (see `Dapper.Execute`). +* `batchSize` – If specified, indicates the number of rows to insert in each batch, even if doing so requires more than 999 command parameters. +* `cancellationToken` – The optional cancellation token (`BulkInsertAsync` only). + +The method returns the total number of rows affected (or, more specifically, the sum of the numbers returned when executing the SQL commands for each batch). diff --git a/Faithlife.Utility.Dapper.md b/Faithlife.Utility.Dapper.md new file mode 100644 index 0000000..1ea379a --- /dev/null +++ b/Faithlife.Utility.Dapper.md @@ -0,0 +1,11 @@ +# Faithlife.Utility.Dapper assembly + +The assembly `Faithlife.Utility.Dapper.dll` has 1 public type in 1 namespace. + +## Faithlife.Utility.Dapper namespace + +| public type | description | +| --- | --- | +| static class [BulkInsertUtility](Faithlife.Utility.Dapper/BulkInsertUtility.md) | Methods for bulk insert with Dapper. | + + diff --git a/Faithlife.Utility.Dapper/BulkInsertUtility.md b/Faithlife.Utility.Dapper/BulkInsertUtility.md new file mode 100644 index 0000000..b843207 --- /dev/null +++ b/Faithlife.Utility.Dapper/BulkInsertUtility.md @@ -0,0 +1,25 @@ +# BulkInsertUtility class + +Methods for bulk insert with Dapper. + +```csharp +public static class BulkInsertUtility +``` + +## Public Members + +| name | description | +| --- | --- | +| static [BulkInsert<TInsert>](BulkInsertUtility/BulkInsert.md)(…) | Efficiently inserts multiple rows, in batches as necessary. | +| static [BulkInsert<TCommon,TInsert>](BulkInsertUtility/BulkInsert.md)(…) | Efficiently inserts multiple rows, in batches as necessary. | +| static [BulkInsertAsync<TInsert>](BulkInsertUtility/BulkInsertAsync.md)(…) | Efficiently inserts multiple rows, in batches as necessary. | +| static [BulkInsertAsync<TCommon,TInsert>](BulkInsertUtility/BulkInsertAsync.md)(…) | Efficiently inserts multiple rows, in batches as necessary. | +| static [GetBulkInsertCommands<TInsert>](BulkInsertUtility/GetBulkInsertCommands.md)(…) | Gets the Dapper `CommandDefinition`s used by `BulkInsert` and `BulkInsertAsync`. | +| static [GetBulkInsertCommands<TCommon,TInsert>](BulkInsertUtility/GetBulkInsertCommands.md)(…) | Gets the Dapper `CommandDefinition`s used by `BulkInsert` and `BulkInsertAsync`. | + +## See Also + +* namespace [Faithlife.Utility.Dapper](../Faithlife.Utility.Dapper.md) +* [BulkInsertUtility.cs](https://github.com/Faithlife/DapperUtility/tree/master/src/Faithlife.Utility.Dapper/BulkInsertUtility.cs) + + diff --git a/Faithlife.Utility.Dapper/BulkInsertUtility/BulkInsert.md b/Faithlife.Utility.Dapper/BulkInsertUtility/BulkInsert.md new file mode 100644 index 0000000..9627cb3 --- /dev/null +++ b/Faithlife.Utility.Dapper/BulkInsertUtility/BulkInsert.md @@ -0,0 +1,31 @@ +# BulkInsertUtility.BulkInsert<TInsert> method (1 of 2) + +Efficiently inserts multiple rows, in batches as necessary. + +```csharp +public static int BulkInsert(this IDbConnection connection, string sql, + IEnumerable insertParams, IDbTransaction transaction = null, int? batchSize = null) +``` + +## See Also + +* class [BulkInsertUtility](../BulkInsertUtility.md) +* namespace [Faithlife.Utility.Dapper](../../Faithlife.Utility.Dapper.md) + +--- + +# BulkInsertUtility.BulkInsert<TCommon,TInsert> method (2 of 2) + +Efficiently inserts multiple rows, in batches as necessary. + +```csharp +public static int BulkInsert(this IDbConnection connection, string sql, TCommon commonParam, + IEnumerable insertParams, IDbTransaction transaction = null, int? batchSize = null) +``` + +## See Also + +* class [BulkInsertUtility](../BulkInsertUtility.md) +* namespace [Faithlife.Utility.Dapper](../../Faithlife.Utility.Dapper.md) + + diff --git a/Faithlife.Utility.Dapper/BulkInsertUtility/BulkInsertAsync.md b/Faithlife.Utility.Dapper/BulkInsertUtility/BulkInsertAsync.md new file mode 100644 index 0000000..4259b0e --- /dev/null +++ b/Faithlife.Utility.Dapper/BulkInsertUtility/BulkInsertAsync.md @@ -0,0 +1,32 @@ +# BulkInsertUtility.BulkInsertAsync<TInsert> method (1 of 2) + +Efficiently inserts multiple rows, in batches as necessary. + +```csharp +public static Task BulkInsertAsync(this IDbConnection connection, string sql, + IEnumerable insertParams, IDbTransaction transaction = null, int? batchSize = null, CancellationToken cancellationToken = default(CancellationToken)) +``` + +## See Also + +* class [BulkInsertUtility](../BulkInsertUtility.md) +* namespace [Faithlife.Utility.Dapper](../../Faithlife.Utility.Dapper.md) + +--- + +# BulkInsertUtility.BulkInsertAsync<TCommon,TInsert> method (2 of 2) + +Efficiently inserts multiple rows, in batches as necessary. + +```csharp +public static Task BulkInsertAsync(this IDbConnection connection, string sql, + TCommon commonParam, IEnumerable insertParams, IDbTransaction transaction = null, + int? batchSize = null, CancellationToken cancellationToken = default(CancellationToken)) +``` + +## See Also + +* class [BulkInsertUtility](../BulkInsertUtility.md) +* namespace [Faithlife.Utility.Dapper](../../Faithlife.Utility.Dapper.md) + + diff --git a/Faithlife.Utility.Dapper/BulkInsertUtility/GetBulkInsertCommands.md b/Faithlife.Utility.Dapper/BulkInsertUtility/GetBulkInsertCommands.md new file mode 100644 index 0000000..c3527ed --- /dev/null +++ b/Faithlife.Utility.Dapper/BulkInsertUtility/GetBulkInsertCommands.md @@ -0,0 +1,32 @@ +# BulkInsertUtility.GetBulkInsertCommands<TInsert> method (1 of 2) + +Gets the Dapper `CommandDefinition`s used by `BulkInsert` and `BulkInsertAsync`. + +```csharp +public static IEnumerable GetBulkInsertCommands(string sql, + IEnumerable insertParams, IDbTransaction transaction = null, int? batchSize = null, CancellationToken cancellationToken = default(CancellationToken)) +``` + +## See Also + +* class [BulkInsertUtility](../BulkInsertUtility.md) +* namespace [Faithlife.Utility.Dapper](../../Faithlife.Utility.Dapper.md) + +--- + +# BulkInsertUtility.GetBulkInsertCommands<TCommon,TInsert> method (2 of 2) + +Gets the Dapper `CommandDefinition`s used by `BulkInsert` and `BulkInsertAsync`. + +```csharp +public static IEnumerable GetBulkInsertCommands(string sql, + TCommon commonParam, IEnumerable insertParams, IDbTransaction transaction = null, + int? batchSize = null, CancellationToken cancellationToken = default(CancellationToken)) +``` + +## See Also + +* class [BulkInsertUtility](../BulkInsertUtility.md) +* namespace [Faithlife.Utility.Dapper](../../Faithlife.Utility.Dapper.md) + + diff --git a/README.md b/README.md index 38d2360..05e6b94 100644 --- a/README.md +++ b/README.md @@ -1 +1 @@ -### https://faithlife.github.io/RepoName/ +### https://faithlife.github.io/DapperUtility/ diff --git a/_layouts/default.html b/_layouts/default.html index 317cac3..cbbdf91 100644 --- a/_layouts/default.html +++ b/_layouts/default.html @@ -19,11 +19,11 @@ - ProjectName + Faithlife.Utility.Dapper