Skip to content

Commit

Permalink
bulk copy (docs) (#70)
Browse files Browse the repository at this point in the history
* bulk copy

* nits
  • Loading branch information
mgravell authored Nov 14, 2023
1 parent 80339d9 commit f344fd0
Show file tree
Hide file tree
Showing 3 changed files with 61 additions and 5 deletions.
55 changes: 55 additions & 0 deletions docs/bulkcopy.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
# Bulk Copy

A common scenario with databases is "bulk copy"; in the case of SQL Server, this is exposed via
[`SqlBulkCopy.WriteToServer`](https://learn.microsoft.com/dotnet/api/system.data.sqlclient.sqlbulkcopy.writetoserver),
which allows a data feed to be pumped into a table very efficiently and rapidly.

The problem is: that data feed needs to be a `DataTable`, a `DbDataReader`, or similar; but your
application code is likely to be using *objects*, for example a `List<Customer>` or a `IEnumerable<Order>` sequence.

What we need is a mechanism to turn `IEnumerable<T>` (or `IAsyncEnumerable<T>`) into a `DbDataReader`,
suitable for use with `SqlBulkCopy`!

This *is not* a feature that vanilla `Dapper` provides; a library from my distant past,
[`FastMember`](https://www.nuget.org/packages/FastMember), can help bridge this gap, but like `Dapper` it is
based on runtime reflection, and is not AOT compatible.

## Introducing `TypeAccessor`

A new API in `Dapper.AOT` gives us what we need - welcome `TypeAccessor`! Usage is simple:

``` csharp
IEnumerable<Customer> customers = ...
var reader = TypeAccessor.CreateDataReader(customers);
```

This gives us a `DbDataReader` that iterates over the provided sequence. We can optionally
specify a subset of instance properties/fields to expose (and their order) - and we can start
from either `IEnumerable<T>` or `IAsyncEnumerable<T>`.

To use this with `SqlBulkCopy`, we probably want to specify columns explicitly:

``` csharp
using var table = new SqlBulkCopy(connection)
{
DestinationTableName = "Customers",
ColumnMappings =
{
{ nameof(Customer.CustomerNumber), nameof(Customer.CustomerNumber) }
{ nameof(Customer.Name), nameof(Customer.Name) }
}
};
table.EnableStreaming = true;
table.WriteToServer(TypeAccessor.CreateDataReader(customers,
[nameof(Customer.Name), nameof(Customer.CustomerNumber)]));
return table.RowsCopied;
```

This writes the data from our `customers` sequence into the `Customers` table. To be explicit,
we've specified the column mappings manually, although you can often omit this. Likewise, we have
restricted the `DbDataReader` to only expose the `Name` and `CustomerNumber` members.

This all works using generated AOT-compatible code, and does not require any additional
libaries other than `Dapper.AOT`.


1 change: 1 addition & 0 deletions docs/readme.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ Index:
- [Getting Started](/gettingstarted)
- [SQL Syntax](/sqlsyntax)
- [Generated Code](/generatedcode)
- [Bulk Copy](/bulkcopy)
- [Frequently Asked Questions](/faq)

Packages:
Expand Down
10 changes: 5 additions & 5 deletions test/UsageBenchmark/BatchInsertBenchmarks.cs
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ namespace Dapper;
public class BatchInsertBenchmarks : IDisposable
{
private readonly SqlConnection connection = new(Program.ConnectionString);
private Customer[] customers = Array.Empty<Customer>();
private Customer[] customers = [];

public void DebugState()
{
Expand Down Expand Up @@ -246,12 +246,12 @@ public int SqlBulkCopyDapper()
{
DestinationTableName = "BenchmarkCustomers",
ColumnMappings =
{
{ nameof(Customer.Name), nameof(Customer.Name) }
}
{
{ nameof(Customer.Name), nameof(Customer.Name) }
}
};
table.EnableStreaming = true;
table.WriteToServer(TypeAccessor.CreateDataReader(customers, new[] { nameof(Customer.Name) }));
table.WriteToServer(TypeAccessor.CreateDataReader(customers, [nameof(Customer.Name)]));
return table.RowsCopied;
}
finally
Expand Down

0 comments on commit f344fd0

Please sign in to comment.