Async File IO APIs mimicking Win32 OVERLAPPED #24847
Comments
|
Hi @alexbudmsft We are currently working on a If I am right, it could mean that we could fulfill your request by adding the following overloads to public partial class FileStream : IO.Stream
{
public override int Read(byte[] buffer, int offset, int count)
public override int Read(Span<byte> buffer)
+ public virtual int Read(Span<byte> buffer, long fileOffset)
public override Task<int> ReadAsync(byte[] buffer, int offset, int count, CancellationToken cancellationToken)
public override ValueTask<int> ReadAsync(Memory<byte> buffer, CancellationToken cancellationToken)
+ public virtual ValueTask<int> ReadAsync(Memory<byte> buffer, long fileOffset, CancellationToken cancellationToken)
public override void Write(byte[] buffer, int offset, int count) { }
public override void Write(ReadOnlySpan<byte> buffer) { }
+ public virtual void Write(ReadOnlySpan<byte> buffer, long fileOffset) { }
public override Task WriteAsync(byte[] buffer, int offset, int count, CancellationToken cancellationToken)
public override ValueTask WriteAsync(ReadOnlyMemory<byte> buffer, CancellationToken cancellationToken)
+ public virtual ValueTask WriteAsync(ReadOnlyMemory<byte> buffer, long fileOffset, CancellationToken cancellationToken)
}Would that be enough for you as a potential end-user of the feature? Edit: I think that we can get What would be the expected behavior for you as an end-user? Would throwing for these methods be OK? Or even expected?
Could you say something more about possible usages? I wonder what kind of problems this feature could allow solving. The more I know the better prioritization I can try to get for this request. |
|
One of the use case I have is about raw data reading/writing where a Stream is not particularly useful: I don't need an object that keeps track of the current |
|
@adamsitnik Otherwise, if I own an API that takes a Stream parameter, I cannot use these higher performance APIs even if the runtime type of my parameter happens to be FileStream, unless I try casting to FileStream and have a dedicated code path for that case. Though, in the case that the FileStream has been wrapped in some delegating Stream, then even casting to FileStream would not work. I'm not sure how much other Stream implementations would benefit from overriding this API. Perhaps something like MemoryStream could improve performance a tiny bit due to not needing to update the position. It could also enable some degree of thread-safety in the case of only readers. public virtual int Read(Span<byte> buffer, long fileOffset)
{
if (!CanSeek)
{
throw new NotSupportedException();
}
int oldPosition = Position;
Position = fileOffset;
int bytesRead = Read(buffer);
Position = oldPosition;
return bytesRead;
} |
|
I don't think these should be on either Stream or FileStream. The purpose of Stream is in reading a sequence of data in a linear manner. Concurrent usage of a file where every request is for a specific position is not a stream. It has dubious interactions with any buffering in the instance. It doesn't work with everything the FileStream could wrap (e.g. a pipe). Suggesting these APIs could be used in parallel but other overloads of the same method can't is confusing and likely to lead to production bugs. And then doing so in a way that overrides a base method that isn't also thread-safe is a recipe for more production bugs. If we want to add APIs for this scenario, I'm against them being instance methods on FileStream. They could be methods related to a SafeFileHandle, on File, or on some new low-level File I/O type. |
|
@stephentoub I totally agree. Today |
|
Maybe there should be a type that contains a file handle and that exposes low-level operations on files. As @stephentoub has pointed out, the It could be an informative exercise to scan the entire catalog of Windows and Linux file API functions that take a file handle. Any such API potentially is a candidate for being exposed on a low-level file handle type. |
👍
To obtain a The other way is to create a Moreover, accessing the handle has side-effects, like Or tracking the file offset: When we switch to tracking the offset only in memory (#49145), we might need to sync it with OS offset for cases where
This would require using handles or opening the file every time we want to perform an operation?
IMO this is the best solution. We could introduce a very simple API and moreover separate the sync and async implementations: public sealed class OverlappedFile : IDisposable
{
public string Path { get; }
public FileAccess FileAccess { get; }
public long Length { get; set; }
public OverlappedFile(string path, FileMode mode = FileMode.Open, FileAccess access = FileAccess.Read, FileShare share = FileShare.Read) { }
public int Read(Span<byte> buffer, long fileOffset) { };
public void Write(ReadOnlySpan<byte> buffer, long fileOffset) { }
}
public sealed class AsynchronousOverlappedFile : IDisposable
{
public string Path { get; }
public FileAccess FileAccess { get; }
public long Length { get; set; }
public AsynchronousOverlappedFile(string path, FileMode mode = FileMode.Open, FileAccess access = FileAccess.Read, FileShare share = FileShare.Read) { }
public ValueTask<int> ReadAsync(Memory<byte> buffer, long fileOffset, CancellationToken cancellationToken) { }
public ValueTask WriteAsync(ReadOnlyMemory<byte> buffer, long fileOffset, CancellationToken cancellationToken) { }
}Since for Windows and Unix the file must be seekable to perform overlapped IO, there is no need for the The biggest problem for me here is... naming. My current ideas:
|
This whole issue is about introducing new APIs. I'm not sure why new APIs that return a SafeFileHandle would be off the table.
Yes, the former. |
One of the problems with On Windows, this is fetched with an additional syscall: Which we of course want to avoid in high-perf APIs (not completely, but just do it once for a given file like we do it in If we skip the file type check and just set the offsets for the https://docs.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-readfile#parameters
So this would give us a silent error. On Linux, there are some edge cases as well: https://man7.org/linux/man-pages/man2/pwrite.2.html
To avoid that, we would have to open the file on our own and ensure that Moreover, having a dedicated type that can be more future-proof because allows for a very easy extension of the type and introduction of a new state. Let's say that we want to use new APIs like |
|
I can only speak for my own use cases, but an OverlappedFile type that cannot be constructed with a file handle would not likely be useful to me. Performance Concerns Other Concerns -File locking issues. If I have to open multiple file handles to do different operations (write, delete, set file info, etc.), and I want these file handles open concurrently, I cannot open these file handles with very restrictive FileShare options, since all of these handles must be compatible with each other. This means that I cannot as effectively lock other applications out of the files that I am currently modifying. -Specifying File Open Options Though I understand that your API above is a very preliminary proposal, it doesn't currently provide for specifying FileSystemRights or FileOptions or FileSecurity like can be done for FileStream today. Personally, I actually go to native code today for even more control of creating the file handle. I understand that some options will necessarily be incompatible with any implementation of OverlappedFile, but I would consider using a file handle a sufficiently advanced scenario that it is acceptable to put the responsibility on the consumer to use compatible options, as long as OverlappedFile fails in an obvious way if the file handle wasn't opened with the correct options. For example, I would consider not respecting the fileOffset parameter at all to be an obvious failure that would surely be noticed quickly. I do recognize your concerns about a lack of encapsulation making future growth more difficult. However, generally, couldn't you just have consumers that pass file handles fall back to an older code path if necessary? I think, it's better to have a usable API with limited growth potential, as opposed to an API that can't be used at all (for my use cases). Thanks. |
adamsitnik
added
api-ready-for-review
Background and MotivationSome high-performance server applications require the possibility to perform parallel reads and|or writes with the overhead being as small as possible. In theory, it's possible to use Both Windows and Unix provide the necessary APIs that allow for that:
Both Windows and Unix support only regular disk files. Unseekable files (pipes, etc.) are not supported. Proposed APIUnderlying OS APIs like The problem with This makes the code very hard to read and maintain. Example: So when introducing new APIs, we would like to separate public sealed class RandomAccessFile : IDisposable
{
public string? Path { get; } // not Name like in FileStream
public FileAccess Access { get; }
public SafeFileHandle SafeHandle { get; }
public ulong Length { get; set; } // ulong, not long
public RandomAccessFile(string filePath, FileMode mode = FileMode.Open, FileAccess access = FileAccess.Read, FileShare share = FileShare.Read)
public RandomAccessFile(SafeFileHandle handle, FileAccess access = FileAccess.Read)
public void Dispose()
public int Read(Span<byte> buffer, ulong fileOffset)
public int Write(ReadOnlySpan<byte> buffer, ulong fileOffset)
}
public sealed class AsynchronousRandomAccessFile : IDisposable
{
public string? Path { get; }
public FileAccess Access { get; }
public SafeFileHandle SafeHandle { get; }
public ulong Length { get; set; }
public AsynchronousRandomAccessFile(string filePath, FileMode mode = FileMode.Open, FileAccess access = FileAccess.Read, FileShare share = FileShare.Read)
public AsynchronousRandomAccessFile(SafeFileHandle handle, FileAccess access = FileAccess.Read)
public void Dispose()
public ValueTask<int> ReadAsync(Memory<byte> buffer, ulong fileOffset, CancellationToken cancellationToken = default)
public ValueTask<int> WriteAsync(ReadOnlyMemory<byte> buffer, ulong fileOffset, CancellationToken cancellationToken = default)
}Alternative names: Note: The APIs are using Usage ExamplesCopying files in parallel: static void CopyParallelSync(string sourcePath, string destinationPath)
{
using (var sourceFile = new RandomAccessFile(sourcePath))
using (var destinationFile = new RandomAccessFile(destinationPath))
{
int bufferLength = (int)(sourceFile.Length / (ulong)Environment.ProcessorCount);
Parallel.For(0, Environment.ProcessorCount, index =>
{
byte[] bytes = ArrayPool<byte>.Shared.Rent(bufferLength);
ulong fileOffset = (ulong)(index * bufferLength);
int bytesRead = sourceFile.Read(bytes.AsSpan(0, bufferLength), fileOffset);
int bytesWritten = destinationFile.Write(bytes.AsSpan(0, bufferLength), fileOffset);
// for the sake of simplicity we assume that bufferLength == bytesRead == bytesWritten
ArrayPool<byte>.Shared.Return(bytes);
});
}
}
static async Task CopyParallelAsync(string sourcePath, string destinationPath)
{
await using (var sourceFile = new AsynchronousRandomAccessFile(sourcePath))
await using (var destinationFile = new AsynchronousRandomAccessFile(destinationPath))
{
int bufferLength = (int)(sourceFile.Length / (ulong)Environment.ProcessorCount);
Task[] tasks = new Task[Environment.ProcessorCount];
for (int index = 0; index < tasks.Length; index++)
{
tasks[index] = CopyPartAsync(index, bufferLength, sourceFile, destinationFile);
}
await Task.WhenAll(tasks);
}
async Task CopyPartAsync(int index, int bufferLength, AsynchronousRandomAccessFile sourceFile, AsynchronousRandomAccessFile destinationFile)
{
byte[] bytes = ArrayPool<byte>.Shared.Rent(bufferLength);
ulong fileOffset = (ulong)(index * bufferLength);
int bytesRead = await sourceFile.ReadAsync(bytes.AsMemory(0, bufferLength), fileOffset);
int bytesWritten = await destinationFile.WriteAsync(bytes.AsMemory(0, bufferLength), fileOffset);
ArrayPool<byte>.Shared.Return(bytes);
}
}Alternative DesignsExtending FileStreamAs @stephentoub wrote:
SafeFileHandle extensionspublic sealed partial class SafeFileHandle : Microsoft.Win32.SafeHandles.SafeHandleZeroOrMinusOneIsInvalid
{
public SafeFileHandle()
public SafeFileHandle(System.IntPtr preexistingHandle, bool ownsHandle)
+ public static SafeFileHandle OpenFile(string path, FileMode mode = FileMode.Open, FileAccess access = FileAccess.Read, FileShare share = FileShare.Read, FileOptions options = FileOptions.None)
public override bool IsInvalid { get; }
+ public bool CanSeek { get; }
protected override bool ReleaseHandle() { throw null; }
}
+public static class SafeFileHandleExtensions
+{
+ public static int Read(this SafeFileHandle fileHandle, Span<byte> buffer, ulong fileOffset)
+ public static int Write(this SafeFileHandle fileHandle, ReadOnlySpan<byte> buffer, ulong fileOffset)
+
+ public static ValueTask<int> ReadAsync(this SafeFileHandle fileHandle, Memory<byte> buffer, ulong fileOffset, CancellationToken cancellationToken = default)
+ public static ValueTask<int> WriteAsync(this SafeFileHandle fileHandle, ReadOnlyMemory<byte> buffer, ulong fileOffset, CancellationToken cancellationToken = default)
+}The advantage would be the lack of introducing new type(s), but the disadvantage is a potential and quite common performance hit which is something that we want to avoid in a high-performance API. There are two ways of obtaining safe file handles. The first one is calling native OS API. It's not cross-platform and it requires a very good understanding of the APIs. As noticed by @jkotas, this could be solved by an introduction of a factory method that creates When the user provides us such a handle, we don't know whether the file is seekable or not. To find out, we have to perform a syscall: This could be mitigated by extending The other way of obtaining When buffering is enabled, the (if the buffers are empty, no syscalls are being performed) And when file is seekable (which should always be true for this API), we need to synchronise the file offset with the OS (this is a mandatory syscall as of today and it reduces the cost of every This could be minimized by accessing the SafeFileHandle safeFileHandle = fileStream.SafeFileHandle;
safeFileHandle.X(..);
safeFileHandle.X(..);
safeFileHandle.X(..);
safeFileHandle.X(..);but there would be nothing to stop the users from calling the property many times and paying for the overhead every time: fileStream.SafeFileHandle.X(..);
fileStream.SafeFileHandle.X(..);
fileStream.SafeFileHandle.X(..);
fileStream.SafeFileHandle.X(..); |
Why ulong? long is much more natural to use within .NET. |
Can this be addressed by introducing an file SafeHandle factory method that takes similar arguments as FileStream constructor? Also, I think it is important that the APIs we expose for this give you access to the OS handle if needed, to let you call any specialized OS-specific methods. Otherwise, one would have to switch to something completely different to do an OS-specific call, or resort to private reflection to fetch the OS handle. |
With
Good point, I am going to add such method to the alternative proposal
Another good point, I am going to add a property that exposes the handle |
|
Is overlapped IO primarily a Windows term? Also, overlapped does not suggest what it is for to anyone not familiar with it. I think your suggestions of RandomAccessFile and AsynchronousRandomAccessFile are good if we create these types. |
|
RandomAccessFile 👍🏾 |
+1. This is the same thing I suggested earlier in the thread: #24847 (comment).
Sounds fine.
Are there known consumers for both? Will FileStream sit on top of these new types?
This should be nullable, unless we require that only SafeFileHandles backed by disk files are used and we have a way of retrieving the original path on all OSes.
What work would DisposeAsync do that's asynchronous?
These should be ValueTask rather than Task. Also, in these APIs, on Windows, Linux, and macOS, is the underlying operation guaranteed to write out all of the data, or might it only do a partial write?
What does this do if there isn't a known length? |
The original author of the feature request asked for
I would rather expect both types to reuse the same helper methods, without one depending on another.
Good point, I am going to fix that.
Another good point. I assume that
I have explained why they are not returning
This is an excellent catch 👍 . https://man7.org/linux/man-pages/man2/pwrite.2.html
I am going to update the proposal to return
Only seekable files will be supported, so I would assume that |
On WIndows they need to be aligned page sized blocks? ReadFileScatter function
|
|
@benaadams yes, the full list of |
Updated proposalpublic sealed partial class SafeFileHandle : Microsoft.Win32.SafeHandles.SafeHandleZeroOrMinusOneIsInvalid
{
public SafeFileHandle()
public SafeFileHandle(System.IntPtr preexistingHandle, bool ownsHandle)
+ public static SafeFileHandle Open(string filePath, FileMode mode = FileMode.Open, FileAccess access = FileAccess.Read, FileShare share = FileShare.Read, FileOptions options = FileOptions.None, long preAllocationSize = 0)
public override bool IsInvalid { get; }
protected override bool ReleaseHandle() { throw null; }
+ public ulong Length { get; }
+ public int ReadAtOffset(Span<byte> buffer, ulong fileOffset);
+ public int WriteAtOffset(ReadOnlySpan<byte> buffer, ulong fileOffset);
+ public long ReadScatterAtOffset(ReadOnlyMemory<Memory<byte>> buffers, ulong fileOffset);
+ public long WriteGatherAtOffset(ReadOnlyMemory<ReadOnlyMemory<byte>> buffers, ulong fileOffset);
+ public ValueTask<int> ReadAtOffsetAsync(Memory<byte> buffer, ulong fileOffset, CancellationToken cancellationToken = default);
+ public ValueTask<int> WriteAtOffsetAsync(ReadOnlyMemory<byte> buffer, ulong fileOffset, CancellationToken cancellationToken = default);
+ public ValueTask<long> ReadScatterAtOffsetAsync(ReadOnlyMemory<Memory<byte>> buffers, ulong fileOffset, CancellationToken cancellationToken = default);
+ public ValueTask<long> WriteGatherAtOffsetAsync(ReadOnlyMemory<ReadOnlyMemory<byte>> buffers, ulong fileOffset, CancellationToken cancellationToken = default);
}Mappings to OS-specific methods:Windows:
Unix:
Linux: as soon as we add Exceptions
|
adamsitnik
added
api-ready-for-review
namespace System.IO
{
public static class RandomAccess
{
public static long GetLength(SafeFileHandle handle) => throw null;
public static int ReadAtOffset(SafeFileHandle handle, Span<byte> buffer, long fileOffset);
public static int ReadAtOffset(SafeFileHandle handle, Span<byte> buffer, ulong fileOffset);
public static void WriteAtOffset(SafeFileHandle handle, ReadOnlySpan<byte> buffer, long fileOffset);
public static void WriteAtOffset(SafeFileHandle handle, ReadOnlySpan<byte> buffer, ulong fileOffset);
public static long ReadScatterAtOffset(SafeFileHandle handle, IReadOnlyList<Memory<byte>> buffers, long fileOffset);
public static long ReadScatterAtOffset(SafeFileHandle handle, IReadOnlyList<Memory<byte>> buffers, ulong fileOffset);
public static void WriteGatherAtOffset(SafeFileHandle handle, IReadOnlyList<ReadOnlyMemory<byte>> buffers, long fileOffset);
public static void WriteGatherAtOffset(SafeFileHandle handle, IReadOnlyList<ReadOnlyMemory<byte>> buffers, ulong fileOffset);
public static ValueTask<int> ReadAtOffsetAsync(SafeFileHandle handle, Memory<byte> buffer, long fileOffset, CancellationToken cancellationToken = default);
public static ValueTask<int> ReadAtOffsetAsync(SafeFileHandle handle, Memory<byte> buffer, ulong fileOffset, CancellationToken cancellationToken = default);
public static ValueTask WriteAtOffsetAsync(SafeFileHandle handle, ReadOnlyMemory<byte> buffer, long fileOffset, CancellationToken cancellationToken = default);
public static ValueTask WriteAtOffsetAsync(SafeFileHandle handle, ReadOnlyMemory<byte> buffer, ulong fileOffset, CancellationToken cancellationToken = default);
public static ValueTask<long> ReadScatterAtOffsetAsync(SafeFileHandle handle, IReadOnlyList<Memory<byte>> buffers, long fileOffset, CancellationToken cancellationToken = default);
public static ValueTask<long> ReadScatterAtOffsetAsync(SafeFileHandle handle, IReadOnlyList<Memory<byte>> buffers, ulong fileOffset, CancellationToken cancellationToken = default);
public static ValueTask WriteGatherAtOffsetAsync(SafeFileHandle handle, IReadOnlyList<ReadOnlyMemory<byte>> buffers, long fileOffset, CancellationToken cancellationToken = default);
public static ValueTask WriteGatherAtOffsetAsync(SafeFileHandle handle, IReadOnlyList<ReadOnlyMemory<byte>> buffers, ulong fileOffset, CancellationToken cancellationToken = default);
}
partial class File
{
public static SafeFileHandle OpenHandle(string filePath, FileMode mode = FileMode.Open, FileAccess access = FileAccess.Read, FileShare share = FileShare.Read, FileOptions options = FileOptions.None, long preAllocationSize = 0)
}
}
namespace Microsoft.Win32.SafeHandles
{
partial class SafeFileHandle
{
public bool IsAsync { get; }
}
} |
bartonjs
added
api-approved
|
Why aren't all of these names just Read/Write{Async}? I don't see the benefit of "AtOffset" when there's already an argument "fileOffset" that's required, and there's no precedent for using the terminology "Scatter"/"Gather" in .NET APIs, e.g. the Socket methods that support such things are just overloads of Receive/Send{Async}. These should just be overloads of "Read/Write{Async}". We're talking about adding such overloads to Stream as well, and there these should definitely just be overloads, not names Including this additional terminology. @bartonjs, @terrajobst, can we revisit this? |
stephentoub
added
api-ready-for-review
Personally, I find it providing clarity that it's entirely disassociated with the file handle's inbuilt notion of position. It's a completely different kind of read from
The impression I got from the review was that we wanted the terms if we added it to Stream. If there's dissent on that point and you think we're setting up for mismatch, that's worth bringing up again. |
The fileOffset argument provides that. What else could it mean? On top of that, these are non-extension static methods on a class named RandomAccess, which itself conveys it's not just going from some built-in location for sequential streaming.
From my perspective, that's a bad direction. We should not do that. Whether the methods use special OS APIs or do multiple read/writes or copy into a buffer with a single read/write is an implementation detail. At the end of the day, the only difference here is the data type of the argument, and whether it's a Memory or a list or a ReadOnlySequence or whatever, it's still the same read/write operation, just with various types holding it. That's what we use overloads for. |
Being extension methods was discussed. If it happens then the call just looks like Even as a static invocation, But I think we spent more time talking about the name when it was proposed as an instance method on SafeFileHandle than after it was a static non-extension method on RandomAccess. So maybe I'm the odd man out. |
I don't see how that disambiguates offset in destination from file offset. If that's really a concern, it should be AtFileOffset rather than AtOffset. Regarding extension methods, as you say, that was discussed and dismissed. As static non- extension methods, this additional wording is unnecessary. For me, it's too verbose and duplicative, saying the same thing multiple times when it's already evident from the RandomAccess class name and the required fileOffset parameter name exactly what this is. We call these methods Read and Write everywhere else, with semantics impacted by what arguments they take; we should do the same here. |
|
(That said, I care much more about not including Scatter and Gather in the names. The mechanism by which these are implemented is an implementation detail, we overload by data type in .NET for these situations, I don't want .NET devs to have to learn the scatter/gather terminology, and we already don't use those words for existing APIs in .NET that provide such semantics, e.g. on Socket.) |
namespace System.IO
{
public static class RandomAccess
{
public static long GetLength(SafeFileHandle handle) => throw null;
public static int Read(SafeFileHandle handle, Span<byte> buffer, long fileOffset);
public static int Read(SafeFileHandle handle, Span<byte> buffer, ulong fileOffset);
public static void Write(SafeFileHandle handle, ReadOnlySpan<byte> buffer, long fileOffset);
public static void Write(SafeFileHandle handle, ReadOnlySpan<byte> buffer, ulong fileOffset);
public static long Read(SafeFileHandle handle, IReadOnlyList<Memory<byte>> buffers, long fileOffset);
public static long Read(SafeFileHandle handle, IReadOnlyList<Memory<byte>> buffers, ulong fileOffset);
public static void Write(SafeFileHandle handle, IReadOnlyList<ReadOnlyMemory<byte>> buffers, long fileOffset);
public static void Write(SafeFileHandle handle, IReadOnlyList<ReadOnlyMemory<byte>> buffers, ulong fileOffset);
public static ValueTask<int> ReadAsync(SafeFileHandle handle, Memory<byte> buffer, long fileOffset, CancellationToken cancellationToken = default);
public static ValueTask<int> ReadAsync(SafeFileHandle handle, Memory<byte> buffer, ulong fileOffset, CancellationToken cancellationToken = default);
public static ValueTask WriteAsync(SafeFileHandle handle, ReadOnlyMemory<byte> buffer, long fileOffset, CancellationToken cancellationToken = default);
public static ValueTask WriteAsync(SafeFileHandle handle, ReadOnlyMemory<byte> buffer, ulong fileOffset, CancellationToken cancellationToken = default);
public static ValueTask<long> ReadAsync(SafeFileHandle handle, IReadOnlyList<Memory<byte>> buffers, long fileOffset, CancellationToken cancellationToken = default);
public static ValueTask<long> ReadAsync(SafeFileHandle handle, IReadOnlyList<Memory<byte>> buffers, ulong fileOffset, CancellationToken cancellationToken = default);
public static ValueTask WriteAsync(SafeFileHandle handle, IReadOnlyList<ReadOnlyMemory<byte>> buffers, long fileOffset, CancellationToken cancellationToken = default);
public static ValueTask WriteAsync(SafeFileHandle handle, IReadOnlyList<ReadOnlyMemory<byte>> buffers, ulong fileOffset, CancellationToken cancellationToken = default);
}
partial class File
{
public static SafeFileHandle OpenHandle(string filePath, FileMode mode = FileMode.Open, FileAccess access = FileAccess.Read, FileShare share = FileShare.Read, FileOptions options = FileOptions.None, long preAllocationSize = 0)
}
}
namespace Microsoft.Win32.SafeHandles
{
partial class SafeFileHandle
{
public bool IsAsync { get; }
}
} |
bartonjs
added
api-approved
ghost
added
the
in-pr
ghost
removed
the
in-pr
adamsitnik
removed
the
blocking
ghost
locked as resolved and limited conversation to collaborators
Edit by @carlossanlop: The updated API Proposal is here.
It would be great if the .NET framework had async file APIs that closely modelled Win32 OVERLAPPED without the need for building one ourselves with Interop (e.g. https://dschenkelman.github.io/2013/10/29/asynchronous-io-in-c-io-completion-ports/)
Today, the APIs do not take an offset parameter and instead operate on the internal file position. While there are some workarounds (creating multiple Filestream objects and changing their file pointer before issuing the IO) this is unwieldy and inefficient. Ideally there'd be a single FILE_OBJECT (in kernel mode) opened with FILE_FLAG_OVERLAPPED against which you can queue many async IOs where .NET would allocate an OVERLAPPED internally and set the OffsetLow/High fields.
You'd have to integrate this somehow with completions but keep it as flexible as possible. In Win32 you can check for completions via polling (looking at OVERLAPPED.Internal for STATUS_IO_PENDING), waiting on an event (OVERLAPPED.hEvent), waiting on the FILE_OBJECT's internal event (by waiting on the file handle itself), associating the handle with an IOCP and calling GetQueuedCompletionStatus(), and finally putting the thread in an alertable wait state via SleepEx et al and using ReadFileEx.
We don't necessarily have to support all of these, but at the very least polling and event based should be supported somehow. The .NET "Task" model already wraps an event-like scheme with Task.WaitAny() for example, so we could try to plug into that. For polling, we'd want a method on the OVERLAPPED wrapper (whether we use the existing .NET Overlapped class or hide it is up to you) that checks the Internal field (i.e. HasOverlappedIoCompleted macro equivalent).
This will allow a lot more high performance server style applications that care about the full functionality of the Win32 IO model to be written in C# without any hand-rolled Interop.
The text was updated successfully, but these errors were encountered: