UI improvements.

Files handling improvements.
i2van · Jun 11, 2021 · f123202 · f123202
1 parent ea3db5c
commit f123202
Show file tree

Hide file tree

Showing 30 changed files with 955 additions and 269 deletions.
diff --git a/Deploy/buildlpx.cmd b/Deploy/buildlpx.cmd
@@ -1,4 +1,4 @@
-@set version=6.11.0
+@set version=6.12.0
 @set zip="%ProgramFiles%\7-Zip\7z.exe"
 @set output="CsvLINQPadDriver.%version%.lpx6"
 

diff --git a/README.md b/README.md
@@ -1,6 +1,7 @@
 [![Latest build](https://github.com/i2van/CsvLINQPadDriver/workflows/build/badge.svg)](https://github.com/i2van/CsvLINQPadDriver/actions)
-[![NuGet](https://buildstats.info/nuget/CsvLINQPadDriver)](https://www.nuget.org/packages/CsvLINQPadDriver)
-[![License](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![NuGet](https://img.shields.io/nuget/v/CsvLINQPadDriver)](https://www.nuget.org/packages/CsvLINQPadDriver)
+[![Downloads](https://img.shields.io/nuget/dt/CsvLINQPadDriver)](https://www.nuget.org/packages/CsvLINQPadDriver)
+[![License](https://img.shields.io/badge/license-MIT-yellow)](https://opensource.org/licenses/MIT)
 
 # CsvLINQPadDriver for LINQPad 6 #
 
@@ -12,7 +13,7 @@
 * [Example](#example)
 * [Prerequisites](#prerequisites)
 * [Installation](#installation)
-  * [NuGet](#nuget-)
+  * [NuGet](#nuget)
   * [Manual](#manual)
 * [Usage](#usage)
 * [Configuration Options](#configuration-options)
@@ -37,7 +38,7 @@
 * [Authors](#authors)
 * [Credits](#credits)
   * [Tools](#tools)
-  * [NuGet](#nuget)
+  * [Libraries](#libraries)
 * [License](#license)
 
 ## Description ##
@@ -122,7 +123,9 @@ select new { author.Name, book.Title }
 
 ## Installation ##
 
-### NuGet [![NuGet](https://buildstats.info/nuget/CsvLINQPadDriver)](https://www.nuget.org/packages/CsvLINQPadDriver) ###
+### NuGet ###
+
+[![NuGet](https://img.shields.io/nuget/v/CsvLINQPadDriver)](https://www.nuget.org/packages/CsvLINQPadDriver)
 
 * Open LINQPad 6.
 * Click `Add connection` main window.
@@ -152,30 +155,33 @@ CSV files connection can be added to LINQPad 6 the same way as any other connect
 
 ### CSV Files ###
 
-* CSV files: list of CSV files and folders. Can be added via files/folder dialogs, dedicated hotkeys, by typing one file/folder per line or by Drag&Drop (`Ctrl` adds files). Wildcards `?` and `*` are supported; `**.csv` searches in folder and its sub-folders.
+* CSV files: list of CSV files and folders. Can be added via files/folder dialogs, dedicated hotkeys, by typing one file/folder per line or by Drag&Drop (`Ctrl` adds files, `Alt` toggles `*` and `**` masks). Wildcards `?` and `*` are supported; `**.csv` searches in folder and its sub-folders.
   * `c:\Books\Books?.csv`: `Books.csv`, `Books1.csv`, etc. files in folder `c:\Books`
   * `c:\Books\*.csv`: all `*.csv` files in folder `c:\Books`
   * `c:\Books\**.csv`: all `*.csv` files in folder `c:\Books` and its sub-folders.
 * Order files by: specifies files sort order. Affects similar files order.
 * No BOM encoding: specifies encoding for files without [BOM](https://en.wikipedia.org/wiki/Byte_order_mark). `UTF-8` is default.
+* Validate file paths: check if file paths are valid.
 * Ignore files with invalid format: files with content which does not resemble CSV will be ignored.
 
 ### Format ###
 
 * CSV separator: character used to separate columns in files. Can be `,`, `\t`, etc. Auto-detected if empty.
-* Use CsvHelper library separator auto-detection. Use with caution: sometimes it fails.
+* Use CsvHelper library separator auto-detection: use CsvHelper library separator auto-detection instead of internal one.
+* Ignore bad data: ignore malformed CSV data.
 * Allow comments: lines starting with `#` will be ignored.
 
 ### Memory ###
 
-* Cache CSV data in memory:
+* Cache data in memory:
   * if checked: parsed rows from file are cached in memory. This cache survives multiple query runs, even when query is changed. Cache is cleared as soon as LINQPad clears query data.
   * if unchecked: disable cache. Multiple enumerations of file content results in multiple reads and parsing of file. Can be significantly slower for complex queries. Significantly reduces memory usage.  Useful when reading very large files.
-* Intern CSV strings: intern strings. Significantly reduce memory consumption when CSV contains repeatable values.
+* Intern strings: intern strings. Significantly reduce memory consumption when CSV contains repeatable values.
 
 ### Generation ###
 
 * Generate single class for similar files: single class will be generated for similar files which allows to query them as a single one. Might not work well for files with relations.
+  * Also show similar files non-grouped: show similar files non-grouped in addition to similar files groups.
 * String comparison: string comparison for `Equals` and `GetHashCode` methods.
 
 ### Relations ###
@@ -185,8 +191,9 @@ CSV files connection can be added to LINQPad 6 the same way as any other connect
 
 ### Misc ##
 
-* Debug info: additional debug information will be available, e.g. generated Data Context source.
-* Remember this connection: connection info will be saved and available after LINQPad restart.
+* Debug info: show additional driver debug info, e.g. generated data context source.
+* Contains production data: files contain production data.
+* Remember this connection: connection will be available on next run.
 
 ## Relations Detection ##
 
@@ -248,7 +255,13 @@ Formats object the way PowerShell [Format-List](https://docs.microsoft.com/en-us
 int GetHashCode();
 ```
 
-Returns object hash code. Hash code is not cached and recalculated each time method is called. Also note that each time driver is reloaded hash codes will be [different](https://andrewlock.net/why-is-string-gethashcode-different-each-time-i-run-my-program-in-net-core/).
+Returns object hash code.
+
+Note that:
+
+* Generated data object is mutable.
+* Hash code is not cached and recalculated each time method is called.
+* Each time driver is reloaded string hash codes will be [different](https://andrewlock.net/why-is-string-gethashcode-different-each-time-i-run-my-program-in-net-core/).
 
 > Depends on string comparison driver setting. Relations are not participated.
 
@@ -381,13 +394,14 @@ TimeSpan? ToTimeSpan(
 * [LINQPad 6](https://www.linqpad.net/LINQPad6.aspx)
 * [LINQPad Command-Line and Scripting](https://www.linqpad.net/lprun.aspx)
 
-### NuGet ###
+### Libraries ###
 
 * [CsvHelper](https://github.com/JoshClose/CsvHelper)
 * [Fluent Assertions](https://github.com/fluentassertions/fluentassertions)
 * [Humanizer](https://github.com/Humanizr/Humanizer)
 * [Moq](https://github.com/moq/moq4)
 * [NUnit](https://github.com/nunit/nunit)
+* [Windows API Code Pack](https://github.com/contre/Windows-API-Code-Pack-1.1)
 
 ## License ##
 

diff --git a/Src/CsvLINQPadDriver/CodeGen/CsvCSharpCodeGenerator.cs b/Src/CsvLINQPadDriver/CodeGen/CsvCSharpCodeGenerator.cs
@@ -37,7 +37,7 @@ public record TypeCodeResult(string TypeName, string Code, string CodeName, stri
         public record Result(string Code, IReadOnlyCollection<IGrouping<string, TypeCodeResult>> CodeGroups);
 
         // ReSharper disable once RedundantAssignment
-        internal static Result GenerateCode(CsvDatabase db, ref string nameSpace, ref string typeName, ICsvDataContextDriverProperties props) =>
+        public static Result GenerateCode(CsvDatabase db, ref string nameSpace, ref string typeName, ICsvDataContextDriverProperties props) =>
             new CsvCSharpCodeGenerator(nameSpace, typeName = DefaultContextTypeName, props).GenerateSrcFile(db);
 
         private Result GenerateSrcFile(CsvDatabase csvDatabase)
@@ -49,12 +49,7 @@ private Result GenerateSrcFile(CsvDatabase csvDatabase)
                     .GroupBy(typeCode => typeCode.TypeName)
                     .ToImmutableList();
 
-            return new Result($@"using System;
-using System.Collections.Generic;
-
-using CsvLINQPadDriver;
-
-namespace {_contextNameSpace}
+            return new Result($@"namespace {_contextNameSpace}
 {{
     /// <summary>CSV Data Context</summary>
     public class {_contextTypeName} : {typeof(CsvDataContextBase).GetCodeTypeClassName()}
@@ -70,8 +65,9 @@ public class {_contextTypeName} : {typeof(CsvDataContextBase).GetCodeTypeClassNa
                 {GetBoolConst(_properties.IsStringInternEnabled)},
                 {GetBoolConst(_properties.IsCacheEnabled)},
                 {table.CsvSeparator.AsValidCSharpCode()},
-                {nameof(NoBomEncoding)}.{_properties.NoBomEncoding},
+                {typeof(NoBomEncoding).GetCodeTypeClassName()}.{_properties.NoBomEncoding},
                 {GetBoolConst(_properties.AllowComments)},
+                {GetBoolConst(_properties.IgnoreBadData)},
                 {table.FilePath.AsValidCSharpCode()},
                 new {typeof(CsvColumnInfoList<>).GetCodeTypeClassName(GetClassName(table))} {{
                     {string.Join(string.Empty, table.Columns.Select(c => $@"{{ {c.Index}, x => x.{c.CodeName} }}, "))}
@@ -109,7 +105,7 @@ public sealed record {className} : {typeof(ICsvRowBase).GetCodeTypeClassName()}
 
         /// <summary>{SecurityElement.Escape(csvRelation.DisplayName)}</summary> {(hideRelationsFromDump ? $@"
         [{typeof(HideFromDumpAttribute).GetCodeTypeClassName()}]" : string.Empty)}
-        public IEnumerable<{csvRelation.TargetTable.GetCodeRowClassName()}> {csvRelation.CodeName} {{ get; set; }}")
+        public System.Collections.Generic.IEnumerable<{csvRelation.TargetTable.GetCodeRowClassName()}> {csvRelation.CodeName} {{ get; set; }}")
             )}
     }}", table.CodeName!, table.FilePath);
 
@@ -142,7 +138,7 @@ private static string GenerateIndexer(IReadOnlyCollection<string> properties, bo
         }}";
 
             static string GenerateIndexerException(bool intIndexer) =>
-                $@"default: throw new IndexOutOfRangeException(string.Format(""There is no property {(intIndexer ? "at index {0}" : "with name \\\"{0}\\\"")}"", index));";
+                $@"default: throw new System.IndexOutOfRangeException(string.Format(""There is no property {(intIndexer ? "at index {0}" : "with name \\\"{0}\\\"")}"", index));";
 
             static string IntToStr(int val) =>
                 val.ToString(CultureInfo.InvariantCulture);
@@ -155,7 +151,7 @@ private static string GenerateToString(IReadOnlyCollection<string> properties)
             return $@"
         public override string ToString()
         {{
-            return string.Format(""{string.Join(string.Empty, properties.Select((v, i) => $"{v.PadRight(namePadding)} : {{{i+1}}}{{0}}"))}"", Environment.NewLine, {string.Join(", ", properties)});
+            return string.Format(""{string.Join(string.Empty, properties.Select((v, i) => $"{v.PadRight(namePadding)} : {{{i+1}}}{{0}}"))}"", System.Environment.NewLine, {string.Join(", ", properties)});
         }}";
         }
 
@@ -166,14 +162,14 @@ public bool Equals({typeName} obj)
             if(obj == null) return false;
             if(ReferenceEquals(this, obj)) return true;
             return  {string.Join($" && {Environment.NewLine}{GetIndent(20)}",
-                properties.Select(property => $"string.Equals({property}, obj.{property}, StringComparison.{stringComparison})"))};
+                properties.Select(property => $"string.Equals({property}, obj.{property}, System.StringComparison.{stringComparison})"))};
         }}
 
         public override int GetHashCode()
         {{
-            var hashCode = new HashCode();
+            var hashCode = new System.HashCode();
 
-{string.Join(Environment.NewLine, properties.Select(property => $"{GetIndent(12)}hashCode.Add({property}, StringComparer.{GetStringComparer(stringComparison)});"))}
+{string.Join(Environment.NewLine, properties.Select(property => $"{GetIndent(12)}hashCode.Add({property}, System.StringComparer.{GetStringComparer(stringComparison)});"))}
 
             return hashCode.ToHashCode();
         }}";

diff --git a/Src/CsvLINQPadDriver/CodeGen/CsvTableBase.cs b/Src/CsvLINQPadDriver/CodeGen/CsvTableBase.cs
@@ -24,23 +24,33 @@ public abstract class CsvTableBase<TRow> : CsvTableBase, IEnumerable<TRow>
         private readonly char? _csvSeparator;
         private readonly NoBomEncoding _noBomEncoding;
         private readonly bool _allowComments;
+        private readonly bool _ignoreBadData;
 
         protected readonly string FilePath;
 
-        protected CsvTableBase(bool isStringInternEnabled, char? csvSeparator, NoBomEncoding noBomEncoding, bool allowComments, string filePath, IEnumerable<CsvColumnInfo> propertiesInfo, Action<TRow> relationsInit)
+        protected CsvTableBase(
+            bool isStringInternEnabled,
+            char? csvSeparator,
+            NoBomEncoding noBomEncoding,
+            bool allowComments,
+            bool ignoreBadData,
+            string filePath,
+            IEnumerable<CsvColumnInfo> propertiesInfo,
+            Action<TRow> relationsInit)
             : base(isStringInternEnabled)
         {
             _csvSeparator = csvSeparator;
             _noBomEncoding = noBomEncoding;
             _allowComments = allowComments;
+            _ignoreBadData = ignoreBadData;
 
             FilePath = filePath;
 
             _cachedCsvRowMappingBase ??= new CsvRowMappingBase<TRow>(propertiesInfo, relationsInit);
         }
 
         protected IEnumerable<TRow> ReadData() =>
-            FilePath.CsvReadRows(_csvSeparator, IsStringInternEnabled, _noBomEncoding, _allowComments, _cachedCsvRowMappingBase!);
+            FilePath.CsvReadRows(_csvSeparator, IsStringInternEnabled, _noBomEncoding, _allowComments, _ignoreBadData, _cachedCsvRowMappingBase!);
 
         // ReSharper disable once UnusedMember.Global
         public abstract IEnumerable<TRow> WhereIndexed(Func<TRow, string> getProperty, string propertyName, params string[] values);

diff --git a/Src/CsvLINQPadDriver/CodeGen/CsvTableEnumerable.cs b/Src/CsvLINQPadDriver/CodeGen/CsvTableEnumerable.cs
@@ -7,8 +7,16 @@ namespace CsvLINQPadDriver.CodeGen
     internal class CsvTableEnumerable<TRow> : CsvTableBase<TRow>
         where TRow : ICsvRowBase, new()
     {
-        public CsvTableEnumerable(bool isStringInternEnabled, char? csvSeparator, NoBomEncoding noBomEncoding, bool allowComments, string filePath, IEnumerable<CsvColumnInfo> propertiesInfo, Action<TRow> relationsInit)
-            : base(isStringInternEnabled, csvSeparator, noBomEncoding, allowComments, filePath, propertiesInfo, relationsInit)
+        public CsvTableEnumerable(
+            bool isStringInternEnabled,
+            char? csvSeparator,
+            NoBomEncoding noBomEncoding,
+            bool allowComments,
+            bool ignoreBadData,
+            string filePath,
+            IEnumerable<CsvColumnInfo> propertiesInfo,
+            Action<TRow> relationsInit)
+            : base(isStringInternEnabled, csvSeparator, noBomEncoding, allowComments, ignoreBadData, filePath, propertiesInfo, relationsInit)
         {
         }
 

diff --git a/Src/CsvLINQPadDriver/CodeGen/CsvTableFactory.cs b/Src/CsvLINQPadDriver/CodeGen/CsvTableFactory.cs
@@ -12,12 +12,13 @@ public static CsvTableBase<TRow> CreateTable<TRow>(
             char? csvSeparator,
             NoBomEncoding noBomEncoding,
             bool allowComments,
+            bool ignoreBadData,
             string filePath,
             IEnumerable<CsvColumnInfo> propertiesInfo,
             Action<TRow> relationsInit)
             where TRow : ICsvRowBase, new() =>
             isCacheEnabled
-                ? new CsvTableList<TRow>(isStringInternEnabled, csvSeparator, noBomEncoding, allowComments, filePath, propertiesInfo, relationsInit)
-                : new CsvTableEnumerable<TRow>(isStringInternEnabled, csvSeparator, noBomEncoding, allowComments, filePath, propertiesInfo, relationsInit);
+                ? new CsvTableList<TRow>(isStringInternEnabled, csvSeparator, noBomEncoding, allowComments, ignoreBadData, filePath, propertiesInfo, relationsInit)
+                : new CsvTableEnumerable<TRow>(isStringInternEnabled, csvSeparator, noBomEncoding, allowComments, ignoreBadData, filePath, propertiesInfo, relationsInit);
     }
 }
diff --git a/Src/CsvLINQPadDriver/CodeGen/CsvTableList.cs b/Src/CsvLINQPadDriver/CodeGen/CsvTableList.cs
@@ -12,8 +12,16 @@ internal class CsvTableList<TRow> : CsvTableBase<TRow>, IList<TRow>
         private readonly IDictionary<string, ILookup<string, TRow>> _indices = new Dictionary<string, ILookup<string, TRow>>();
         private readonly Lazy<IList<TRow>> _dataCache;
 
-        public CsvTableList(bool isStringInternEnabled, char? csvSeparator, NoBomEncoding noBomEncoding, bool allowComments, string filePath, IEnumerable<CsvColumnInfo> propertiesInfo, Action<TRow> relationsInit)
-            : base(isStringInternEnabled, csvSeparator, noBomEncoding, allowComments, filePath, propertiesInfo, relationsInit) =>
+        public CsvTableList(
+            bool isStringInternEnabled,
+            char? csvSeparator,
+            NoBomEncoding noBomEncoding,
+            bool allowComments,
+            bool ignoreBadData,
+            string filePath,
+            IEnumerable<CsvColumnInfo> propertiesInfo,
+            Action<TRow> relationsInit)
+            : base(isStringInternEnabled, csvSeparator, noBomEncoding, allowComments, ignoreBadData, filePath, propertiesInfo, relationsInit) =>
             _dataCache = new Lazy<IList<TRow>>(() => ReadData().Cache($"{typeof(TRow).Name}:{FilePath}"));
 
         private IList<TRow> DataCache =>