[APIPUB-52] - Serilog formatting (#56)

* Add a TextFormatter class to use it primarly in AWS cloudwatch

* Add the SmartFormat library to create better templates in the TextFormatter

* Add unit test to the TextFormatter

* Improve of the TextFormatter

* Add more unit tests for the TextFormatter class

* Add the readme files for the TextFormatter class

* Remove unused usings

* Fix the path in the readme file

* Add the path to the TextFormatter in README.

README.md

@@ -123,6 +123,7 @@ When you're ready to look further, review these other topics:
 * [API Publisher Configuration](docs/API-Publisher-Configuration.md)
 * [Considerations for API Hosts](docs/Considerations-for-API-Hosts.md)
 * [Considerations for Docker Configuration and Execution](docs/Considerations-docker-configuration-and-execution.md)
+* [How to use the TextFormatter class](docs/Use-TextFormatter-Serilog.md).
 
 ## Support
 

docs/CloudWatch-configuration.md

@@ -14,4 +14,8 @@ Configure the AWS log storage parameters in the configuration file.
 
 Export AWS credentials to consume AWS parameters store inside the container
 
-![](../images/docker/setup-aws-credentials.png)
+![](../images/docker/setup-aws-credentials.png)
+
+## Looking for a different output format?
+The default sink for AWSCloudWatch publish the log as a JsonFormat, if you want to use a different one, check the following link:
+[How to use the TextFormatter class](Use-TextFormatter-Serilog.md).

docs/Use-TextFormatter-Serilog.md

@@ -0,0 +1,106 @@
+# How to use the TextFormatter class
+
+API Publisher uses by default the following output template in the _logging.json_ configuration:
+```
+[{Timestamp:yyyy-MM-dd HH:mm:ss,fff}] [{Level:u4}] [{ThreadId:00}] {SourceContext} - {Message} {Exception} {NewLine}
+```
+
+This template uses the format provided by the **Serilog** library, you can find more details on how to create your specific format:
+[Formatting Output](https://github.com/serilog/serilog/wiki/Formatting-Output)
+
+The sinks available in API Publisher are Console, File, and Aws CloudWatch. Some of these sinks have compatibility with the default output template, but in some cases that doesn't work, for example the Aws CloudWatch sink. If you run into this problem, we provide our TextFormatter which is similar to the default. Here we are going to see how to use it:
+
+## How to use it in the AWSCloudWatch:
+
+In the _logging.json_ file
+
+- Using the default format
+```
+"WriteTo:AWSCloudWatch": {
+  "Name": "AmazonCloudWatch",
+  "Args": {
+    "logGroup": "Ed-Fi-Publisher",
+    "logStreamPrefix": "Ed-Fi-Tools",
+    "restrictedToMinimumLevel": "Verbose",
+    "textFormatter": "EdFi.Tools.ApiPublisher.Core.Configuration.Serilog.TextFormatter, EdFi.Tools.ApiPublisher.Core"
+  }
+}
+```
+
+> [!NOTE]
+> The default template format provided by the TextFormatter is: ```[{Timestamp:yyyy-MM-dd HH:mm:ss,fff}] [{Level}] [{ThreadId:00}] [{SourceContext}] - {Message} {Exception} {NewLine}```
+
+
+
+- Using a custom format
+```
+"WriteTo:AWSCloudWatch": {
+  "Name": "AmazonCloudWatch",
+  "Args": {
+    "logGroup": "Ed-Fi-Publisher",
+    "logStreamPrefix": "Ed-Fi-Tools",
+    "restrictedToMinimumLevel": "Verbose",
+    "textFormatter": {
+      "type": "EdFi.Tools.ApiPublisher.Core.Configuration.Serilog.TextFormatter, EdFi.Tools.ApiPublisher.Core",
+      "format": "[{Timestamp:MM-dd HH:mm:ss}] {Level} - {Message} {Exception} {NewLine}" 
+    }
+  }
+}
+```
+
+## Values accepted in the template format:
+
+1. You should defined each one between braces, eg. ```{Level}```. 
+2. Some of them can be formatted using the .NET framework, eg. ```{Timestamp:dd-MM-yy}```  
+3. The following values are accepted by the implementation.
+
+   - **Timestamp:** Date and time of the event. It can be formatted using the [Custom date and time format string]( https://learn.microsoft.com/en-us/dotnet/standard/base-types/custom-date-and-time-format-strings) provided by .NET.
+   - **Level:** Level of the event. The values will be display as follows and cannot be formatted 
+        - Verbose => "ALL"
+        - Debug => "DEBUG"
+        - Information => "INFO"
+        - Warning => "WARN"
+        - Error => "ERROR"
+        - Fatal => "FATAL"
+   - **ThreadId:** Thread where the event was triggered. Is an Integer and can be formatted using [Custom numeric format strings](https://learn.microsoft.com/en-us/dotnet/standard/base-types/custom-numeric-format-strings) provided by .NET 
+   - **SourceContext:** Class name where the event was triggered. 
+   - **Message:** Message of the event.
+   - **Exception:** Contains the message and stacktrace of the event. Cannot be formatted.
+   - **NewLine:** A property with the value of System.Environment.NewLine.
+
+## More examples
+You can also use this format in other sinks:
+
+1. Console
+```
+"WriteTo:Console": {
+  "Name": "Console",
+  "Args": {
+    "formatter": {
+      "type": "EdFi.Tools.ApiPublisher.Core.Configuration.Serilog.TextFormatter, EdFi.Tools.ApiPublisher.Core",
+      "format": "[{Level}] - {Message} {Exception} {NewLine}"
+    }
+  }
+}
+```
+
+2. File
+```
+"WriteTo:File": {
+  "Name": "File",
+  "Args": {
+    "formatter": {
+      "type": "EdFi.Tools.ApiPublisher.Core.Configuration.Serilog.TextFormatter, EdFi.Tools.ApiPublisher.Core",
+      "format": "[{Level}] - {Message} {Exception} {NewLine}"
+    },
+    "path": "C:\\ProgramData\\Ed-Fi-API-Publisher\\Ed-Fi-API-PublisherSerilog.log"
+  }
+}
+```
+
+
+
+
+
+
+

src/EdFi.Tools.ApiPublisher.Core/Configuration/Serilog/TextFormatter.cs

@@ -0,0 +1,106 @@
+// SPDX-License-Identifier: Apache-2.0
+// Licensed to the Ed-Fi Alliance under one or more agreements.
+// The Ed-Fi Alliance licenses this file to you under the Apache License, Version 2.0.
+// See the LICENSE and NOTICES files in the project root for more information.
+
+using Serilog.Events;
+using Serilog.Formatting;
+using SmartFormat;
+using SmartFormat.Core.Settings;
+using System;
+using System.Collections.Generic;
+using System.IO;
+
+namespace EdFi.Tools.ApiPublisher.Core.Configuration.Serilog;
+
+public class TextFormatter : ITextFormatter
+{
+    private readonly string _format;
+    public const string DefaultFormat = "[{Timestamp:yyyy-MM-dd HH:mm:ss,fff}] [{Level}] [{ThreadId:00}] [{SourceContext}] - {Message} {Exception} {NewLine}";
+    public TextFormatter(string format = DefaultFormat)
+    {
+        _format = format;
+    }
+    public void Format(LogEvent logEvent, TextWriter output)
+    {
+        try
+        {
+            var sf = Smart.CreateDefaultSmartFormat(new SmartSettings
+            {
+                StringFormatCompatibility = true
+            });
+            var logEventFormatValues = new LogEventFormatValues(logEvent);
+            var formatted = sf.Format(_format, logEventFormatValues);
+            output.Write(formatted);
+        }
+        catch (Exception ex)
+        {
+            output.Write($"Unable to render log message. Reason was {ex}");
+        }
+    }
+}
+
+public class LogEventFormatValues
+{
+    private const string ThreadIdSerilogPropertyName = "ThreadId";
+    private const string SourceContextSerilogPropertyName = "SourceContext";
+
+    public DateTime Timestamp => _logEvent.Timestamp.DateTime;
+    public string Level => GetShortFormatLevel(_logEvent.Level);
+    public string SourceContext => GetStringValueFromProperty(_logEvent.Properties.GetValueOrDefault(SourceContextSerilogPropertyName));
+    public string Message => _logEvent.MessageTemplate.Render(_logEvent.Properties);
+    public string Exception => $"{_logEvent.Exception?.Message} {_logEvent.Exception?.StackTrace}";
+    public double ThreadId => GetIntValueFromProperty((_logEvent.Properties.GetValueOrDefault(ThreadIdSerilogPropertyName)));
+
+    public string NewLine => Environment.NewLine;
+
+    private readonly LogEvent _logEvent;
+
+    public LogEventFormatValues(LogEvent logEvent)
+    {
+        _logEvent = logEvent;
+    }
+
+    private string GetShortFormatLevel(LogEventLevel logEventLevel)
+    {
+        var value = logEventLevel switch
+        {
+            LogEventLevel.Verbose => "ALL",
+            LogEventLevel.Debug => "DEBUG",
+            LogEventLevel.Information => "INFO",
+            LogEventLevel.Warning => "WARN",
+            LogEventLevel.Error => "ERROR",
+            LogEventLevel.Fatal => "FATAL",
+            _ => throw new ArgumentException("Unexpected value for LogEvent.Level", nameof(logEventLevel))
+        };
+
+        return value.ToString();
+    }
+
+    private int GetIntValueFromProperty(LogEventPropertyValue logEventPropertyValue)
+    {
+        int result = 0;
+        if (logEventPropertyValue is ScalarValue scalar && scalar.Value != null)
+        {
+            if (scalar.Value is int intValue)
+            {
+                result = intValue;
+            }
+        }
+        return result;
+    }
+
+    private string GetStringValueFromProperty(LogEventPropertyValue logEventPropertyValue)
+    {
+        string result = String.Empty;
+        if (logEventPropertyValue is ScalarValue scalar && scalar.Value != null)
+        {
+            if (scalar.Value is string stringValue)
+            {
+                result = stringValue;
+            }
+        }
+        return result;
+    }
+
+}

src/EdFi.Tools.ApiPublisher.Core/EdFi.Tools.ApiPublisher.Core.csproj

@@ -18,6 +18,7 @@
     <PackageReference Include="Microsoft.Extensions.Configuration.Json" Version="8.0.0" />
     <PackageReference Include="Microsoft.Extensions.DependencyInjection" Version="8.0.0" />
     <PackageReference Include="Newtonsoft.Json" Version="13.0.3" />
+    <PackageReference Include="SmartFormat" Version="3.4.0" />
     <PackageReference Include="System.Threading.Tasks.Dataflow" Version="8.0.0" />
   </ItemGroup>
 </Project>