19 Aug 01:53

russcam

50a7e34

6.8.2

6.8.1...6.8.2

Features & Enhancements

#4005 Add a CloudConnectionPool IConnectionPool type
This makes it easier than ever to work with Elasticsearch service on Elastic Cloud 🎉
#4002 make sure types that implement DebuggerDisplay override ToString()
#4015 Add if_seq_no and if_primary_term to bulk updates.
#4024 Add Missing method to Date Range Aggregation. thanks @fontourafernando 👍

Bug Fixes

#3977 MoreLikeThis Query not working with artificial document
#3981 JsonNetSerializer doesn't work for GeoLocation
#3997 Serialize GeoLocation with internal serializer
#3999 Fix 6.x query validity in integration tests
#4014 BulkAll() improvements

Misc

#4008 Fix Composite Aggregation integration test
#4022 Delete existing docs when generating new docs

View the full list of issues and PRs

Assets 2

16 Aug 09:10

codebrain

7.2.0

58f806a

7.2.0

7.1.0...7.2.0

Features & Enhancements

#3979 Update API REST specs to 7.2.0
#3851 Map failures on NodeStatistics
#3943 Support for if_seq_no and if_primary_term in the bulk update request
#3963 GetTaskResponse does not expose Failures for reindex task
#3966 Expose Response on GetTaskResponse
#3986 Add 7.2 to the azure devops matrix
#4002 Ensure types that implement DebuggerDisplay override ToString
#4005 Add a CloudConnectionPool
#4015 Add if_seq_no and if_primary_term to bulk updates.
#4024 Add missing property to DateRangeAggregation
#4014 BulkAll() improvements
#3951 BulkAll() requires DefaultIndex or DefaultMappingFor with index
#3983 Implement distance_feature query
#3984 Introduce the columnar option for SQL REST requests.
#3989 AuditTrail integration assertion fix
#3992 Expose external refreshes through the stats API
#3995 Add packaging to cluster stats response.
#3996 Add created_by info to usage stats.
#3998 Add numeric_type sorting option.
#4019 Map node failures.

Deprecations

#3988 Force selection of calendar or fixed intervals in date histo aggregations. Adding [Obsolete] attributes as required.

Bug Fixes

#3970 Bitnami image does not work without DisablePing
#3971 Indices.Exists fails in Full .NET framework, works in .NET Core
#3972 Fix for #3971, change check for mimeType to include empty string
#3981 BUG JsonNetSerializer doesn't work for GeoLocation
#4026 Ensure GeoLocationFormatter returns GeoLocation type
#4027 Normalize exception from cancelling an observable coordinated operation
#4007 Fixed formatting typos (’ --> ')

Docs

#4022 Delete existing docs when generating new docs
#3968 Improve Index name inference docs
#3976 Doc generation should remove docs directory contents

View the full list of issues and PRs

Assets 2

24 Jul 10:27

russcam

6.8.1

178d354

6.8.1

6.8.0...6.8.1

Features

#3752 Add net461 target framework for NEST.JsonNetSerializer nuget package
#3784 Add User-Agent string to API calls to Elasticsearch
#3823 and #3827 Add index name to ICreateIndexResponse

Docs

#3914 and #3969 Improve Index name inference docs

Bug Fixes

#3778 HandleNestTypesOnSourceJsonConverter does not serialize/deserialize Nest.DateRange
#3819 and #3820 Use CultureInfo.InvariantCulture when writing with GeoWKTWriter

View the full list of issues and PRs

Assets 2

19 Jul 07:57

codebrain

7.1.0

c3f2796

7.1.0

7.0.1...7.1.0

Features & Enhancements

#3843, #3919 Implement Script Score Query
#3911 Include size of snapshot in snapshot metadata
#3912 Implement Api Key authorisation on requests
#3927 Add CCR Follow Info API
#3932 Implement GeoTileGrid aggregation
#3933 date_nanos type implementation
#3934 Implement adjust_offsets on word delimiter graph token filter
#3937 Additions to Invalidate Token API
#3938, #3941 Add rank_feature, rank_features field datatypes and rank_feature query
#3939 Implement Forget Follower API
#3940 Implement Freeze and Unfreeze API

Bug Fixes

#3899 Anchored DateMath no longer serializes time zone information for local DateTime
#3904 Using weight parameter in script_score resulted in an invalid request
#3905 TypeMappingDescriptor<T> still supports setting AllField
#3906 TrackTotalHits is ignored during MultiSearch call
#3907 UseSocketsHttpHandler fix
#3909 Serialize only script in script_score function
#3910 Include time zone information for Local and Utc DateTimeKind
#3917 Add track_total_hits to search request body
#3918 Obsolete AllField and IndexField
#3920 Issue in deserializing WKT Geoshape
#3922 Support numbers with exponent in WKT
#3926 JsonSerializerSettings not honored for Field Expressions
#3936 Exception thrown when using cat APIs with nonexistent index
#3942 Honour IPropertyMappingProvider on custom IElastisearchSerializer
#3944 Handle cat API 404 responses
#3945 NEST requires explicit installation of System.Diagnostics.DiagnosticSource nuget package.
#3948 Add System.Diagnostics.DiagnosticSource reference for NetStandard 2.0
#3952 Transfer-Encoding: chunked is enabled by default
#3958 Fix default chunked transfer encoding and expose as configurable

Deprecations

#3905 TypeMappingDescriptor still supports setting AllField

View the full list of issues and PRs

Assets 2

05 Jul 04:43

russcam

7.0.1

a2ac5c0

7.0.1

7.0.0...7.0.1

Bug Fixes

#3913 Enabling http compression results in empty http request body (PR #3921)

View the full list of issues and PRs

Assets 2

27 Jun 14:17

russcam

7.0.0

00eaaa9

7.0.0

7.0.0-beta1...7.0.0

NEST & Elasticsearch.Net 7.0: Now GA!

After many months of work, two alphas and a beta, we are pleased to announce the GA release of the NEST and Elasticsearch.Net 7.0 clients.

The overall themes of this release have been based around faster serialization, performance improvements, codebase simplification, and ensuring parity with the many new features available in Elasticsearch 7.0.

Types removal

Specifying types within the .NET clients is now deprecated in 7.0, in line with the overall Elasticsearch type removal strategy.

In instances where your index contains type information and you need to preserve this information, one recommendation is to introduce a property to describe the document type (similar to a table per class with discriminator field in the ORM world) and then implement a custom serialization / deserialization implementation for that class.

This Elasticsearch page details some other approaches.

Faster Serialization

After internalizing the serialization routines, and IL-merging the Newtonsoft.Json package in 6.x, we are pleased to announce that the next stage of serialization improvements have been completed in 7.0.

Both SimpleJson and Newtonsoft.Json have been completely removed and replaced with an implementation of Utf8Json, a fast serializer that works directly with UTF-8 binary. This has yielded a significant performance improvement, which we will be sharing in more detail in a later blog post.

With the move to Utf8Json, we have removed some features that were available in the previous JSON libraries that have proven too onerous to carry forward at this stage.

JSON in the request is never indented, even if SerializationFormatting.Indented is specified. The serialization routines generated by Utf8Json never generate an IJsonFormatter<T> that will indent JSON, for performance reasons. We are considering options for exposing indented JSON for development and debugging purposes.
NEST types cannot be extended by inheritance. With NEST 6.x, additional properties can be included for a type by deriving from that type and annotating these new properties. With the current implementation of serialization with Utf8Json, this approach will not work.
Serializer uses Reflection.Emit. Utf8Json uses Reflection.Emit to generate efficient formatters for serializing types that it sees. Reflection.Emit is not supported on all platforms, for example, UWP, Xamarin.iOS, and Xamarin.Android.
Elasticsearch.Net.DynamicResponse deserializes JSON arrays to List<object>. SimpleJson deserialized JSON arrays to object[], but Utf8Json deserializes them to List<object>. This change is preferred for allocation and performance reasons.
Utf8Json is much stricter when deserializing JSON object field names to C# POCO properties. With the internal Json.NET serializer in 6.x, JSON object field names would attempt to be matched with C# POCO property names first by an exact match, falling back to a case insensitive match. With Utf8Json in 7.x however, JSON object field names must match exactly the name configured for the C# POCO property name.

We believe that the trade-off of features vs. GA release has been worthwhile at this stage. We hold a view to address some of these missing features in a later release.

High to Low level client dispatch changes

In 6.x, the process of an API call within NEST looked roughly like this

client.Search()
  => Dispatch()
    => LowLevelDispatch.SearchDispatch()
      => lowlevelClient.Search()
        => lowlevelClient.DoRequest()

With 7.x, this process has been changed to remove dispatching to the low-level client methods. The new process looks like this

client.Search()
  => lowlevelClient.DoRequest()

This means that in the high-level client IRequest now builds its own URLs, with the upside that the call chain is shorter and allocates fewer closures. The downside is that there are now two URL building mechanisms, one in the low-level client and a new one in the high-level client. In practice, this area of the codebase is kept up to date via code generation, so it does not place any additional burden on development.

Given the simplified call chain and debugging experience, we believe this is an improvement worth making.

Namespaced API methods and Upgrade Assistant

As the API surface of Elasticsearch has grown to well over 200 endpoints, so has the number of client methods exposed, leading to an almost overwhelming number to navigate and explore through in an IDE. This is further exacerbated by the fact that the .NET client exposes both synchronous and asynchronous API methods for both the fluent API syntax as well as the object initializer syntax.

To address this, the APIs are now accessible through sub-properties on the client instance.

For example, in 6.x, to create a machine learning job

var putJobResponse = client.PutJob<Metric>("id", c => c
    .Description("Lab 1 - Simple example")
    .ResultsIndexName("server-metrics")
	.AnalysisConfig(a => a
        .BucketSpan("30m")
        .Latency("0s")
    	.Detectors(d => d.Sum(c => c.FieldName(r => r.Total)))
	)
	.DataDescription(d => d.TimeField(r => r.Timestamp))
);

This has changed to

var putJobResponse = client.MachineLearning.PutJob<Metric>("id", c => c
    .Description("Lab 1 - Simple example")
    .ResultsIndexName("server-metrics")
	.AnalysisConfig(a => a
        .BucketSpan("30m")
        .Latency("0s")
    	.Detectors(d => d.Sum(c => c.FieldName(r => r.Total)))
	)
	.DataDescription(d => d.TimeField(r => r.Timestamp))
);

Notice the client.MachineLearning.PutJob method call in 7.0, as opposed to client.PutJob in 6.x.

We believe this grouping of functionality leads to a better discoverability experience when writing your code, and improved readability when reviewing somebody else's.

The Upgrade Assistant

To assist developers in migrating from 6.x, we have published the Nest.7xUpgradeAssistant Nuget package. When included in your project and the using Nest.ElasticClientExtensions; directive is added, calls will be redirected from the old API method names to the new API method names in 7.0. The result is that your project will compile and you won't need to immediately update your code to use the namespaced methods; instead you'll see compiler warnings indicating the location of the new API methods in 7.0.

This package is to assist developers migrating from 6.x to 7.0 and is limited in scope to this purpose. It is recommended that you observe the compiler warnings and adjust your code as indicated.

Observability and DiagnosticSource

7.0 introduces emitting System.Diagnostics.DiagnosticSource information from the client, during a request. The goal is to enable rich information exchange with the Elastic APM .NET agent and other monitoring libraries.

We emit DiagnosticSource information for key parts of a client request via an Activity event, shipping support for Id, ParentId, RootId, as well as request Duration.

To facilitate wiring this up to DiagnosticListener.AllListeners, we ship both with static access to the publisher names and events through Elasticsearch.Net.Diagnostics.DiagnosticSources as well as strongly typed listeners, removing the need to cast the object passed to activity start/stop events.

An example listener implementation that writes events to the console is given below

private class ListenerObserver : IObserver<DiagnosticListener>
{
  public void OnCompleted() => Console.WriteLine("Completed");
  
  public void OnError(Exception error) => Console.Error.WriteLine(error.Message);
	
  public void OnNext(DiagnosticListener value)
  {
		void WriteToConsole<T>(string eventName, T data)
		{
			var a = Activity.Current;
			Console.WriteLine($"{eventName?.PadRight(30)} {a.Id?.PadRight(32)} {a.ParentId?.PadRight(32)} {data?.ToString().PadRight(10)}");
		}
		if (value.Name == DiagnosticSources.AuditTrailEvents.SourceName)
			value.Subscribe(new AuditDiagnosticObserver(v => WriteToConsole(v.Key, v.Value)));

		if (value.Name == DiagnosticSources.RequestPipeline.SourceName)
			value.Subscribe(new RequestPipelineDiagnosticObserver(
				v => WriteToConsole(v.Key, v.Value),
				v => WriteToConsole(v.Key, v.Value))
			);

		if (value.Name == DiagnosticSources.HttpConnection.SourceName)
			value.Subscribe(new HttpConnectionDiagnosticObserver(
				v => WriteToConsole(v.Key, v.Value),
				v => WriteToConsole(v.Key, v.Value)
			));

		if (value.Name == DiagnosticSources.Serializer.SourceName)
			value.Subscribe(new SerializerDiagnosticObserver(v => WriteToConsole(v.Key, v.Value)));
  }
}

Using the following example

var pool = new SniffingConnectionPool(new[] { node.NodesUris().First() });
var settings = new ConnectionSettings(pool).SniffOnStartup();
var client = new ElasticClient(settings);
var x = client.Search<object>(s=>s.AllIndices());
               	
Console.WriteLine(new string('-', Console.WindowWidth - 1));
                                  	
var y = client.Search<object>(s=>s.Index("does-not-exist"));

Emits the following console output:

SniffOnStartup.Start       	|59e275e-4f9c835d189eb14a.    	Event: SniffOnStartup
Sniff.Start                	|59e275e-4f9c835d189eb14a.1. 	|59e275e-4f9c835d189eb14a.   	GET _nodes/http,settings
Sniff.Start                	|59e...

Assets 2

12 Jun 13:11

codebrain

7.0.0-beta1

a9b2fb1

7.0.0-beta1 Pre-release

Pre-release

7.0.0-alpha2...7.0.0-beta1

Following the advice given in 7.0.0-alpha1 the namespacing change has been implemented for this release.

Namespaced API methods

In both Elasticsearch.Net and NEST clients, API methods are made available on the root of the client. As the API surface of Elasticsearch has grown to well over 200 endpoints, so has the number of API methods exposed, leading to an almost overwhelming number of methods to navigate through to choose the method aligning with the desired API call. This is further exacerbated by the fact that the client exposes synchronous and asynchronous API methods for both the fluent API syntax as well as the object initializer syntax.

To address this, the APIs are now accessible through sub-properties on the client instance.

For example, in 6.x, to create a Machine Learning job the code is:

var putJobResponse = client.PutJob<Metric>("id", c => c
    .Description("Lab 1 - Simple example")
    .ResultsIndexName("server-metrics")
    .AnalysisConfig(a => a
        .BucketSpan("30m")
        .Detectors(d => d.Sum(c => c.FieldName(r => r.Total)))
    )
    .DataDescription(d => d.TimeField(r => r.Timestamp));
);

This has changed to:

var putJobResponse = client.MachineLearning.PutJob<Metric>("id", c => c
    .Description("Lab 1 - Simple example")
    .ResultsIndexName("server-metrics")
    .AnalysisConfig(a => a
        .BucketSpan("30m")
        .Detectors(d => d.Sum(c => c.FieldName(r => r.Total)))
    )
    .DataDescription(d => d.TimeField(r => r.Timestamp));
);

Notice the client.MachineLearning.PutJob method call in 7.0.0-beta1, as opposed to client.PutJob in 6.x

Features & Enhancements

#3809 Add overloads of SerializeToString and SerializeToBytes
#3765 Clean up GET /_cluster/stats
#3807 Use indexers to set default mapping values
#3806 User SecureString for Basic and Proxy passwords

Bug Fixes

#3771 Breaking formatter backwards compatibility
#3805 Routing must be specified when doc used
#3706 Calling ConnectionSettings.DefaultMappingFor throws ArgumentException when the Type has been previously added
#3719 Wrong converting datetime fields
#3751 Unable to get raw document JSON without deserializing

View the full list of issues and PRs

Assets 2

29 May 06:31

codebrain

6.8.0

fd19c75

6.8.0

6.7.0...6.8.0

Features & Enhancements

#3757 Include stored_scripts response in cluster metadata response.
#3758 Add buckets to AutoDateHistogramAggregation.
#3759 Additional Rollup Stats
#3763 Attribute SuggestBucket for serialization
#3764 XPack info and usage improvements
#3766 Add pretty names (and other properties) to the cluster stats response
#3767 Add doc’s sequence number + primary term to get responses

Deprecations

#3770 Set obsolete warning on GeoShape properties.

Bug fixes

#3775 Convert rollover lifecycle action to accept max size as a string and not long?
#3776 Use a custom converter for GeoOrientation to tolerate alternative options in Elasticsearch

View the full list of issues and PRs

Assets 2

09 May 08:09

russcam

6.7.0

d17b00a

6.7.0

Major feature highlights include support for the Security Api Keys and Index Lifecycle Management endpoints.

6.6.0...6.7.0

Features & Enhancements

#3684 Security API Key functionality
#3671 Index Lifecycle functionality
#3627 Add Median Absolute Deviation Aggregation
#3632 Add Machine Learning filter APIs and detector custom rules
#3634 Add User Dictionary Rules to NoriTokenizer
#3635 Add prebuilt ICU Analyzer
#3686 & #3704 Add support for is_write_index for the create index API - thanks @KPStolk 👍
#3631 Remove delegate allocations in Fluent.Assign
#3653 Use ContractJsonConverterAttribute where possible
#3665 Remove superfluous pragma directives
#3727 Additional details on CCR auto follow stats
#3725 Add realm information for Authenticate API
#3723 Add password_hash parameter to PutUser
#3718 Allow geo_shape types to be coerced into valid shapes.

Bug Fixes

#3619 ignore_missing added to all ingest processors
#3629 Set the timeout on the bulk call to use the request timeout
#3696 More Like This Query Conditionless behaviour
#3699 Update ClusterHealthResponse.cs to align with ClusterHealthResponse.java - thanks @SpencerLN 👍
#3698 Add properties missing from ClusterHealthResponse.cs - thanks @SpencerLN 👍

View the full list of issues and PRs

Assets 2

08 May 08:57

russcam

7.0.0-alpha2

5eb459b

7.0.0-alpha2 Pre-release

Pre-release

This is another alpha release of the .NET client, as we move a step closer to GA. Thank you for all your feedback on the 7.0.0-alpha1 client, it has been incredibly helpful in squashing bugs in the new serializer implementation.

7.0.0-alpha1...7.0.0-alpha2

Features

#3680 Implement Intervals query
#3698 Add properties currently missing on ClusterHealthResponse. Thanks @SpencerLN 👍
#3714 Add SequenceNumber and PrimaryTerm to Hits, Get and MulitGetHit
#3704 Add is_write_index to aliases on create index API
#3668 Improve field resolver performance

Breaking Changes

#3670 Majority of response interfaces are gone

This reduces the interface type explosion within the client and simplifies the async API code paths. There is a discussion to keep interfaces for the most commonly mocked responses, like ISearchResponse<T>
#3685 Strongly typed property expressions

Benchmarking demonstrates a performance improvement in using Expression<Func<T, TValue>> instead of `Expression<Func<T, object>>, particularly on .NET Framework.

Bug Fixes

#3697 Fix MoreLikeThis query conditionless behaviour
#3715 Fix ExtendedStats and Percentiles aggregation deserialization
#3717 Fix empty geo_bounds aggregation
#3678 Fix DateHistogramBucket deserialization when there are no sub aggregations
#3700 Generate API paths without leading forward slashes
#3726 Remove _source_include and _source_exclude from generated types
#3679 Fix utf8json escape characters

Misc

#3730 Use the MemoryStream buffers, where possible, instead of copying byte array. Thanks @Henr1k80 👍
#3677 Refactor ServerError, Error and ErrorCause
#3702 Support deprecated Name property on ElasticsearchTypeAttribute
#3666 Rename SynchronizedCollection

View the full list of issues and PRs

Assets 2

Releases: elastic/elasticsearch-net

6.8.2

Features & Enhancements

Bug Fixes

Misc

7.2.0

Features & Enhancements

Deprecations

Bug Fixes

Docs

6.8.1

Features

Docs

Bug Fixes

7.1.0

Features & Enhancements

Bug Fixes

Deprecations

7.0.1

Bug Fixes

7.0.0

NEST & Elasticsearch.Net 7.0: Now GA!

Types removal

Faster Serialization

High to Low level client dispatch changes

Namespaced API methods and Upgrade Assistant

The Upgrade Assistant

Observability and DiagnosticSource

7.0.0-beta1

Namespaced API methods

Features & Enhancements

Bug Fixes

6.8.0

Features & Enhancements

Deprecations

Bug fixes

6.7.0

Features & Enhancements

Bug Fixes

7.0.0-alpha2

Features

Breaking Changes

Bug Fixes

Misc