GeoShape visualisation from ES

[DavidKHE]

Hi All,

I am using for sometime time now Vega with Kibana and find it very powerful.
In our current implementation I am visualising various KPIs over US counties.
essentially so far using the URL directly from open source.
///Data
{
"name": "ESlinkCounties",
"url": "https://vector.maps.elastic.co/blob/1543537167969055?elastic_tile_service_tos=agree",
"format": {"type": "topojson", "feature": "data"},
"transform": [...]
}
///Mark
{
"type": "shape",
"name": "UScountyOVERVIEW",
"from": {"data": "ESlinkCounties"},
"encode": {
"enter": {"opacity": {"value": 0.4}},
"update": {
"fill": {
"scale": "color",
"field": {"signal": "KPIs"}
},
"tooltip": [{"field": {"signal": "KPIs"}}]
},
"hover": {"fill": {"value": "black"}}
},
"transform": [
{"type": "geoshape", "projection": "projection"}
]
}

so far it was working fine. But now, we have a restriction coming from Elastic Cloud that no external URL could be used.

hence I created additional index with GeoShape type mapping for the US-Counties.
as validation, tested this with Kibana Maps and it looks OK.

But, now I could not find the way how to visualise the same GeoShapes in vega.

Can someone give some hints how can I visualise GeoShape (JSON), in Vega?

thanks,
David

[nyurik]

@DavidKHE if i remember correctly, you can change kibana yaml config settings in the Elastic cloud, enabling external access in Vega

[thomasneirynck]

hi @DavidKHE

tested this with Kibana Maps and it looks OK.

Are there any limitations you're experiencing with the Maps-app, that you cannot use the Maps-app for your purposes? This sort of choropleth map (where you color shapes based on some metric) should be doable out-of-the-box (for some examples, see https://www.elastic.co/guide/en/kibana/current/vector-style.html)

wrt. avoiding external access, there's a few other approaches to avoid having your users use CORS for external access to this data.

• you could proxy the request through Kibana (as long as Kibana has outbound access to the outside internet). https://www.elastic.co/guide/en/kibana/current/maps-connect-to-ems.html Set the map.proxyElasticMapsServiceInMaps flag to true.
• index the US-counties directly into Elasticsearch. The Maps-app can visualize individual documents as well (download that file here https://maps.elastic.co/#file/usa_counties)

[DavidKHE]

Hi,

thanks for the quick feedback.

@nyurik - you are referring to the vega.enableExternalURL: true? if yes, than unfortunately this option is locked. Moreover there is an issue that because of this HREF would not work in VEGA to allow even dashboards navigation.

@thomasneirynck - there are two main reasons for me to use VEGA instead of the Maps-app.

1. when using Vega I can make weighted average. that is more accurate and useful for me. currently the Map-App support avg/min/max/count
2. I can make in one "shot" multiple KPIs and provide selection for user to select various KPIs and map will be updated automatically. in case using the Map-App need to create layer and this is more complicated to end user to "Hide"/"Unhide" each time they like to see specific KPI.

I am sharing below the sample code that is essentially not working. my feeling this is something with the "formatting" and/or transform

{
"$schema": "https://vega.github.io/schema/vega/v4.3.0.json",
"config": {
"kibana": {
"type": "map",
"latitude": 40.7,
"longitude": -74,
"zoom": 7,
"mapStyle": "default",
"minZoom": 1,
"maxZoom": 13,
"zoomControl": true,
"scrollWheelZoom": true,
"delayRepaint": true
}
},
"data": [
{
"name": "ES",
"values":[
{
"_index" : "us_state_geojson_local",
"_type" : "_doc",
"_id" : "wMSQxm4BjvdUS135E5ZF",
"_score" : 1.0,
"_source" : {
"coordinates" : {
"type" : "polygon",
"coordinates" : [
[
[
-89.954995,
47.290458
],
[
-90.09901,
47.039972
],
[
-90.423042,
46.547639
],
[
-90.387039,
46.539001
],
[
-90.315032,
46.521726
],
[
-90.243024,
46.504452
],
[
-90.207021,
46.469902
],
[
-90.207021,
46.461264
],
[
-90.171017,
46.461264
],
[
-90.171017,
46.452627
],
[
-90.171017,
46.44399
],
[
-90.171017,
46.435352
],
[
-90.171017,
46.400802
],
[
-90.135014,
46.400802
],
[
-90.135014,
46.392165
],
[
-90.135014,
46.366253
],
[
-90.135014,
46.357615
],
[
-90.09901,
46.357615
],
[
-90.135014,
46.348978
],
[
-90.135014,
46.34034
],
[
-89.018902,
46.115767
],
[
-88.80288,
46.029393
],
[
-88.766877,
46.029393
],
[
-88.766877,
46.020756
],
[
-88.730873,
46.020756
],
[
-88.694869,
46.012118
],
[
-88.658866,
45.994843
],
[
-88.658866,
45.986206
],
[
-88.586859,
46.012118
],
[
-88.550855,
46.020756
],
[
-88.514851,
46.020756
],
[
-88.478848,
45.994843
],
[
-88.442844,
45.994843
],
[
-88.442844,
45.986206
],
[
-88.406841,
45.986206
],
[
-88.370837,
45.986206
],
[
-88.334833,
45.960294
],
[
-88.262826,
45.951656
],
[
-88.226823,
45.951656
],
[
-88.226823,
45.943019
],
[
-88.190819,
45.943019
],
[
-88.154815,
45.943019
],
[
-88.154815,
45.934381
],
[
-88.118812,
45.925744
],
[
-88.118812,
45.917106
],
[
-88.118812,
45.787545
],
[
-87.938794,
45.761633
],
[
-87.938794,
45.752995
],
[
-87.90279,
45.752995
],
[
-87.794779,
45.709808
],
[
-87.830783,
45.562972
],
[
-87.794779,
45.46796
],
[
-87.866787,
45.364311
],
[
-87.866787,
45.347036
],
[
-87.830783,
45.347036
],
[
-87.830783,
45.355673
],
[
-87.794779,
45.347036
],
[
-87.686769,
45.381586
],
[
-87.650765,
45.312486
],
[
-87.722772,
45.269299
],
[
-87.722772,
45.157013
],
[
-87.614761,
45.096551
],
[
-87.434743,
45.079276
],
[
-87.39874,
45.2002
],
[
-87.182718,
45.329761
],
[
-87.110711,
45.442048
],
[
-86.750675,
45.442048
],
[
-86.246625,
45.234749
],
[
-86.534653,
45.053363
],
[
-86.642664,
44.949714
],
[
-86.750675,
44.776966
],
[
-86.786679,
44.621492
],
[
-86.858686,
44.517843
],
[
-86.858686,
44.440106
],
[
-87.038704,
44.103246
],
[
-87.038704,
43.99096
],
[
-87.110711,
43.818211
],
[
-87.110711,
43.628188
],
[
-87.146715,
43.610913
],
[
-87.146715,
43.299965
],
[
-87.110711,
43.248141
],
[
-87.110711,
43.023568
],
[
-87.074707,
42.997655
],
[
-87.074707,
42.764445
],
[
-87.038704,
42.74717
],
[
-87.038704,
42.496684
],
[
-87.0027,
42.496684
],
[
-87.110711,
42.099363
],
[
-87.182718,
41.926614
],
[
-87.218722,
41.762503
],
[
-85.490549,
41.762503
],
[
-84.806481,
41.762503
],
[
-84.806481,
41.693403
],
[
-83.474347,
41.727953
],
[
-83.40234,
41.736591
],
[
-83.114311,
41.961164
],
[
-83.114311,
42.289386
],
[
-83.042304,
42.332573
],
[
-82.826283,
42.37576
],
[
-82.646265,
42.557146
],
[
-82.50225,
42.608971
],
[
-82.466247,
42.721257
],
[
-82.466247,
42.937193
],
[
-82.322232,
43.170404
],
[
-82.106211,
43.593638
],
[
-82.178218,
43.913223
],
[
-82.286229,
44.250082
],
[
-82.322232,
44.509205
],
[
-82.50225,
45.338399
],
[
-83.582358,
45.822095
],
[
-83.438344,
45.994843
],
[
-83.582358,
46.10713
],
[
-83.834383,
46.10713
],
[
-83.942394,
46.055305
],
[
-84.122412,
46.245329
],
[
-84.122412,
46.530364
],
[
-84.266427,
46.495814
],
[
-84.374437,
46.513089
],
[
-84.554455,
46.461264
],
[
-84.770477,
46.634013
],
[
-84.842484,
46.893136
],
[
-85.310531,
47.083159
],
[
-88.190819,
48.231937
],
[
-88.370837,
48.309674
],
[
-88.694869,
48.249212
],
[
-88.766877,
48.206025
],
[
-89.342934,
47.972815
],
[
-89.486949,
48.016002
],
[
-89.666967,
47.748241
],
[
-89.774977,
47.540943
],
[
-89.954995,
47.290458
]
]
]
},
"iso_3166_2" : "US-MI",
"postal" : "MI",
"name" : "Michigan",
"label_en" : "Michigan"
}
},
{
"_index" : "us_state_geojson_local",
"_type" : "_doc",
"_id" : "wcSQxm4BjvdUS135E5ZF",
"_score" : 1.0,
"_source" : {
"coordinates" : {
"type" : "multipolygon",
"coordinates" : [
[
[
[
-64.968497,
17.681349
],
[
-64.932493,
17.629525
],
[
-64.788479,
17.620887
],
[
-64.536454,
17.698624
],
[
-64.50045,
17.741811
],
[
-64.572457,
17.836823
],
[
-64.824482,
17.828186
],
[
-64.932493,
17.802273
],
[
-64.968497,
17.681349
]
]
],
[
[
[
-65.112511,
18.432806
],
[
-65.148515,
18.355069
],
[
-65.148515,
18.225507
],
[
-65.076508,
18.225507
],
[
-64.932493,
18.260057
],
[
-64.89649,
18.190958
],
[
-64.824482,
18.18232
],
[
-64.788479,
18.260057
],
[
-64.644464,
18.268695
],
[
-64.644464,
18.363706
],
[
-64.860486,
18.398256
],
[
-64.89649,
18.458718
],
[
-65.076508,
18.458718
],
[
-65.112511,
18.432806
]
]
]
]
},
"iso_3166_2" : "US-VI",
"postal" : "VI",
"name" : "United States Virgin Islands",
"label_en" : "United States Virgin Islands"
}
}
],
"format": { "type": "json"}
}
],
"projections": [{"name": "projection", "type": "albersUsa"}],
"marks": [
{
"type": "shape",
"from": {"data": "ES"},
"encode": {
"update": {
"strokeWidth": {"value": 0.5},
"stroke": {"value": "darkblue"},
"fill": {"value": "lightblue"},
"fillOpacity": {"value": 0.5}
},
"hover": {
"fill": {"value": "#66C2A5"},
"strokeWidth": {"value": 2},
"stroke": {"value": "#FC8D62"}
}
},
"transform": [
{
"type":"geoshape",
"projection": "projection"
}
]
}
]

}

[thomasneirynck]

@DavidKHE thx for the explanation, that's a great help.

Note that wrt. (2), you could think about creating an input-control that would create a filter https://www.elastic.co/guide/en/kibana/current/add-input-controls.html). You'd have a single layer with all the data, but the control would filter based on some class-value (which I assume is how you are doing this now).

For (1), could you create enhancement requests in the Kibana-repo (https://github.com/elastic/kibana/issues/new), perhaps with some examples?

For your follow-up question, one thing to consider is that when shapes are returned from Elasticsearch, their formatting diverges a little from valid Geojson. In this case, note how all the geometry-types (polygon, multipolygon) are lowercase, instead of camel-cased. This might be the reason Vega is not being able to display the geometries.

[DavidKHE]

Hi @thomasneirynck,

Thanks for the feedback. I will try to test those camel-cased and update for the results.

regarding usage of filter. this is an option, but not really "clean". if I find no other better method. this is likely what will be done.
basically in Vega using signal, that will visualise the KPI, that is specific field from the same document that already been queried. same document is containing multiple KPIs in different fields.
If we are using the filters from add-input-controls this meaning that need to re-create the documents. more documents and than making that each document has only one KPI.

I still in favour of Vega. my feeling this is really something more of formatting.
hopefully could make it. :-)

thanks,
David

[nyurik]

@DavidKHE i just confirmed - Elastic currently doesn't enable that flag except for premium accounts (in which case you can just request it via support request). I think they are working on a system to allow it, but don't know how long it will be.

[DavidKHE]

@nyurik - thanks. opened ticket.

I will update on the results.

thanks,
David

[DavidKHE]

@nyurik - unfortunately the ticket we raised was rejected.

Case Update for #00464038 Changes on Kibana Configuration [ ref:_00Db0H5KI._5004MYnr1Q:ref ].

do you have any contact internally maybe to help changing this property?

thanks,
David