Read full config with gnmi get #143

ganglyu · 2023-08-15T08:26:34Z

Why I did it

Need to read full config for SONiC

How I did it

Read CONFIG DB and convert to json.

How to verify it

Verified with unit test and end2end test.

Which release branch to backport (provide reason below if selected)

Description for the changelog

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

qiluo-msft · 2023-08-17T06:37:56Z

sonic_data_client/mixed_db_client.go

@@ -337,7 +337,9 @@ func (c *MixedDbClient) populateDbtablePath(path *gnmipb.Path, value *gnmipb.Typ

 	tblPath.dbNamespace = dbNamespace
 	tblPath.dbName = targetDbName
-	tblPath.tableName = stringSlice[1]
+	if len(stringSlice) > 1 {


stringSlice

Suggest move together the code related to len(stringSlice) and tableName. #Closed

qiluo-msft · 2023-08-17T06:40:44Z

sonic_data_client/mixed_db_client.go

+		// Did no provide table name
+		// Get all tables in the DB
+		if tblPath.dbName == "COUNTERS_DB" {
+			return fmt.Errorf("Can not read all tables in COUNTERS_DB")


Can not

This is interesting limitation. Is there a reason? Is it in any design doc? Or just not implemented today. #Closed

tables in COUNTERS_DB other than COUNTERS table doesn't have keys.
So, I prefer not to implement it today.

I have Added TODO and new unit test.

qiluo-msft · 2023-08-17T06:46:03Z

sonic_data_client/mixed_db_client.go

@@ -430,7 +434,9 @@ func (c *MixedDbClient) makeJSON_redis(msi *map[string]interface{}, key *string,
 	// TODO: Use Yang model to identify leaf-list
 	if key == nil && op == nil {
 		for f, v := range mfv {
-			if strings.HasSuffix(f, "@") {
+			if f == "NULL" {
+				continue


Hard to understand the new branch. Could you add some code comment? #Closed

… read_full_config

avl-dev · 2023-08-17T13:59:33Z

Can template like "/*" be the part solution?

gnmi_get ... -xpath "/sonic-db:CONFIG_DB/*"

It will get all the fields but suddenly (bug or feature) delete table keys (like PORT, FLEX_COUNTER_TABLE etc.).

So, for example, all Ethernet ports from PORT are in same area with other keys:
(example of output from gnmic cli command, same with gnmi_get)

[
{
"source": "10.199.199.205:50051",
"timestamp": 1692108429229533512,
"time": "2023-08-15T14:07:09.229533512Z",
"target": "CONFIG_DB",
"updates": [
{
"Path": "*",
"values": {
"*": {
"ACL": {
"FLEX_COUNTER_DELAY_STATUS": "false",
"FLEX_COUNTER_STATUS": "disable",
"POLL_INTERVAL": "10000"
},
"BUFFER_POOL_WATERMARK": {
"FLEX_COUNTER_DELAY_STATUS": "false",
"FLEX_COUNTER_STATUS": "enable"
},
"Config": {
"acl_counter_high_threshold": "85",
"acl_counter_low_threshold": "70",
"acl_counter_threshold_type": "percentage",
"acl_entry_high_threshold": "85",
"acl_entry_low_threshold": "70",
"acl_entry_threshold_type": "percentage",
"acl_group_high_threshold": "85",
"acl_group_low_threshold": "70",
"acl_group_threshold_type": "percentage",
"acl_table_high_threshold": "85",
"acl_table_low_threshold": "70",
"acl_table_threshold_type": "percentage",
"dnat_entry_high_threshold": "85",
"dnat_entry_low_threshold": "70",
"dnat_entry_threshold_type": "percentage",
"fdb_entry_high_threshold": "85",
"fdb_entry_low_threshold": "70",
"fdb_entry_threshold_type": "percentage",
"ipmc_entry_high_threshold": "85",
"ipmc_entry_low_threshold": "70",
"ipmc_entry_threshold_type": "percentage",
"ipv4_neighbor_high_threshold": "85",
"ipv4_neighbor_low_threshold": "70",
"ipv4_neighbor_threshold_type": "percentage",
"ipv4_nexthop_high_threshold": "85",
"ipv4_nexthop_low_threshold": "70",
"ipv4_nexthop_threshold_type": "percentage",
"ipv4_route_high_threshold": "85",
"ipv4_route_low_threshold": "70",
"ipv4_route_threshold_type": "percentage",
"ipv6_neighbor_high_threshold": "85",
"ipv6_neighbor_low_threshold": "70",
"ipv6_neighbor_threshold_type": "percentage",
"ipv6_nexthop_high_threshold": "85",
"ipv6_nexthop_low_threshold": "70",
"ipv6_nexthop_threshold_type": "percentage",
"ipv6_route_high_threshold": "85",
"ipv6_route_low_threshold": "70",
"ipv6_route_threshold_type": "percentage",
"mpls_inseg_high_threshold": "85",
"mpls_inseg_low_threshold": "70",
"mpls_inseg_threshold_type": "percentage",
"mpls_nexthop_high_threshold": "85",
"mpls_nexthop_low_threshold": "70",
"mpls_nexthop_threshold_type": "percentage",
"nexthop_group_high_threshold": "85",
"nexthop_group_low_threshold": "70",
"nexthop_group_member_high_threshold": "85",
"nexthop_group_member_low_threshold": "70",
"nexthop_group_member_threshold_type": "percentage",
"nexthop_group_threshold_type": "percentage",
"polling_interval": "300",
"snat_entry_high_threshold": "85",
"snat_entry_low_threshold": "70",
"snat_entry_threshold_type": "percentage"
},
"DATABASE": {
"VERSION": "version_4_0_2"
},
"Ethernet0": {
"admin_status": "up",
"alias": "fortyGigE0/0",
"index": "0",
"lanes": "25,26,27,28",
"mtu": "9100",
"speed": "40000"
},
"Ethernet100": {
"admin_status": "up",
"alias": "fortyGigE0/100",
"index": "25",
"lanes": "121,122,123,124",
"mtu": "9100",
"speed": "40000"
},
"Ethernet104": {
"admin_status": "up",
"alias": "fortyGigE0/104",
"index": "26",
"lanes": "81,82,83,84",
"mtu": "9100",
"speed": "40000"
},
...

So up here: ACL and BUFFER_POOL_WATERMARK – are from FLEX_COUNTER_TABLE; Ethernet* – from PORT; "config" – from CRM.

I see kind of solution to run multiple get request with different tables like "/sonic-db:CONFIG_DB/PORT", "/sonic-db:CONFIG_DB/CRM", "/sonic-db:CONFIG_DB/FLEX_COUNTER_TABLE" and so on. And then aggregate these data in one JSON file.

ganglyu · 2023-08-18T00:57:58Z

Can template like "/*" be the part solution?

gnmi_get ... -xpath "/sonic-db:CONFIG_DB/*"

It will get all the fields but suddenly (bug or feature) delete table keys (like PORT, FLEX_COUNTER_TABLE etc.).

So, for example, all Ethernet ports from PORT are in same area with other keys: (example of output from gnmic cli command, same with gnmi_get)

[
{
"source": "10.199.199.205:50051",
"timestamp": 1692108429229533512,
"time": "2023-08-15T14:07:09.229533512Z",
"target": "CONFIG_DB",
"updates": [
{
"Path": "",
"values": {
"": {
"ACL": {
"FLEX_COUNTER_DELAY_STATUS": "false",
"FLEX_COUNTER_STATUS": "disable",
"POLL_INTERVAL": "10000"
},
"BUFFER_POOL_WATERMARK": {
"FLEX_COUNTER_DELAY_STATUS": "false",
"FLEX_COUNTER_STATUS": "enable"
},
"Config": {
"acl_counter_high_threshold": "85",
"acl_counter_low_threshold": "70",
"acl_counter_threshold_type": "percentage",
"acl_entry_high_threshold": "85",
"acl_entry_low_threshold": "70",
"acl_entry_threshold_type": "percentage",
"acl_group_high_threshold": "85",
"acl_group_low_threshold": "70",
"acl_group_threshold_type": "percentage",
"acl_table_high_threshold": "85",
"acl_table_low_threshold": "70",
"acl_table_threshold_type": "percentage",
"dnat_entry_high_threshold": "85",
"dnat_entry_low_threshold": "70",
"dnat_entry_threshold_type": "percentage",
"fdb_entry_high_threshold": "85",
"fdb_entry_low_threshold": "70",
"fdb_entry_threshold_type": "percentage",
"ipmc_entry_high_threshold": "85",
"ipmc_entry_low_threshold": "70",
"ipmc_entry_threshold_type": "percentage",
"ipv4_neighbor_high_threshold": "85",
"ipv4_neighbor_low_threshold": "70",
"ipv4_neighbor_threshold_type": "percentage",
"ipv4_nexthop_high_threshold": "85",
"ipv4_nexthop_low_threshold": "70",
"ipv4_nexthop_threshold_type": "percentage",
"ipv4_route_high_threshold": "85",
"ipv4_route_low_threshold": "70",
"ipv4_route_threshold_type": "percentage",
"ipv6_neighbor_high_threshold": "85",
"ipv6_neighbor_low_threshold": "70",
"ipv6_neighbor_threshold_type": "percentage",
"ipv6_nexthop_high_threshold": "85",
"ipv6_nexthop_low_threshold": "70",
"ipv6_nexthop_threshold_type": "percentage",
"ipv6_route_high_threshold": "85",
"ipv6_route_low_threshold": "70",
"ipv6_route_threshold_type": "percentage",
"mpls_inseg_high_threshold": "85",
"mpls_inseg_low_threshold": "70",
"mpls_inseg_threshold_type": "percentage",
"mpls_nexthop_high_threshold": "85",
"mpls_nexthop_low_threshold": "70",
"mpls_nexthop_threshold_type": "percentage",
"nexthop_group_high_threshold": "85",
"nexthop_group_low_threshold": "70",
"nexthop_group_member_high_threshold": "85",
"nexthop_group_member_low_threshold": "70",
"nexthop_group_member_threshold_type": "percentage",
"nexthop_group_threshold_type": "percentage",
"polling_interval": "300",
"snat_entry_high_threshold": "85",
"snat_entry_low_threshold": "70",
"snat_entry_threshold_type": "percentage"
},
"DATABASE": {
"VERSION": "version_4_0_2"
},
"Ethernet0": {
"admin_status": "up",
"alias": "fortyGigE0/0",
"index": "0",
"lanes": "25,26,27,28",
"mtu": "9100",
"speed": "40000"
},
"Ethernet100": {
"admin_status": "up",
"alias": "fortyGigE0/100",
"index": "25",
"lanes": "121,122,123,124",
"mtu": "9100",
"speed": "40000"
},
"Ethernet104": {
"admin_status": "up",
"alias": "fortyGigE0/104",
"index": "26",
"lanes": "81,82,83,84",
"mtu": "9100",
"speed": "40000"
},
...

So up here: ACL and BUFFER_POOL_WATERMARK – are from FLEX_COUNTER_TABLE; Ethernet* – from PORT; "config" – from CRM.

I see kind of solution to run multiple get request with different tables like "/sonic-db:CONFIG_DB/PORT", "/sonic-db:CONFIG_DB/CRM", "/sonic-db:CONFIG_DB/FLEX_COUNTER_TABLE" and so on. And then aggregate these data in one JSON file.

We will support wildcard in the future.

ganglyu added 4 commits August 15, 2023 16:24

Read full config db

86c9a64

Update switch

584286f

Improve logic

b89bfbf

Fix coverage

f6608ae

ganglyu requested a review from qiluo-msft August 16, 2023 04:21

Merge branch 'master' into read_full_config

28b974d

qiluo-msft reviewed Aug 17, 2023

View reviewed changes

ganglyu added 2 commits August 17, 2023 15:47

Address review comment

27aefee

Merge branch 'read_full_config' of github.com:ganglyu/sonic-gnmi into…

b32c172

… read_full_config

ganglyu requested a review from qiluo-msft August 17, 2023 08:19

Merge branch 'master' into read_full_config

82d65e9

ganglyu added 2 commits August 18, 2023 15:13

Fix unit test

8435d4d

Fix unit test

88de555

qiluo-msft approved these changes Aug 21, 2023

View reviewed changes

ganglyu merged commit fdb94da into sonic-net:master Aug 21, 2023
4 checks passed

avl-dev mentioned this pull request Aug 24, 2023

help/bug: The way to read full config (CONFIG_DB) #147

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Read full config with gnmi get #143

Read full config with gnmi get #143

ganglyu commented Aug 15, 2023

qiluo-msft Aug 17, 2023 •

edited

Loading

ganglyu Aug 17, 2023

qiluo-msft Aug 17, 2023 •

edited

Loading

ganglyu Aug 17, 2023

ganglyu Aug 17, 2023

qiluo-msft Aug 17, 2023 •

edited

Loading

ganglyu Aug 17, 2023

avl-dev commented Aug 17, 2023 •

edited

Loading

ganglyu commented Aug 18, 2023

Read full config with gnmi get #143

Read full config with gnmi get #143

Conversation

ganglyu commented Aug 15, 2023

Why I did it

How I did it

How to verify it

Which release branch to backport (provide reason below if selected)

Description for the changelog

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

qiluo-msft Aug 17, 2023 • edited Loading

Choose a reason for hiding this comment

ganglyu Aug 17, 2023

Choose a reason for hiding this comment

qiluo-msft Aug 17, 2023 • edited Loading

Choose a reason for hiding this comment

ganglyu Aug 17, 2023

Choose a reason for hiding this comment

ganglyu Aug 17, 2023

Choose a reason for hiding this comment

qiluo-msft Aug 17, 2023 • edited Loading

Choose a reason for hiding this comment

ganglyu Aug 17, 2023

Choose a reason for hiding this comment

avl-dev commented Aug 17, 2023 • edited Loading

ganglyu commented Aug 18, 2023

qiluo-msft Aug 17, 2023 •

edited

Loading

qiluo-msft Aug 17, 2023 •

edited

Loading

qiluo-msft Aug 17, 2023 •

edited

Loading

avl-dev commented Aug 17, 2023 •

edited

Loading