Enable Multi DB #20305

Pan-XT · 2024-09-19T17:47:26Z

Why I did it

Enable Multi DB for improving routing loading performance. The detail performancee data could be found in the following HLD
[HLD] MultiDB HLD
sonic-net/SONiC#1855

How I did it

modify [dockers/docker-database/Dockerfile.j2] and add [multi_database_config.json.j2]
modify [dockers/docker-database/docker-database-init.sh] to enable Multi DB if /etc/sonic/enable_multidb exists and ensure that the redis service has the necessary permissions to access /var/lib/$inst
modify [Makefile.work] [slave.mk] [files/build_templates/sonic_debian_extension.j2 ] for setting ENABLE_MULTIDB=y when building SONiC image.

some fix PR for multiple database process, otherwise redis instance cannot be launched correctly.
#20726 #20545 sonic-net/sonic-swss#3292

How I verified it

sonic-net/sonic-mgmt#15084

…into multidb

eddieruan-alibaba · 2024-09-24T05:09:07Z

Double committed to Phoenixwing eddieruan-alibaba@9f9f4d8

and verified on vSONiC

admin@PE3:~$ ps aux | grep redis
message+ 1011 1.4 0.2 86764 22372 pts/0 Sl 04:30 0:31 /usr/bin/redis-server 127.0.0.1:6379
message+ 1012 0.6 0.2 77548 22316 pts/0 Sl 04:30 0:14 /usr/bin/redis-server 127.0.0.1:6380
message+ 1015 0.5 0.2 64748 17696 pts/0 Sl 04:30 0:11 /usr/bin/redis-server 127.0.0.1:6381
message+ 1016 1.9 0.3 71404 22532 pts/0 Sl 04:30 0:43 /usr/bin/redis-server 127.0.0.1:6382
message+ 1018 0.2 0.2 62188 16456 pts/0 Sl 04:30 0:05 /usr/bin/redis-server 127.0.0.1:6383

eddieruan-alibaba · 2024-09-24T05:10:05Z

/azpw ms_conflict

eddieruan-alibaba · 2024-09-24T05:11:29Z

@StormLiangMS, @liushilongbuaa Can you trigger the following command?

Github Branch not ready
Please wait a few minutes to run again!
'/azpw ms_conflict'

eddieruan-alibaba

This change is enabled in Phoenix wing and verified in Phoenix wing's daily sanity.

http://phoenixwing.com.cn/vsonic

…ends on /etc/sonic/enable_multidb

eddieruan-alibaba · 2024-10-11T19:51:31Z

Validated with phoenix wing vsonic image

admin@PE3:$ ls -l /etc/sonic/enable_multidb
-rw-r--r-- 1 root root 0 Oct 11 01:46 /etc/sonic/enable_multidb
admin@PE3:$ ps aux | grep redis
message+ 990 0.8 0.5 86764 39228 pts/0 Sl 06:28 7:06 /usr/bin/redis-server 127.0.0.1:6379
message+ 991 0.8 0.2 71404 21248 pts/0 Rl 06:28 6:50 /usr/bin/redis-server 127.0.0.1:6380
message+ 992 0.3 0.2 64748 17384 pts/0 Sl 06:28 2:28 /usr/bin/redis-server 127.0.0.1:6381
message+ 994 2.9 0.3 71404 24000 pts/0 Sl 06:28 23:44 /usr/bin/redis-server 127.0.0.1:6382
message+ 996 0.1 0.2 62188 16904 pts/0 Sl 06:28 1:22 /usr/bin/redis-server 127.0.0.1:6383
admin 111122 0.0 0.0 6868 648 pts/0 S+ 19:51 0:00 grep redis

liuh-80 · 2024-11-06T08:29:49Z

dockers/docker-database/critical_processes.j2

@@ -1,5 +1,6 @@
 {% if INSTANCES %}
 {%     for redis_inst, redis_items in INSTANCES.items() %}
+


Remove this unnecessary change?

If there is no blank line here, all program:{{ redis_inst }} of critical_processes will be concentrated on one line, which will cause subsequent regular expression matching to fail.

Does changing {%- endfor %} to {% endfor %} work? That will make the intention a bit clearer.

Changing {%- endfor %} to {% endfor %} works. I apply this change instead of adding blank line.

and revert chmod -f 0760 $sock_file

…into multidb

yuezhoujk · 2024-11-08T06:22:13Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2024-11-08T06:22:26Z

Azure Pipelines successfully started running 1 pipeline(s).

yuezhoujk · 2024-11-09T15:45:46Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2024-11-09T15:45:57Z

Azure Pipelines successfully started running 1 pipeline(s).

yuezhoujk · 2024-11-13T01:18:06Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2024-11-13T01:18:17Z

Azure Pipelines successfully started running 1 pipeline(s).

…into multidb

kperumalbfn · 2024-11-15T18:07:31Z

@Pan-XT Could you add this build option ENABLE_MULTIDB in the multi DB HLD doc?

kperumalbfn · 2024-11-15T18:08:28Z

@Pan-XT Could you also sonic-mgmt tests for multi DB. Though this option is not enabled by default, will be good to have these tests in sonic-mgmt.

eddieruan-alibaba · 2024-11-16T02:01:04Z

@Pan-XT Could you also sonic-mgmt tests for multi DB. Though this option is not enabled by default, will be good to have these tests in sonic-mgmt.

@kperumalbfn he replied it before. sonic-net/sonic-mgmt#15084. We need to build a new image with this compile flag is enabled, then reuse existing test cases. No need to create new one since we don't expect any differences except performance gain.

Pan-XT · 2024-11-16T03:57:13Z

@Pan-XT Could you add this build option ENABLE_MULTIDB in the multi DB HLD doc?

I raise a PR sonic-net/SONiC#1855

Pan-XT · 2024-11-19T07:32:18Z

@lguohan

Can you help to review this PR， I have already updated the description based on your suggestion

…into multidb

kperumalbfn

Thanks for the changes. Please update the HLD with the performance numbers.

FengPan-Frank · 2024-11-22T08:59:12Z

dockers/docker-database/multi_database_config.json.j2

+        },
+        "redis1": {
+           "hostname" : "{{HOST_IP}}",
+            "port" : 6378,


pls note that env REDIS_PORT/BMP_DB_PORT need to be reserved, and port > 6400 is not used currently.

FengPan-Frank · 2024-11-22T09:09:33Z

also some fix PR for multiple database process, otherwise redis instance cannot be launched correctly.

#20545
#20726

Pan-XT · 2024-11-22T09:32:13Z

also some fix PR for multiple database process, otherwise redis instance cannot be launched correctly.

#20545 #20726

Thanks for the suggestion. These PRs are merged and I link them in description.

Enable Multi DB

Pan-XT added 14 commits August 30, 2024 18:26

Updated supervisord.conf.j2 to support multidb

aaefb71

Merge branch 'multidb' of https://github.com/Pan-XT/sonic-buildimage …

c5c3e59

…into multidb

chown redis

289f8e6

update multi db json.j2

f51867e

[sonic-swss] update submodule for sonic-swss

42fc418

Updated docker-database-init.sh to support multidb

67a9804

update multi_database_config.json.j2

4c6527e

update docker-database-init.sh

46939a0

update multi_database_config.json

5b3c458

update critical_processes.j2

2bc006a

update docker-database-init.sh

614bb83

update docker-database-init.sh

e42bfc3

update docker-database-init.sh

f51eedb

update docker-database-init.sh

2b121de

Pan-XT requested a review from lguohan as a code owner September 19, 2024 17:47

Enable Multi DB

397c6fa

Revert changes to src/sonic-swss

81fdc48

eddieruan-alibaba approved these changes Sep 25, 2024

View reviewed changes

Pan-XT added 2 commits September 26, 2024 13:11

enable multi db for cisco-8101-p4-32x100-vs

a220f2c

Set ENABLE_MULTIDB=y when building SONiC image and enable Multidb dep…

e53cc25

…ends on /etc/sonic/enable_multidb

Pan-XT requested review from qiluo-msft and xumia as code owners October 9, 2024 11:44

Pan-XT and others added 2 commits October 10, 2024 14:57

update sonic_debian_extension.j2

87aab3e

Merge branch 'sonic-net:master' into multidb

b158823

eddieruan-alibaba requested review from venkatmahalingam and hasan-brcm October 11, 2024 19:51

liuh-80 reviewed Nov 6, 2024

View reviewed changes

revert changes to supervisord.conf.j2

3b30645

and revert chmod -f 0760 $sock_file

Pan-XT requested a review from oleksandrivantsiv November 6, 2024 15:27

Pan-XT added 5 commits November 7, 2024 10:29

Merge branch 'master' of https://github.com/sonic-net/sonic-buildimage …

f402859

…into multidb

add comments on chown -R redis:redis /var/lib/$inst

55808b5

Merge branch 'master' of https://github.com/sonic-net/sonic-buildimage …

90ace92

…into multidb

Merge branch 'master' of https://github.com/sonic-net/sonic-buildimage …

de605aa

…into multidb

change {%- endfor %} to {% endfor %} in critical_processes.j2

f947e14

Pan-XT added 2 commits November 13, 2024 20:46

Merge branch 'master' of https://github.com/sonic-net/sonic-buildimage …

f4913db

…into multidb

Merge branch 'master' of https://github.com/sonic-net/sonic-buildimage …

944f71a

…into multidb

Pan-XT mentioned this pull request Nov 16, 2024

Introduce a compile flag ENABLE_MULTIDB sonic-net/SONiC#1855

Open

liuh-80 approved these changes Nov 18, 2024

View reviewed changes

Merge branch 'master' of https://github.com/sonic-net/sonic-buildimage …

f083e18

…into multidb

kperumalbfn approved these changes Nov 21, 2024

View reviewed changes

kperumalbfn merged commit 9e888d4 into sonic-net:master Nov 22, 2024
22 checks passed

FengPan-Frank reviewed Nov 22, 2024

View reviewed changes

VladimirKuk pushed a commit to Marvell-switching/sonic-buildimage that referenced this pull request Jan 21, 2025

Enable Multi DB (sonic-net#20305)

4dc46f2

Enable Multi DB

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable Multi DB #20305

Enable Multi DB #20305

Pan-XT commented Sep 19, 2024 •

edited

Loading

eddieruan-alibaba commented Sep 24, 2024

eddieruan-alibaba commented Sep 24, 2024

eddieruan-alibaba commented Sep 24, 2024

eddieruan-alibaba left a comment

eddieruan-alibaba commented Oct 11, 2024

liuh-80 Nov 6, 2024

Pan-XT Nov 6, 2024

liuh-80 Nov 7, 2024

saiarcot895 Nov 7, 2024

Pan-XT Nov 8, 2024

yuezhoujk commented Nov 8, 2024

azure-pipelines bot commented Nov 8, 2024

yuezhoujk commented Nov 9, 2024

azure-pipelines bot commented Nov 9, 2024

yuezhoujk commented Nov 13, 2024

azure-pipelines bot commented Nov 13, 2024

kperumalbfn commented Nov 15, 2024

kperumalbfn commented Nov 15, 2024

eddieruan-alibaba commented Nov 16, 2024

Pan-XT commented Nov 16, 2024

Pan-XT commented Nov 19, 2024 •

edited

Loading

kperumalbfn left a comment

FengPan-Frank Nov 22, 2024

FengPan-Frank commented Nov 22, 2024

Pan-XT commented Nov 22, 2024

		@@ -1,5 +1,6 @@
		{% if INSTANCES %}
		{% for redis_inst, redis_items in INSTANCES.items() %}

Enable Multi DB #20305

Enable Multi DB #20305

Conversation

Pan-XT commented Sep 19, 2024 • edited Loading

Why I did it

How I did it

How I verified it

eddieruan-alibaba commented Sep 24, 2024

eddieruan-alibaba commented Sep 24, 2024

eddieruan-alibaba commented Sep 24, 2024

eddieruan-alibaba left a comment

Choose a reason for hiding this comment

eddieruan-alibaba commented Oct 11, 2024

liuh-80 Nov 6, 2024

Choose a reason for hiding this comment

Pan-XT Nov 6, 2024

Choose a reason for hiding this comment

liuh-80 Nov 7, 2024

Choose a reason for hiding this comment

saiarcot895 Nov 7, 2024

Choose a reason for hiding this comment

Pan-XT Nov 8, 2024

Choose a reason for hiding this comment

yuezhoujk commented Nov 8, 2024

azure-pipelines bot commented Nov 8, 2024

yuezhoujk commented Nov 9, 2024

azure-pipelines bot commented Nov 9, 2024

yuezhoujk commented Nov 13, 2024

azure-pipelines bot commented Nov 13, 2024

kperumalbfn commented Nov 15, 2024

kperumalbfn commented Nov 15, 2024

eddieruan-alibaba commented Nov 16, 2024

Pan-XT commented Nov 16, 2024

Pan-XT commented Nov 19, 2024 • edited Loading

kperumalbfn left a comment

Choose a reason for hiding this comment

FengPan-Frank Nov 22, 2024

Choose a reason for hiding this comment

FengPan-Frank commented Nov 22, 2024

Pan-XT commented Nov 22, 2024

Pan-XT commented Sep 19, 2024 •

edited

Loading

Pan-XT commented Nov 19, 2024 •

edited

Loading