open source version cannot run with SCM HA #7491

julienlau · 2024-11-26T17:16:34Z

julienlau
Nov 26, 2024

Hello,

I did not manage to start a full HA cluster with ozone.scm.ratis.enable=true & ozone.om.ratis.enable=true.

I have the following error at bootstrap :

2024-11-26 16:21:17,262 [main] INFO server.StorageContainerManagerStarter: registered UNIX signal handlers for [TERM, HUP, INT]
2024-11-26 16:21:17,520 [main] INFO reflections.Reflections: Reflections took 157 ms to scan 3 urls, producing 134 keys and 291 values 
2024-11-26 16:21:17,595 [main] INFO ha.SCMHANodeDetails: ServiceID for StorageContainerManager is null
Invalid Config ozone.scm.ratis.enable Provided ConfigValue: true, Expected Config Value: false. Current State of SCM: SCM is running in Non-HA without Ratis Ratis SCM -> Non Ratis SCM or Non HA SCM -> HA SCM is not supported
2024-11-26 16:21:17,598 [shutdown-hook-0] INFO server.StorageContainerManagerStarter: SHUTDOWN_MSG:

I tried to run the docker compose project from the example project : compose/ozone-ha from version 1.4.0 and it appears that the parameter ozone.scm.ratis.enable=true is not set.

Is it even possible to make it run HA without cloudera-manager ?
How to do it please ?

Regards
JL

Answered by adoroszlai

Nov 27, 2024

   <property>
      <name>ozone.scm.ratis.enable</name>
      <value>false</value>
   </property>

seems to be the problem, it should be true.

View full answer

julienlau · 2024-11-26T17:21:37Z

julienlau
Nov 26, 2024
Author

here is a screenshot from ozone-manager of the docker compose project compose/ozone-ha from version 1.4.0

0 replies

adoroszlai · 2024-11-26T17:33:41Z

adoroszlai
Nov 26, 2024
Collaborator

I tried to run the docker compose project from the example project : compose/ozone-ha from version 1.4.0 and it appears that the parameter ozone.scm.ratis.enable=true is not set.

It is set here, along few other settings needed for SCM HA:

ozone/hadoop-ozone/dist/src/main/compose/ozone-ha/docker-config

Lines 31 to 36 in e4b6007

    
           OZONE-SITE.XML_ozone.scm.service.ids=scmservice 
        
           OZONE-SITE.XML_ozone.scm.nodes.scmservice=scm1,scm2,scm3 
        
           OZONE-SITE.XML_ozone.scm.address.scmservice.scm1=scm1 
        
           OZONE-SITE.XML_ozone.scm.address.scmservice.scm2=scm2 
        
           OZONE-SITE.XML_ozone.scm.address.scmservice.scm3=scm3 
        
           OZONE-SITE.XML_ozone.scm.ratis.enable=true

0 replies

julienlau · 2024-11-26T17:52:50Z

julienlau
Nov 26, 2024
Author

then why it does not appear in the configuration in ozone-manager ?

3 replies

adoroszlai Nov 26, 2024
Collaborator

Please provide steps to reproduce the problem.

julienlau Nov 27, 2024
Author

I first tried to run the HA setup on a cluster of servers but failed with SCM bootstrap.

I then tried the example HA setup from the docker compose of version 1.4.0 in order to reverse engineers it.
I run it on a single server with :

cd compose/ozone-ha
docker compose up -d

All services are running, but if I open the ozone manager UI, it does not show ozone.scm.ratis.enable=true and the default value being ozone.scm.ratis.enable=false, I think it may be a bug either in the parameter handling either in the ozone manager.

adoroszlai Nov 27, 2024
Collaborator

Thanks. I don't know why ozone.scm.ratis.enable=true is not shown, will check. You can confirm SCM is running in HA mode by opening its web UI (http://localhost:9876/ for SCM 1, check actual port mapping for other nodes using docker compose ps) and scrolling to the section SCM Roles (HA).

BTW, you might want to run more datanodes by docker compose up -d --no-recreate --scale datanode=3.

julienlau · 2024-11-27T16:01:48Z

julienlau
Nov 27, 2024
Author

ok thanks.

I still struggle to start SCM outside of docker compose.

If I look at the docker compose it seems that ozone scm --bootstrap and ozone scm --init are never used.
In fact, the command used is directly ozone scm.

If I try to do the same on my bare metal setup I have the following error on scm1 :

2024-11-27 16:51:31,300 [main] INFO ha.SCMHANodeDetails: Setting configuration key ozone.scm.address with value of key ozone.scm.address.scmservice.scm1: xxx
2024-11-27 16:51:31,307 [main] ERROR server.StorageContainerManager: Please make sure you have run 'ozone scm --init' command to generate all the required metadata to /data/2/scm-db/scm.
2024-11-27 16:51:31,309 [main] ERROR server.StorageContainerManagerStarter: SCM start failed with exception
org.apache.hadoop.hdds.scm.exceptions.SCMException: SCM not initialized due to storage config failure.

and now if I look at the documentation here : https://ozone.apache.org/docs/1.4.0/feature/scm-ha.html
It is said that I should run first on primordial node ozone scm --bootstrap.
However this command fails with error :

2024-11-27 16:58:05,248 [main] ERROR server.StorageContainerManager: Bootstrap is not supported without SCM HA.
scm bootstrap failed

Lastly, if I skip completely the bootstrap, I manage to start 3 scm instances by running on each node:

ozone scm --init
ozone scm

However, when I open SCM UI there are 0 nodes registered and HA status is standalone:

5 replies

adoroszlai Nov 27, 2024
Collaborator

If I look at the docker compose it seems that ozone scm --bootstrap and ozone scm --init are never used.

In the Docker setup those commands are triggered behind the scenes by the presence of ENSURE_SCM_INITIALIZED and ENSURE_SCM_BOOTSTRAPPED variables.

ozone/hadoop-ozone/dist/src/main/compose/ozone-ha/docker-compose.yaml

Line 79 in e4b6007

ENSURE_SCM_INITIALIZED: /data/metadata/scm/current/VERSION

ozone/hadoop-ozone/dist/src/main/compose/ozone-ha/docker-compose.yaml

Line 89 in e4b6007

ENSURE_SCM_BOOTSTRAPPED: /data/metadata/scm/current/VERSION

ozone/hadoop-ozone/dist/src/main/dockerlibexec/entrypoint.sh

Lines 113 to 123 in e4b6007

    
           if [ -n "$ENSURE_SCM_INITIALIZED" ]; then 
        
             if [ ! -f "$ENSURE_SCM_INITIALIZED" ]; then 
        
               # Improve om and scm start up options 
        
               /opt/hadoop/bin/ozone scm --init || /opt/hadoop/bin/ozone scm -init 
        
             fi 
        
           elif [ -n "$ENSURE_SCM_BOOTSTRAPPED" ]; then 
        
             if [ ! -f "$ENSURE_SCM_BOOTSTRAPPED" ]; then 
        
               # Improve om and scm start up options 
        
               /opt/hadoop/bin/ozone scm --bootstrap 
        
             fi 
        
           fi

command fails with error:
Bootstrap is not supported without SCM HA

Looks like some configs are missing. Please share your ozone-site.xml.

julienlau Nov 27, 2024
Author

ok thanks I missed that.

regarding my ozone config, here it is :

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<configuration>
   <property>
      <name>ozone.metadata.dirs</name>
      <value>/data/1/metadata</value>
   </property>
   <property>
      <name>ozone.om.db.dirs</name>
      <value>/data/1/om-db</value>
   </property>

   <property>
      <name>ozone.scm.db.dirs</name>
      <value>/data/2/scm-db</value>
   </property>
    <property>
        <name>ozone.scm.client.address</name>
        <value>localhost</value>
        <tag>OZONE, SCM, REQUIRED</tag>
        <description>
      The address of the Ozone SCM client service. This is a required setting.

      It is a string in the host:port format. The port number is optional
      and defaults to 9860.
    </description>
    </property>
    <property>
        <name>ozone.scm.names</name>
        <value>localhost</value>
        <tag>OZONE, REQUIRED</tag>
        <description>
      The value of this property is a set of DNS | DNS:PORT | IP
      Address | IP:PORT. Written as a comma separated string. e.g. scm1,
      scm2:8020, 7.7.7.7:7777.
      This property allows datanodes to discover where SCM is, so that
      datanodes can send heartbeat to SCM.
    </description>
    </property>
   <property>
      <name>ozone.om.ratis.enable</name>
      <value>true</value>
   </property>
   <property>
   <name>ozone.om.service.ids</name>
   <value>omservice</value>
   </property>
   <property>
      <name>ozone.om.nodes.omservice</name>
      <value>om1,om2,om3</value>
   </property>
   <property>
      <name>ozone.om.address.omservice.om1</name>
      <value>server76</value>
   </property>
   <property>
      <name>ozone.om.address.omservice.om2</name>
      <value>server77</value>
   </property>
   <property>
      <name>ozone.om.address.omservice.om3</name>
      <value>server78</value>
   </property>

   <property>
      <name>ozone.scm.ratis.enable</name>
      <value>false</value>
   </property>
   <property>
   <name>ozone.scm.service.ids</name>
   <value>scmservice</value>
   </property>
   <property>
      <name>ozone.scm.nodes.scmservice</name>
      <value>scm1,scm2,scm3</value>
   </property>
   <property>
      <name>ozone.scm.address.scmservice.scm1</name>
      <value>server76</value>
   </property>
   <property>
      <name>ozone.scm.address.scmservice.scm2</name>
      <value>server77</value>
   </property>
   <property>
      <name>ozone.scm.address.scmservice.scm3</name>
      <value>server78</value>
   </property>
   <property>
      <name>ozone.scm.primordial.node.id</name>
      <value>scm1</value>
   </property>
   <property>
      <name>ozone.server.default.replication.type</name>
      <value>EC</value>
   </property>
   <property>
      <name>ozone.server.default.replication</name>
      <value>RS-3-2-1024k</value>
   </property>

   <property>
      <name>ozone.om.address</name>
      <value>localhost</value>
   </property>

   <property>
      <name>hdds.scm.safemode.min.datanode</name>
      <value>1</value>
   </property>

   <property>
      <name>hdds.container.ratis.datanode.storage.dir</name>
      <value>/data/1/container-ratis</value>
   </property>
   <property>
      <name>hdds.datanode.dir</name>
      <value>/data/2/hdds,/data/3/hdds,/data/4/hdds,/data/5/hdds,/data/6/hdds,/data/7/hdds,/data/8/hdds,/data/9/hdds,/data/10/hdds,/data/11/hdds,/data/12/hdds,/data/13/hdds,/data/14/hdds,/data/15/hdds,/data/16/hdds,/data/17/hdds,/
data/18/hdds,/data/19/hdds,/data/20/hdds,/data/21/hdds</value>
   </property>
   <property>
      <name>dfs.container.ratis.datastream.enabled</name>
      <value>true</value>
   </property>
   <property>
      <name>ozone.recon.db.dir</name>
      <value>/data/1/recon</value>
   </property>
</configuration>

adoroszlai Nov 27, 2024
Collaborator

   <property>
      <name>ozone.scm.ratis.enable</name>
      <value>false</value>
   </property>

seems to be the problem, it should be true.

Answer selected by julienlau

julienlau Nov 27, 2024
Author

thank you very much.
That's way better

dombizita Nov 27, 2024
Collaborator

You may also need to change this property.

    <property>
        <name>ozone.scm.names</name>
        <value>localhost</value>
        <tag>OZONE, REQUIRED</tag>
        <description>
      The value of this property is a set of DNS | DNS:PORT | IP
      Address | IP:PORT. Written as a comma separated string. e.g. scm1,
      scm2:8020, [7.7.7.7:7777](http://7.7.7.7:7777/).
      This property allows datanodes to discover where SCM is, so that
      datanodes can send heartbeat to SCM.
    </description>
    </property>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

open source version cannot run with SCM HA #7491

{{title}}

Replies: 4 comments 8 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

open source version cannot run with SCM HA #7491

julienlau Nov 26, 2024

Replies: 4 comments · 8 replies

julienlau Nov 26, 2024 Author

adoroszlai Nov 26, 2024 Collaborator

julienlau Nov 26, 2024 Author

adoroszlai Nov 26, 2024 Collaborator

julienlau Nov 27, 2024 Author

adoroszlai Nov 27, 2024 Collaborator

julienlau Nov 27, 2024 Author

adoroszlai Nov 27, 2024 Collaborator

julienlau Nov 27, 2024 Author

adoroszlai Nov 27, 2024 Collaborator

julienlau Nov 27, 2024 Author

dombizita Nov 27, 2024 Collaborator

julienlau
Nov 26, 2024

Replies: 4 comments 8 replies

julienlau
Nov 26, 2024
Author

adoroszlai
Nov 26, 2024
Collaborator

julienlau
Nov 26, 2024
Author

adoroszlai Nov 26, 2024
Collaborator

julienlau Nov 27, 2024
Author

adoroszlai Nov 27, 2024
Collaborator

julienlau
Nov 27, 2024
Author

adoroszlai Nov 27, 2024
Collaborator

julienlau Nov 27, 2024
Author

adoroszlai Nov 27, 2024
Collaborator

julienlau Nov 27, 2024
Author

dombizita Nov 27, 2024
Collaborator