Seeking asssitance with base deployment errors using arcgis-enterprise-primary.json file

[J-J-E]

Hello everyone! I'm having a little bit of trouble correctly configuring this base deployment. For some reason, the datastore component fails to configure and says it cant reach the arcgis server component on localhost. I have validated the url below with the ip:port is indeed accessible, and both the portal and server components are in place and running and appear to be working fine. Im attaching the stack trace as well as the config file (with passwords removed). Any help is appreciated as I've kind of hit a roadblock. Thanks in advance!

STDERR: Error encountered: Could not connect to server on machine 'localhost'. ArcGIS Data Store or ArcGIS Server on that machine may not be running or the machine is not reachable at this time.
---- End output of "/opt/arcgis/datastore/tools/configuredatastore.sh" "https://10.168.0.23:6443/arcgis/admin" "PSAArcGISServer" "" "/opt/arcgis/gisdata/arcgisdatastore" --stores relational ----
Ran "/opt/arcgis/datastore/tools/configuredatastore.sh" "https://10.168.0.23:6443/arcgis/admin" "PSAArcGISServer" "" "/opt/arcgis/gisdata/arcgisdatastore" --stores relational returned 1

cinc-stacktrace.txt
arcgis-enterprise-primary.json

[cameronkroeker]

Hi @J-J-E,

One thing I notice is using IP for server and portal but datastore is set to hostname. In the datastore json block, try changing "preferredidentifier": "hostname", to "preferredidentifier": "ip",:

"data_store": {
      "install_dir": "/opt",
      "setup_options": "-f Relational,TileCache",
      "data_dir": "/opt/arcgis/gisdata/arcgisdatastore",
      "preferredidentifier": "ip",
      "install_system_requirements": true,
      "types": "relational",
      "tilecache": {
        "backup_type": "none",
        "backup_location": "/net/FILESERVER/gisdata/arcgisbackup/tilecache"
      },
      "relational": {
        "backup_type": "fs",
        "backup_location": "/opt/arcgis/gisdata/arcgisbackup/relational"
      }
    }

The other thing is may want to change FILESERVER to the actual file server hostname, or ensure there is an /etc/hosts entry for it. Or if not wanting to create the backup for the tilecache, set "backup_type": "fs", to "backup_type": "none",.

Thanks,
Cameron K.

[J-J-E]

Cameron,

Thank you very much for the suggestion.

1. I updated the identifier from hostname -> ip as suggested.
2. I removed the tilecache component as we actually have no need for one in this base deployment
   2.1 I also removed the reference to the FILERSERVER, as we have no plans to add a standby machine and per the docs, local paths should be used instead of a file server in that case
3. I created my own recipe to create local directories, paths are defined in [arcgis][localdirectories][directories]. I updated the enterprise-primary.json file with these paths and the call to the recipe (also attached) as they were in the initial 10.9.1 deployment but I didn't copy them into the 11.3 json file because they already existed. I had to upload the localdirectories.rb file as a .txt file as github wouldn't accept the .rb

This is still failing at the configure portal. The datastore is installed and running, but for some reason cannot reach the ArcGIS Server component on its local host. I have validated I can access the server services and admin pages at the url in the stacktrace that the datastore cannot connect to.

arcgis-enterprise-primary1103.json
localdirectories.txt
stacktrace.txt
chef-enterprise1103-clioutput.txt

[cameronkroeker]

@J-J-E thanks for the additional logs and information.

One observation is in the localdirectories.rb recipe the following is defined ['arcgis']['localserver']['directories'], however in the json file its defined as ['arcgis']['localdirectories']['directories']. So I am not quite sure I understand how that part succeeded.

localdirectories.rb:

#
# Cookbook Name:: arcgis-enterprise
# Recipe:: createlocaldirs
#
# Copyright 2023 John Ehlen
#

# Create local server directories

node['arcgis']['localserver']['directories'].each do |dir|
  directory dir do
    owner node['arcgis']['run_as_user']
    mode '0755' if node['platform'] != 'windows'
    recursive true
    action :create
  end
end

I would expect the json to look like this to match the localdirectories.rb:

"localserver": {
   "directories": [
      "/opt/arcgis/gisdata/arcgisserver",
      "/opt/arcgis/gisdata/arcgisbackup",
      "/opt/arcgis/gisdata/arcgisbackup/relational",
      "/opt/arcgis/gisdata/arcgisportal",
      "/opt/arcgis/gisdata/arcgisportal/content"
   ]
},

Alternatively update the localdirectories.rb to match the json:

node['arcgis']['localdirectories']['directories'].each do |dir|
  directory dir do
    owner node['arcgis']['run_as_user']
    mode '0755' if node['platform'] != 'windows'
    recursive true
    action :create
  end
end

At any rate I suppose it worked somehow and its likely not related to the relational data store failing to register with server. Let's check to ensure both the arcgis server and arcgis data store services are started. As a sudo user run the following two commands on share the result:

• sudo systemctl status arcgisserver.service
• sudo systemctl status arcgisdatastore.service

Thanks,
Cameron K.

[cameronkroeker]

Thanks. After adding hostidentifier, did it set hostidentifier=10.168.0.23 in /opt/arcgis/datastore/framework/etc/hostidentifier.properties?

The only other thing I can think of is perhaps when data store reaches out to https://10.168.0.23:6443/arcgis/admin endpoint its getting redirected to https://gis-test.filmla.com/server (since this is server's webcontext url). Is this url accessible from the server/datstore node or only accessible outside? Perhaps /etc/hosts needs to be modified to:

10.168.0.23 gis-test.filmla.com instead of 35.235.116.48 gis-test.filmla.com. Jus a thought Im not sure to be honest.

Update: Just realized you have already tried this! I am not quite sure what the culprit is here. From what I can tell the chef configuration all looks correct and valid.

[J-J-E]

1. did it set hostidentifier=10.168.0.23 in /opt/arcgis/datastore/framework/etc/hostidentifier.properties?

• The ip address was actually inside this file prior to adding the [hostidentifier] in the json file. IT stayed the same after adding that config

2. when data store reaches out to https://10.168.0.23:6443/arcgis/admin endpoint its getting redirected to https://gis-test.filmla.com/server

• this redirect isn't happening from my local machine (screenshot below). This is a headless linux vm so I don't have a desktop browser to try accessing it from the machine, but I can't imagine the url isn't accessible as the datastore node is on the same machine as the server node and shares the same IP? I am able to ping the ip address from the cli successfully and I also verified no firewall was active

3. Yes, I have tried configuring with the hostfile pointing to the internal and to the external IP with the same results. I'm kind of at a loss as well here. Been troubleshooting this for two weeks before I posted anything and I've hit a roadblock in this trying to get it configured and running :/

[cameronkroeker]

I wonder if it being federated already is causing an issue. Can you unfederate, and then try registering the datastore?

I imagine manually running the following command yields the same result?

/opt/arcgis/datastore/tools/configuredatastore.sh "https://10.168.0.23:6443/arcgis/admin" "PSAArcGISServer" "*****" "/opt/arcgis/gisdata/arcgisdatastore" --stores relational

[cameronkroeker]

Actually, I just thought of something. Let's check to ensure the hostname.properties file for ArcGIS Server is configured with the correct ip and not set to hostname. I wonder if since originally the /etc/hosts file contained the entry 10.168.0.23 gis-test.filmla.com when server installed it picked gis-test.filmla.com as its hostname/machine name.

/opt/arcgis/server/framework/etc/hostname.properties

Also sign into https://10.168.0.23:6443/arcgis/admin and select machine and see what the machine name is set to.

• node['arcgis']['server']['hostname'] = Host name or IP address of ArcGIS Server machine. Default value is ''.

As well as for portal:

• node['arcgis']['portal']['hostname'] = Host name or IP address of Portal for ArcGIS machine specified in the hostname.properties file. Default value is ''.
• node['arcgis']['portal']['preferredidentifier'] = Portal for ArcGIS preferred identifier method <hostname|ip>. Default method used is hostname.

We may need to start completely over and ensure those attributes are set prior to the installation.

[J-J-E]

Cameron,

Thank you again for all the time you're spending helping me with this. I greatly appreciate it.

1. Same issue running the command manually

2. arcgis server admin page is configured using IP
   

3. Arcgis portal appears to be configured using machine fully qualified internal GCP hostname, which is strange, as this isn't in any config file and when I check hostname on the machine, its not the fully qualified hostname but rather just the first part "gis-test-enterprise"
   
   

I think you're probably right in that the best bet is to blow this machine away and redploy a new instance fully from scratch. On that note, can you validate I added the hostname and identifier properties correctly in this updated config? Do I need to add the ssl cert into the [arcgis][server] component? Anything else I should change before I deploy with this script?

arcgis-enterprise-primary1103.json

[cameronkroeker]

Just need to add a comma to line 43 and 78 of the json file. Other than that is looks good to me. Adding the ssl cert into [arcgis][server] component imports the certificate into arcgis server which is used for the admin url 6443. If you have a cert issued to the ip then I'd say its worth it, otherwise I dont see the need to import a cert that doesn't match https://10.168.0.23:6443/arcgis because then you would get an annoying mismatched address cert error everytime navigating to that url.

[J-J-E]

ahh good catch on the commas haha. I did not add the ssl cert to the arcgis server component, ran the install and everything appears to be working correctly with both the server and portal showing the ip address as machine name, and the datastore configured correctly. Thanks again for all your help. I suppose now I'm off to republish 200 services haha!

[J-J-E]

@cameronkroeker Hey Cameron, thanks again for your previous help. I am experiencing another issue in this environment configuration and I believe its related to the ssl somehow. I've found this help article, and Esri Support indicates they believe this is an issue with the SSL yet we cannot find a resolution since everything at the portal endpoints shows a valid cert. They recommended we reconfigure the env using hostnames and not IP addresses, and I was wondering if you could tell me which hostname/FQDNI should use for the URLs, as well as the identifier settings.

When looking at the machine itself, the hostname is just "gis-test-enterprise", and its fully qualified URL within out LAN in Google Cloud is "https://gis-test-enterprise.us-west2-c.c.ops-gcp.internal".