Dead Script?

[SenorKarlos]

I started this last night and I'm not sure what the hell. I know it can take a while, but this is a but much I think

It is now 04:46 in my time zone, log timestamped

processor is barely moving, ram barely touched, it made a flat nodes file, which seems to have stopped at 90GB or so. I'm not seeing any lines about US post code import either... is the 4.4 dockerfile messed up somewhere?

I only think it might be doing something with the top screen and the pg_wal folder of the data volume keeps having its modified updted, but I don't know shit about Postgress.

  10364 senorka+  20   0   89.0g  44.9g  44.7g D  22.3  63.5 144:02.18 osm2pgsql
  10370 systemd+  20   0 2260096   2.0g   2.0g S  18.3   2.8  22:46.33 postgres
  5700 lxd       20   0   16.8g   1.4g  61996 S  15.6   2.0  26:28.15 node

I'm just checking it while awake, gonna take action in a few hours when it's actually time to get up. I don't want to kill it if it's suddenly gonna start going through steps. compose for reference as well

version: "3.7"

services:
  nominatim:
    container_name: nominatim
    image: mediagis/nominatim:4.4
    logging:
      driver: "json-file"
      options:
        max-size: "250m"
    restart: always
    ports:
      - '7070:8080'
    environment:
      PBF_URL: https://download.geofabrik.de/north-america-latest.osm.pbf
      REPLICATION_URL: https://download.geofabrik.de/north-america-updates/
      USER_AGENT: redacted/present
      IMPORT_WIKIPEDIA: 'true'
      IMPORT_US_POSTCODES: 'true'
      THREADS: 16
    volumes:
      - ./nominatim-data:/var/lib/postgresql/14/main
      - ./nominatim-flatnode:/nominatim/flatnode
    shm_size: '4gb'
networks:
  default:
    name: nominatim

$ docker compose up -d && docker compose logs -f
WARN[0000] /home/senorkarlos/nominatim/docker-compose.yml: `version` is obsolete
[+] Running 4/4
 ✔ nominatim 3 layers [⣿⣿⣿]      0B/0B      Pulled
   ✔ a73181b1281a Pull complete
   ✔ 4f4fb700ef54 Pull complete
   ✔ 00d7b91676a5 Pull complete
[+] Running 1/2
 ⠹ Network nominatim    Created
 ✔ Container nominatim  Started
WARN[0000] /home/senorkarlos/nominatim/docker-compose.yml: `version` is obsolete
nominatim  | + tailpid=0
nominatim  | + replicationpid=0
nominatim  | + trap stopServices SIGTERM TERM INT
nominatim  | + /app/config.sh
nominatim  | + id nominatim
nominatim  | + useradd -m -p qaIACxO6wMR3 nominatim
nominatim  | + IMPORT_FINISHED=/var/lib/postgresql/14/main/import-finished
nominatim  | + '[' '!' -f /var/lib/postgresql/14/main/import-finished ']'
nominatim  | + /app/init.sh
nominatim  | + OSMFILE=/nominatim/data.osm.pbf
nominatim  | + CURL=("curl" "-L" "-A" "${USER_AGENT}" "--fail-with-body")
nominatim  | + SCP='sshpass -p DMg5bmLPY7npHL2Q scp -o StrictHostKeyChecking=no u355874-sub1@u355874-sub1.your-storagebox.de'
nominatim  | + '[' -z 16 ']'
nominatim  | + '[' true = true ']'
nominatim  | + echo 'Downloading Wikipedia importance dump'
nominatim  | Downloading Wikipedia importance dump
nominatim  | + sshpass -p DMg5bmLPY7npHL2Q scp -o StrictHostKeyChecking=no u355874-sub1@u355874-sub1.your-storagebox.de:wikimedia-importance.sql.gz /nominati
nominatim  | Warning: Permanently added 'u355874-sub1.your-storagebox.de' (RSA) to the list of known hosts.
nominatim  | + '[' '' = true ']'
nominatim  | Skipping optional GB postcode import
nominatim  | + '[' -f '' ']'
nominatim  | + echo 'Skipping optional GB postcode import'
nominatim  | + '[' true = true ']'
nominatim  | + sshpass -p DMg5bmLPY7npHL2Q scp -o StrictHostKeyChecking=no u355874-sub1@u355874-sub1.your-storagebox.de:us_postcodes.csv.gz /nominatim/us_pos
nominatim  | Skipping optional Tiger addresses import
nominatim  | Downloading OSM extract from https://download.geofabrik.de/north-america-latest.osm.pbf
nominatim  | + '[' '' = true ']'
nominatim  | + '[' -f '' ']'
nominatim  | + echo 'Skipping optional Tiger addresses import'
nominatim  | + '[' https://download.geofabrik.de/north-america-latest.osm.pbf '!=' '' ']'
nominatim  | + echo Downloading OSM extract from https://download.geofabrik.de/north-america-latest.osm.pbf
nominatim  | + curl -L -A SenorKarlos/PokeMap.ca --fail-with-body https://download.geofabrik.de/north-america-latest.osm.pbf -C - --create-dirs -o /nominatim
nominatim  |   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
nominatim  |                                  Dload  Upload   Total   Spent    Left  Speed
100 13.9G  100 13.9G    0     0  15.1M      0  0:15:41  0:15:41 --:--:-- 13.5M
nominatim  | + '[' '' '!=' '' ']'
nominatim  | + '[' '!' -f /var/lib/postgresql/14/main/PG_VERSION ']'
nominatim  | + chown postgres /var/lib/postgresql/14/main
nominatim  | + sudo -u postgres /usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main
nominatim  | The files belonging to this database system will be owned by user "postgres".
nominatim  | This user must also own the server process.
nominatim  |
nominatim  | The database cluster will be initialized with locale "en_US.UTF-8".
nominatim  | The default database encoding has accordingly been set to "UTF8".
nominatim  | The default text search configuration will be set to "english".
nominatim  |
nominatim  | Data page checksums are disabled.
nominatim  |
nominatim  | fixing permissions on existing directory /var/lib/postgresql/14/main ... ok
nominatim  | creating subdirectories ... ok
nominatim  | selecting dynamic shared memory implementation ... posix
nominatim  | selecting default max_connections ... 100
nominatim  | selecting default shared_buffers ... 128MB
nominatim  | selecting default time zone ... Etc/UTC
nominatim  | creating configuration files ... ok
nominatim  | running bootstrap script ... ok
nominatim  | performing post-bootstrap initialization ... ok
nominatim  | syncing data to disk ... ok
nominatim  |
nominatim  |
nominatim  | Success. You can now start the database server using:
nominatim  |
nominatim  |     /usr/lib/postgresql/14/bin/pg_ctl -D /var/lib/postgresql/14/main -l logfile start
nominatim  |
nominatim  | initdb: warning: enabling "trust" authentication for local connections
nominatim  | You can change this by editing pg_hba.conf or using the option -A, or
nominatim  | --auth-local and --auth-host, the next time you run initdb.
nominatim  | + cp /etc/postgresql/14/main/conf.d/postgres-import.conf.disabled /etc/postgresql/14/main/conf.d/postgres-import.conf
nominatim  | + sudo service postgresql start
nominatim  |  * Starting PostgreSQL 14 database server
nominatim  |    ...done.
nominatim  | + sudo -E -u postgres psql postgres -tAc 'SELECT 1 FROM pg_roles WHERE rolname='\''nominatim'\'''
nominatim  | + grep -q 1
nominatim  | + sudo -E -u postgres createuser -s nominatim
nominatim  | + sudo -E -u postgres psql postgres -tAc 'SELECT 1 FROM pg_roles WHERE rolname='\''www-data'\'''
nominatim  | + grep -q 1
nominatim  | + sudo -E -u postgres createuser -SDR www-data
nominatim  | + sudo -E -u postgres psql postgres -tAc 'ALTER USER nominatim WITH ENCRYPTED PASSWORD '\''qaIACxO6wMR3'\'''
nominatim  | ALTER ROLE
nominatim  | + sudo -E -u postgres psql postgres -tAc 'ALTER USER "www-data" WITH ENCRYPTED PASSWORD '\''qaIACxO6wMR3'\'''
nominatim  | ALTER ROLE
nominatim  | + sudo -E -u postgres psql postgres -c 'DROP DATABASE IF EXISTS nominatim'
nominatim  | DROP DATABASE
nominatim  | NOTICE:  database "nominatim" does not exist, skipping
nominatim  | + chown -R nominatim:nominatim /nominatim
nominatim  | + cd /nominatim
nominatim  | + '[' '' = true ']'
nominatim  | + sudo -E -u nominatim nominatim import --osm-file /nominatim/data.osm.pbf --threads 16
nominatim  | 2024-03-29 22:16:55: Using project directory: /nominatim
nominatim  | 2024-03-29 22:16:59: Creating database
nominatim  | 2024-03-29 22:17:00: Setting up country tables
nominatim  | 2024-03-29 22:17:03: Importing OSM data file
nominatim  | 2024-03-29 22:17:03  osm2pgsql version 1.11.0
nominatim  | 2024-03-29 22:17:03  Database version: 14.11 (Ubuntu 14.11-0ubuntu0.22.04.1)
nominatim  | 2024-03-29 22:17:03  PostGIS version: 3.2
nominatim  | 2024-03-29 22:17:03  Storing properties to table '"public"."osm2pgsql_properties"'.

[mtmail]

I started this last night ... It is now 04:46 in my time zone

How many hours is that?

processor is barely moving, ram barely touched, it made a flat nodes file, which seems to have stopped at 90GB or so.

The osm2pgsql step imports the *.pbf file into the postgres tables planet_osm_rels, planet_osm_ways, planet_osm_nodes. If a flatnode file is requested then the data for table planet_osm_nodes goes into that file instead.

osm2pgsql initialises the file with the full size and then starts adding data. It neither grows or shrinks. That's why it's only useful for big imports (continents, the whole planet).

I'm not seeing any lines about US post code import either...

That step is much later in the setup. If in doubt import a small region first, like a small US state, so get accustomed to the import steps.

I'm surprised you didn't configure to import the US TIGER address data. It's additional data useful with US imports.

is the 4.4 dockerfile messed up somewhere?

Let's start with the assumption that the Dockerfile works for others users.

Can you list what hardware you're using? Is the storage network mounted, inside a VM itself (e.g. VMware) or such?

shm_size: '4gb'

The Readme file lists "Half the size of your available RAM is recommended". If you have only 8GB RAM then the import will likely not work. Setting the values listed in "PostgreSQL Tuning" helps with the osm2pgsql step.

The osm2pgsql step used to be much more verbose, now it only prints when it finishes. You should be able to see records being inserted into the planet_* tables ("select count(*) from $tablename;"). osm2pgsql first works on the nodes, then ways, then relations.

[mtmail]

processor is barely moving, ram barely touched

Then it seems to be either IO-bound or constantly waiting for the database to insert records. osm2pgsql works single-threaded, so on systems with many CPU cores it looks weird but data in PBF files can't be processed in parallel. osm2pgsql doesn't use a lot of RAM when flatnode file is present.

[SenorKarlos]

I'm basing off of two or three year old memories from 4.2, and not using flat nodes. It is most definitely running now :D all my cpus are pinned.

I was actually going to skip US post codes but forgot to remove the line, I don't even service US other than bordering with Canada and only need reverse coord searches lol. wups.

Wishing I remembered to increase the memory though, system only has 72 GB of it 🤦‍♂️

I checked this briefly when I got up and left it alone, and glad I did LOL :) thanks for the reply!!

I just realized the logs are in UTC, so I would have started the run at about 18:00 local FYI.