Nominatim Docker 4.0 error while importing country file to database #338

cormanttech · 2022-07-04T02:15:40Z

cormanttech
Jul 4, 2022

Hi All,

I am getting this error when the automatic import process is on importing the country file to the database,

nominatim@ip-172-31-1-226:~/nominatim-docker/4.0$ sudo docker run -it --rm \

-e PBF_URL=https://download.geofabrik.de/asia/philippines-latest.osm.pbf
-e REPLICATION_URL=https://download.geofabrik.de/asia/philippines-updates/
-p 8080:8080
--name nominatim
mediagis/nominatim:4.0

trap stopServices TERM
/app/config.sh
id nominatim
useradd -m -p qaIACxO6wMR3 nominatim
IMPORT_FINISHED=/var/lib/postgresql/12/main/import-finished
TOKENIZER_DIR=/nominatim/tokenizer
'[' '!' -f /var/lib/postgresql/12/main/import-finished ']'
/app/init.sh
OSMFILE=/nominatim/data.osm.pbf
'[' '' = true ']'
'[' -f '' ']'
echo 'Skipping optional Wikipedia importance import'
Skipping optional Wikipedia importance import
'[' '' = true ']'
'[' -f '' ']'
echo 'Skipping optional GB postcode import'
Skipping optional GB postcode import
'[' '' = true ']'
'[' -f '' ']'
echo 'Skipping optional US postcode import'
Skipping optional US postcode import
'[' '' = true ']'
'[' -f '' ']'
echo 'Skipping optional Tiger addresses import'
Skipping optional Tiger addresses import
'[' https://download.geofabrik.de/asia/philippines-latest.osm.pbf '!=' '' ']'
echo Downloading OSM extract from https://download.geofabrik.de/asia/philippines-latest.osm.pbf
Downloading OSM extract from https://download.geofabrik.de/asia/philippines-latest.osm.pbf
curl -L https://download.geofabrik.de/asia/philippines-latest.osm.pbf -C - --create-dirs -o /nominatim/data.osm.pbf
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 475M 100 475M 0 0 8373k 0 0:00:58 0:00:58 --:--:-- 8611k
'[' '' '!=' '' ']'
'[' '!' -f /var/lib/postgresql/12/main/PG_VERSION ']'
sudo service postgresql start

Starting PostgreSQL 12 database server [ OK ]

sudo -E -u postgres psql postgres -tAc 'SELECT 1 FROM pg_roles WHERE rolname='''nominatim''''
grep -q 1
sudo -E -u postgres createuser -s nominatim
grep -q 1
sudo -E -u postgres psql postgres -tAc 'SELECT 1 FROM pg_roles WHERE rolname='''www-data''''
sudo -E -u postgres createuser -SDR www-data
sudo -E -u postgres psql postgres -tAc 'ALTER USER nominatim WITH ENCRYPTED PASSWORD '''qaIACxO6wMR3''''
ALTER ROLE
sudo -E -u postgres psql postgres -tAc 'ALTER USER "www-data" WITH ENCRYPTED PASSWORD '''qaIACxO6wMR3''''
ALTER ROLE
sudo -E -u postgres psql postgres -c 'DROP DATABASE IF EXISTS nominatim'
NOTICE: database "nominatim" does not exist, skipping
DROP DATABASE
chown -R nominatim:nominatim /nominatim
cd /nominatim
sudo -E -u nominatim nominatim import --osm-file /nominatim/data.osm.pbf --threads 16
2022-07-04 01:56:05: Using project directory: /nominatim
2022-07-04 01:56:05: Creating database
2022-07-04 01:56:07: Setting up country tables
2022-07-04 01:56:09: Importing OSM data file
2022-07-04 01:56:09 osm2pgsql version 1.5.1
2022-07-04 01:56:09 Database version: 12.11 (Ubuntu 12.11-0ubuntu0.20.04.1)
2022-07-04 01:56:09 PostGIS version: 3.0
2022-07-04 01:56:09 Parsing gazetteer style file '/usr/local/etc/nominatim/import-full.style'.
NOTICE: table "place" does not exist, skipping
Processing: Node(10360k 647.5k/s) Way(0k 0.00k/s) Relation(0 0.0/s)Traceback (most recent call last):
File "/usr/local/bin/nominatim", line 11, in
exit(cli.nominatim(module_dir='/usr/local/lib/nominatim/module',
File "/usr/local/lib/nominatim/lib-python/nominatim/cli.py", line 235, in nominatim
return parser.run(**kwargs)
File "/usr/local/lib/nominatim/lib-python/nominatim/cli.py", line 96, in run
return args.command.run(args)
File "/usr/local/lib/nominatim/lib-python/nominatim/clicmd/setup.py", line 77, in run
database_import.import_osm_data(files,
File "/usr/local/lib/nominatim/lib-python/nominatim/tools/database_import.py", line 98, in import_osm_data
run_osm2pgsql(options)
File "/usr/local/lib/nominatim/lib-python/nominatim/tools/exec_utils.py", line 139, in run_osm2pgsql
subprocess.run(cmd, cwd=options.get('cwd', '.'),
File "/usr/lib/python3.8/subprocess.py", line 516, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/local/lib/nominatim/osm2pgsql', '--hstore', '--latlon', '--slim', '--with-forward-dependencies', 'false', '--log-progress', 'true', '--number-processes', '1', '--cache', '730', '--output', 'gazetteer', '--style', '/usr/local/etc/nominatim/import-full.style', '--create', '/nominatim/data.osm.pbf']' died with <Signals.SIGKILL: 9>.
nominatim@ip-172-31-1-226:~/nominatim-docker/4.0$

Thank you.

leonardehrenfried · 2022-07-04T05:41:25Z

leonardehrenfried
Jul 4, 2022
Collaborator

It looks like osm2pgsql is running out of memory. How much do you have?

0 replies

cormanttech · 2022-07-04T05:53:16Z

cormanttech
Jul 4, 2022
Author

Hi Sir,

This is the total memory, the available is only 1GB

We just need the Philippines map, is that enough for a 475 MB of .osm.pbf data?

1 reply

cormanttech Jul 4, 2022
Author

I am also getting this error now:

nominatim@ip-172-31-1-226:~/nominatim-docker/4.0$ sudo docker run -it --rm -e PBF_URL=https://download.geofabrik.de/asia/philippines-latest.osm.pbf -e REPLI CATION_URL=https://download.geofabrik.de/asia/philippines-updates/ -p 8080:808 0 --name nominatim mediagis/nominatim:4.0

trap stopServices TERM
/app/config.sh
id nominatim
useradd -m -p qaIACxO6wMR3 nominatim
IMPORT_FINISHED=/var/lib/postgresql/12/main/import-finished
TOKENIZER_DIR=/nominatim/tokenizer
'[' '!' -f /var/lib/postgresql/12/main/import-finished ']'
/app/init.sh
OSMFILE=/nominatim/data.osm.pbf
'[' '' = true ']'
'[' -f '' ']'
echo 'Skipping optional Wikipedia importance import'
Skipping optional Wikipedia importance import
'[' '' = true ']'
'[' -f '' ']'
echo 'Skipping optional GB postcode import'
Skipping optional GB postcode import
'[' '' = true ']'
'[' -f '' ']'
echo 'Skipping optional US postcode import'
Skipping optional US postcode import
'[' '' = true ']'
'[' -f '' ']'
echo 'Skipping optional Tiger addresses import'
Skipping optional Tiger addresses import
'[' https://download.geofabrik.de/asia/philippines-latest.osm.pbf '!=' '' ']'
echo Downloading OSM extract from https://download.geofabrik.de/asia/philippin es-latest.osm.pbf
Downloading OSM extract from https://download.geofabrik.de/asia/philippines-late st.osm.pbf
curl -L https://download.geofabrik.de/asia/philippines-latest.osm.pbf -C - --c reate-dirs -o /nominatim/data.osm.pbf
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 475M 100 475M 0 0 9356k 0 0:00:52 0:00:52 --:--:-- 9672k
'[' '' '!=' '' ']'
'[' '!' -f /var/lib/postgresql/12/main/PG_VERSION ']'
sudo service postgresql start

Starting PostgreSQL 12 database server [ OK ]

sudo -E -u postgres psql postgres -tAc 'SELECT 1 FROM pg_roles WHERE rolname=' ''nominatim''''
grep -q 1
sudo -E -u postgres createuser -s nominatim
grep -q 1
sudo -E -u postgres psql postgres -tAc 'SELECT 1 FROM pg_roles WHERE rolname=' ''www-data''''
sudo -E -u postgres createuser -SDR www-data
sudo -E -u postgres psql postgres -tAc 'ALTER USER nominatim WITH ENCRYPTED PA SSWORD '''qaIACxO6wMR3''''
ALTER ROLE
sudo -E -u postgres psql postgres -tAc 'ALTER USER "www-data" WITH ENCRYPTED P ASSWORD '''qaIACxO6wMR3''''
ALTER ROLE
sudo -E -u postgres psql postgres -c 'DROP DATABASE IF EXISTS nominatim'
NOTICE: database "nominatim" does not exist, skipping
DROP DATABASE
chown -R nominatim:nominatim /nominatim
cd /nominatim
sudo -E -u nominatim nominatim import --osm-file /nominatim/data.osm.pbf --thr eads 16
2022-07-04 07:45:47: Using project directory: /nominatim
2022-07-04 07:45:47: Creating database
2022-07-04 07:45:48: Setting up country tables
2022-07-04 07:45:50: Importing OSM data file
2022-07-04 07:45:51 osm2pgsql version 1.5.1
2022-07-04 07:45:51 Database version: 12.11 (Ubuntu 12.11-0ubuntu0.20.04.1)
2022-07-04 07:45:51 PostGIS version: 3.0
2022-07-04 07:45:51 Parsing gazetteer style file '/usr/local/etc/nominatim/impo rt-full.style'.
NOTICE: table "place" does not exist, skipping
Processing: Node(29740k 210.9k/s) Way(0k 0.00k/s) Relation(0 0.0/s)2022-07-04 07 :48:12 ERROR: planet_osm_nodes - error on COPY: server closed the connection un expectedly
This probably means the server terminated abnormally
before or while processing the request.

2022-07-04 07:48:12 ERROR: Data: 4671365258 139882576 1211682556
4671365259 139882583 1211683143
4671365260 139883378 1211683134
4671365261 139883371 1211682547
4671365262 139883970 1211684039
4671365263 139882884 1211684206
4671365264 139882750 1211683286
4671365265 139883837 1211683118
4671365266 139883916 1211683323
4671365267 139884137 1211684423
4671365268 139885034 1211684232
4671365269 139884813 1211683132
4671365270 139883302 1211684955
4671365271 139883380 1211685504
4671365272 139884351 1211685358
4671365273 139884273
...
45721003 1205895177
4682802177 145721467 1205895037
4682802178 145720505 1205892619
4682802179 145719827 1205892817
4682802180 145719763 1205892596
4682802181 145720464 1205892432
4682802182 145720151 1205892449
4682802183 145719690 1205892573
4682802184 145719614 1205892271
4682802185 145720074 1205892147
4682802186 145719929 1205891783
4682802187 145719598 1205891897
4682802188 145719701 1205892217
4682802489 145720032 1205892103
4682802490 145720361 1205891213
4682802491 145719929 1205891354

2022-07-04 07:48:12 ERROR: DB copy thread failed: COPYing data to Postgresql.
Processing: Node(29900k 210.6k/s) Way(0k 0.00k/s) Relation(0 0.0/s)Traceback (mo st recent call last):
File "/usr/local/bin/nominatim", line 11, in
exit(cli.nominatim(module_dir='/usr/local/lib/nominatim/module',
File "/usr/local/lib/nominatim/lib-python/nominatim/cli.py", line 235, in nomi natim
return parser.run(**kwargs)
File "/usr/local/lib/nominatim/lib-python/nominatim/cli.py", line 96, in run
return args.command.run(args)
File "/usr/local/lib/nominatim/lib-python/nominatim/clicmd/setup.py", line 77, in run
database_import.import_osm_data(files,
File "/usr/local/lib/nominatim/lib-python/nominatim/tools/database_import.py", line 98, in import_osm_data
run_osm2pgsql(options)
File "/usr/local/lib/nominatim/lib-python/nominatim/tools/exec_utils.py", line 139, in run_osm2pgsql
subprocess.run(cmd, cwd=options.get('cwd', '.'),
File "/usr/lib/python3.8/subprocess.py", line 516, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/local/lib/nominatim/osm2pgsql', ' --hstore', '--latlon', '--slim', '--with-forward-dependencies', 'false', '--log- progress', 'true', '--number-processes', '1', '--cache', '952', '--output', 'gaz etteer', '--style', '/usr/local/etc/nominatim/import-full.style', '--create', '/ nominatim/data.osm.pbf']' returned non-zero exit status 2.
nominatim@ip-172-31-1-226:~/nominatim-docker/4.0$

I already lowered my shared buffers to 200MB because I only have 4GB of RAM in total

mtmail · 2022-07-04T13:37:20Z

mtmail
Jul 4, 2022

I think you need at least 2GB of RAM. With 1GB of RAM I wasn't able to import Monaco and that country is less than 1 megabyte of *.pbf data.

2 replies

cormanttech Jul 5, 2022
Author

Thanks for this sir! I will try to free some memory. Thank you.

cormanttech Jul 6, 2022
Author

Hi All,

I think I successfully installed the Nominatim OSM 4.0 docker container. But when I am accessing my localhost/, this is what it only shows:

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Nominatim Docker 4.0 error while importing country file to database #338

{{title}}

Replies: 3 comments 3 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Nominatim Docker 4.0 error while importing country file to database #338

cormanttech Jul 4, 2022

Replies: 3 comments · 3 replies

leonardehrenfried Jul 4, 2022 Collaborator

cormanttech Jul 4, 2022 Author

cormanttech Jul 4, 2022 Author

mtmail Jul 4, 2022

cormanttech Jul 5, 2022 Author

cormanttech Jul 6, 2022 Author

cormanttech
Jul 4, 2022

Replies: 3 comments 3 replies

leonardehrenfried
Jul 4, 2022
Collaborator

cormanttech
Jul 4, 2022
Author

cormanttech Jul 4, 2022
Author

mtmail
Jul 4, 2022

cormanttech Jul 5, 2022
Author

cormanttech Jul 6, 2022
Author