Problems with reindex on 3.7.8

[Wildcard5970]

Hi,

Im having problem to upgrade from 3.2.5 to 3.7.8. When I run: PYTHON=/usr/bin/python3.11 ./upgrade.sh as netbox user i get this output. Something is not right with the database and I want some help figuring it out.

----------------upgrade.sh Output---------------------------
Collecting static files (python3 netbox/manage.py collectstatic --no-input)...

513 static files copied to '/opt/netbox-3.7.8/netbox/static', 24 unmodified.
Removing stale content types (python3 netbox/manage.py remove_stale_contenttypes --no-input)...
Rebuilding search cache (python3 netbox/manage.py reindex --lazy)...
Reindexing 74 models.
Indexing models
core.datasource... No objects found.
core.datafile... No objects found.
circuits.circuit... Traceback (most recent call last):
File "/opt/netbox-3.7.8/venv/lib64/python3.11/site-packages/django/db/backends/utils.py", line 89, in execute
return self.cursor.execute(sql, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/netbox-3.7.8/venv/lib64/python3.11/site-packages/psycopg/server_cursor.py", line 292, in execute
raise ex.with_traceback(None)
psycopg.errors.UndefinedColumn: column circuits_circuit.provider_account_id does not exist
LINE 1: ...") LEFT OUTER JOIN "circuits_provideraccount" ON ("circuits...
^

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/opt/netbox-3.7.8/netbox/manage.py", line 10, in
execute_from_command_line(sys.argv)
File "/opt/netbox-3.7.8/venv/lib64/python3.11/site-packages/django/core/management/init.py", line 442, in execute_from_command_line
utility.execute()
File "/opt/netbox-3.7.8/venv/lib64/python3.11/site-packages/django/core/management/init.py", line 436, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/opt/netbox-3.7.8/venv/lib64/python3.11/site-packages/django/core/management/base.py", line 412, in run_from_argv
self.execute(*args, **cmd_options)
File "/opt/netbox-3.7.8/venv/lib64/python3.11/site-packages/django/core/management/base.py", line 458, in execute
output = self.handle(*args, **options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/netbox-3.7.8/netbox/extras/management/commands/reindex.py", line 90, in handle
i = search_backend.cache(model.objects.iterator(), remove_existing=False)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/netbox-3.7.8/netbox/netbox/search/backends.py", line 194, in cache
for instance in instances:
File "/opt/netbox-3.7.8/venv/lib64/python3.11/site-packages/django/db/models/query.py", line 516, in _iterator
yield from iterable
File "/opt/netbox-3.7.8/venv/lib64/python3.11/site-packages/django/db/models/query.py", line 91, in iter
results = compiler.execute_sql(
^^^^^^^^^^^^^^^^^^^^^
File "/opt/netbox-3.7.8/venv/lib64/python3.11/site-packages/django/db/models/sql/compiler.py", line 1562, in execute_sql
cursor.execute(sql, params)
File "/opt/netbox-3.7.8/venv/lib64/python3.11/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/netbox-3.7.8/venv/lib64/python3.11/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
return executor(sql, params, many, context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/netbox-3.7.8/venv/lib64/python3.11/site-packages/django/db/backends/utils.py", line 84, in _execute
with self.db.wrap_database_errors:
File "/opt/netbox-3.7.8/venv/lib64/python3.11/site-packages/django/db/utils.py", line 91, in exit
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/opt/netbox-3.7.8/venv/lib64/python3.11/site-packages/django/db/backends/utils.py", line 89, in execute
return self.cursor.execute(sql, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/netbox-3.7.8/venv/lib64/python3.11/site-packages/psycopg/server_cursor.py", line 292, in execute
raise ex.with_traceback(None)
django.db.utils.ProgrammingError: column circuits_circuit.provider_account_id does not exist
LINE 1: ...") LEFT OUTER JOIN "circuits_provideraccount" ON ("circuits...

[candlerb]

You omitted the earlier output of upgrade.sh. Did it say anything about migrations? It seems they haven't been applied properly (although I would expect it to fail on error).

[Wildcard5970]

Here is the full output:

`
(venv) [netbox@mgxnetwinf11 netbox]$ python3 netbox/manage.py migrate
Operations to perform:
Apply all migrations: account, admin, auth, circuits, contenttypes, core, dcim, django_rq, extras, ipam, sessions, social_django, taggit, tenancy, users, virtualization, vpn, wireless
Running migrations:
Applying dcim.0155_interface_poe_mode_type...Traceback (most recent call last):
File "/opt/netbox-3.7.8/venv/lib64/python3.11/site-packages/django/db/backends/utils.py", line 87, in _execute
return self.cursor.execute(sql)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/netbox-3.7.8/venv/lib64/python3.11/site-packages/psycopg/cursor.py", line 732, in execute
raise ex.with_traceback(None)
psycopg.errors.DuplicateColumn: column "poe_mode" of relation "dcim_interface" already exists

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/opt/netbox-3.7.8/netbox/manage.py", line 10, in
execute_from_command_line(sys.argv)
File "/opt/netbox-3.7.8/venv/lib64/python3.11/site-packages/django/core/management/init.py", line 442, in execute_from_command_line
utility.execute()
File "/opt/netbox-3.7.8/venv/lib64/python3.11/site-packages/django/core/management/init.py", line 436, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/opt/netbox-3.7.8/venv/lib64/python3.11/site-packages/django/core/management/base.py", line 412, in run_from_argv
self.execute(*args, **cmd_options)
File "/opt/netbox-3.7.8/venv/lib64/python3.11/site-packages/django/core/management/base.py", line 458, in execute
output = self.handle(*args, **options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/netbox-3.7.8/venv/lib64/python3.11/site-packages/django/core/management/base.py", line 106, in wrapper
res = handle_func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/netbox-3.7.8/venv/lib64/python3.11/site-packages/django/core/management/commands/migrate.py", line 356, in handle
post_migrate_state = executor.migrate(
^^^^^^^^^^^^^^^^^
File "/opt/netbox-3.7.8/venv/lib64/python3.11/site-packages/django/db/migrations/executor.py", line 135, in migrate
state = self._migrate_all_forwards(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/netbox-3.7.8/venv/lib64/python3.11/site-packages/django/db/migrations/executor.py", line 167, in _migrate_all_forwards
state = self.apply_migration(
^^^^^^^^^^^^^^^^^^^^^
File "/opt/netbox-3.7.8/venv/lib64/python3.11/site-packages/django/db/migrations/executor.py", line 252, in apply_migration
state = migration.apply(state, schema_editor)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/netbox-3.7.8/venv/lib64/python3.11/site-packages/django/db/migrations/migration.py", line 132, in apply
operation.database_forwards(
File "/opt/netbox-3.7.8/venv/lib64/python3.11/site-packages/django/db/migrations/operations/fields.py", line 108, in database_forwards
schema_editor.add_field(
File "/opt/netbox-3.7.8/venv/lib64/python3.11/site-packages/django/db/backends/base/schema.py", line 713, in add_field
self.execute(sql, params)
File "/opt/netbox-3.7.8/venv/lib64/python3.11/site-packages/django/db/backends/postgresql/schema.py", line 48, in execute
return super().execute(sql, None)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/netbox-3.7.8/venv/lib64/python3.11/site-packages/django/db/backends/base/schema.py", line 201, in execute
cursor.execute(sql, params)
File "/opt/netbox-3.7.8/venv/lib64/python3.11/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/netbox-3.7.8/venv/lib64/python3.11/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
return executor(sql, params, many, context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/netbox-3.7.8/venv/lib64/python3.11/site-packages/django/db/backends/utils.py", line 84, in _execute
with self.db.wrap_database_errors:
File "/opt/netbox-3.7.8/venv/lib64/python3.11/site-packages/django/db/utils.py", line 91, in exit
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/opt/netbox-3.7.8/venv/lib64/python3.11/site-packages/django/db/backends/utils.py", line 87, in _execute
return self.cursor.execute(sql)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/netbox-3.7.8/venv/lib64/python3.11/site-packages/psycopg/cursor.py", line 732, in execute
raise ex.with_traceback(None)
django.db.utils.ProgrammingError: column "poe_mode" of relation "dcim_interface" already exists
`

[candlerb]

It appears your database has become corrupted somehow: I suspect partial migrations had been previously applied. Without seeing the errors from those previous migrations it's hard to know what went wrong.

If you have a backup of the database when you were running v3.2.5, taken before you made any attempt to upgrade to v3.7.8, then you can restore from that backup, run ./upgrade.sh again, and report any errors you see - it may be possible to diagnose.

Otherwise I can't really help you; you will have to unpick this yourself. If you're a SQL expert I guess you could apply schema changes manually to align your schema with that from a fresh v3.7.8 installation, and mark the migrations as completed.

I also don't understand why you didn't get those errors before. ./upgrade.sh should also run the migrations.

[Wildcard5970]

It seems like it. This is our test system and has a sync from production. I have tried the pg_dump pg_restore with the —clean tag. Without any luck.

Is there any way you can reset the database from netbox cli? As the database with its values is not very important.

[candlerb]

As the database with its values is not very important.

To start with an empty netbox database, just drop it, recreate it, then run ./upgrade.sh. This will create all the tables correctly. (Same as when you do an initial installation).