[Merged by Bors] - sql: improve database schema handling #6003
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Use consistent package/folder structure for local.sql and state.sql databases. When a new database is created, use schema script instead of running all the migrations. Also, check that there's no schema drift by diffing database schema against the schema script after running migrations.
ivan4th
force-pushed
the
feature/schema-snapshot
branch
from
f33a283 to
7a97f7a
Compare
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## develop #6003 +/- ##
=========================================
- Coverage 82.0% 82.0% -0.1%
=========================================
Files 307 312 +5
Lines 34106 34327 +221
=========================================
+ Hits 27996 28153 +157
- Misses 4332 4389 +57
- Partials 1778 1785 +7 ☔ View full report in Codecov by Sentry. |
poszu
reviewed
Jun 3, 2024
fasmat
requested changes
Jun 3, 2024
fasmat
reviewed
Jun 4, 2024
fasmat
reviewed
Jun 4, 2024
fasmat
reviewed
Jun 4, 2024
fasmat
reviewed
Jun 5, 2024
fasmat
reviewed
Jun 5, 2024
Also, add a migration that removes stray tables from quicksynced DBs
tryBuild failed: |
|
Last |
|
bors try |
tryBuild failed: |
|
bors try |
tryBuild succeeded: |
fasmat
reviewed
Aug 19, 2024
|
Took me a bit but I made it through all changes, there is also still this comment by @poszu that is open: #6003 (comment) |
Also fixed |
fasmat
reviewed
Aug 20, 2024
|
CI rerun due: Likely unrelated |
fasmat
approved these changes
Aug 20, 2024
|
bors merge |
spacemesh-bors bot
pushed a commit
that referenced
this pull request
Aug 20, 2024
## Motivation Currently, the database schema is always built by running a series of migrations, even on a fresh database. There's no place in the source code to look for the full database schema in a single place. If schema drift happens for whatever reason (running dev version, a bug in a coded migration, etc.), it may go undetected causing issues at some later point in time. Another problem is that `state.sql` and `local.sql` databases aren't being handled in a consistent manner, with `sql.Open` / `sql.InMemory` being used for the state database with its associated migrations, and `localsql.Open` / `localsql.InMemory` being derived from them. This has some unpleasant consequences, most importantly that it is hard to implement coded migrations as they'll have to be referred explicitly from `node/node.go` and possibly in other places. `syncv2` will require some coded migration(s) for storing synctrees in the database, and ahead of these changes, it would be nice to have this kind of refactoring done in `go-spacemesh`.
|
Pull request successfully merged into develop. Build succeeded: |
spacemesh-bors
bot
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
Currently, the database schema is always built by running a series of migrations, even on a fresh database. There's no place in the source code to look for the full database schema in a single place. If schema drift happens for whatever reason (running dev version, a bug in a coded migration, etc.), it may go undetected causing issues at some later point in time.
Another problem is that
state.sqlandlocal.sqldatabases aren't being handled in a consistent manner, withsql.Open/sql.InMemorybeing used for the state database with its associated migrations, andlocalsql.Open/localsql.InMemorybeing derived from them. This has some unpleasant consequences, most importantly that it is hard to implement coded migrations as they'll have to be referred explicitly fromnode/node.goand possibly in other places.syncv2will require some coded migration(s) for storing synctrees in the database, and ahead of these changes, it would be nice to have this kind of refactoring done ingo-spacemesh.Description
When a new database is created, use schema script instead of running all the migrations. Also, check that there's no schema drift by diffing database schema against the schema script after running migrations.
Use consistent package/folder structure for
local.sqlandstate.sqldatabases:sql/statesqlandsql/localsqlhandle the state db and the local db, respectively, withSchema,InMemoryandOpenfunctions provided. Coded migrations can be added to these packages, toosql/statesql/schema/schema.sqlandsql/statesql/schema/schema.sqlcontain current database schemassql/statesql/schema/migrations/*.sqlandsql/statesql/schema/migrations/*.sqlcontain database migrationsWhen a new database is created, the schema snapshot from
schema.sqlis used to initialize it. If the database already exists, its version is checked and if it's old, the migrations are run. In both cases, the schema is retrieved from the database afterwards and diffed against the schema snapshot, logging warnings about any detected differences (schema drift).Fixes #6026.
Test Plan
Verify on testnet and mainnet nodes
TODO