I want to switch to use the make-migrations script and drop all tables from any previous schema version. Previously used manual override migrations. Best way to do this?

[kfieldi5]

I guess do they both get run if I use dart run drift_dev make-migrations moving forward?

@override
  int get schemaVersion => 5;
  
MigrationStrategy get migration {
    return MigrationStrategy(
      onCreate: (Migrator m) async {
        await m.createAll();
      },
      onUpgrade: (Migrator m, int from, int to) async {
        // disable foreign_keys before migrations
        await customStatement('PRAGMA foreign_keys = OFF');

        await transaction(() async {
          if (from < 5) {
            await m.drop(xxx);
            await m.drop(yyy);
            await m.drop(zzz);
            await m.createAll();
          }
        });

        // Assert that the schema is valid after migrations
        if (kDebugMode) {
          final wrongForeignKeys =
              await customSelect('PRAGMA foreign_key_check').get();
          assert(wrongForeignKeys.isEmpty,
              '${wrongForeignKeys.map((e) => e.data)}');
        }
      },
      beforeOpen: (details) async {
        await customStatement('PRAGMA foreign_keys = ON');
        if (kDebugMode) {
          // This check pulls in a fair amount of code that's not needed
          // anywhere else, so we recommend only doing it in debug builds.
          //await validateDatabaseSchema();
        }
      },
    );
  }

any guidance appreciated thanks.

[simolus3]

You can keep using your existing migration logic and only use the new form for later migrations, is that what you were looking for?

If all of your migrations involve dropping and then re-creating all tables, there's also not too much use in most what make-migrations has to offer. It's still useful to test that these migrations are doing what they're supposed to do, but you don't really need versioned schema helpers in that case.

[kfieldi5]

Most future migrations will be a lot more intricate than this. I am only dropping all tables on this version so that we have a clean-slate.

My first schema file is drift_schema_v4.json

Before that I did not have schema files.

To utilize the new make-migrations script I need to call runMigrationSteps and still manually put in my addColumn's, removeColumn's, and alterTable commands? I guess I'm unclear on how to use the new automatic way.

Here is my attempt, please correct as needed:

@override
  MigrationStrategy get migration {
    return MigrationStrategy(
      onCreate: (Migrator m) async {
        await m.createAll();
      },
      onUpgrade: (Migrator m, int from, int to) async {
        // disable foreign_keys before migrations
        await customStatement('PRAGMA foreign_keys = OFF');

        // Manually running migrations up to schema version 4, after which we've
        // enabled step-by-step migrations.
        if (from <= 4) {
          await transaction(() async {
            await m.drop(xxx);
            await m.drop(yyy);
            await m.drop(zzz);
          });
        }

        // At this point, we should be migrated to schema 4. For future schema
        // changes, we will "start" at schema 5.
        await m.runMigrationSteps(
          from: max(4, from),
          to: to,
          // ignore: missing_required_argument
          steps: migrationSteps(
            from4To5: (m, schema) async {
              // For future upgrades, this will be more intricate:
              // adding/deleting specific columns, altering tables etc.
              await m.createAll();
            },
          ),
        );

        // Assert that the schema is valid after migrations
        if (kDebugMode) {
          final wrongForeignKeys =
              await customSelect("PRAGMA foreign_key_check").get();
          assert(wrongForeignKeys.isEmpty,
              "${wrongForeignKeys.map((e) => e.data)}");
        }
      },
      beforeOpen: (details) async {
        await customStatement("PRAGMA foreign_keys = ON");
        if (kDebugMode) {
          // This check pulls in a fair amount of code that's not needed
          // anywhere else, so we recommend only doing it in debug builds.
          //await validateDatabaseSchema();
        }
      },
    );
  }

[simolus3]

That code looks correct to me.

    // changes, we will "start" at schema 5.

The steps runner would start at schema 4, with from4To5 being the first migration.,

My first schema file is drift_schema_v4.json

Then you can also remove the missing_required_argument ignore. It's only part of the documentation for internal reasons (because we're testing that snippet against generated code that does have all of the versions), I'll remove it from there as well.

[kfieldi5]

Thanks @simolus3. I think I'm in the clear.

We are having a production release-only bug in the apk's we're making and our remote logging was setup poorly so all these precautions are to prevent any of this from potentially biting us in the ass next release. I've added a lot more logging but I've also been reading through a lot of issues with locking DBs. I do not call db.close() anywhere except in test code. Should I be? Also I call setupSqlCipher twice (once before init).. not sure if thats necessary.

This is the init impl for android:

import "dart:io";

import "package:drift/drift.dart";
import "package:drift/native.dart";
import "package:flutter/services.dart";
import "package:path/path.dart" as p;
import "package:path_provider/path_provider.dart";
import "package:sqlcipher_flutter_libs/sqlcipher_flutter_libs.dart";
import "package:sqlite3/open.dart";
import "package:sqlite3/sqlite3.dart";

import "../../utils/logging.dart";
import "database_service.dart";

DatabaseService init(final String dbPassphrase, bool deletePreviousDb) {
  setupSqlCipher();
  return DatabaseService(_createLazyDb(dbPassphrase, deletePreviousDb));
}

LazyDatabase _createLazyDb(String dbPassphrase, bool deletePrevious) {
  return LazyDatabase(() async {
    logger.info("DRIFT DeviceOnlyDatabase::_createLazyDb "
        "deletePrevious: $deletePrevious");

    final dbFolder = await getApplicationDocumentsDirectory();
    final file = File(p.join(dbFolder.path, "db.sqlite"));

    if (deletePrevious && await file.exists()) {
      await file.delete();
    }

    final token = RootIsolateToken.instance;

    return NativeDatabase.createInBackground(
      file,
      isolateSetup: () async {
        BackgroundIsolateBinaryMessenger.ensureInitialized(token!);
        await setupSqlCipher();
      },
      setup: (rawDb) {
        assert(_debugCheckHasCipher(rawDb));
        rawDb.execute("PRAGMA key = '$dbPassphrase';");
      },
    );
  });
}

bool _debugCheckHasCipher(Database database) {
  return database.select("PRAGMA cipher_version;").isNotEmpty;
}

Future<void> setupSqlCipher() async {
  await applyWorkaroundToOpenSqlCipherOnOldAndroidVersions();
  open.overrideFor(OperatingSystem.android, openCipherOnAndroid);
}

[simolus3]

I do not call db.close() anywhere except in test code. Should I be?

If you use a database temporarily, it can make sense to close it explicitly. If your database essentially has to stay open as long as the application (and most databases fit this criteria), there's no need to close it explicitly. After your application process gets killed, all completed database operations will be persisted and the SQLite was designed to never leave the database file in an invalid state.

Also I call setupSqlCipher twice (once before init).. not sure if thats necessary.

It's not, but it also doesn't make anything worse. The setupSqlCipher call in isolateSetup is sufficient.

[kfieldi5]

@simolus3 I am upping my schemaVersion to 6 (first schema bump since using migration manager) and, though from5to6 is generated fine, its erasing the from4to5 in my migrationSteps auto-gen function. Not sure why this is happening.

Also I wish we could have a flag param on dart run drift_dev make-migrations so that we could say the first version is v5 or something. Everytime I run that script I have to change the v1 and v2 on these lines to v5 and v6 in the migration_test.dart auto gen file.

createOld: v5.DatabaseAtV5.new,
createNew: v6.DatabaseAtV6.new,

Thanks,
Kev

[simolus3]

Not sure why this is happening.

Could you post the contents of the migrations folder (the one containing all the jsons)? Are the files in there named correctly (e.g. is there a v4, a v5 and a v6 file)?

Everytime I run that script I have to change the v1 and v2 on these lines to v5 and v6 in the migration_test.dart auto gen file.

The tool should be able to detect the first migration based what schema versions you have. So if the first file is a v4, the example should suggest a v4 to v5 test.
But the tool also shouldn't overwrite your existing migration test file, are you deleting that file manually before running make-migrations?

[kfieldi5]

Before schemaVersion bump, we only have drift_schemas/db_sqlite/drift_schema_v5.json, and database_service.steps.dart has from4to5 included in it.

After version bump we have drift_schema_v5.json and drift_schema_v6.json in that folder and we only have from5to6 in the database_service.steps.dart class

Hypothesis:

The first run of make-migrations may have been run when (temporarily) a drift_schema_v4.json existed? But then maybe due to it not being needed I deleted the file after .steps.dart file was genned?

[simolus3]

The first run of make-migrations may have been run when (temporarily) a drift_schema_v4.json existed?

That would explain this. But drift definitely needs all json definitions for all schema versions when generating code. We use to it to determine which schema versions there are (relevant for test code). We also need to know the "source" so that we can continue emitting correct Dart code after a possible major release with breaking changes (if you only keep the generated code around, that might then become invalid with no way of updating it).