validateDatabaseSchema in runtime asks for database, but when I provide one it fails

[farguello]

Hi! I'm trying to set up migrations and a simple change in a column. Changed NOT NULL to nullable in the schema definition.

I set up database step-by-step migration with dart run drift_dev make-migrations and generated necesary code.

onUpgrade: (m, from, to) async {
        await customStatement('PRAGMA foreign_keys = OFF');

        await m.runMigrationSteps(
          from: from,
          to: to,
          steps: migrationSteps(
            from1To2: (m, schema) async {
              await m.alterTable(
                TableMigration(schema.animals),
              );
            },
          ),
        );

        if (kDebugMode) {
          // Fail if the migration broke foreign keys
          final wrongForeignKeys = await customSelect('PRAGMA foreign_key_check').get();
          assert(wrongForeignKeys.isEmpty, '${wrongForeignKeys.map((e) => e.data)}');
        }

        await customStatement('PRAGMA foreign_keys = ON;');
      },

The problem comes after it, and I think I don't understand how this works.
I wanted to make a schema validation on runtime

beforeOpen: (details) async {
        // your existing beforeOpen callback, enable foreign keys, etc.

        if (kDebugMode) {
          // This check pulls in a fair amount of code that's not needed
          // anywhere else, so we recommend only doing it in debug builds.
          await validateDatabaseSchema(this); //<---- notice the `this` parameter
        }
      },

First of all I had to pass this to the function signature because it was required, and the validator works, but it fails with this message

Schema does not match
      animals:
       columns:
        gender:
         Not equal: `NOT NULL` (expected) and `NULL` (actual)

If I inspect the database it did run the migration just fine, but what bugs me is that error when I want to validate...
Am I missing something here?

[dickermoshe]

Does this happen in the app or when you run the generated tests?

[farguello]

This happens right when I start the app and the db instance gets created

[dickermoshe]

what happens when you run the generated tests?

[dickermoshe]

If those pass it probably means that you updated the schema version before writing the actual migration while the app was running.
So an empty migration was ran. writing the migration after has no effect because the database thinks it's already at the latest version.

[farguello]

Thanks @dickermoshe I posted an answer that relates to thi on @simolus3 response!

[simolus3]

If I inspect the database it did run the migration just fine, but what bugs me is that error when I want to validate...

This is a good catch that hasn't occurred to me: Migration tests might instantiate the schema of your database to an older version of the schema, but validateDatabaseSchema will match it against the current version. So that might fail in schema tests.

But since this also happens when running the app, I suspect something else is going wrong too. It would be good to know whether the tests pass - if they do, it might be that only your existing app instance is in a bad state due to e.g. an incomplete migration after changing the schema as @dickermoshe said.

[farguello]

Adding TestWidgetsFlutterBinding.ensureInitialized(); didn't work.
Upon changing to flutter_test package I tried it and I see it has some problems with getApplicationDocumentsDirectory helper

MissingPluginException(No implementation found for method getApplicationDocumentsDirectory on channel plugins.flutter.io/path_provider)
package:flutter/src/services/platform_channel.dart 332:7        MethodChannel._invokeMethod
===== asynchronous gap ===========================
dart:async                                                      Future.then
package:drift/src/utils/lazy_database.dart 47:27                LazyDatabase._awaitOpened
package:drift/src/utils/lazy_database.dart 64:12                LazyDatabase.ensureOpen
package:drift_dev/src/services/schema/verifier_impl.dart 41:23  VerifierImplementation.migrateAndValidate
===== asynchronous gap ===========================
dart:async                                                      _CustomZone.registerUnaryCallback
package:drift_dev/src/services/schema/verifier_impl.dart 29:29  VerifierImplementation.migrateAndValidate
test/drift/insights_app/migration_test.dart 32:28               main.<fn>.<fn>.<fn>
===== asynchronous gap ===========================
dart:async                                                      _CustomZone.registerUnaryCallback
test/drift/insights_app/migration_test.dart 30:28               main.<fn>.<fn>.<fn>

This is the code in AppDatabase class

class AppDatabase extends _$AppDatabase {
  AppDatabase() : super(_openConnection());

  @override
  int get schemaVersion => 2;

  @override
  MigrationStrategy get migration {
    return MigrationStrategy(
      onCreate: (Migrator m) async {
        await m.createAll();
      },
      onUpgrade: (m, from, to) async {
        await customStatement('PRAGMA foreign_keys = OFF');

        await m.runMigrationSteps(
          from: from,
          to: to,
          steps: migrationSteps(
            from1To2: (m, schema) async {
              await m.alterTable(
                TableMigration(schema.animals),
              );
            },
          ),
        );

        if (kDebugMode) {
          // Fail if the migration broke foreign keys
          final wrongForeignKeys = await customSelect('PRAGMA foreign_key_check').get();
          assert(wrongForeignKeys.isEmpty, '${wrongForeignKeys.map((e) => e.data)}');
        }

        await customStatement('PRAGMA foreign_keys = ON;');
      },
      beforeOpen: (details) async {
        // your existing beforeOpen callback, enable foreign keys, etc.

        if (kDebugMode) {
          // This check pulls in a fair amount of code that's not needed
          // anywhere else, so we recommend only doing it in debug builds.
          await validateDatabaseSchema(this);
        }
      },
    );
  }
}

LazyDatabase _openConnection() {
  // the LazyDatabase util lets us find the right location for the file async.
  return LazyDatabase(
    () async {
      // put the database file, called db.sqlite here, into the documents folder
      // for your app.
      final dbFolder = await getApplicationDocumentsDirectory();
      log(dbFolder.toString());
      final file = File(p.join(dbFolder.path, 'mydb.sqlite'));

      // Also work around limitations on old Android versions
      if (Platform.isAndroid) {
        await applyWorkaroundToOpenSqlite3OnOldAndroidVersions();
      }

      // Make sqlite3 pick a more suitable location for temporary files - the
      // one from the system may be inaccessible due to sandboxing.
      final cachebase = (await getTemporaryDirectory()).path;
      // We can't access /tmp on Android, which sqlite3 would try by default.
      // Explicitly tell it about the correct temporary directory.
      sqlite3.tempDirectory = cachebase;

      return NativeDatabase.createInBackground(file);
    },
  );
}

So I'm assuming now that the poblem comes from that helper that defines where the db will be stored...

[simolus3]

So I'm assuming now that the poblem comes from that helper that defines where the db will be stored...

Indeed, that code shouldn't run in unit tests. In these tests, we want to use in-memory databases instead of storing them on disk. The schemaAt() test utility allows returning a connection for you to use, but you need to change your database class for that to be possible:

class AppDatabase extends _$AppDatabase {
  AppDatabase([QueryExecutor e]) : super(e ?? _openConnection());

Then you can use AppDatabase(schema.newConnection()) to use the database in tests.

[farguello]

Thanks for that, it helped me a lot!
So now, after running tests I get
All tests passed!
This means that my existing app instance is in a bad state, right?

The problem is still unclear to me: why would then the inspected database in the documents folder be reflecting the correct database migration but still get that error?

I followed the migration guide word by word. I will try re-doing it.
Should I try to fix that problem if it persists or should I keep myself from debugging it if tests do pass?

[farguello]

Okay this is kind of embarrassing.
All this trouble was because I hadn't run yet build_runner, so the .g files hadn't been updated.

Thank you so much for your time and patience!