engine.Reload(): read InnoDB tables sizes including FULLTEXT index volume

[shlomi-noach]

Description

Followup to #17066, and see #17119.

The changes in #17066 solve a performance issue. We note, however, that the query TablesWithSize80 does not include the file size / allocated size for InnoDB FULLTEXT indexes.

InnoDB's FULLTEXT index isn't a BTREE, and is implemented using hidden InnoDB tables. Those are visible in information_schema.innodb_tablespaces. However, in that table they are not named after the original table. Instead, they look something like mydb/fts_000000000000075e_00000000000005f9_index_2.

The key to unlocking these table identities is in information_schema.innodb_tables, which introduces this table as a third table in the TablesWithSize80, and this introduces two problems:

• Performance: with large table counts (hundreds, 1000+) the query performs poorly.
• Correctness: innodb_table names are encoded. The matching my-db/my-table entry in innodb_tablespaces will look like my@002ddb/my@002dtable in innodb_tables. This makes it impossible to join between the three tables. (Note: MySQL has a filename charset that can translate the two. But it was made internal only and inaccessible to queries).

In between the two problems, this PR takes a completely different approach on MySQL 8.0:

• We introduce a new query which reads InnoDB tables with their sizes.
• This query joins innodb_tables and innodb_tablespaces using valid conditions (which are not the table name).
• The result is a list of InnoDB table sizes, for both external and hidden tables (in fact, the query will return two rows for a table that has FULLTEXT keys)
• The list uses the encoded InnoDB names
• On MySQL 8.0, schema.Engine reads table data without sizes, reads the InnoDB table sizes as explained, then programmatically joins the two.
• To join the two we have reimplemented the MySQL filename encoding function.
• The programmatic join makes sure to correctly handle partitioned tables
• The programmatic join makes sure to cumulate table sizes for non-FTS plus FTS on same table

On non MySQL 8.0 the behavior remains unchanged.

This PR should pass all existing tests. We also add a unit test designed for schema.Engine.Reload() that validates the operation of the new logic.

Related Issue(s)

• Fixes schema.Engine: need to improve on tables-with-sizes query #17119
• Improve Schema Engine's TablesWithSize80 query #17066
• Bug Report: OnlineDDL migrations fail when keyspace has thousands of tables #17059
• Addresses Request: re-introduce MySQL 5.7 testing for schema.Engine #17168

Checklist

• "Backport to:" labels have been added if this change should be back-ported to release branches
• If this change is to be back-ported to previous releases, a justification is included in the PR description
• Tests were added or are not required
• Did the new or modified tests pass consistently locally and on CI?
• Documentation was added or is not required

Deployment Notes

[vitess-bot]

Review Checklist

Hello reviewers! 👋 Please follow this checklist when reviewing this Pull Request.

General

• Ensure that the Pull Request has a descriptive title.
• Ensure there is a link to an issue (except for internal cleanup and flaky test fixes), new features should have an RFC that documents use cases and test cases.

Tests

• Bug fixes should have at least one unit or end-to-end test, enhancement and new features should have a sufficient number of tests.

Documentation

• Apply the release notes (needs details) label if users need to know about this change.
• New features should be documented.
• There should be some code comments as to why things are implemented the way they are.
• There should be a comment at the top of each new or modified test to explain what the test does.

New flags

• Is this flag really necessary?
• Flag names must be clear and intuitive, use dashes (-), and have a clear help text.

If a workflow is added or modified:

• Each item in Jobs should be named in order to mark it as required.
• If the workflow needs to be marked as required, the maintainer team must be notified.

Backward compatibility

• Protobuf changes should be wire-compatible.
• Changes to _vt tables and RPCs need to be backward compatible.
• RPC changes should be compatible with vitess-operator
• If a flag is removed, then it should also be removed from vitess-operator and arewefastyet, if used there.
• vtctl command output order should be stable and awk-able.

[shlomi-noach]

Working on the testing failures. We have pretty strict testing!

[shlomi-noach]

Converting to draft while working on all unit tests.

[shlomi-noach]

Good to review! All unit tests have been adapted (note: I'm impressed by just how much nitty gritty coverage they do for engine.Reload).

[codecov]

Codecov Report

Attention: Patch coverage is 75.22124% with 28 lines in your changes missing coverage. Please review.

Project coverage is 67.44%. Comparing base (4550640) to head (25949d4).
Report is 19 commits behind head on main.

Files with missing lines	Patch %	Lines
go/vt/vttablet/tabletserver/schema/engine.go	76.00%	12 Missing ⚠️
go/mysql/collations/charset/filename.go	73.33%	8 Missing ⚠️
go/mysql/flavor_mysqlgr.go	0.00%	3 Missing ⚠️
go/mysql/flavor_filepos.go	0.00%	2 Missing ⚠️
go/mysql/flavor_mariadb_binlog_playback.go	0.00%	2 Missing ⚠️
go/mysql/flavor_mysql.go	80.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #17118      +/-   ##
==========================================
+ Coverage   67.15%   67.44%   +0.28%     
==========================================
  Files        1571     1570       -1     
  Lines      252250   252285      +35     
==========================================
+ Hits       169409   170148     +739     
+ Misses      82841    82137     -704     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[dbussink]

Nice fix to ensure we have the right information here.

[arthurschreiber]

Nice! This is looking great.

[mattlord]

Nice work on this! Just had some minor nits and questions.

One larger thing that wasn't clear to me though... what role does TablesWithSize80 now play? Could/should we simplify that further so that it's not concerned with sizes at all?

[mattlord]

Shouldn't this be:

mysqlFlavor{}.baseShowInnodbTableSizes()

? I think so...

[shlomi-noach]

Just making sure I understand correctly, did you mean that filePosFlavor. baseShowInnodbTableSizes () should return the new query proposed in this PR, as opposed to an empty string? Why, then does filePosFlavor.baseShowTablesWithSizes() return a TablesWithSize56 result as opposed to TablesWithSize80?

In accordance with the rest of filePosFlavor behavior, I don't think it should be using 8.0-grade queries?

[mattlord]

I'm not sure. I don't think file/position is really tied to a version like that. We can leave it like it is.

[mattlord]

Same for this one, I think:

mysqlFlavor{}.baseShowInnodbTableSizes()

[mattlord]

mysqlgr (go/vt/vtgr) is gone (#13308), but looks like we missed removing the flavor. Are you OK removing it here? We can also do that separately.

[shlomi-noach]

Let's please do that separately!

[mattlord]

Not specific to this new work, but what does a Vitess user do if they have more than 10,000 tables? Seems like we do not support that today? I wonder if we need this limitation today (it was added 12 years ago, so while Vitess was only an internal YouTube project where exceeding that limit may have been unthinkable/impossible).

[shlomi-noach]

Excellent question. Let's take it off this thread -- there's so much code in Vitess where we limit results to 10000 -- the good news is that it's relatively easy to find!

[mattlord]

Nice! Confirmed that with the encoded names we don't have to worry about tables that actually have these special chars in them as this example is encoded as te@0023p@0023st:

mysql> create table `te#p#st` (id int);
Query OK, 0 rows affected (0.01 sec)

mysql> SELECT it.name, its.file_size as normal_tables_sum_file_size, its.allocated_size as normal_tables_sum_allocated_size FROM information_schema.innodb_tables it JOIN information_schema.innodb_tablespaces its ON (its.space = it.space) WHERE its.name LIKE CONCAT(database(), '/%') AND its.name NOT LIKE CONCAT(database(), '/fts_%') UNION ALL SELECT it.name, SUM(its.file_size) as hidden_tables_sum_file_size, SUM(its.allocated_size) as hidden_tables_sum_allocated_size FROM information_schema.innodb_tables it JOIN information_schema.innodb_tablespaces its ON (  its.name LIKE CONCAT(database(), '/fts_', CONVERT(LPAD(HEX(table_id), 16, '0') USING utf8mb3) COLLATE utf8mb3_general_ci, '_%') ) WHERE  its.name LIKE CONCAT(database(), '/fts_%') GROUP BY it.name;
+-----------------------------+-----------------------------+----------------------------------+
| name                        | normal_tables_sum_file_size | normal_tables_sum_allocated_size |
+-----------------------------+-----------------------------+----------------------------------+
| vt_commerce/product         |                      114688 |                           114688 |
| vt_commerce/customer        |                      114688 |                           114688 |
| vt_commerce/corder          |                      114688 |                           114688 |
| vt_commerce/te@0023p@0023st |                      114688 |                           114688 |
| vt_commerce/members#p#p0    |                      114688 |                           114688 |
| vt_commerce/members#p#p1    |                      114688 |                           114688 |
| vt_commerce/members#p#p2    |                      114688 |                           114688 |
| vt_commerce/members#p#p3    |                      114688 |                           114688 |
| vt_commerce/members#p#p4    |                      114688 |                           114688 |
| vt_commerce/members#p#p5    |                      114688 |                           114688 |
+-----------------------------+-----------------------------+----------------------------------+
10 rows in set, 1 warning (0.01 sec)

[shlomi-noach]

what role does TablesWithSize80 now play?

Great question. It does not play any role at all and will never be used in 8.0. I should remove it altogether!

[shlomi-noach]

Eradicated TablesWithSize80.

[shlomi-noach]

Addendum: this PR introduces a logic difference between MySQL 8.0 and legacy versions (5.7). The code path for engine.Reload() splits and issues different queries and code based on the version.

As such, we notice our unit testing fakes 8.0 only and does not cover the now separate 5.7 path. Moreover, 5.7 endtoend testing is long since remvoed, and so we end up without 5.7 tests.

25949d4 introduces the ability to set up a "legacy" test environment, and splits/duplicates some unit testing for the different environments: (

• "Legacy" tests are just copies of existing main code, but with 5.7 environment
• Other tests assume 8.0 and have been modified to accommodate this PR's changes.

See this commit for the relevant changes.

See also #17168