Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: align space_index_test with the transactional DDL #157

Merged
merged 1 commit into from
Sep 15, 2023
Merged

test: align space_index_test with the transactional DDL #157

merged 1 commit into from
Sep 15, 2023

Conversation

mkostoevr
Copy link
Contributor

@mkostoevr mkostoevr commented Sep 15, 2023

If the tarantool/tarantool#8751 is merged, the space drop is transactional, so it only yields once on the space drop operation finish. This causes the test fail [1] because of unexpected state in the expirationd task.

This patch fixes the problem by wrapping space drop and recreation into a transaction.

More informative description of what the issue is:

Consider the following facts:

  1. If the PR mentioned above is merged, the space drop yields on the full DDL operation end. After that point, the space is dropped entirely (no indexes or entries in the _space space exist).
  2. If the expirationd task's atomic_iteration option is set to false (which is the default used in the test), each tuple drop causes the expirationd task fiber yield.

So here's what happens if the fact 1 is false (the PR mentioned above is not merged):

  1. The test fiber creates the expirationd task and yields into it.
  2. The expirationd task drops the first tuple from the given space and yields back to the test fiber.
  3. The test fiber does the space:drop(), which is not atomic, so
  4. the drop yields on the primary index drop. Control is returned to the expirationd task.
  5. The expirationd task:
    1. Finishes the space iteration (there was only one tuple in the given space)
    2. Uses the space (box.space[task.space].engine == "vinyl") we wanted to drop in the test fiber.
    3. Goes sleep, which yields to the test fiber.
  6. The test fiber continues the space drop, finishes it (the next yields happening during the space drop do not return the control back to the expirationd task though, because it sleeps and it's sleep timer hasn't expired yet).
  7. The test fiber recreates the space, and finally, when it's time to switch back to the expirationd task, the space exists again like no space drop happened.

But if the space drop is atomic things change starting from the step 3: the space drop only yields on the whole DDL transaction commit. After that point, the space does not exist anymore. So:

  1. The atomic space drop yields to the expirationd task on commit.
  2. The expirationd task:
    1. Finishes the iteration and checks if the space's engine is vinyl in order to update the vinyl_assumed_space_len, but the space does not exist anymore, so it fails. This causes the expirationd task restart.

The last mentioned step increments the restart counter so the test check fails.

In order to mitigate that the patch wraps the space drop and recreation into a transaction, so the test fiber only yields when all the required changes are applied, so the expirationd task does not see the dropped space and only works with the original or recreated space.

The tranactioning is only done for Tarantool version 2.2.1 or newer because in the version the single-yield transactional DDL had been introduced.

  1. https://github.com/tarantool/tarantool/actions/runs/6186475380/job/16795081554

@mkostoevr mkostoevr marked this pull request as draft September 15, 2023 10:38
@mkostoevr mkostoevr marked this pull request as ready for review September 15, 2023 12:05
Copy link
Contributor

@oleg-jukovec oleg-jukovec left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please, add a short changelog entry here:

And update a Tarantool version for the workaround, see @olegrok comment.

test/unit/space_index_test.lua Outdated Show resolved Hide resolved
test/unit/space_index_test.lua Outdated Show resolved Hide resolved
test/unit/space_index_test.lua Outdated Show resolved Hide resolved
Copy link
Contributor

@oleg-jukovec oleg-jukovec left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the patch!

CHANGELOG.md Outdated Show resolved Hide resolved
If the tarantool/tarantool#8751 is merged, the space drop is
transactional, so it only yields once on the space drop operation
finish. This causes the test fail [1] because of unexpected state
in the expirationd task.

This patch fixes the problem by wrapping space drop and recreation
into a transaction.

More informative description of what the issue is:

Consider the following facts:
a. If the PR mentioned above is merged, the space drop yields on
   the full DDL operation end. After that point, the space is
   dropped entirely (no indexes or entries in the `_space` space
   exist).
b. If the expirationd task's `atomic_iteration` option is set to
   false (which is the default used in the test), each tuple drop
   causes the expirationd task fiber yield.

So here's what happens if the fact 'a' is false (the PR mentioned
above is not merged):
1. The test fiber creates the expirationd task and yields into it.
2. The expirationd task drops the first tuple from the given space
   and yields back to the test fiber.
3. The test fiber does the `space:drop()`, which is not atomic, so
4. the drop yields on the primary index drop. Control is returned
   to the expirationd task.
5. The expirationd task:
   1. Finishes the space iteration (there was only one tuple in the
      given space)
   2. Uses the space (`box.space[task.space].engine == "vinyl"`) we
      wanted to drop in the test fiber.
   3. Goes sleep, which yields to the test fiber.
6. The test fiber continues the space drop, finishes it (the next
   yields happening during the space drop do not return the control
   back to the expirationd task though, because it sleeps and it's
   sleep timer hasn't expired yet).
7. The test fiber recreates the space, and finally, when it's time
   to switch back to the expirationd task, the space exists again
   like no space drop happened.

But if the space drop is atomic things change starting from the
step 3: the space drop only yields on the whole DDL transaction
commit. After that point, the space does not exist anymore. So:

4. The atomic space drop yields to the expirationd task on commit.
5. The expirationd task:
   1. Finishes the iteration and checks if the space's engine is
      vinyl in order to update the vinyl_assumed_space_len, but
      the space does not exist anymore, so it fails. This causes
      the expirationd task restart.

The last mentioned step increments the restart counter so the test
check fails.

In order to mitigate that the patch wraps the space drop and
recreation into a transaction, so the test fiber only yields
when all the required changes are applied, so the expirationd
task does not see the dropped space and only works with the
original or recreated space.

The tranactioning is only done for Tarantool version 2.2.1 or
newer because in the version the single-yield transactional DDL
had been introduced.

1. https://github.com/tarantool/tarantool/actions/runs/6186475380/job/16795081554
@oleg-jukovec oleg-jukovec merged commit ca8a7a6 into tarantool:master Sep 15, 2023
76 checks passed
oleg-jukovec added a commit that referenced this pull request Mar 25, 2024
Overview

    The release introduces a role for Tarantool 3.0.

Breaking changes

    None.

New features

    Tarantool 3.0 role for expirationd (#160).

Testing

    Updated the 'space_index_test.lua' to drop and recreate the
    test space atomically. This prevents the space access failure
    in the expirationd task fiber if the `space:drop` function is
    transactional (#157).

    Updated version of `luatest` in `make deps` to 1.0.1 to support
    Tarantool 3.0 role tests (#160).
@oleg-jukovec oleg-jukovec mentioned this pull request Mar 25, 2024
oleg-jukovec added a commit that referenced this pull request Mar 25, 2024
Overview

    The release introduces a role for Tarantool 3.0.

Breaking changes

    None.

New features

    Tarantool 3.0 role (#160).

Testing

    Updated the 'space_index_test.lua' to drop and recreate the
    test space atomically. This prevents the space access failure
    in the expirationd task fiber if the `space:drop` function is
    transactional (#157).

    Updated version of `luatest` in `make deps` to 1.0.1 to support
    Tarantool 3.0 role tests (#160).
oleg-jukovec added a commit that referenced this pull request Mar 25, 2024
Overview

    The release introduces a role for Tarantool 3.0.

Breaking changes

    None.

New features

    Tarantool 3.0 role (#160).

Testing

    Updated the 'space_index_test.lua' to drop and recreate the
    test space atomically. This prevents the space access failure
    in the expirationd task fiber if the `space:drop` function is
    transactional (#157).

    Updated version of `luatest` in `make deps` to 1.0.1 to support
    Tarantool 3.0 role tests (#160).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants