Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Batch Deletes #447

Merged
merged 25 commits into from
Nov 1, 2024
Merged
Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
170f8d5
Initial Push for Batch Deletes
PineappleIOnic Sep 10, 2024
79e65da
Update src/Database/Adapter/MariaDB.php
PineappleIOnic Sep 10, 2024
b079870
Address comments, add MongoDB and Postgres Support
PineappleIOnic Sep 17, 2024
a931aab
Make event test the last test
PineappleIOnic Sep 17, 2024
4f3e91e
Run Linter
PineappleIOnic Sep 17, 2024
0fe42d5
Merge branch 'main' into feat-batch-delete
PineappleIOnic Sep 17, 2024
c3115b5
Update Authorisation Validator
PineappleIOnic Sep 17, 2024
04d8015
Use ID based deletion, add more tests
PineappleIOnic Oct 1, 2024
87a6e3a
Linter and CodeQL
PineappleIOnic Oct 1, 2024
22abde7
CodeQL
PineappleIOnic Oct 1, 2024
19e6029
Continue to add more tests
PineappleIOnic Oct 2, 2024
d3264dc
Fix Cascade
PineappleIOnic Oct 2, 2024
1241b0a
Run Linter
PineappleIOnic Oct 2, 2024
e6eae09
Remove unnecessary cleanup
PineappleIOnic Oct 4, 2024
f1a9eb5
Disable debug in compose file
PineappleIOnic Oct 4, 2024
dbe77e9
Add more tests
PineappleIOnic Oct 10, 2024
c75b4bf
Remove debug flag in compose file
PineappleIOnic Oct 10, 2024
4121f7f
Run Linter
PineappleIOnic Oct 10, 2024
66e0736
Use new event for bulk delete
PineappleIOnic Oct 16, 2024
d34b44d
Address comments
PineappleIOnic Oct 18, 2024
afd0d9c
Run Linter
PineappleIOnic Oct 18, 2024
a65da29
Fix mongoDB attempting to delete an empty array of documents
PineappleIOnic Oct 28, 2024
b3cab3b
Move empty affected document check to database.php
PineappleIOnic Oct 28, 2024
a1a5f53
Address comments
PineappleIOnic Nov 1, 2024
82eecce
Merge branch 'main' into feat-batch-delete
PineappleIOnic Nov 1, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions src/Database/Adapter.php
Original file line number Diff line number Diff line change
Expand Up @@ -617,6 +617,16 @@ abstract public function updateDocuments(string $collection, array $documents, i
*/
abstract public function deleteDocument(string $collection, string $id): bool;

/**
* Delete Documents
*
* @param string $collection
* @param array<\Utopia\Database\Query> $queries
*
* @return bool
*/
abstract public function deleteDocuments(string $collection, array $queries): bool;

/**
* Find Documents
*
Expand Down
51 changes: 51 additions & 0 deletions src/Database/Adapter/MariaDB.php
Original file line number Diff line number Diff line change
Expand Up @@ -1632,6 +1632,57 @@ public function deleteDocument(string $collection, string $id): bool
return $deleted;
}

/**
* Delete Documents
*
* @param string $collection
* @param array<\Utopia\Database\Query> $queries
*
* @return bool
*/
public function deleteDocuments(string $collection, array $queries): bool
{
$name = $this->filter($collection);
$where = [];

$queries = array_map(fn ($query) => clone $query, $queries);

$conditions = $this->getSQLConditions($queries);
if (!empty($conditions)) {
$where[] = $conditions;
}

if ($this->sharedTables) {
$where[] = "table_main._tenant = :_tenant";
}

$sqlWhere = !empty($where) ? 'WHERE ' . implode(' AND ', $where) : '';

$sql = "
DELETE FROM {$this->getSQLTable($name)}
{$sqlWhere};
";

$sql = $this->trigger(Database::EVENT_DOCUMENT_DELETE, $sql);

$stmt = $this->getPDO()->prepare($sql);

foreach ($queries as $query) {
$this->bindConditionValue($stmt, $query);
}
if ($this->sharedTables) {
$stmt->bindValue(':_tenant', $this->tenant);
}

try {
$stmt->execute();
} catch (PDOException $e) {
$this->processException($e);
}

return true;
abnegate marked this conversation as resolved.
Show resolved Hide resolved
}

/**
* Find Documents
*
Expand Down
38 changes: 38 additions & 0 deletions src/Database/Adapter/Mongo.php
Original file line number Diff line number Diff line change
Expand Up @@ -908,6 +908,44 @@ public function deleteDocument(string $collection, string $id): bool
return (!!$result);
}

/**
* Delete Documents
*
* @param string $collection
* @param array<\Utopia\Database\Query> $queries
*
* @return bool
*/
public function deleteDocuments(string $collection, array $queries): bool
{
$name = $this->getNamespace() . '_' . $this->filter($collection);
$queries = array_map(fn ($query) => clone $query, $queries);

$filters = $this->buildFilters($queries);

if ($this->sharedTables) {
$filters['_tenant'] = (string)$this->getTenant();
}

$filters = $this->replaceInternalIdsKeys($filters, '$', '_', $this->operators);
$filters = $this->timeFilter($filters);

$options = [];

try {
$res = $this->client->delete(
abnegate marked this conversation as resolved.
Show resolved Hide resolved
collection: $name,
filters: $filters,
options: $options,
limit: 0
);
} catch (MongoException $e) {
$this->processException($e);
}

return true;
}

/**
* Update Attribute.
* @param string $collection
Expand Down
52 changes: 52 additions & 0 deletions src/Database/Adapter/Postgres.php
Original file line number Diff line number Diff line change
Expand Up @@ -1556,6 +1556,58 @@ public function deleteDocument(string $collection, string $id): bool
return $deleted;
}


/**
* Delete Documents
*
* @param string $collection
* @param array<\Utopia\Database\Query> $queries
*
* @return bool
*/
public function deleteDocuments(string $collection, array $queries): bool
{
$name = $this->filter($collection);
$where = [];

$queries = array_map(fn ($query) => clone $query, $queries);

$conditions = $this->getSQLConditions($queries);
if (!empty($conditions)) {
$where[] = $conditions;
}

if ($this->sharedTables) {
$where[] = "table_main._tenant = :_tenant";
}

$sqlWhere = !empty($where) ? 'WHERE ' . implode(' AND ', $where) : '';

$sql = "
DELETE FROM {$this->getSQLTable($name)}
{$sqlWhere}
";

$sql = $this->trigger(Database::EVENT_DOCUMENT_DELETE, $sql);

$stmt = $this->getPDO()->prepare($sql);

foreach ($queries as $query) {
$this->bindConditionValue($stmt, $query);
}
if ($this->sharedTables) {
$stmt->bindValue(':_tenant', $this->tenant);
}

try {
$stmt->execute();
} catch (PDOException $e) {
$this->processException($e);
}

return true;
}

/**
* Find Documents
*
Expand Down
79 changes: 79 additions & 0 deletions src/Database/Database.php
Original file line number Diff line number Diff line change
Expand Up @@ -4970,6 +4970,85 @@ private function deleteCascade(Document $collection, Document $relatedCollection
}
}

/**
* Delete Documents
*
* Deletes all documents which match the given query, will respect the relationship's onDelete optin.
*
* @param string $collection
* @param array<Query> $queries
* @param int $batchSize
*
* @return bool
*
* @throws AuthorizationException
* @throws DatabaseException
* @throws RestrictedException
*/
public function deleteDocuments(string $collection, array $queries = [], int $batchSize = 100): bool
PineappleIOnic marked this conversation as resolved.
Show resolved Hide resolved
{
if ($this->adapter->getSharedTables() && empty($this->adapter->getTenant())) {
throw new DatabaseException('Missing tenant. Tenant must be set when table sharing is enabled.');
}

$queries = Query::groupByType($queries)['filters'];
$collection = $this->silent(fn () => $this->getCollection($collection));

$deleted = $this->withTransaction(function () use ($collection, $queries, $batchSize) {
$lastDocument = null;
while (true) {
$affectedDocuments = $this->find($collection->getId(), array_merge(
PineappleIOnic marked this conversation as resolved.
Show resolved Hide resolved
empty($lastDocument) ? [
Query::limit($batchSize),
] : [
Query::limit($batchSize),
Query::cursorAfter($lastDocument),
abnegate marked this conversation as resolved.
Show resolved Hide resolved
],
$queries,
));

if (empty($affectedDocuments)) {
break;
}

foreach ($affectedDocuments as $document) {
PineappleIOnic marked this conversation as resolved.
Show resolved Hide resolved
if ($collection->getId() !== self::METADATA) {
$documentSecurity = $collection->getAttribute('documentSecurity', false);
$isValid = $this->authorization->isValid(new Input(self::PERMISSION_DELETE, [
...$collection->getDelete(),
...($documentSecurity ? $document->getDelete() : [])
]));
if (!$isValid) {
throw new AuthorizationException($this->authorization->getDescription());
}
PineappleIOnic marked this conversation as resolved.
Show resolved Hide resolved
}

// Delete Relationships
if ($this->resolveRelationships) {
$document = $this->silent(fn () => $this->deleteDocumentRelationships($collection, $document));
}

// Fire events
$this->trigger(self::EVENT_DOCUMENT_DELETE, $document);

$this->purgeRelatedDocuments($collection, $document->getId());
$this->purgeCachedDocument($collection->getId(), $document->getId());
}

if (count($affectedDocuments) < 100) {
PineappleIOnic marked this conversation as resolved.
Show resolved Hide resolved
break;
} else {
$lastDocument = end($affectedDocuments);
}
}

// Mass delete using adapter with query
return $this->adapter->deleteDocuments($collection->getId(), $queries);
PineappleIOnic marked this conversation as resolved.
Show resolved Hide resolved
});

return $deleted;
}

/**
* Cleans the all the collection's documents from the cache
* And the all related cached documents.
Expand Down
Loading
Loading