Merge pull request #8781 from RusJaI/master

Add Missing registry cleanup DB scripts
wso2 · Nov 11, 2024 · 4b81025 · 4b81025
2 parents e23fcfe + f6cebcc
commit 4b81025
Showing 1 changed file with 265 additions and 0 deletions.
diff --git a/...up/setup/setting-up-databases/managing-data-growth-and-improving-performance.md b/...up/setup/setting-up-databases/managing-data-growth-and-improving-performance.md
@@ -129,3 +129,268 @@ This will remove the old and invalid tokens, sessions, and Auth codes, which can
     ```
 
 Replace `WSO2AM_DB` with the name of your WSO2 API Manager database in the above script.
+
+## Registry database cleanup
+
+Creating and updating APIs and adding tags and ratings cause different registry properties to be added to the Registry database (Registry DB) in WSO2 API-M. When `Registry Versioning` is enabled, older properties are persisted in the database leading to unnecessary record growth in the Registry tables in the database. This directly affects the performance of the product if the number of records increase beyond a certain amount. Due to this limitation, registry versioning is disabled by default in WSO2 API Manager 3.0.0 onwards to prevent unnecessary database growth.
+
+In WSO2 API-M versions preceding 3.0.0 (2.x.x and 1.x.x), `Registry Versioning` is enabled by default, therefore you need to cleanup the Registry DB if there are large tables with millions of records.
+
+You can cleanup the following tables in the Registry DB without affecting the functionality of the product.
+-   REG_LOG
+-   REG_PROPERTY
+-   REG_RESOURCE_PROPERTY
+-   REG_TAG
+-   REG_RESOURCE_TAG
+
+!!! info
+    You can always take a backup of the Registry database before the cleanup as a precaution for any issues that might occur during cleanup.
+
+!!! note "Before you begin..."
+    Remember to shut down the server and stop any DB transactions before the database cleanup. This is to prevent any inconsistencies that could happen with new data that gets added to the DB during the cleanup.
+
+The Cleanup scripts for the Registry DB tables are given below.
+
+#### REG_LOG Table Cleanup
+
+!!! example "DB Types: H2, DB2, MySQL, MSSQL, Oracle and Postgresql."
+    ``` sql
+    CREATE TABLE REG_LOG_IDS_TO_KEEP (
+             REG_LOG_ID INTEGER,
+             REG_TENANT_ID INTEGER
+    );
+    
+    INSERT INTO REG_LOG_IDS_TO_KEEP (REG_LOG_ID, REG_TENANT_ID)
+    SELECT MAX(REG_LOG_ID) AS REG_LOG_ID, REG_TENANT_ID FROM REG_LOG GROUP BY REG_PATH, REG_TENANT_ID;
+    
+    DELETE FROM REG_LOG WHERE REG_LOG_ID NOT IN (SELECT REG_LOG_ID FROM REG_LOG_IDS_TO_KEEP);
+    DROP TABLE REG_LOG_IDS_TO_KEEP;
+    
+    DELETE FROM REG_LOG WHERE REG_ACTION = 7;
+    ```
+
+
+
+#### REG_PROPERTY and REG_RESOURCE_PROPERTY Table Cleanup
+
+!!! example "DB types: H2, MSSQL and Postgresql"
+
+    ``` sql
+    CREATE TABLE TEMP_REG_RESOURCE_PROPERTY_ID(REG_PROPERTY_ID INTEGER);
+    -- Extract resource property (ID) created when versioning is disabled --
+    INSERT INTO TEMP_REG_RESOURCE_PROPERTY_ID(REG_PROPERTY_ID)
+    SELECT REG_PROPERTY_ID
+    FROM REG_RESOURCE_PROPERTY
+    WHERE REG_PATH_ID IN
+        (SELECT REG_PATH_ID
+            FROM REG_RESOURCE);
+    -- Extract resource property (ID) created when versioning is enabled --
+    INSERT INTO TEMP_REG_RESOURCE_PROPERTY_ID(REG_PROPERTY_ID)
+    SELECT REG_PROPERTY_ID
+    FROM REG_RESOURCE_PROPERTY
+    WHERE REG_VERSION IN
+        (SELECT REG_VERSION
+            FROM REG_RESOURCE);
+    -- Drop the foreign key constraint --
+    ALTER TABLE REG_RESOURCE_PROPERTY DROP CONSTRAINT REG_RESOURCE_PROPERTY_FK_BY_TAG_ID;
+    -- delete all unwanted REG_RESOURCE_PROPERTY entries --
+    DELETE
+    FROM REG_RESOURCE_PROPERTY
+    WHERE REG_PROPERTY_ID NOT IN
+        (SELECT REG_PROPERTY_ID
+        FROM TEMP_REG_RESOURCE_PROPERTY_ID);
+    -- delete all unwanted REG_PROPERTY entries --
+    DELETE
+    FROM REG_PROPERTY
+    WHERE REG_ID NOT IN
+        (SELECT REG_PROPERTY_ID
+        FROM TEMP_REG_RESOURCE_PROPERTY_ID);
+    -- Insert back the foreign key constraint --
+    ALTER TABLE REG_RESOURCE_PROPERTY ADD CONSTRAINT REG_RESOURCE_PROPERTY_FK_BY_TAG_ID FOREIGN KEY (REG_PROPERTY_ID, REG_TENANT_ID) REFERENCES REG_PROPERTY (REG_ID, REG_TENANT_ID);
+    -- drop temporary table --
+    DROP TABLE TEMP_REG_RESOURCE_PROPERTY_ID;
+    ```
+
+!!! example "DB types: MySQL"
+
+    ``` sql
+    CREATE TABLE TEMP_REG_RESOURCE_PROPERTY_ID(REG_PROPERTY_ID INTEGER);
+    -- Extract resource property (ID) created when versioning is disabled --
+    INSERT INTO TEMP_REG_RESOURCE_PROPERTY_ID(REG_PROPERTY_ID)
+    SELECT REG_PROPERTY_ID
+    FROM REG_RESOURCE_PROPERTY
+    WHERE REG_PATH_ID IN
+        (SELECT REG_PATH_ID
+            FROM REG_RESOURCE);
+    -- Extract resource property (ID) created when versioning is enabled --
+    INSERT INTO TEMP_REG_RESOURCE_PROPERTY_ID(REG_PROPERTY_ID)
+    SELECT REG_PROPERTY_ID
+    FROM REG_RESOURCE_PROPERTY
+    WHERE REG_VERSION IN
+        (SELECT REG_VERSION
+            FROM REG_RESOURCE);
+    -- Drop the foreign key constraint --
+    ALTER TABLE REG_RESOURCE_PROPERTY DROP FOREIGN KEY REG_RESOURCE_PROPERTY_FK_BY_TAG_ID;
+    -- delete all unwanted REG_RESOURCE_PROPERTY entries --
+    DELETE
+    FROM REG_RESOURCE_PROPERTY
+    WHERE REG_PROPERTY_ID NOT IN
+        (SELECT REG_PROPERTY_ID
+        FROM TEMP_REG_RESOURCE_PROPERTY_ID);
+    -- delete all unwanted REG_PROPERTY entries --
+    DELETE
+    FROM REG_PROPERTY
+    WHERE REG_ID NOT IN
+        (SELECT REG_PROPERTY_ID
+        FROM TEMP_REG_RESOURCE_PROPERTY_ID);
+    -- Insert back the foreign key constraint --
+    ALTER TABLE REG_RESOURCE_PROPERTY ADD CONSTRAINT REG_RESOURCE_PROPERTY_FK_BY_TAG_ID FOREIGN KEY (REG_PROPERTY_ID, REG_TENANT_ID) REFERENCES REG_PROPERTY (REG_ID, REG_TENANT_ID);
+    -- drop temporary table --
+    DROP TABLE TEMP_REG_RESOURCE_PROPERTY_ID;
+    ```
+
+!!! example "DB types: DB2 and Oracle"
+
+    ``` sql
+    CREATE TABLE TEMP_REG_RESOURCE_PROPERTY_ID(REG_PROPERTY_ID INTEGER);
+    -- Extract resource property (ID) created when versioning is disabled --
+    INSERT INTO TEMP_REG_RESOURCE_PROPERTY_ID(REG_PROPERTY_ID)
+    SELECT REG_PROPERTY_ID
+    FROM REG_RESOURCE_PROPERTY
+    WHERE REG_PATH_ID IN
+        (SELECT REG_PATH_ID
+            FROM REG_RESOURCE);
+    -- Extract resource property (ID) created when versioning is enabled --
+    INSERT INTO TEMP_REG_RESOURCE_PROPERTY_ID(REG_PROPERTY_ID)
+    SELECT REG_PROPERTY_ID
+    FROM REG_RESOURCE_PROPERTY
+    WHERE REG_VERSION IN
+        (SELECT REG_VERSION
+            FROM REG_RESOURCE);
+    -- delete all unwanted REG_RESOURCE_PROPERTY entries --
+    DELETE
+    FROM REG_RESOURCE_PROPERTY
+    WHERE REG_PROPERTY_ID NOT IN
+        (SELECT REG_PROPERTY_ID
+        FROM TEMP_REG_RESOURCE_PROPERTY_ID);
+    -- delete all unwanted REG_PROPERTY entries --
+    DELETE
+    FROM REG_PROPERTY
+    WHERE REG_ID NOT IN
+        (SELECT REG_PROPERTY_ID
+        FROM TEMP_REG_RESOURCE_PROPERTY_ID);
+    -- drop temporary table --
+    DROP TABLE TEMP_REG_RESOURCE_PROPERTY_ID;
+    ```
+
+#### REG_TAG and REG_RESOURCE_TAG Table Cleanup
+
+!!! example "DB types: H2, MSSQL and Postgresql"
+
+    ``` sql
+    CREATE TABLE TEMP_REG_RESOURCE_TAG_ID(REG_TAG_ID INTEGER);
+    -- Extract resource tag (ID) created when versioning is disabled --
+    INSERT INTO TEMP_REG_RESOURCE_TAG_ID(REG_TAG_ID)
+    SELECT REG_TAG_ID
+    FROM REG_RESOURCE_TAG
+    WHERE REG_PATH_ID IN
+        (SELECT REG_PATH_ID
+            FROM REG_RESOURCE);
+    -- Extract resource tag (ID) created when versioning is enabled --
+    INSERT INTO TEMP_REG_RESOURCE_TAG_ID(REG_TAG_ID)
+    SELECT REG_TAG_ID
+    FROM REG_RESOURCE_TAG
+    WHERE REG_VERSION IN
+        (SELECT REG_VERSION
+            FROM REG_RESOURCE);
+    -- Remove the foreign key constraint --
+    ALTER TABLE REG_RESOURCE_TAG DROP CONSTRAINT REG_RESOURCE_TAG_FK_BY_TAG_ID;
+    -- delete all unwanted REG_RESOURCE_TAG entries --
+    DELETE
+    FROM REG_RESOURCE_TAG
+    WHERE REG_TAG_ID NOT IN
+        (SELECT REG_TAG_ID
+        FROM TEMP_REG_RESOURCE_TAG_ID);
+    -- delete all unwanted REG_TAG entries --
+    DELETE
+    FROM REG_TAG
+    WHERE REG_ID NOT IN
+        (SELECT REG_TAG_ID
+        FROM TEMP_REG_RESOURCE_TAG_ID);
+    -- add the foreign key constraint back --
+    ALTER TABLE REG_RESOURCE_TAG ADD CONSTRAINT REG_RESOURCE_TAG_FK_BY_TAG_ID FOREIGN KEY (REG_TAG_ID, REG_TENANT_ID) REFERENCES REG_TAG (REG_ID, REG_TENANT_ID);
+    -- drop temporary table --
+    DROP TABLE TEMP_REG_RESOURCE_TAG_ID;
+    ```
+
+!!! example "DB types: MySQL"
+
+    ``` sql
+    CREATE TABLE TEMP_REG_RESOURCE_TAG_ID(REG_TAG_ID INTEGER);
+    -- Extract resource tag (ID) created when versioning is disabled --
+    INSERT INTO TEMP_REG_RESOURCE_TAG_ID(REG_TAG_ID)
+    SELECT REG_TAG_ID
+    FROM REG_RESOURCE_TAG
+    WHERE REG_PATH_ID IN
+        (SELECT REG_PATH_ID
+            FROM REG_RESOURCE);
+    -- Extract resource tag (ID) created when versioning is enabled --
+    INSERT INTO TEMP_REG_RESOURCE_TAG_ID(REG_TAG_ID)
+    SELECT REG_TAG_ID
+    FROM REG_RESOURCE_TAG
+    WHERE REG_VERSION IN
+        (SELECT REG_VERSION
+            FROM REG_RESOURCE);
+    -- Remove the foreign key constraint --
+    ALTER TABLE REG_RESOURCE_TAG DROP FOREIGN KEY REG_RESOURCE_TAG_FK_BY_TAG_ID;
+    -- delete all unwanted REG_RESOURCE_TAG entries --
+    DELETE
+    FROM REG_RESOURCE_TAG
+    WHERE REG_TAG_ID NOT IN
+        (SELECT REG_TAG_ID
+        FROM TEMP_REG_RESOURCE_TAG_ID);
+    -- delete all unwanted REG_TAG entries --
+    DELETE
+    FROM REG_TAG
+    WHERE REG_ID NOT IN
+        (SELECT REG_TAG_ID
+        FROM TEMP_REG_RESOURCE_TAG_ID);
+    -- add the foreign key constraint back --
+    ALTER TABLE REG_RESOURCE_TAG ADD CONSTRAINT REG_RESOURCE_TAG_FK_BY_TAG_ID FOREIGN KEY (REG_TAG_ID, REG_TENANT_ID) REFERENCES REG_TAG (REG_ID, REG_TENANT_ID);
+    -- drop temporary table --
+    DROP TABLE TEMP_REG_RESOURCE_TAG_ID;
+    ```
+
+!!! example "DB types: DB2 and Oracle"
+
+    ``` sql
+    CREATE TABLE TEMP_REG_RESOURCE_TAG_ID(REG_TAG_ID INTEGER);
+    -- Extract resource tag (ID) created when versioning is disabled --
+    INSERT INTO TEMP_REG_RESOURCE_TAG_ID(REG_TAG_ID)
+    SELECT REG_TAG_ID
+    FROM REG_RESOURCE_TAG
+    WHERE REG_PATH_ID IN
+        (SELECT REG_PATH_ID
+            FROM REG_RESOURCE);
+    -- Extract resource tag (ID) created when versioning is enabled --
+    INSERT INTO TEMP_REG_RESOURCE_TAG_ID(REG_TAG_ID)
+    SELECT REG_TAG_ID
+    FROM REG_RESOURCE_TAG
+    WHERE REG_VERSION IN
+        (SELECT REG_VERSION
+            FROM REG_RESOURCE);
+    -- delete all unwanted REG_RESOURCE_TAG entries --
+    DELETE
+    FROM REG_RESOURCE_TAG
+    WHERE REG_TAG_ID NOT IN
+        (SELECT REG_TAG_ID
+        FROM TEMP_REG_RESOURCE_TAG_ID);
+    -- delete all unwanted REG_TAG entries --
+    DELETE
+    FROM REG_TAG
+    WHERE REG_ID NOT IN
+        (SELECT REG_TAG_ID
+        FROM TEMP_REG_RESOURCE_TAG_ID);
+    -- drop temporary table --
+    DROP TABLE TEMP_REG_RESOURCE_TAG_ID;
+    ```