diff --git a/docs/BindingsOverview.md b/docs/BindingsOverview.md index 43667987..2dd88ed5 100644 --- a/docs/BindingsOverview.md +++ b/docs/BindingsOverview.md @@ -22,6 +22,7 @@ - [Startup retries](#startup-retries) - [Broken connection retries](#broken-connection-retries) - [Function exception retries](#function-exception-retries) + - [Lease Tables clean up](#lease-tables-clean-up) ## Input Binding @@ -147,7 +148,7 @@ If an exception occurs in the user function when processing changes then the bat If the function execution fails 5 times in a row for a given row then that row is completely ignored for all future changes. Because the rows in a batch are not deterministic, rows in a failed batch may end up in different batches in subsequent invocations. This means that not all rows in the failed batch will necessarily be ignored. If other rows in the batch were the ones causing the exception, the "good" rows may end up in a different batch that doesn't fail in future invocations. -You can run this query to see what rows have failed 5 times and are currently ignored, see [Leases table](#az_funcleases_) documentation for how to get the correct Leases table to query for your function. +You can run this query to see what rows have failed 5 times and are currently ignored, see [Leases table](./TriggerBinding.md#az_funcleases_) documentation for how to get the correct Leases table to query for your function. ```sql SELECT * FROM [az_func].[Leases__] WHERE _az_func_AttemptCount = 5 @@ -168,3 +169,101 @@ e.g. ```sql UPDATE [Products].[az_func].[Leases__] SET _az_func_AttemptCount = 0 WHERE ProductId = 123 ``` + +#### Lease Tables clean up + +Before clean up, please see [Leases table](./TriggerBinding.md#az_funcleases_) documentation for understanding how they are created and used. + +Why clean up? +1. You renamed your function/class/method name, which causes a new lease table to be created and the old one to be obsolete. +2. You created a trigger function that you no longer need and wish to clean up its associated data. +3. You want to reset your environment. +The Azure SQL Trigger does not currently handle automatically cleaning up any leftover objects, and so we have provided the below scripts to help guide you through doing that. + +- Delete all the lease tables that haven't been accessed in `@CleanupAgeDays` days: + +```sql +-- Deletes all the lease tables that haven't been accessed in @CleanupAgeDays days (set below) +-- and removes them from the GlobalState table. +USE [] +GO +DECLARE @TableName NVARCHAR(MAX); +DECLARE @UserFunctionId char(16); +DECLARE @UserTableId int; +DECLARE @CleanupAgeDays int = ; +DECLARE LeaseTable_Cursor CURSOR FOR + +SELECT 'az_func.Leases_'+UserFunctionId+'_'+convert(varchar(100),UserTableID) as TABLE_NAME, UserFunctionID, UserTableID +FROM az_func.GlobalState +WHERE DATEDIFF(day, LastAccessTime, GETDATE()) > @CleanupAgeDays + +OPEN LeaseTable_Cursor; + +FETCH NEXT FROM LeaseTable_Cursor INTO @TableName, @UserFunctionId, @UserTableId; + +WHILE @@FETCH_STATUS = 0 +BEGIN + PRINT N'Dropping table ' + @TableName; + EXEC ('DROP TABLE IF EXISTS ' + @TableName); + PRINT 'Removing row from GlobalState for UserFunctionID = ' + RTRIM(CAST(@UserFunctionId AS NVARCHAR(30))) + ' and UserTableID = ' + RTRIM(CAST(@UserTableId AS NVARCHAR(30))); + DELETE FROM az_func.GlobalState WHERE UserFunctionID = @UserFunctionId and UserTableID = @UserTableId + FETCH NEXT FROM LeaseTable_Cursor INTO @TableName, @UserFunctionId, @UserTableId; +END; + +CLOSE LeaseTable_Cursor; + +DEALLOCATE LeaseTable_Cursor; +``` + +- Clean up a specific lease table: + +To find the name of the lease table associated with your function, look in the log output for a line such as this which is emitted when the trigger is started. + +`SQL trigger Leases table: [az_func].[Leases_84d975fca0f7441a_901578250]` + +This log message is at the `Information` level, so make sure your log level is set correctly. + +```sql +-- Deletes the specified lease table and removes it from GlobalState table. +USE [] +GO +DECLARE @TableName NVARCHAR(MAX) = ; -- e.g. '[az_func].[Leases_84d975fca0f7441a_901578250] +DECLARE @UserFunctionId char(16) = ; -- e.g. '84d975fca0f7441a' the first section of the lease table name [Leases_84d975fca0f7441a_901578250]. +DECLARE @UserTableId int = ; -- e.g. '901578250' the second section of the lease table name [Leases_84d975fca0f7441a_901578250]. +PRINT N'Dropping table ' + @TableName; +EXEC ('DROP TABLE IF EXISTS ' + @TableName); +PRINT 'Removing row from GlobalState for UserFunctionID = ' + RTRIM(CAST(@UserFunctionId AS NVARCHAR(30))) + ' and UserTableID = ' + RTRIM(CAST(@UserTableId AS NVARCHAR(30))); +DELETE FROM az_func.GlobalState WHERE UserFunctionID = @UserFunctionId and UserTableID = @UserTableId +``` + +- Clear all trigger related data for a reset: + +```sql +-- Deletes all the lease tables and clears them from the GlobalState table. +USE [] +GO +DECLARE @TableName NVARCHAR(MAX); +DECLARE @UserFunctionId char(16); +DECLARE @UserTableId int; +DECLARE LeaseTable_Cursor CURSOR FOR + +SELECT 'az_func.Leases_'+UserFunctionId+'_'+convert(varchar(100),UserTableID) as TABLE_NAME, UserFunctionID, UserTableID +FROM az_func.GlobalState + +OPEN LeaseTable_Cursor; + +FETCH NEXT FROM LeaseTable_Cursor INTO @TableName, @UserFunctionId, @UserTableId; + +WHILE @@FETCH_STATUS = 0 +BEGIN + PRINT N'Dropping table ' + @TableName; + EXEC ('DROP TABLE IF EXISTS ' + @TableName); + PRINT 'Removing row from GlobalState for UserFunctionID = ' + RTRIM(CAST(@UserFunctionId AS NVARCHAR(30))) + ' and UserTableID = ' + RTRIM(CAST(@UserTableId AS NVARCHAR(30))); + DELETE FROM az_func.GlobalState WHERE UserFunctionID = @UserFunctionId and UserTableID = @UserTableId + FETCH NEXT FROM LeaseTable_Cursor INTO @TableName, @UserFunctionId, @UserTableId; +END; + +CLOSE LeaseTable_Cursor; + +DEALLOCATE LeaseTable_Cursor; +``` \ No newline at end of file diff --git a/src/TriggerBinding/SqlTableChangeMonitor.cs b/src/TriggerBinding/SqlTableChangeMonitor.cs index 0af9b36c..0ff9c153 100644 --- a/src/TriggerBinding/SqlTableChangeMonitor.cs +++ b/src/TriggerBinding/SqlTableChangeMonitor.cs @@ -955,7 +955,7 @@ LEFT OUTER JOIN {this._leasesTableName} AS l ON {leasesTableJoinCondition} IF @unprocessed_changes = 0 AND @current_last_sync_version < {newLastSyncVersion} BEGIN UPDATE {GlobalStateTableName} - SET LastSyncVersion = {newLastSyncVersion} + SET LastSyncVersion = {newLastSyncVersion}, LastAccessTime = GETUTCDATE() WHERE UserFunctionID = '{this._userFunctionId}' AND UserTableID = {this._userTableId}; DELETE FROM {this._leasesTableName} WHERE {LeasesTableChangeVersionColumnName} <= {newLastSyncVersion}; diff --git a/src/TriggerBinding/SqlTriggerListener.cs b/src/TriggerBinding/SqlTriggerListener.cs index 411dd766..83a6a7ac 100644 --- a/src/TriggerBinding/SqlTriggerListener.cs +++ b/src/TriggerBinding/SqlTriggerListener.cs @@ -321,8 +321,12 @@ IF OBJECT_ID(N'{GlobalStateTableName}', 'U') IS NULL UserFunctionID char(16) NOT NULL, UserTableID int NOT NULL, LastSyncVersion bigint NOT NULL, + LastAccessTime Datetime NOT NULL DEFAULT GETUTCDATE(), PRIMARY KEY (UserFunctionID, UserTableID) ); + ELSE IF NOT EXISTS(SELECT 1 FROM sys.columns WHERE Name = N'LastAccessTime' + AND Object_ID = Object_ID(N'{GlobalStateTableName}')) + ALTER TABLE {GlobalStateTableName} ADD LastAccessTime Datetime NOT NULL DEFAULT GETUTCDATE(); "; using (var createGlobalStateTableCommand = new SqlCommand(createGlobalStateTableQuery, connection, transaction)) @@ -390,7 +394,7 @@ IF NOT EXISTS ( WHERE UserFunctionID = '{this._userFunctionId}' AND UserTableID = {userTableId} ) INSERT INTO {GlobalStateTableName} - VALUES ('{this._userFunctionId}', {userTableId}, {(long)minValidVersion}); + VALUES ('{this._userFunctionId}', {userTableId}, {(long)minValidVersion}, GETUTCDATE()); "; using (var insertRowGlobalStateTableCommand = new SqlCommand(insertRowGlobalStateTableQuery, connection, transaction))