[FEATURE REQ] Filter version sets / Keep only specific version set during extractor pipeline

[JamesProant]

Please describe the feature.

Hello,

My team and I have been working on a way to filter the version sets extracted from the portal. At the moment the extractor pipeline extracts ALL version sets. (I have read from the developers that this is done on purpose due to: "Version sets (can) have a 1 to many relationship with APIs. They can be created independently, associated with multiple APIs, etc.") issues 452-(#452 (comment))

We work with other teams and we only want them to have the version sets associated with their specific API extracted. And at the moment we don't have 1 to many relationship with version sets and APIs.

I read that the developers are working on a filter mechanism in a future release, but in the meantime I have written a PowerShell script to filter out unwanted version sets during the extractor pipeline process which I would like to share with others who might also be searching for a solution to this issue (as the only other solution at the moment is adding a gitignore file, but this requires manual work) My script automates this process.

Some things to note with my script:

• It only filters out a single version set, so if you have a 1 to many relationship with a version set and API, this will not work for you.
• If there are no version sets found with the displayName of the version set to search for (a new parameter you will add to the pipeline), all version sets will be deleted from the artifacts folder.

Instructions to use the PowerShell script filter task:

1. Copy the task script below and paste it into the run-extractor.yaml file right after the Run extractor task. It's important that this filter task comes before the PublishPipelineArtifact@1 task.
2. Add an extra parameter to the run-extractor.yaml file called: API_VERSION_SET_NAME

- name: API_VERSION_SET_NAME
    type: string
    default: versionSetName
    displayName: Display name of the API version set to be saved.

Filter version sets task:

         - task: PowerShell@2
            displayName: Filtering version sets
            inputs:
              targetType: "inline"
              script: |
                Set-StrictMode -Version Latest
                $ErrorActionPreference = "Stop"
                $VerbosePreference = "Continue"
                $InformationPreference = "Continue"

                Write-Information "Set the path to the artificats version sets folder"
                $versionSetsFolderPath = "$(Build.ArtifactStagingDirectory)/${{ parameters.API_MANAGEMENT_SERVICE_OUTPUT_FOLDER_PATH }}/version sets"
                Write-Information "Path set to: $versionSetsFolderPath"

                Write-Information "Getting all folders inside the $versionSetsFolderPath"

                $folders = Get-ChildItem -Path $versionSetsFolderPath -Directory

                Write-Information "Looping through each folder"

                foreach ($folder in $folders) {
                    # Get the path to the json file inside the current folder
                    $jsonFilePath = Join-Path -Path $folder.FullName -ChildPath "apiVersionSetInformation.json"
                    
                    $deleteFolder = $true

                    # Check if the json file exists
                    if (Test-Path -Path $jsonFilePath -PathType Leaf) {
                        # Read the contents of the json file
                        $jsonContent = Get-Content -Path $jsonFilePath | ConvertFrom-Json
                        
                        # Check if the displayName property is API_VERSION_SET_NAME
                        if ($jsonContent.properties.displayName -eq "${{ parameters.API_VERSION_SET_NAME}}") {
                            # If displayName is API_VERSION_SET_NAME, do not delete the folder
                            $deleteFolder = $false
                            Write-Host "Keeping folder: $($folder.Name) because it has displayName $($jsonContent.properties.displayName)"
                        }

                        # If the delete flag is still true, delete the folder
                        if ($deleteFolder) {
                            Write-Host "Deleting folder: $($folder.Name) because it does not have correct version set name. Version set deleted: $($jsonContent.properties.displayName)"
                            Remove-Item -Path $folder.FullName -Recurse -Force
                        }
                    } else {
                        Write-Host "Deleting folder: $($folder.Name) because no json file found in folder"
                        Remove-Item -Path $folder.FullName -Recurse -Force
                    }
                }

                $folders = Get-ChildItem -Path $versionSetsFolderPath -Directory

                Write-Information "Looping through each folder again to show what has been kept"

                foreach ($folder in $folders) {
                    # Get the path to the json file inside the current folder
                    $jsonFilePath = Join-Path -Path $folder.FullName -ChildPath "apiVersionSetInformation.json"

                    # Check if the json file exists
                    if (Test-Path -Path $jsonFilePath -PathType Leaf) {
                        # Read the contents of the json file
                        $jsonContent = Get-Content -Path $jsonFilePath | ConvertFrom-Json

                        Write-Host "Kept folder: $($folder.Name) because it has displayName: $($jsonContent.properties.displayName)"
                    }
                }

                Write-Information "Execution complete."

And that's it. After the extractor pipeline runs, only the version sets which have the same name as the parameter API_VERSION_SET_NAME will be kept. All other version sets will be removed from the artifacts folder.

There is logging added to the script so you can see in the pipeline console which version sets have been kept and removed. This way if there is a typo or there were no version sets found with the value you give to API_VERSION_SET_NAME, you can check the logs for the correct version set name and run the pipeline again.

Example output logs:

I hope this can help others with this issue until the developers release their own filter mechanism.

[github-actions]

  Thank you for opening this issue! Please be patient while we will look into it and get back to you as this is an open source project. In the meantime make sure you take a look at the [closed issues](https://github.com/Azure/apiops/issues?q=is%3Aissue+is%3Aclosed) in case your question has already been answered. Don't forget to provide any additional information if needed (e.g. scrubbed logs, detailed feature requests,etc.).
  Whenever it's feasible, please don't hesitate to send a Pull Request (PR) our way. We'd greatly appreciate it, and we'll gladly assess and incorporate your changes.

[jeroenmaes]

Hi @JamesProant, this looks great! Thangs for sharing your implementation.

My first test results are very satisfying. I'm using Azure DevOps and a Linux Agent.
On a Windows Agent however I had some strange behaviour where all the versionSets still got published in the automated Pull Request. Not yet sure where the issue lies.

[JamesProant]

Hi @jeroenmaes thanks for the feedback. I am also using Azure DevOps and a Linux Agent. What I guess might be causing the issue on Windows is possibly the file paths. When you test it on a Windows agent do you see the correct file path in the logs? Write-Information "Path set to: $versionSetsFolderPath" should show the parent folder of the version sets.

[JamesProant]

Just wanted to give an update on a small change I made to the script. I added a .Trim() and .ToLower() to the versionSetName and the displayName, so that the requested version set will be kept even if there is an extra space or a mistaken capital or lowercase letter in the name. I will share the revised code here below:

        - task: PowerShell@2
          displayName: Filtering version sets
          inputs:
            targetType: "inline"
            script: |
              Set-StrictMode -Version Latest
              $ErrorActionPreference = "Stop"
              $VerbosePreference = "Continue"
              $InformationPreference = "Continue"

              $versionSetName = "${{ parameters.API_VERSION_SET_NAME }}".Trim().ToLower()

              Write-Information "Set the path to the artificats version sets folder"
              $versionSetsFolderPath = "$(Build.ArtifactStagingDirectory)/${{ parameters.API_MANAGEMENT_SERVICE_OUTPUT_FOLDER_PATH }}/version sets"
              Write-Information "Path set to: $versionSetsFolderPath"

              Write-Information "Getting all folders inside the $versionSetsFolderPath folder"

              $folders = Get-ChildItem -Path $versionSetsFolderPath -Directory

              Write-Information "Looping through each folder"

              foreach ($folder in $folders) {
                  # Get the path to the json file inside the current folder
                  $jsonFilePath = Join-Path -Path $folder.FullName -ChildPath "apiVersionSetInformation.json"
                  
                  $deleteFolder = $true

                  # Check if the json file exists
                  if (Test-Path -Path $jsonFilePath -PathType Leaf) {
                      # Read the contents of the json file
                      $jsonContent = Get-Content -Path $jsonFilePath | ConvertFrom-Json
                      
                      $jsonDisplayName = $jsonContent.properties.displayName.Trim().ToLower()

                      # Check if the displayName property is equal to API_VERSION_SET_NAME parameter
                      if ($jsonDisplayName -eq $versionSetName) {
                          # If displayName is API_VERSION_SET_NAME, do not delete the folder
                          $deleteFolder = $false
                          Write-Host "Keeping folder: $($folder.Name) because it has displayName $($jsonContent.properties.displayName)"
                      }

                      # If the delete flag is still true, delete the folder
                      if ($deleteFolder) {
                          Write-Host "Deleting folder: $($folder.Name) because it does not have correct version set name. Version set deleted: $($jsonContent.properties.displayName)"
                          Remove-Item -Path $folder.FullName -Recurse -Force
                      }
                  } else {
                      Write-Host "Deleting folder: $($folder.Name) because no json file found in folder"
                      Remove-Item -Path $folder.FullName -Recurse -Force
                  }
              }

              $folders = Get-ChildItem -Path $versionSetsFolderPath -Directory

              Write-Information "Looping through each folder again to show what has been kept"

              foreach ($folder in $folders) {
                  # Get the path to the json file inside the current folder
                  $jsonFilePath = Join-Path -Path $folder.FullName -ChildPath "apiVersionSetInformation.json"

                  # Check if the json file exists
                  if (Test-Path -Path $jsonFilePath -PathType Leaf) {
                      # Read the contents of the json file
                      $jsonContent = Get-Content -Path $jsonFilePath | ConvertFrom-Json

                      Write-Host "Kept folder: $($folder.Name) because it has displayName: $($jsonContent.properties.displayName)"
                  }
              }

              Write-Information "Execution complete."
              

[waelkdouh]

we just tested under v6.0.0-rc1 and the issue has been resolved. Please test and let us know.

[JamesProant]

Hi @waelkdouh ,
I have tested with version 6.0.1 but I don't see any version set filtering happening.
If I extract an API then I still get every version set for every API in our APIM instance, instead of just for the API I am extracting.

Is there a setting I need to add so that the extractor knows which version sets to extract?