Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[master] Release v15! Query and CommonEnrichersConfig changes #1913

Merged
merged 30 commits into from
Mar 4, 2024
Merged

Conversation

MrHinsh
Copy link
Member

@MrHinsh MrHinsh commented Feb 6, 2024

This update is a +semver: major change. There are many config changes!

Change 1: Updates all Processors to use only CommonEnrichersConfig and removes inline options. Only options set in CommonEnrichersConfig will be honored.

This will minimise the number of configuration options for processors, and reduce confusion and support tickets.

  1. TfsNodeStructureOptions has been made the default so AreaMaps, IterationMaps, NodeBasePaths, PrefixProjectToNodes, ShouldCreateMissingRevisionPaths, and ReplicateAllExistingNodes are all removed from WorkItemMigrationContext and must instead be added to CommonEnrichersConfig in line with TfsNodeStructure. The documentation for AreaMaps, IterationMaps, NodeBasePaths has all been moved here.
  2. TfsRevisionManagerOptions has been made default so ReplayRevisions and MaxRevisions is now only configurable by adding a TfsRevisionManagerOptions section to CommonEnrichersConfig. See TfsRevisionManagerOptions
  3. 'TfsWorkItemLinkEnricherOptions' has been made the default moving its option FilterIfLinkCountMatches, and SaveAfterEachLinkIsAdded to CommonEnrichersConfig. See TfsWorkItemLinkEnricherOptions
  "CommonEnrichersConfig": [
    {
      "$type": "TfsNodeStructureOptions",
      "PrefixProjectToNodes": false,
      "NodeBasePaths": [],
      "AreaMaps": {
        "^Skypoint Cloud$": "MigrationTest5"
      },
      "IterationMaps": {
        "^Skypoint Cloud\\\\Sprint 1$": "MigrationTest5\\Sprint 1"
      },
      "ShouldCreateMissingRevisionPaths": true,
      "ReplicateAllExistingNodes": true
    },
    {
      "$type": "TfsWorkItemLinkEnricherOptions",
      "Enabled": true,
      "FilterIfLinkCountMatches": true,
      "SaveAfterEachLinkIsAdded": false
    },
    {
      "$type": "TfsRevisionManagerOptions",
      "Enabled": true,
      "ReplayRevisions": true,
      "MaxRevisions": 0
    }
  ],

Change 2: Update the Query system to no longer have two parts that are added together. All queries are now the full query. There are no docs on this but a typical query now looks like:

 "WIQLQuery": "SELECT [System.Id] FROM WorkItems WHERE [System.TeamProject] = @TeamProject AND [System.WorkItemType] NOT IN ('Test Suite', 'Test Plan','Shared Steps','Shared Parameter','Feedback Request') ORDER BY [System.ChangedDate] desc",

Change 3: Added a String Manipulator to combat the new 100000 max string length and the inability of the ObjectModel to support special chars:

    {
      "$type": "StringManipulatorEnricherOptions",
      "Enabled": true,
      "MaxStringLength": 1000000,
      "Manipulators": [
        {
          "$type": "RegexStringManipulator",
          "Enabled": false,
          "Pattern": "[^( -~)\n\r\t]+",
          "Replacement": "",
          "Description": "Remove invalid characters from the end of the string"
        }
      ]
    }

Real version of #1889

norschel and others added 13 commits January 25, 2024 13:14
The current bug issue template does not yet include the latest AzD
Server version 2022.
This update is a +semver: major change. There are many config changes!

**Change 1:** Updates all Processors to use only `CommonEnrichersConfig`
and removes inline options. _Only options set in `CommonEnrichersConfig`
will be honored._

This will minimise the number of configuration options for processors,
and reduce confusion and support tickets.

1. `TfsNodeStructureOptions` has been made the default so `AreaMaps`,
`IterationMaps`, `NodeBasePaths`, `PrefixProjectToNodes`,
`ShouldCreateMissingRevisionPaths`, and `ReplicateAllExistingNodes` are
all removed from `WorkItemMigrationContext` and must instead be added to
`CommonEnrichersConfig` in line with
[TfsNodeStructure](https://nkdagility.com/learn/azure-devops-migration-tools/Reference/v2/ProcessorEnrichers/TfsNodeStructure/).
The documentation for `AreaMaps`, `IterationMaps`, `NodeBasePaths` has
all been moved here.
2. `TfsRevisionManagerOptions` has been made default so
`ReplayRevisions` and `MaxRevisions` is now only configurable by adding
a `TfsRevisionManagerOptions` section to `CommonEnrichersConfig`. See
[TfsRevisionManagerOptions](https://nkdagility.com/learn/azure-devops-migration-tools/Reference/v2/ProcessorEnrichers/TfsRevisionManager/)
3. 'TfsWorkItemLinkEnricherOptions' has been made the default moving its
option `FilterIfLinkCountMatches`, and `SaveAfterEachLinkIsAdded` to
`CommonEnrichersConfig`. See
[TfsWorkItemLinkEnricherOptions](https://nkdagility.com/learn/azure-devops-migration-tools/Reference/v2/ProcessorEnrichers/TfsWorkItemLinkEnricher/)

```
  "CommonEnrichersConfig": [
    {
      "$type": "TfsNodeStructureOptions",
      "PrefixProjectToNodes": false,
      "NodeBasePaths": [],
      "AreaMaps": {
        "^Skypoint Cloud$": "MigrationTest5"
      },
      "IterationMaps": {
        "^Skypoint Cloud\\\\Sprint 1$": "MigrationTest5\\Sprint 1"
      },
      "ShouldCreateMissingRevisionPaths": true,
      "ReplicateAllExistingNodes": true
    },
    {
      "$type": "TfsWorkItemLinkEnricherOptions",
      "Enabled": true,
      "FilterIfLinkCountMatches": true,
      "SaveAfterEachLinkIsAdded": false
    },
    {
      "$type": "TfsRevisionManagerOptions",
      "Enabled": true,
      "ReplayRevisions": true,
      "MaxRevisions": 0
    }
  ],
```


**Change 2:** Update the Query system to no longer have two parts that
are added together. All queries are now the full query. There are no
docs on this but a typical query now looks like:

```
 "WIQLQuery": "SELECT [System.Id] FROM WorkItems WHERE [System.TeamProject] = @teamproject AND [System.WorkItemType] NOT IN ('Test Suite', 'Test Plan','Shared Steps','Shared Parameter','Feedback Request') ORDER BY [System.ChangedDate] desc",
```
This should truncate longTest fields to the max length and resolve #1861
With the support and help of @colutti we have added a "force set" on
ClosedDate at validation time. Thsi adds an additional test below that
if closedDate on the target is blank, but it is set on the source that
we copy it across again... it may have been skipped before because the
system thinks its not editable!

This may result in an error if it really is not editable.

```
         if (sourceWorkItem.ToWorkItem().Fields.Contains(closedDateField) && sourceWorkItem.ToWorkItem().Fields[closedDateField].Value != null)
         {
             Log.LogDebug("CheckClosedDateIsValid::Setting Closed Date [#{sourceId}][Rev{sourceRev}]: {sourceClosedDate} ", sourceWorkItem.ToWorkItem().Id, sourceWorkItem.ToWorkItem().Rev, sourceWorkItem.ToWorkItem().Fields[closedDateField].Value);
             targetWorkItem.ToWorkItem().Fields[closedDateField].Value = sourceWorkItem.ToWorkItem().Fields[closedDateField].Value;
         }
         else
         {
             Log.LogWarning("The field {closedDateField} is set to Null and will revert to the current date on save! ", closedDateField);
             Log.LogWarning("Source Closed Date [#{sourceId}][Rev{sourceRev}]: {sourceClosedDate} ", sourceWorkItem.ToWorkItem().Id, sourceWorkItem.ToWorkItem().Rev, sourceWorkItem.ToWorkItem().Fields[closedDateField].Value);
         }               
     }
```
@MrHinsh MrHinsh self-assigned this Feb 6, 2024
@MrHinsh MrHinsh temporarily deployed to nakedalmweb-githubSDK February 6, 2024 14:38 — with GitHub Actions Inactive
@MrHinsh MrHinsh temporarily deployed to nakedalmweb-githubSDK February 6, 2024 14:44 — with GitHub Actions Inactive
@MrHinsh MrHinsh marked this pull request as draft February 6, 2024 14:54
@MrHinsh MrHinsh temporarily deployed to nakedalmweb-githubSDK February 6, 2024 16:56 — with GitHub Actions Inactive
@MrHinsh MrHinsh temporarily deployed to nakedalmweb-githubSDK February 6, 2024 16:58 — with GitHub Actions Inactive
@MrHinsh MrHinsh temporarily deployed to nakedalmweb-githubSDK February 6, 2024 20:11 — with GitHub Actions Inactive
@MrHinsh MrHinsh changed the title Release v15! Query and CommonEnrichersConfig changes [master] Release v15! Query and CommonEnrichersConfig changes Feb 6, 2024
@MrHinsh MrHinsh temporarily deployed to nakedalmweb-githubSDK February 6, 2024 20:57 — with GitHub Actions Inactive
@MrHinsh MrHinsh temporarily deployed to nakedalmweb-githubSDK February 8, 2024 12:47 — with GitHub Actions Inactive
@MrHinsh MrHinsh temporarily deployed to nakedalmweb-githubSDK February 12, 2024 21:38 — with GitHub Actions Inactive
@MrHinsh MrHinsh linked an issue Feb 13, 2024 that may be closed by this pull request
@MrHinsh MrHinsh temporarily deployed to nakedalmweb-githubSDK February 13, 2024 09:30 — with GitHub Actions Inactive
@MrHinsh MrHinsh temporarily deployed to nakedalmweb-githubSDK February 13, 2024 15:16 — with GitHub Actions Inactive
@MrHinsh MrHinsh temporarily deployed to nakedalmweb-githubSDK February 14, 2024 12:12 — with GitHub Actions Inactive
@MrHinsh MrHinsh temporarily deployed to nakedalmweb-githubSDK February 15, 2024 13:55 — with GitHub Actions Inactive
@MrHinsh MrHinsh temporarily deployed to nakedalmweb-githubSDK February 16, 2024 13:03 — with GitHub Actions Inactive
- Added a new error message and warning when a workItem is checked and
gets a `DeniedOrNotExistException`. This work item is skiped, but the
specific error is still written and logged.
- Updated `DeniedOrNotExistException` for better error message.
+telemitry for the error.
- Tried to make
`Engine.TypeDefinitionMaps.Items.ContainsKey(missingWorkItemType)` case
agnostic for #1940
@MrHinsh MrHinsh temporarily deployed to nakedalmweb-githubSDK February 17, 2024 18:00 — with GitHub Actions Inactive
Added 4 Unit tests...

1) Create StringManipulatorEnricher
2) String field too long
3) String field is shortter than max
4) Apply regex manipulator

Should resolve #1950
@MrHinsh MrHinsh temporarily deployed to nakedalmweb-githubSDK February 22, 2024 23:21 — with GitHub Actions Inactive
Adds a workaround for the missing repository properties when migration
build pipelines.
I couldn't find the root of this issue but this should fix the
exceptions.

Fixes #1879, #1631, #1565

Co-authored-by: Martin Hinshelwood nkdAgility.com <martin@nkdagility.com>
@MrHinsh MrHinsh temporarily deployed to nakedalmweb-githubSDK February 24, 2024 08:48 — with GitHub Actions Inactive
It looks like the follwoing was not clear and continues to result in
support requests:

> There are 4 Nodes (Area or Iteration) found in the history of the
Source that are missing from the Target! These MUST be added or mapped
before we can continue using the instructions on
>
https://nkdagility.com/learn/azure-devops-migration-tools/Reference/v1/Processors/WorkItemMigrationContext/#iteration-maps-and-area-maps

Trying with the following:

> !! There are MISSING Area or Iteration Paths"
> NOTE: It is NOT possible to migrate a work item if the Area or
Iteration path does not exist on the target project. This is because the
work item will be created with the same Area and Iteration path as the
source work item with the project name swapped. The work item will not
be created if the path does not exist. The only way to resolve this is
to follow the instructions:
> !! There are {missingAreaPaths} Nodes (Area or Iteration) found in the
history of the Source that need to be added to the Target! These MUST be
added or mapped before we can continue using the instructions on
https://nkdagility.com/learn/azure-devops-migration-tools//Reference/v2/ProcessorEnrichers/TfsNodeStructure/#iteration-maps-and-area-maps

Lets see if this reduces these requests...
@MrHinsh MrHinsh temporarily deployed to nakedalmweb-githubSDK February 27, 2024 19:17 — with GitHub Actions Inactive
@MrHinsh MrHinsh marked this pull request as ready for review February 27, 2024 19:18
@MrHinsh MrHinsh merged commit 9594196 into master Mar 4, 2024
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

String is not supported by database collation
3 participants