[Bug]: ShouldCreateMissingRevisionPaths setting is ignored / always set to false

[norschel]

Version

• I confirm that I am using the latest version

Source Version

Azure DevOps Server 2020

Target Version

Azure DevOps Server 2020

Relevant configuration

{
  "ChangeSetMappingFile": null,
  "Source": {
    "$type": "TfsTeamProjectConfig",
    "Collection": "https://<servername>.global/tfs/OT1TFS/",
    "Project": "<SourceTeamProject>",
    "ReflectedWorkItemIDFieldName": "TfsMigrationTool.ReflectedWorkItemId",
    "AllowCrossProjectLinking": false,
    "AuthenticationMode": "Prompt",
    "PersonalAccessToken": "",
    "PersonalAccessTokenVariableName": "",
    "LanguageMaps": {
      "AreaPath": "Area",
      "IterationPath": "Iteration"
    }
  },
  "Target": {
    "$type": "TfsTeamProjectConfig",
    "Collection": "https://<servername>.global/tfs/OT1TFS/",
    "Project": "NicoTest004",
    "ReflectedWorkItemIDFieldName": "TfsMigrationTool.ReflectedWorkItemId",
    "AllowCrossProjectLinking": false,
    "AuthenticationMode": "Prompt",
    "PersonalAccessToken": "",
    "PersonalAccessTokenVariableName": "",
    "LanguageMaps": {
      "AreaPath": "Area",
      "IterationPath": "Iteration"
    }
  },
  "CommonEnrichersConfig": [],
  "FieldMaps": [],
  "GitRepoMapping": null,
  "LogLevel": "Debug",
  "Processors": [
    {
      "$type": "WorkItemMigrationConfig",
      "Enabled": true,
      "ReplayRevisions": true,
      "PrefixProjectToNodes": false,
      "UpdateCreatedDate": true,
      "UpdateCreatedBy": true,
      "WIQLQueryBit": "and [System.WorkItemType] IN ('Test Suite', 'Test Plan','Shared Steps','Shared Parameter','Test Case')",
      "WIQLOrderBit": "[System.ChangedDate] desc",
      "LinkMigration": true,
      "AttachmentMigration": true,
      "AttachmentWorkingPath": "c:\\temp\\WorkItemAttachmentWorkingFolder\\",
      "FixHtmlAttachmentLinks": true,
      "SkipToFinalRevisedWorkItemType": false,
      "WorkItemCreateRetryLimit": 5,
      "FilterWorkItemsThatAlreadyExistInTarget": true,
      "PauseAfterEachWorkItem": false,
      "AttachmentMaxSize": 480000000,
      "AttachRevisionHistory": false,
      "LinkMigrationSaveEachAsAdded": false,
      "GenerateMigrationComment": true,
      "WorkItemIDs": null,
      "MaxRevisions": 0,
      "UseCommonNodeStructureEnricherConfig": false,
      "NodeBasePaths": [],
      "AreaMaps": {},
      "IterationMaps": {},
      "MaxGracefulFailures": 0,
      "SkipRevisionWithInvalidIterationPath": false,
      "SkipRevisionWithInvalidAreaPath": false,
      "ShouldCreateMissingRevisionPaths": true,
      "ShouldCreateNodesUpFront": false,
          "MigrateAllNodeStructures": true,
	  "ReplicateAllExistingNodes": true
    },
    {
      "$type": "TestVariablesMigrationConfig",
      "Enabled": true
    },
    {
      "$type": "TestConfigurationsMigrationConfig",
      "Enabled": true
    },
    {
      "$type": "TestPlansAndSuitesMigrationConfig",
      "Enabled": true,
      "PrefixProjectToNodes": false,
      "OnlyElementsWithTag": null,
      "TestPlanQueryBit": null,
      "RemoveAllLinks": false,
      "MigrationDelay": 0,
      "UseCommonNodeStructureEnricherConfig": false,
      "NodeBasePaths": null,
      "AreaMaps": null,
      "IterationMaps": null,
      "RemoveInvalidTestSuiteLinks": false,
      "FilterCompleted": false
    }
  ],
  "Version": "14.3",
  "workaroundForQuerySOAPBugEnabled": false,
  "WorkItemTypeDefinition": {
    "sourceWorkItemTypeName": "targetWorkItemTypeName"
  },
  "Endpoints": {
    "InMemoryWorkItemEndpoints": [
      {
        "Name": "Source",
        "EndpointEnrichers": null
      },
      {
        "Name": "Target",
        "EndpointEnrichers": null
      }
    ]
  }
}

Relevant log output

No response

What happened?

Repo Steps:

1. Created a new team project based on Scrum Template
2. Uploaded modified work item templates (only added reflected-id field)
3. Started migration using the above config
4. Exception is thrown -> "you need to create or map missing area-/iteration paths"

Expected behavior:
My expectation is that because ShouldCreateMissingRevisionPaths is set to true, all missing area paths will be created as part of the migration.

Debugging analysis:
As part of my debugging session, I've figured out that in file TfsNodeStructure -> ProcessorExecutionBegin -> Options.ShouldCreateMissingRevisionPaths is always set false even if the value is set to true in my mapping file.
If you set the option to true in source code then everything works as expected.
As the moment my test code looks like:

public override void ProcessorExecutionBegin(IProcessor processor)
{
    if (Options.Enabled)
    {
        Log.LogInformation("Migrating all Nodes before the Processor run.");
        EntryForProcessorType(processor);
        Options.ShouldCreateMissingRevisionPaths = true;

        if (Options.ReplicateAllExistingNodes)
        {
            MigrateAllNodeStructures();
        }
        RefreshForProcessorType(processor);
    }
}

Debug in Visual Studio

• Visual Studio Debug

[MrHinsh]

I don't see anywhere in the logic that it gets set to "false". ADO is down right now so I will have to debug later.

[MrHinsh]

Sooooo.... the ReplicateAllExistingNodes value does not get passed.

[norschel]

Sooooo.... the ReplicateAllExistingNodes value does not get passed.

That is another issue, but I've found out a few more details.

It looks like in my case the TestPlansAndSuitesMigration context is overriding the property (line 87).

{
    Enabled = true,
    NodeBasePaths = _config.NodeBasePaths,
    PrefixProjectToNodes = _config.PrefixProjectToNodes,
    AreaMaps = _config.AreaMaps ?? new Dictionary<string, string>(),
    IterationMaps = _config.IterationMaps ?? new Dictionary<string, string>()
    //,ShouldCreateMissingRevisionPaths = true
});

I've added the static ShouldCreateMissingRevisionPaths property and then property is never changing to false.

[MrHinsh]

TestPlansAndSuitesMigration does not have any interaction with ShouldCreateMissingRevisionPaths... but it also does not pass the value.

You can use the UseCommonNodeStructureEnricherConfig to have a single setup for the settings.

"CommonEnrichersConfig": [
  {
    "$type": "TfsNodeStructureOptions",
    "PrefixProjectToNodes": false,
    "NodeBasePaths": [],
    "AreaMaps": {},
    "IterationMaps": {},
    "ShouldCreateMissingRevisionPaths": true,
    "ReplicateAllExistingNodes":  true,
  }
],

and then on the processors use:

"UseCommonNodeStructureEnricherConfig": true,

This does not happen when you use these processors separately; they are not expected to be run together.

We should probably move to only using CommonEnrichersConfig

[MrHinsh]

#1881

[MrHinsh]

This is now resolved in v15