Workitems not supporting danish letters "øæå" in usernames and titles, and user mapping are not working.

[xci-mrt]

Hello again,
After migrating our workitems, all titles and "assigned to" is missing danish letters "æøå"

when searching a users workitems, they are assigned to a user, but not mapped to the domain user that has been added to the collection.
Users are from same domain as the old collection.
any way to enable utf-8 encoding for usernames and workitems, and get the user mapping correct?

[xci-mrt]

[xci-mrt]

query showing items assigned to "me" but cannot be searched when in boards and assigned to @me

[xci-mrt]

[xci-mrt]

assigned to workitems, with danish name without ø

[xci-mrt]

[xci-mrt]

Screenshot of the user added to the collection with domain name

[MrHinsh]

Hi @xci-mrt, we added a filtering tool earlier in the year as we had a lot of errors with emojies, other chars, and the length of strings that are not supported in the Object Model. This is configurable, but you are right that the default is greedy... its not an encoding issue...

Check out the StringManipulatorTool. You can just turn it off... or you can configure the pattern to include danish... what you probably want is:

{
  "MigrationTools": {
    "Version": "16.0",
    "CommonTools": {
      "StringManipulatorTool": {
        "Enabled": "True",
        "Manipulators": [
          {
            "$type": "RegexStringManipulator",
            "Description": "Remove invalid chars... but keep danish.",
            "Enabled": "True",
            "Pattern": "[^( -~æøåÆØÅ)\n\r\t]+",
            "Replacement": ""
          }
        ],
        "MaxStringLength": "1000000"
      }
    }
  }
}

Update: There is a bug in the current release and the fix should be in v16.0.5-Preview.2 where it looks like your manipulator is ignored. This is not quite the case, its just added to the list that includes the default. Same result...

Workaround in v16.0.4

You can edit the appsettings.json and remove the default manipulator (or just edit it)!

[xci-mrt]

awesome, will try this out.. any idea on the user mapping issue?

[xci-mrt]

@MrHinsh I have just tried a test run again, and usernames still missing æøå,

[MrHinsh]

I think it might still be adding the default... can you check your logs for:

StringManipulatorTool::ProcessorExecutionWithFieldItem::Running::{Description} with {pattern}

The default one has the description of Remove invalid characters from the end of the string and the pattern of [^( -~)\n\r\t]+! If they are both being run then that's the problem! I'd need to look at the code and see how we can override...

[MrHinsh]

OK, im moving the default to code rather than the appsettings.json... but you can disable the default by editing the appsettings.json and setting the default manipulator to false.

[MrHinsh]

Fix should be in v16.0.5-Preview.2 once it passes the gates