Can I Replace or Strip characters from Strings?

[bldcm]

I am trying to replace "invalid characters" so they can be saved in paragraph fields like description. Below is a snipit of a V1 configuration with character maps. If only one of the characters is present then the mapping works fine. If multiple characters are present it does not work and logging seems to indicate that each mapping starts with the original source data to do the mapping, meaning only the last found mapping takes affect.

Can multiple mappings be done on a single field?
Can multiple fields use the same mapping (i.e. have multiple source fields and targetfields) with repeating the configuration for each field?

"FieldMaps": [
{
"$type": "RegexFieldMapConfig",
"WorkItemTypeName": "",
"sourceField": "System.Description",
"targetField": "System.Description",
"pattern": "\uFFFF",
"replacement": "U+FFFF"
},
{
"$type": "RegexFieldMapConfig",
"WorkItemTypeName": "",
"sourceField": "System.Description",
"targetField": "System.Description",
"pattern": "\uF1A4",
"replacement": "U+F1A4"
},
{
"$type": "RegexFieldMapConfig",
"WorkItemTypeName": "*",
"sourceField": "System.Description",
"targetField": "System.Description",
"pattern": "\uF210",
"replacement": "U+F210"
},

[MrHinsh]

Correct. this is not the tool for that job. Use the StringManipulatorTool instead!

 {
  "MigrationTools": {
    "Version": "16.0",
    "CommonTools": {
      "StringManipulatorTool": {
        "Enabled": "True",
        "Manipulators": [
          {
            "$type": "RegexStringManipulator",
            "Description": "Remove invalid characters from string",
            "Enabled": "True",
            "Pattern": "[^( -~)\n\r\t]+",
            "Replacement": ""
          }
        ],
        "MaxStringLength": "1000000"
      }
    }
  }
}

You can add multiple manipulators or create one uber pattern... there is also a fix in the latest preview to enable you to override the default rather than merge it in... note that the Tfs Object Model does not support some characters...

note: The workaround is to change the appsettings.json and remove the default manipulator if you don't have the version with the fix.

[bldcm]

Thank you, I will try.

This checks characters in all fields (e.g. Description, Title, Comments, etc.)? Can it be limited to certain fields to try and not slow down migration?

[bldcm]

I think I am getting close, but the StringManipulatorTool is not working for me. I'm using version 16.0.4. I would like to search and replace individual characters so that the config can be used in all migrations to handle the individual characters wherever they appear, even it not contiguous. My last test was for a contiguous character sequence and it didn't work either. I have attached full configurations and logs.

This is the text I am trying to scrub:
------------
W10 64, Office 2016 64 or Office 2016 32

Launch Outlook and execute an archiving for a folder

Result:
"��O펧甛O" folder occurs next to PDF Favorites during archiving in C:\Users<actual user>\Documents

issued folder contains 325 numbered (test set consists of 493 mail) for subfolders and all of them includes

Archive.rtf
Convert.cmd
------------

The hex in memory for the characters are as follows.
ff ff \uffff
ff ff \uffff
a4 f1 
4f 00 O
a7 d3 펧
1b 75 甛
10 f2 
4f 00 O

When I try the regex with the text to scrub and a regex tester it does identify the characters. My StringManipulatorTool pattern configurations I have tried are:

  "StringManipulatorTool": {
    "Enabled": "True",
    "Manipulators": [
      {
        "$type": "RegexStringManipulator",
        "Description": "Remove invalid characters from string",
        "Enabled": "True",
        //"Pattern": "[^( -~)\n\r\t]+",
    //"Pattern": "[\uffff\u004f\uf1a4\ud3a7\u751b\uf210]+",
   "Pattern": "(\uffff)?\uf1a4\u004f\ud3a7\u751b\uf210\u004f",
        "Replacement": ""
      }
    ],
    "MaxStringLength": "1000000"
  },	

migration-errors20241009.log
migration2024100911.log
configuration-TFS2ADOPreProd-PowerPDF-NOTO-upgraded-DIAGS.json