Special characters in description and revision cause errors

[bldcm]

Version

• I confirm that I am using the latest version

Source Version

Team Foundation Server 2017

Target Version

Azure DevOps Server 2020

Relevant configuration

Version 14.4.4.  Looked at 14.4.5 and .6 and it doesn't appear to have addressed this

Relevant log output

2024-01-27 11:18:20.120 -05:00 [INF] [                 Bug][Complete:     1/1][sid: 37715|Rev: 22][tid:  null | System.History (History) | 
2024-01-27 11:18:20.128 -05:00 [INF] [                 Bug][Complete:     1/1][sid: 37715|Rev: 22][tid:  null | System.Description (Description) | This batch from prod us got stuck because of invalid characters<div>9ff7ca1b4f97421988287adb126c079b<br></div><div>attached it as a zip</div><div>If you upload it using iceboxtestrunner it should get stuck</div><div><br></div><div>Exception from prod</div><div><span style="font-family:Monaco, Menlo, Consolas, &quot;Courier New&quot;, monospace;white-space:pre-wrap;">System.ArgumentException: '�', hexadecimal value 0xFFFF, is an invalid character.</span><br></div>


2024-01-27 11:18:20.181 -05:00 [ERR] Microsoft.TeamFoundation.TeamFoundationServerInvalidRequestException: The request was rejected by the server.
Technical information:
  HTTP code 400: Bad Request ---> System.Net.WebException: The remote server returned an error: (400) Bad Request.
   at System.Net.HttpWebRequest.GetResponse()
   at Microsoft.TeamFoundation.Client.Channels.TfsHttpWebRequest.SendRequestAndGetResponse(HttpWebRequest webRequest, WebException& webException)
   --- End of inner exception stack trace ---
   at Microsoft.TeamFoundation.Client.Channels.TfsHttpWebRequest.ReadResponse(HttpWebResponse webResponse, WebException webException)
   at Microsoft.TeamFoundation.Client.Channels.TfsHttpWebRequest.SendRequest()
   at Microsoft.TeamFoundation.Client.Channels.TfsHttpRequestChannel.Request(TfsMessage message, TimeSpan timeout)
   at Microsoft.TeamFoundation.Client.Channels.TfsHttpClientBase.Invoke(TfsClientOperation operation, Object[] parameters, TimeSpan timeout, Object[]& outputs)
   at Microsoft.TeamFoundation.WorkItemTracking.Proxy.ClientService.Update(XmlNode package, XmlNode& result, MetadataTableHaveEntry[] metadataHave, String& dbStamp, RowSetCollection& metadata)
   at Microsoft.TeamFoundation.WorkItemTracking.Proxy.WorkItemServer.Update(String requestId, XmlElement package, XmlElement& result, MetadataTableHaveEntry[] metadataHave, String& dbStamp, IMetadataRowSets& metadata)
   at Microsoft.TeamFoundation.WorkItemTracking.Client.WorkItemStore.SendUpdatePackage(XmlElement package, XmlElement& result, Boolean bulk)
   at Microsoft.TeamFoundation.WorkItemTracking.Client.WorkItemStore.SaveWorkItems(WorkItem[] workitems, Boolean bulk, SaveFlags saveFlags)
   at Microsoft.TeamFoundation.WorkItemTracking.Client.WorkItem.Save(SaveFlags saveFlags)
   at MigrationTools.TfsExtensions.SaveToAzureDevOps(WorkItemData context) in C:\Projects_KofaxGitTFS\Products\TFS\VSTSSyncMigration\14.4.4_source\src\MigrationTools.Clients.AzureDevops.ObjectModel\TfsExtensions.cs:line 78
   at VstsSyncMigrator.Engine.WorkItemMigrationContext.ReplayRevisions(List`1 revisionsToMigrate, WorkItemData sourceWorkItem, WorkItemData targetWorkItem) in C:\Projects_KofaxGitTFS\Products\TFS\VSTSSyncMigration\14.4.4_source\src\VstsSyncMigrator.Core\Execution\MigrationContext\WorkItemMigrationContext.cs:line 850
2024-01-27 11:18:20.209 -05:00 [INF] ===============================================================
2024-01-27 11:18:20.130 -05:00 [INF] [                 Bug][Complete:     1/1][sid: 37715|Rev: 22][tid:  null | System.CreatedBy (Created By) | Marcus Jansson <READSOFT\Marcus Jansson>

What happened?

exception being thrown during save:

Microsoft.TeamFoundation.TeamFoundationServerInvalidRequestException: The request was rejected by the server.
Technical information:
HTTP code 400: Bad Request ---> System.Net.WebException: The remote server returned an error: (400) Bad Request

Have tracked it down to "bad data" in the description and hence the revision during playback. Below is what it looks like in the description

System.ArgumentException: '�', hexadecimal value 0xFFFF, is an invalid

Microsoft.TeamFoundation.WorkItemTracking.Client.WorkItem class version 16.205.1 appears to be latest, others are preview
.Validate() not catching a problem with the data and
.Save() throws an exception

The log snipit shows the description/history data as the utility sees it
migration.log

Debug in Visual Studio

• Visual Studio Debug

[MrHinsh]

You have an invalid character in the contents of a field, this is not checked with the Validate!

Look like you may need to check 37715 and see what the data is. Perhaps its an emoji or other character. This is also an unsupported version of the code.

[bldcm]

Your description of the problem is exactly what I said in my description, so I don't think you have added any new info except to say I am on an unsupported version which I believe was the supported version until last week. If I update to 14.4.6 will you then look at the issue? I will update to 14.4.6 anyway because I have the issue that was fixed.

Yes, There "appears" to bad data in 37715. We did check and yes it is "bad data". The problem is we cannot fix it in the source work item because it is in revision history for revision playback.

The only way I have found migrate that item is to remove the playback; remove the character from source; and run the migration on just that WI.

Another interesting fact is I can add the "bad data" directly to the work item as it sits in ADO 2020
System.ArgumentException: '�', hexadecimal value 0xFFFF, is an invalid

Is there any way to scrub the data before hand if the MS Validate won't catch it?

[MrHinsh]

We have an open request to be able to strip items on the fly and limit the length of the long text fields: #1863

The version that you are using is around two months old and we have had 10-15 releases since then. We constantly fix bugs and adapt the system. (not always in a good way)... as an OSS tool we cant support old versions and often issues are fixed as reported on a daily bases.

This particular issue with invalid characters has only been reported twice and it will be a significant fix. Please do add more details and upvote the issue above...

[bldcm]

Sounds good.

I must have misread the history, sorry. Will update to .15 and make sure it still happens, but sounds like it will.

I'll comment / upvote in #1863

[MrHinsh]

I think is will since we have not yet added any code to adapt the content of text fields.

If you know the specific field and character you would be able to use a REGEX Fieldmap!

[MrHinsh]

Related to #1815

[MrHinsh]

@bldcm did a Regex work?

How would you envisage this working? Its something that I can for sure squeeze into v15, that's in preview right now.

[bldcm]

I have not had a chance to try the REGEX field map yet, but I will try.

I'm confused about your version comment earlier when I went back and looked:
I'm on version 14.4.4
Site shows:
v15.0.0.Preview.74
v15.0.0.Preview.73
14.4.6 (Latest)

You said I was two months old, but 14.4.6 was released last week according to your comments:

So were are the 10-15 other releases you mentioned. I want to upgrade to latest, but I want to get the correct version.

[MrHinsh]

I may have read what you typed wrong! I think I read 14.3 and not 14.4!