Special characters in description and revision cause errors

[bldcm]

Version

• I confirm that I am using the latest version

Source Version

Team Foundation Server 2017

Target Version

Azure DevOps Server 2020

Relevant configuration

Version 14.4.4.  Looked at 14.4.5 and .6 and it doesn't appear to have addressed this

Relevant log output

2024-01-27 11:18:20.120 -05:00 [INF] [                 Bug][Complete:     1/1][sid: 37715|Rev: 22][tid:  null | System.History (History) | 
2024-01-27 11:18:20.128 -05:00 [INF] [                 Bug][Complete:     1/1][sid: 37715|Rev: 22][tid:  null | System.Description (Description) | This batch from prod us got stuck because of invalid characters<div>9ff7ca1b4f97421988287adb126c079b<br></div><div>attached it as a zip</div><div>If you upload it using iceboxtestrunner it should get stuck</div><div><br></div><div>Exception from prod</div><div><span style="font-family:Monaco, Menlo, Consolas, &quot;Courier New&quot;, monospace;white-space:pre-wrap;">System.ArgumentException: '�', hexadecimal value 0xFFFF, is an invalid character.</span><br></div>


2024-01-27 11:18:20.181 -05:00 [ERR] Microsoft.TeamFoundation.TeamFoundationServerInvalidRequestException: The request was rejected by the server.
Technical information:
  HTTP code 400: Bad Request ---> System.Net.WebException: The remote server returned an error: (400) Bad Request.
   at System.Net.HttpWebRequest.GetResponse()
   at Microsoft.TeamFoundation.Client.Channels.TfsHttpWebRequest.SendRequestAndGetResponse(HttpWebRequest webRequest, WebException& webException)
   --- End of inner exception stack trace ---
   at Microsoft.TeamFoundation.Client.Channels.TfsHttpWebRequest.ReadResponse(HttpWebResponse webResponse, WebException webException)
   at Microsoft.TeamFoundation.Client.Channels.TfsHttpWebRequest.SendRequest()
   at Microsoft.TeamFoundation.Client.Channels.TfsHttpRequestChannel.Request(TfsMessage message, TimeSpan timeout)
   at Microsoft.TeamFoundation.Client.Channels.TfsHttpClientBase.Invoke(TfsClientOperation operation, Object[] parameters, TimeSpan timeout, Object[]& outputs)
   at Microsoft.TeamFoundation.WorkItemTracking.Proxy.ClientService.Update(XmlNode package, XmlNode& result, MetadataTableHaveEntry[] metadataHave, String& dbStamp, RowSetCollection& metadata)
   at Microsoft.TeamFoundation.WorkItemTracking.Proxy.WorkItemServer.Update(String requestId, XmlElement package, XmlElement& result, MetadataTableHaveEntry[] metadataHave, String& dbStamp, IMetadataRowSets& metadata)
   at Microsoft.TeamFoundation.WorkItemTracking.Client.WorkItemStore.SendUpdatePackage(XmlElement package, XmlElement& result, Boolean bulk)
   at Microsoft.TeamFoundation.WorkItemTracking.Client.WorkItemStore.SaveWorkItems(WorkItem[] workitems, Boolean bulk, SaveFlags saveFlags)
   at Microsoft.TeamFoundation.WorkItemTracking.Client.WorkItem.Save(SaveFlags saveFlags)
   at MigrationTools.TfsExtensions.SaveToAzureDevOps(WorkItemData context) in C:\Projects_KofaxGitTFS\Products\TFS\VSTSSyncMigration\14.4.4_source\src\MigrationTools.Clients.AzureDevops.ObjectModel\TfsExtensions.cs:line 78
   at VstsSyncMigrator.Engine.WorkItemMigrationContext.ReplayRevisions(List`1 revisionsToMigrate, WorkItemData sourceWorkItem, WorkItemData targetWorkItem) in C:\Projects_KofaxGitTFS\Products\TFS\VSTSSyncMigration\14.4.4_source\src\VstsSyncMigrator.Core\Execution\MigrationContext\WorkItemMigrationContext.cs:line 850
2024-01-27 11:18:20.209 -05:00 [INF] ===============================================================
2024-01-27 11:18:20.130 -05:00 [INF] [                 Bug][Complete:     1/1][sid: 37715|Rev: 22][tid:  null | System.CreatedBy (Created By) | Marcus Jansson <READSOFT\Marcus Jansson>

What happened?

exception being thrown during save:

Microsoft.TeamFoundation.TeamFoundationServerInvalidRequestException: The request was rejected by the server.
Technical information:
HTTP code 400: Bad Request ---> System.Net.WebException: The remote server returned an error: (400) Bad Request

Have tracked it down to "bad data" in the description and hence the revision during playback. Below is what it looks like in the description

System.ArgumentException: '�', hexadecimal value 0xFFFF, is an invalid

Microsoft.TeamFoundation.WorkItemTracking.Client.WorkItem class version 16.205.1 appears to be latest, others are preview
.Validate() not catching a problem with the data and
.Save() throws an exception

The log snipit shows the description/history data as the utility sees it
migration.log

Debug in Visual Studio

• Visual Studio Debug

[MrHinsh]

You have an invalid character in the contents of a field, this is not checked with the Validate!

Look like you may need to check 37715 and see what the data is. Perhaps its an emoji or other character. This is also an unsupported version of the code.

[bldcm]

Your description of the problem is exactly what I said in my description, so I don't think you have added any new info except to say I am on an unsupported version which I believe was the supported version until last week. If I update to 14.4.6 will you then look at the issue? I will update to 14.4.6 anyway because I have the issue that was fixed.

Yes, There "appears" to bad data in 37715. We did check and yes it is "bad data". The problem is we cannot fix it in the source work item because it is in revision history for revision playback.

The only way I have found migrate that item is to remove the playback; remove the character from source; and run the migration on just that WI.

Another interesting fact is I can add the "bad data" directly to the work item as it sits in ADO 2020
System.ArgumentException: '�', hexadecimal value 0xFFFF, is an invalid

Is there any way to scrub the data before hand if the MS Validate won't catch it?

[MrHinsh]

We have an open request to be able to strip items on the fly and limit the length of the long text fields: #1863

The version that you are using is around two months old and we have had 10-15 releases since then. We constantly fix bugs and adapt the system. (not always in a good way)... as an OSS tool we cant support old versions and often issues are fixed as reported on a daily bases.

This particular issue with invalid characters has only been reported twice and it will be a significant fix. Please do add more details and upvote the issue above...

[bldcm]

Sounds good.

I must have misread the history, sorry. Will update to .15 and make sure it still happens, but sounds like it will.

I'll comment / upvote in #1863

[MrHinsh]

I think is will since we have not yet added any code to adapt the content of text fields.

If you know the specific field and character you would be able to use a REGEX Fieldmap!

[MrHinsh]

Related to #1815

[MrHinsh]

@bldcm did a Regex work?

How would you envisage this working? Its something that I can for sure squeeze into v15, that's in preview right now.

[bldcm]

I have not had a chance to try the REGEX field map yet, but I will try.

I'm confused about your version comment earlier when I went back and looked:
I'm on version 14.4.4
Site shows:
v15.0.0.Preview.74
v15.0.0.Preview.73
14.4.6 (Latest)

You said I was two months old, but 14.4.6 was released last week according to your comments:

So were are the 10-15 other releases you mentioned. I want to upgrade to latest, but I want to get the correct version.

[MrHinsh]

I may have read what you typed wrong! I think I read 14.3 and not 14.4!

[bldcm]

I seem to have taken some weird step backwards as I have tried to move to 14.4.6.

The tool is no longer able to connect to the source TFS collection with the Access token. Getting the error:

"Unable to configure store: Check persmissions and credentials for AccessToken"

I've tried using the exact same token in 14.4.6 as I am still able to use in 14.4.4 (recompiled) and I have tried creating a new token with rights for everything. The configurations appear to be identical. I am not sure what is going on.

Any insights?

[MrHinsh]

This is the first report of this issue, so I'm not sure. We use Access Tokens and a live ADO in out automated testing, which is working.

What's the actual exception that's logged along with that message?

It would be useful to have the log...

[bldcm]

Thanks for helping, the actual exception is below. I have also attached the log and the 14.4.4 (ver 0.0 because locally built) and the 14.4.6 config I am trying to use. The PATs being used are the same in both config files so it isn't a matter of the PAT. These are being run on different VMs though, but I don't know how that
configuration-14.4.4.json
configuration-14.4.6.json
migration-14.4.6.log
would make a difference.

2024-02-04 14:08:06.235 -08:00 [INF] MigrationClient: Connecting to "http://tfs.kofax.com:8080/tfs/Products2"
2024-02-04 14:08:06.314 -08:00 [ERR] Unable to configure store: Check persmissions and credentials for "AccessToken"
System.InvalidOperationException: Basic authentication requires a secure connection to the server.
at Microsoft.VisualStudio.Services.Common.VssBasicCredential.OnCreateTokenProvider(Uri serverUrl, IHttpResponse response)
at Microsoft.VisualStudio.Services.Common.IssuedTokenCredential.CreateTokenProvider(Uri serverUrl, IHttpResponse response, IssuedToken failedToken)
at Microsoft.VisualStudio.Services.Common.VssCredentials.TryGetTokenProvider(Uri serverUrl, IssuedTokenProvider& provider)
at Microsoft.TeamFoundation.Client.Channels.TfsHttpRequestHelpers.PrepareWebRequest(HttpWebRequest webRequest, Guid sessionId, String operationName, CultureInfo cultureInfo, TfsRequestSettings settings, VssCredentials credentials, IdentityDescriptor impersonate, Boolean applyICredentialsToWebProxy, IssuedToken& currentToken, IssuedTokenProvider& tokenProvider)
at Microsoft.TeamFoundation.Client.Channels.TfsHttpRequestHelpers.CreateSoapRequest(Uri requestUri, Guid sessionId, String soapAction, String operationName, CultureInfo cultureInfo, TfsRequestSettings settings, VssCredentials credentials, IdentityDescriptor impersonate, Boolean applyICredentialsToWebProxy, IssuedToken& currentToken, IssuedTokenProvider& tokenProvider)
at Microsoft.TeamFoundation.Client.Channels.TfsHttpWebRequest.CreateWebRequest()
at Microsoft.TeamFoundation.Client.Channels.TfsHttpWebRequest.SendRequest()
at Microsoft.TeamFoundation.Client.Channels.TfsHttpRequestChannel.Request(TfsMessage message, TimeSpan timeout)
at Microsoft.TeamFoundation.Client.Channels.TfsHttpClientBase.Invoke(TfsClientOperation operation, Object[] parameters, TimeSpan timeout, Object[]& outputs)
at Microsoft.TeamFoundation.Framework.Client.LocationWebService.Connect(Int32 connectOptions, Int32 lastChangeId, Int32 features)
at Microsoft.TeamFoundation.Framework.Client.FrameworkServerDataProvider.Connect(ConnectOptions connectOptions)
at Microsoft.TeamFoundation.Framework.Client.FrameworkServerDataProvider.EnsureConnected(ConnectOptions optionsNeeded)
at Microsoft.TeamFoundation.Framework.Client.FrameworkServerDataProvider.get_AuthorizedIdentity()
at MigrationTools._EngineV1.Clients.TfsMigrationClient.GetDependantTfsCollection(NetworkCredential credentials) in D:\a\1\s\src\MigrationTools.Clients.AzureDevops.ObjectModel_EngineV1\Clients\TfsMigrationClient.cs:line 151

[bldcm]

Good News
The RegexFieldMapConfig appears to work for that character:

{
  "$type": "RegexFieldMapConfig",
  "WorkItemTypeName": "*",
  "sourceField": "System.Description",
  "targetField": "System.Description",
  "pattern": "\uFFFF",
  "replacement": "U+FFFF"
},    

However, there is still a general problem of WI.Validate() succeeding and WI.Save() failing without any meaningful feedback on the exception thrown. Possibly not much we can do but keep identifying them and try and find out how to handle them.

Bad News.

I still cannot run with 14.4.6 version of the tool as I am getting authentication errors when I am using the same config on the same VM with a recompiled 14.4.4. Only difference I can see is the version specification, '0.0' for compiled tool vs '14.4' for 14.4.6 out of the box.

Troy

[MrHinsh]

http://tfs.kofax.com:8080/tfs/Products2 is not an HTTPS connection! Only HTTPS is supported.