[BUG] Publish pipeline fails on push_changes_to_Dev_APIM when APIM configured with App Insights

[Alexey-Zheltov]

Release version

APIOPS v5.0.0

Describe the bug

Publish pipeline fails with error message below when publishing artifacts to Dev environment before applying settings to Prod. When skipping publishing to Dev stage, publishing to Prod works fine.

System.Net.Http.HttpRequestException: HTTP request to URI https://management.azure.com/subscriptions/***/resourceGroups/UK-Infrastructure/providers/Microsoft.ApiManagement/service/dev-apim-uk-01/diagnostics/applicationinsights?api-version=2022-04-01-preview failed with status code 400. Content is '{"error":{"code":"ValidationError","message":"One or more fields contain incorrect values:","details":[{"code":"ValidationError","target":"loggerId","message":"LoggerId must be specified."}]}}'.
2023-12-17T17:16:08.6694390Z at common.HttpPipelineExtensions.Validate(Response response, Uri requestUri)
2023-12-17T17:16:08.6694832Z at common.HttpPipelineExtensions.PutResource(HttpPipeline pipeline, Uri uri, JsonObject resource, CancellationToken cancellationToken)
2023-12-17T17:16:08.6695292Z at publisher.Program.<>c__DisplayClass13_0.<b__0>d.MoveNext()
2023-12-17T17:16:08.6696593Z --- End of stack trace from previous location ---
2023-12-17T17:16:08.6697127Z at publisher.Diagnostic.PutDiagnostic(DiagnosticName diagnosticName, JsonObject json, ServiceUri serviceUri, PutRestResource putRestResource, ILogger logger, CancellationToken cancellationToken)
2023-12-17T17:16:08.6697753Z at publisher.Diagnostic.<>c__DisplayClass8_0.<b__0>d.MoveNext()
2023-12-17T17:16:08.6699484Z --- End of stack trace from previous location ---
2023-12-17T17:16:08.6699767Z at System.Threading.Tasks.Parallel.<>c__501.<<ForEachAsync>b__50_0>d.MoveNext() 2023-12-17T17:16:08.6700045Z --- End of stack trace from previous location --- 2023-12-17T17:16:08.6730667Z at common.IEnumerableExtensions.ForEachParallel[T](IEnumerable1 enumerable, Func2 action, CancellationToken cancellationToken) 2023-12-17T17:16:08.6731064Z at publisher.Diagnostic.ProcessArtifactsToPut(IReadOnlyCollection1 files, JsonObject configurationJson, ServiceDirectory serviceDirectory, ServiceUri serviceUri, PutRestResource putRestResource, ILogger logger, CancellationToken cancellationToken)
2023-12-17T17:16:08.6731641Z at publisher.Service.ProcessArtifactsToPut(IReadOnlyCollection`1 files, JsonObject configurationJson, ServiceDirectory serviceDirectory, ServiceUri serviceUri, ListRestResources listRestResources, GetRestResource getRestResource, PutRestResource putRestResource, DeleteRestResource deleteRestResource, ILogger logger, CancellationToken cancellationToken)
2023-12-17T17:16:08.6732078Z at publisher.Publisher.RunWithoutCommitId(CancellationToken cancellationToken)
2023-12-17T17:16:08.6732301Z at publisher.Publisher.Run(CancellationToken cancellationToken)
2023-12-17T17:16:08.6732511Z at publisher.Publisher.ExecuteAsync(CancellationToken cancellationToken)
2023-12-17T17:16:08.6732697Z info: Microsoft.Hosting.Lifetime[0]
2023-12-17T17:16:08.6732851Z Application is shutting down...
2023-12-17T17:16:08.6733105Z fail: Microsoft.Extensions.Hosting.Internal.Host[9]
2023-12-17T17:16:08.6733268Z BackgroundService failed

Expected behavior

Publish pipeline shouldn't fail when pushing changes to lower-level environment.

Actual behavior

Adjustments of pipeline required in order to get it to work

Reproduction Steps

1. Extract all API's.
2. Run publishing pipeline.
3. Publishing pipeline fails on stage push_changes_to_Dev_APIM

[github-actions]

  Thank you for opening this issue! Please be patient while we will look into it and get back to you as this is an open source project. In the meantime make sure you take a look at the [closed issues](https://github.com/Azure/apiops/issues?q=is%3Aissue+is%3Aclosed) in case your question has already been answered. Don't forget to provide any additional information if needed (e.g. scrubbed logs, detailed feature requests,etc.).
  Whenever it's feasible, please don't hesitate to send a Pull Request (PR) our way. We'd greatly appreciate it, and we'll gladly assess and incorporate your changes.

[waelkdouh]

Is the logger artifact being extracted? For some reason it's not being included when publishing which is throwing a 400. Also make sure you are not trying publish against a logger that doesn't exist.

[Alexey-Zheltov]

Logger artifact are getting created in repository:

{
"properties": {
"credentials": {
"instrumentationKey": "{{Logger-Credentials--6566f39dfb426808c40853a6}}"
},
"isBuffered": true,
"loggerType": "applicationInsights",
"resourceId": "/subscriptions/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXX/resourceGroups/XXXXXXXXX/providers/microsoft.insights/components/APIM-UK-Dev-AppInsights"
}
}

I've added config file with following settings and after that publish pipeline also worked successfully for Dev environment. Looks like publisher fails to parse some settings from logger artifact.

apimServiceName: dev-apim-uk-01
loggers:
- name: apim-uk-appinsights
properties:
loggerType: applicationInsights
description: prod application insights
resourceId: "/subscriptions/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXX/resourceGroups/XXXXXXXXX/providers/microsoft.insights/components/APIM-UK-Dev-AppInsights"
credentials:
instrumentationKey: "{{Logger-Credentials--6566f39dfb426808c40853a6}}"
isBuffered: true
diagnostics:

• name: applicationinsights
  properties:
  verbosity: information
  loggerId: "/subscriptions/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXX/resourceGroups/XXXXXXXXX/providers/Microsoft.ApiManagement/service/dev-apim-uk-01/loggers/apim-uk-appinsights"
  apis:
• name: demo-conference-api
  diagnostics:
  • name: applicationinsights
    properties:
    verbosity: information
    loggerId: "/subscriptions/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXX/resourceGroups/UK-Infrastructure/providers/Microsoft.ApiManagement/service/dev-apim-uk-01/loggers/apim-uk-appinsights"

[waelkdouh]

Yeah the behavior you are experiencing is odd. Usually you only need congifuruation.[env].yaml files for higher environments as you promote your changes and not for the lower environment (typically dev). In your case it seems like you are having to provide a configuration file for the lower environment that has configurations that match what you have already extracted which is unnecessary.

@guythetechie any idea what could be causing this behavior?

[guythetechie]

Can you show the contents of your DEV diagnosticInformation.json file? Should be in a path like /artifacts/diagnostics/applicationinsights/diagnosticInformation.json.

[Alexey-Zheltov]

{
"properties": {
"alwaysLog": "allErrors",
"httpCorrelationProtocol": "Legacy",
"logClientIp": true,
"sampling": {
"percentage": 100,
"samplingType": "Fixed"
}
}
}

[Alexey-Zheltov]

I've tried to enable Trace logging and re-run pipeline again to get more details regarding error when there is no config file for Dev environment, but eventually it worked.

I will try to do one more test with fresh API.

[waelkdouh]

Please remember to close the issue once you are ready.