[Question] Extractor pipeline with configuration.extractor.yaml is failing in git action

[DibyaRanjan1]

Release version

v5.1.4

Question Details

Extractor pipeline with configuration.extractor.yaml is failing in git action ?

Expected behavior

It should extract mentioned api.

Actual behavior

Pipleine is failing

Reproduction Steps

Run Extractor pipeline to extract few api.

Note: With the introduction of workspaces we recommend using workspaces with APIOps when possible to limit what can be extracted whenever possible.

More information about workspaces can be found here https://learn.microsoft.com/en-us/azure/api-management/workspaces-overview

apiNames:

• dibya3

backendNames: [ignore]

diagnosticNames: [ignore]

loggerNames: [ignore]

namedValueNames: [ignore]

productNames:

• dibyaproduct3

subscriptionNames:

• dibya3

tagNames: [ignore]

policyFragmentNames: [ignore]

[github-actions]

  Thank you for opening this issue! Please be patient while we will look into it and get back to you as this is an open source project. In the meantime make sure you take a look at the [closed issues](https://github.com/Azure/apiops/issues?q=is%3Aissue+is%3Aclosed) in case your question has already been answered. Don't forget to provide any additional information if needed (e.g. scrubbed logs, detailed feature requests,etc.).
  Whenever it's feasible, please don't hesitate to send a Pull Request (PR) our way. We'd greatly appreciate it, and we'll gladly assess and incorporate your changes.

[waelkdouh]

This seems to be an issue specific to your GitHub environment. Just out of curiosity, are you able to run the extractor without specifying the configuration file? Basically where your able to have a successful run when you extract everything?

[DibyaRanjan1]

Yes. without the configuration.yaml, It's downloading all content. Could you guide what could be the issue ?

[waelkdouh]

Did you make sure the format of the yaml file is correct (indentation, property names, using dashes instead of bullets for example, etc)?

@guythetechie does anything wrong stand out to you in the sample yaml file provided?

[guythetechie]

If I'm looking at it correctly, the error is related to Copilot/code vulnerability scans. We don't do anything Copilot-related in ApiOps, so it's probably specific to your environment.

[DibyaRanjan1]

It seems without extractor coniguration, the http request content type is application/octet-stream but with extractor config option the content type is text/html . This could be a change from your backend side or due to some github policy. I feel, this issue will be faced by many new users.

I have created a new gitrepo without any additional rules or customization. I don't think it could be due to local setup.

[guythetechie]

Interesting. When setting the configuration path, it fails to download the extractor. I'm guessing the download URL gets malformed somehow.

Would you mind adding this here:
Write-Host "Downloading extractor from $uri to $destinationFilePath" after this line in your run_extractor.yaml. This should output the download URL and its destination.

[DibyaRanjan1]

Here is the value.

[DibyaRanjan1]

I guess, The url is wrong and that's why the error is 404. Do you mind checking the your server where the .exe is hosted. Something must have changed there and github is not able to connect to your server with the given url.

[guythetechie]

That URL hasn't existed since version 5.1.0. The Linux executables don't end with .exe.

You mentioned you're using release v5.1.4, but I'm guessing your pipeline is from an older version before v5.1.0. There were pipeline changes in v5.1.0 to use the correct URL.

Can you download the updated pipeline in v5.1.4 and try?