Artifact not getting generated for the API's from extractor

[RameshNallasamy]

Release version

v5.1.4

Question Details

I'm not seeing the API artifacts while running the extractor from version v5.1.4. I even tried with the latest version Azure_DevOps_v6.0.0-alpha.1.0.4 and several other previous versions. No luck. Only the Spectral-result.xml is getting created in the artifacts from portal.

Expected behavior

API artifacts should be downloaded to target folder and pull request should be created

Actual behavior

Only spectral-result.xml file is getting generated and stored in the target folder. No API artifacts are getting saved.

Reproduction Steps

I'm using v5.1.4 to run the extractor.

[github-actions]

  Thank you for opening this issue! Please be patient while we will look into it and get back to you as this is an open source project. In the meantime make sure you take a look at the [closed issues](https://github.com/Azure/apiops/issues?q=is%3Aissue+is%3Aclosed) in case your question has already been answered. Don't forget to provide any additional information if needed (e.g. scrubbed logs, detailed feature requests,etc.).
  Whenever it's feasible, please don't hesitate to send a Pull Request (PR) our way. We'd greatly appreciate it, and we'll gladly assess and incorporate your changes.

[waelkdouh]

Please don't use the alpha version as it's broken.

Looking at the error you shared seems like it can't find the folder to use to pull the artifacts form to merge the pr. Make sure you don't have some misconfigured paths. also the extractor seems to be doing this job and extracting the artifacts for you which you could clearly see in the first stage of the pipeline. If you click on the artifacts icon generated in the first stage, you will see all the extracted artifacts so hopefully once for the next stage, then the pool request will go ahead and take these artifacts and merge them into repositories main branch.

[RameshNallasamy]

As i showed in the screenshot, the extracted artifacts only has spectral-result.xml. I'm not changing anything in the run-extractor.yml. I'm running it as is. For the run extractor logs i can see that it is reading the values from APIM but it's not publishing the artifact to the folder.

I'm thinking the api's should be inside the artifacts-from-portal right?

[guythetechie]

Can you include the output of step Publish pipeline artifact in job Create artifact from portal?

[RameshNallasamy]

Hi @guythetechie , Here it is.

Artifact name input: artifacts-from-portal
Uploading pipeline artifact from /home/vsts/work/1/a for build #864759
Using default max parallelism.
Max dedup parallelism: 192
DomainId: 0
Verbose, ArtifactHttpClientFactory.CreateVssHttpClient: DedupStoreHttpClient with BaseUri: https://vsblobprodcus3.vsblob.visualstudio.com/Af7dc6c9d-b432-4f79-b635-57141ddb4914/, MaxRetries:5, SendTimeout:00:01:40
Information, ApplicationInsightsTelemetrySender will correlate events with X-TFS-Session fc78b283-dd85-4a0b-a976-fb9da68d7c6f
Verbose, Started ITelemetrySender BlobStoreApplicationInsightsTelemetrySender.
Hashtype: Dedup64K
Information, DedupManifestArtifactClient will correlate http requests with X-TFS-Session fc78b283-dd85-4a0b-a976-fb9da68d7c6f
Verbose, Globbing completed in: 00:00:00.0145755
Verbose, Pages to upload computed in 00:00:00.0071938
Information, 1 files processed.
Information, Processed 1 files from /home/vsts/work/1/a successfully.
Verbose, Trying to put node 9902 of 1 children, 0 receipts used. (Children: EE01 [None])
Verbose, Added node 9902 of 1 children and got 1 receipts back. (Receipts: 9902 [2024-05-20T02:24:02Z])
Verbose, ManifestId: B801
Verbose, ContentNode: 9902
Verbose, RootId: 1F02
Verbose, Trying to put node 1F02 of 2 children, 1 receipts used. (Children: 9902 [2024-05-20T02:24:02Z], B801 [None])
Verbose, Added node 1F02 of 2 children and got 1 receipts back. (Receipts: 1F02 [2024-05-20T02:24:02Z])
Information, Uploaded 0 out of 251 bytes
Information, Content upload is done!
Information,
Content upload statistics:
Total Content: 0.3 KB
Physical Content Uploaded: 0.0 bytes
Logical Content Uploaded: 0.0 bytes
Compression Saved: 0.0 bytes
Deduplication Saved: 0.3 KB
Number of Chunks Uploaded: 0
Total Number of Chunks: 3

##[debug]Processed: ##vso[telemetry.publish area=AzurePipelinesAgent;feature=PipelineArtifact]{"FileCount":"1","PlanId":"d90ed454-aed0-4454-9334-d0e63664167d","JobId":"82eded09-40cb-5e35-5d9c-30947af734a8","TaskInstanceId":"c7844f98-e1d9-5dba-790a-50564360692b","ActionDurationMs":"463","ActionName":"PipelineArtifact.UploadAsync","ActionResult":"Success","AttemptNumber":"1","ItemCount":"0","Level":"ThirdParty","CreatedUtcNow":"2024-05-17T14:46:40.5018454Z","SentUtcNow":"2024-05-17T14:46:40.9683927Z","BaseAddress":"https://vsblobprodcus3.vsblob.visualstudio.com/Af7dc6c9d-b432-4f79-b635-57141ddb4914/","X_TFS_Session":"fc78b283-dd85-4a0b-a976-fb9da68d7c6f","DeploymentEnvironment":"PRODUCTION","DeploymentEnvironmentIsProduction":"True","VSOAccount":"vsblobprodcus3","OSName":"Linux","OSVersion":"2024","FrameworkDescription":".NET ","ProcessName":"Agent.PluginHost","DotNetReleaseDword":"-1","Version":"19.237.34720.0","ExceptionCount":"0"}
Verbose, ApplicationInsightsTelemetrySender sent DedupUpload.UploadAsync telemetry
Associated artifact 436768 with build 864759
Verbose, ApplicationInsightsTelemetrySender waiting for 0 of 2 TrackEvent operations to complete...
Verbose, ApplicationInsightsTelemetrySender sent PipelineArtifact.UploadAsync telemetry
Verbose, ApplicationInsightsTelemetrySender operations completed in 1 ms
Information, ApplicationInsightsTelemetrySender correlated 2 events with X-TFS-Session fc78b283-dd85-4a0b-a976-fb9da68d7c6f
Verbose, Stopped ITelemetrySender BlobStoreApplicationInsightsTelemetrySender.
Uploading pipeline artifact finished.
Finishing: Publish pipeline artifact

[waelkdouh]

I see you are referring to artifacts folder. Make sure the defaults are what is expected.

[RameshNallasamy]

I've set it the same way as described in your video. I'm not sure what i am missing here.

[RameshNallasamy]

The only difference i see is your branch is main and mine is master. But i've also updated that accordingly. So i'm stuck here.

[guythetechie]

We've seen this issue before.

In your Run extractor step, the artifacts get written to /home/vsts/work/1/s.

That folder is defined in this line of code:
API_MANAGEMENT_SERVICE_OUTPUT_FOLDER_PATH: $(Build.ArtifactStagingDirectory)/${{ parameters.API_MANAGEMENT_SERVICE_OUTPUT_FOLDER_PATH }}

In your Publish pipeline artifacts step though, the folder /home/vsts/work/1/a gets published as a pipeline artifact.

That folder is defined in this line of code:
targetPath: "$(Build.ArtifactStagingDirectory)"

Both steps refer to "$(Build.ArtifactStagingDirectory)", but it resolves to /home/vsts/work/1/a in one step and /home/vsts/work/1/s in the other.

It's an ADO issue, and we never found a root cause for it. Workaround would be to hardcode a path in your run-extractor.yaml file, or use a different directory such as "$(Agent.TempDirectory)".

You can read more about it in #170 and #113.

[RameshNallasamy]

@guythetechie & @waelkdouh , Thank you both. The copyFiles worked for me from #170 . I'm going with that for now. Thank you for your time in helping this resolve so quickly. Closing this thread.