API are getting deleted in higher enviornments

[DibyaRanjan1]

Release version

all version

Describe the bug

Generally speaking, how could we support a scenario in which in dev 2 apis (api-1, api-2) were updated but only 1 (api-1) needs to be promoted?
I thought by using the config extract.
So I put api-1 in the config extract and I saw I could "extract" one specific api, api-1, and create PR-A.

Now, my prod apim is already in place and tracked in the git repository. When I look to PR-A this is proposing to delete all the other Apis (which are missing in the branch created by the run - extract pipeline) and update api-1. If I understand this correctly, this will give me in the end a prod apim with api-1 updated and all the other ones deleted.
What I would have expected was to keep all the other apis and update api-1. I hope I made myself clear.

Expected behavior

Only selective api should be deployed to higher environments.

Actual behavior

API are getting deleted in higher enviornments

Reproduction Steps

Generally speaking, how could we support a scenario in which in dev 2 apis (api-1, api-2) were updated but only 1 (api-1) needs to be promoted?
I thought by using the config extract.
So I put api-1 in the config extract and I saw I could "extract" one specific api, api-1, and create PR-A.

Now, my prod apim is already in place and tracked in the git repository. When I look to PR-A this is proposing to delete all the other Apis (which are missing in the branch created by the run - extract pipeline) and update api-1. If I understand this correctly, this will give me in the end a prod apim with api-1 updated and all the other ones deleted.
What I would have expected was to keep all the other apis and update api-1. I hope I made myself clear.

[github-actions]

  Thank you for opening this issue! Please be patient while we will look into it and get back to you as this is an open source project. In the meantime make sure you take a look at the [closed issues](https://github.com/Azure/apiops/issues?q=is%3Aissue+is%3Aclosed) in case your question has already been answered. Don't forget to provide any additional information if needed (e.g. scrubbed logs, detailed feature requests,etc.).
  Whenever it's feasible, please don't hesitate to send a Pull Request (PR) our way. We'd greatly appreciate it, and we'll gladly assess and incorporate your changes.

[waelkdouh]

I remember seeing the same question being brought up recently in another issue. Please check the other issues as your question has been answered there. Take a look at both the open and closed issues please.

[DibyaRanjan1]

I saw previous issues and read all the documentation. Is there a way to extract only a single API and migrate to higher environments with out deleting old APIs ? Why does the pipeline delete the existing API ? It does not make sense to me.

[waelkdouh]

The publisher picks up the changes in your last commit. If you only modified a single API then that is the only change that will be published. Nothing should be deleted if you didn't delete explicitly. Something else most be going on.

[DibyaRanjan1]

Thank you for the reply. I have another question. Is It recomended to have branch for each APIM environments ? I mean, the namedvalues and many other values are specific to environments. How to store envirnment specific metadata in git repo ? What is the best way to store the API metdata ?(Single main branch or multiple branches followed by one main branch)

[waelkdouh]

When we built APIops, we envisioned that you would have one set of artifacts, which usually represent the artifacts of the lowest APIm environment, and then, as you promote to higher environments, you would override using environment specific configuration files (configuration.EnV.yaml. If you follow that set up, then you would need a single git repository, which would store the artifacts of the lowest environment.

Remember that at the end of the day, you are at liberty to extract from any git environment to any git repository as you see fit.

[waelkdouh]

Just to add, it's not recommended that you have multiple git repositories with artifacts, extracted from different environments as it would make it hard to keep them in sync. Instead, as I mentioned before, it is recommended to extract from the lowest environment and then start using the apiops to promote across higher environments while overriding using your environment specific configuration files.

[DibyaRanjan1]

When I extract an API or a named value using extractor config, Why it is proposing to delete all other api and all named values ? If this is a feature, How should we promote only required changes to QA and PROD ?

[DibyaRanjan1]

The publisher picks up the changes in your last commit. If you only modified a single API then that is the only change that will be published. Nothing should be deleted if you didn't delete explicitly. Something else most be going on.

My last commit was modifying the extractor.yaml to extract only one namedvalue. When I ran extractor tool, It is propsing to delete all extsting namedvalues .

configuration.extractor.yaml
namedValueNames:

• test4

[waelkdouh]

@guythetechie any thoughts on this?

[DibyaRanjan1]

To fix this, We are doing currently the below steps.

1. git checkout origin/artifacts-from-portal-build-668876 (We get the branch in local) . This brach is created by the extractor pipeline. Extractor pipeline only respects what is mentioned in the extractor.yaml file and delete everything which are not mentioned in the .yaml config. For me It is a code smell and It should not delete all other files.
2. We need to then switch to main branch. git checkout main
3. Now merge main branch with origin/artifacts-from-portal-build-668876 . git merge origin/artifacts-from-portal-build-668876
4. After merging , It is deleting all other files which are not mentioned in extractor.yaml config which is scary. If some will accidently push the changes, It will delete all api, namedvalues and other in the lower environment.
5. To prevent above, we need to write few commands.
   git reset --soft HEAD~ (In main branch)
   manually cheery pick the changes which are relevent and do git restore --staged (to prevent delete all the files).

Could you provide any best approach , so that extractor pipeline should include all old files and add new files which are mentioned in the extractor.yaml file.

[waelkdouh]

In this case you may want to temporarily use gitignore to ignore api2 in this example. As for the extractor its behaving as expected as the moment you include select pais in the configuration.extractor.yaml file you are telling the tool that the only apis you are interested in are those on the list.

Also check issue #401 for more information