Allow multiple URLs for a project

[tejlmand]

Remotes are specified in the manifest as:

remotes:
- name: zephyrproject-rtos
url: https://github.com/zephyrproject-rtos
or
remotes:
- name: zephyrproject-rtos
url: git@github.com:zephyrproject-rtos/zephyr.git
but there cannot be two url for one remote.

the https version is easy to start out with, but often ssh with keys are nicer to work with (imho).

Could it be made possible to maybe be able to specify some like:
remotes:
- name: zephyrproject-rtos
url: https://github.com/zephyrproject-rtos
url-ssh: git@github.com:zephyrproject-rtos/zephyr.git
the type specified under url is used per default but if url-ssh is present and users specify, e.g.
west fetch --ssh
uses url-ssh if present (otherwise fallback is url)

[tejlmand]

As update on the IRC discussion Sep 12 it is important to consider that ssh could have different styles on different remotes, as example:

Consider a company who pulls code from both the internal git servers, as well as different remote sources, could have an internal manifest similar to:

remotes:
- name: zephyrproject-rtos
url: https://github.com/zephyrproject-rtos
ssh: ssh://git@github.com/zephyrproject-rtos

-name: remote-gerrit
url: https://remote-gerrit.somewhere.com/gerrit
ssh: ssh://ssh@remote-gerrit.somewhere.com:29418

-name: mycompany-bitbucket
url: https://mycomp/bitbucket
ssh: ssh://git@bitbucket.mycomp:1234/

projects:
-name: proj1.git
remote: zephyrproject-rtos
-name: proj2.git
remote: remote-gerrit
-name: proj3.git
remote: mycompany-bitbucket

As there can be multiple remotes in the same company, and those can use different user@server syntax, e.g. git and bitbucket seems to use git@server, whereas if I remember correctly from when I worked with gerrit, I think the default syntax there is ssh@server.
But this is most likely configurable.
Also git-servers tends to use different ports.

All of this this may make it trickier to make the solution with just an overlay on command line.

So splitting to allow to specify bort a url and a ssh on remotes that supports both could be a clear way/
Then on west fetch it can be specified what is preferred.

[mbolivar]

As discussed on IRC, I think the URL specifying the transport is a feature, not a bug, and that a potentially better way to handle this would be through adding support for "manifest overlays" similar to device tree overlays; these would be files in west/manifest-overlays/xxx.yml which could be used to override the behavior in the main manifest.

With that feature, you could write a file west/manifest-overlays/remotes.yml containing this:

manifest:
    remotes:
        - name: zephyrproject
        - url: git@github.com:zephyrproject-rtos

The main manifest entry fragment from west/manifest/default.yml:

remotes:
    - name: zephyrproject
      url: https://github.com/zephyrproject-rtos

Would then have its URL overridden from the https:// to the git+ssh version.

To ensure determinism, overlays would be applied in sorted order, allowing users to name them 00-first.yml, 01-second.yml, etc. as desired.

Seem reasonable?

[tejlmand]

sounds reasonable so far, although I need to see the full picture.
To avoid everyone having to write their own overlays, I can then create a repo, containing both:
default.yml

remotes:
    - name: zephyrproject
      url: https://github.com/zephyrproject-rtos

and an overlay: 00-ssh-remote.yml:

manifest:
    remotes:
        - name: zephyrproject
        - url: git@github.com:zephyrproject-rtos

then how does the user of west decides that the overlay should / should not be used ?
I assume, as an argument on the command line, like:
west --overlay 00-ssh-remote.yml

[mbolivar]

To avoid everyone having to write their own overlays, I can then create a repo, containing both:

If you're creating your own repo, why not just update the remote URLs in the main manifest? You are already committing to maintaining a copy at that point.

[tejlmand]

The point that I'm trying to make, is that Github itself provides to possibility to choose between:

• https
• ssh

Therefore there is a possibility for any github developer to use any of the two (and they might have a personal preference for one of the methods).

As seen in just our little reference group, there is already two who default seems to be using SSH.

Therefore, I believe it would make sense that both the https and ssh where described in the manifest for Zephyr so that developers can easily choose their preferred method.
(defaulting to https version is fine with me)

As well as other companies forking Zephyr has the ability to described both a https and ssh version in their manifest.
The overlay proposal as I understand, means that everyone who prefers ssh would need to write that overlay, correct ?

[carlescufi]

CC @nashif

[tejlmand]

@mbolivar did some thinking on this.

remotes:
- name: <name-of-remote>
  url: <default-url-of-remote>[;<optional-url>]

as example:

remotes:
- name: remotes:
  url: https://github.com/zephyrproject-rtos;ssh://git@github.com/zephyrproject-rtos

then default url and transport will be first in list, but user can request other preference, e.g.
west pull --transport ssh which will use ssh is described.
If not available, it will just default to first in list.

[mbolivar]

url: <default-url-of-remote>[;<optional-url>]

Hm, I see what you mean.

I think it would be better to use a real YAML list for this, though. Semicolons are reserved sub-delimiters in URLs. Using the native list type also lets us keep pykwalify schema checking, and add more metadata.

Something like:

remotes:
  - name: my-cool-remote
    urls:
      - transport: git
        url-base: git://my-cool-remote.com/git
      - transport: git+ssh
        url-base: git@my-cool-remote.com:git

Commentary:

• The urls key in a remote entry would be in addition to the current url, not a replacement.
• If the user chooses to use url, the value would be either a simple string like today or a transport: x/ base: y dictionary like an entry in urls.
• It would be an error to specify both urls and url for a single remote.
• If url-base is too verbose, we could shorten it to url or base.
• I think the transport key has some problems (it mixes the VCS, like git, with the protocol used by the VCS, like SSH) and deserves more thought, especially since we've been asked to support non-git systems.

To clarify, the following url fragments would both be valid after this change:

url: https://my-cool-remote.com/git

url:
  transport: git+https
  url-base: https://my-cool-remote.com/git

We would need to come up with some default assumptions about the transport if not given in a url. This is another reason I think it is not quite right.

west pull --transport ssh which will use ssh is described.

Perhaps this could be a configuration option in addition to a manifest defaults entry. I think this is more of a permanent setting than something to specify on an invocation-by-invocation basis.

In defaults:

defaults:
  - transport: git+https

In west/config, something like:

[manifest.defaults]
transport = git+https

[tejlmand]

url: <default-url-of-remote>[;<optional-url>]

Hm, I see what you mean.

I think it would be better to use a real YAML list for this, though. Semicolons are reserved sub-delimiters in URLs.

Correct, but that could be handled by requiring such characters to be escaped. Just as you would need to do if trying to use a url containing ; on command line in a shell.
In a shell the semi-colon would be treated as separation of commands unless escaped.

Using the native list type also lets us keep pykwalify schema checking, and add more metadata.

Now, that is an argument I accept ;)
Sticking to default yaml might be best option.

Been reading up on yaml, and seems this is an allowed sequence, close to your example:

remotes:
  - name: my-cool-remote
    urls:
      - git://my-cool-remote.com/git
      - git@my-cool-remote.com:git
      - ssh://git@my-cool-remote.com/git
      - https://my-cool-remote.com/git

and we don't need to specify transport, as it is given by the URL:
git:// and git@my-cool-remote.com:git defines git transport (technically ssh without encryption)
ssh:// defines ssh
https:// defines https

west pull --transport ssh which will use ssh is described.

Perhaps this could be a configuration option in addition to a manifest defaults entry. I think this is more of a permanent setting than something to specify on an invocation-by-invocation basis.

it could be provided west init, like west init --transport git;https;ssh and then written to west/config
Runing re-init or writing to west/config can thus be used to update the setting.
Question: should we consider a west config command for the future, as I personally don't like to put this with west init

If no setting is given or the transport is not in urls, the first item in urls sequence is picked.

[mbolivar]

and we don't need to specify transport, as it is given by the URL

Sorry again for choosing such a poor name as "transport". I'm trying with this field to specify both the VCS (git, for now) and the actual network transport. So I'm mixing application layer and L4 in the OSI sense in my above proposal, which is why I think it's not quite right.

My point is that if we are going to make this change, we should do so in a way that allows the schema to encode non-git source access methods, even if west itself cannot handle anything other than git, in order to avoid a schema compatibility break later.

It should also be noted that /some/path and file:///some/path are treated differently by Git - the former is more efficient for remotes on systems which support hard links. So some non-URL encodings may be desirable.

[mbolivar]

@tejlmand @carlescufi I found a workaround you might be interested in:

https://wiki.archlinux.org/index.php/git#Protocol_defaults

[mbolivar-nordic]

@tejlmand I am going to close this since west has been available for years and we still haven't discovered a compelling use case for it.

[marc-hb]

#615 looks similar.