x/vgo: use vcs tools instead of https APIs of code hosting sites

[rsc]

In https://research.swtch.com/vgo-module I wrote that

We want to move away from invoking version control tools such as bzr, fossil, git, hg, and svn to download source code. These fragment the ecosystem: packages developed using Bazaar or Fossil, for example, are effectively unavailable to users who cannot or choose not to install these tools. The version control tools have also been a source of exciting security problems. It would be good to move them outside the security perimeter.

We still do, but it seems clear that this will be a large burden especially for corporate users. It is probably better to separate the introduction of versioning from the imposition of this restriction. We can focus on making the migration to versioned packages as painless as possible and do drop version control tool support at some later date, if at all. (One possibility is that if we get a reliable global proxy then you'd have to explicitly ask not to use the proxy, at which point the security and fragmentation concerns are significantly lessened.)

[gopherbot]

Change https://golang.org/cl/107657 mentions this issue: cmd/go/internal/modfetch: update for new gitrepo backend

[gopherbot]

Change https://golang.org/cl/107656 mentions this issue: cmd/vgo/internal/modfetch/gitrepo: add general git repo access

[lrewega]

Has there been any consideration given to the idea of allowing a command to be invoked in lieu of an HTTP proxy? E.g.: GOPROXY_CMD=foo vgo get rsc.io/[email protected] could perhaps invoke a binary foo with the request URL that vgo would usually make:

foo rsc.io/sampler/@v/1.31.zip

In the minimal case, this would allow for use of tools such as curl to inject cookies into the request: GOPROXY_CMD="curl -fsSL -b my-cookie:sekret"

In the more complex case, a script or program could be invoked and decide to use version control or other tools for certain domains. E.g. if foo was a script containing:

#!/bin/sh
case "$1" in
  rsc.io/*)    curl -sSL "$1";;
  code.corp/*) ssh code.corp -- cat "/some/corporate/repos/$1";;
  *)           echo >&2 "no fetching of 3rd party code: $1"; exit 1;;
esac

Git's GIT_SSH and GIT_SSH_COMMAND environment variables inspired this concept.

[sdwarwick]

Why does it seem like the whole concept of URLs and URNs is being re-invented? URLs include b access protocol, and URNs provide a broadly understood concept of uniqueness.

[rsc]

@lrewega At least to start, no, we don't plan to do that. Let's see how things go with just ordinary network-level proxy support instead of command-level.

@sdwarwick, I don't understand what you're getting at. The notation here is the same as go get always has been. We're not changing that.

[flibustenet]

Could be possible to keep both http api and vcs to remove issues like #25102 and begin now to experiment with http api ?

[paulbdavis]

Does this mean that modules in private repos can be fetched over ssh? I am having trouble getting things to build that use private repos as dependencies.

[rsc]

Everything is using git again. Not sure why this wasn't closed.

@paulbdavis, The git invocations always specify https://. They don't fall back to ssh:// like old "go get" used to. If you need to avoid https:// my suggestion would be to use git's insteadOf configuration feature in your $HOME/.gitconfig, so that 'https://' is accepted and understood by git to mean ssh.

[paulbdavis]

@rsc thanks, I had tried this before, but that was before using vcs again.

Are there any plans to continue to allow git+ssh access going forward? (via some env var, git config as above, fallbacks instead of failure, etc.). Without SSH, I am using a combo of vendoring, GOPROXY, and @myitcv's modpub, then compiling in the GOPATH with "vanilla" go in my CI. Tedious, but works so I can use vgo now.

Going forward I would expect the proxy availability to be more robust for handling this use case, but simply allowing git+ssh to be available at least as an option would simplify using private git repos now and in the future (for git users at least).

[myitcv]

@paulbdavis to use git with ssh, you'll need a config setting along the lines of:

git config [email protected]:.insteadof https://github.com/

See https://git-scm.com/docs/git-config for more details.

[paulbdavis]

@rsc thanks, I had tried this before, but that was before using vcs again.

Are there any plans to continue to allow git+ssh access going forward? (via some env var, git config as above, fallbacks instead of failure, etc.). Without SSH, I am using a combo of vendoring, GOPROXY, and @myitcv's modpub, then compiling in the GOPATH with "vanilla" go in my CI. Tedious, but works so I can use vgo now.

Going forward I would expect the proxy availability to be more robust for handling this use case, but simply allowing git+ssh to be available at least as an option would simplify using private git repos now and in the future (for git users at least).

[myitcv]

@paulbdavis did you see my response #24915 (comment)?

Looks like you reposted the same comment (#24915 (comment)) in response?

[paulbdavis]

Sorry, not sure how that got posted again four days later.

Yes, I got git+ssh working as you described.

What I am trying to ask is if using vcs or allowing vcs to be used is going to remain a part of vgo going forward into to 1.11 preview implementation.

From reading the blog posts, eliminating vcs usage seems to be a goal. The use case of having private repositories is a fairly common one, and using vcs tooling makes it easy to get working on any machine that has your ssh keys versus making sure you have a working proxy set up for each set of private repos being used (for example, private libraries for multiple client projects)

[myitcv]

Yes, I got git+ssh working as you described.

Great.

What I am trying to ask is if using vcs or allowing vcs to be used is going to remain a part of vgo going forward into to 1.11 preview implementation.

Yes, it will. Indeed per #24915 I quote:

We still do, but it seems clear that this will be a large burden especially for corporate users. It is probably better to separate the introduction of versioning from the imposition of this restriction. We can focus on making the migration to versioned packages as painless as possible and do drop version control tool support at some later date, if at all.