cmd/go: add support for GitHub Enterprise import path

[matope]

For now, go get does not fetch packages hosted on GitHub Enterprise correctly without any workarounds.

$ go get -v ghe.in.my.company/user/repo/sub
Fetching ghe.in.my.company/user/repo/sub?go-get=1
Parsing meta tags from https://ghe.in.my.company/user/repo/sub?go-get=1 (status code 404)
package ghe.in.my.company/user/repo/sub: unrecognized import path "ghe.in.my.company/user/repo/sub" (parse https://ghe.in.my.company/user/repo/sub?go-get=1: no go-import meta tags)

When I execute go get, it issues an HTTP request to find go-import meta-tags.
But GHE does not return any meta-tag because ghe.in.my.company/user/repo/sub is 404 for GHE ( ghe.in.my.company/user/repo/tree/master/sub would work fine).

So, in order to use golang with GHE, we have to

• a) Use import path with .vcs. ex) ghe.in.my.company/user/repo.git/sub
• b) or use a redirect server that attach go-import meta-tags.

But, a) would be a error prone especially for newbies. Many people at first download repo into ghe.in.my.company/user/repo but, after that, go get would download it's sub packages into ghe.in.my.company/user.git/repo. I think this would make people confused.
b) This would work fine, but I think managing a server only for dependency resolution is not very efficient.

So, I think it would be great if cmd/go natively support GitHub Enterprise.

[matope]

My patch is here. Could anybody please review this?
https://go-review.googlesource.com/#/c/33171/

[gopherbot]

CL https://golang.org/cl/33171 mentions this issue.

[gopherbot]

CL https://golang.org/cl/37176 mentions this issue.

[matope]

Hi, @bradfitz. As go cmd had a major refactoring, I've abandoned the previous CL and created new one which is rebased on Go1.8 with some cleanup.

I'd appreciate you if you could review my new one.

[rsc]

It sounds like maybe the solution should be to support ?go-get=1 in the GHE 404 page, but we'd have to write down exactly what that would mean before approaching GitHub (or suggesting that their paying customers do that).

[blaskovicz]

Here is the Gitlab implementation, for reference. It seems that there was a similar issue with Golang subpackages for it as well.

[jijojv]

ok 1.9 is out and disappointed that it still doesn't work with GitHub Enterprise over ssh as-is.

git clone [email protected]:org/repo.git does not create a .git directory but go decided not to allow any git url schemes and not to use the standard git cloning convention and create a directory with .git suffix which provides a confusing experience for everyone involved...

[ianlancetaylor]

@jijojv I encourage you to send a patch, so that this can be fixed for 1.10.

[jijojv]

@ianlancetaylor sorry just a lowly github admin, but i can in python if it'll get accepted. Looks like someone attempted above, so shouldn't be too hard for a real go user/fan.

[matope]

@rsc Hi, my colleague asked Github Enterprise team about this issue.

They said that they respond <meta> tag from repository page but do not respond from 404 page intentionally due to preventing to leak private repos.Also they suggest a) adding .git into import path, b) using git clone instead of go get or c) using some proxy.

[matope]

I think their intention is reasonable. And as I mentioned in the description of this issue, all of the suggested way are painful a little. So I'd like to propose adding support for Github Enterprise in go get.

[jameschensmith]

@matope, our team is getting 200 response (via git config --global url."https://${GITHUB_TOKEN}:[email protected]/".insteadOf "https://my.ghe.domain/"), but since our repo is private, GitHub Enterprise doesn't return the meta tag unless you're authenticated. Is there a separate issue that needs to be created for this, or is there a workaround other than the options (a - c) that you suggested above? Thanks

[matope]

@poeia As far as I've tested briefly, my patch works for private repositories too.

The CL I sent regards any domains as github instance if the servers return "Server: GitHub.com" header. It doesn't depend on meta-tag. And because Github Eenterprise (probably) returns the header from every pages, my CL could treat private repos well too.

# go1.8
$ go get -v my.ghe.domain/matope/private-repo
Fetching https://my.ghe.domain/matope/private-repo?go-get=1
Parsing meta tags from https://my.ghe.domain/matope/private-repo?go-get=1 (status code 404)
package my.ghe.domain/matope/private-repo: unrecognized import path "my.ghe.domain/matope/private-repo" (parse https://my.ghe.domain/matope/private-repo?go-get=1: no go-import meta tags ())

# go1.8 with CL37176
$ ./git/go/bin/go get -v my.ghe.domain/matope/private-repo
Fetching https://my.ghe.domain/matope/private-repo?go-get=1
Parsing meta tags from https://my.ghe.domain/matope/private-repo?go-get=1 (status code 404)
my.ghe.domain/matope/private-repo (download)
my.ghe.domain/matope/private-repo/sub2
my.ghe.domain/matope/private-repo/sub
my.ghe.domain/matope/private-repo

Note that I'm using not tokens but certificates to authenticate.

[jameschensmith]

@matope, my concern was specifically for a 200 response, and whether or not it is linked with the same issue the 404 response gives. Do they both send you to the login page for private repos? I'm wondering if there is a way to bypass the login without the options (a-c) that you provided, or if we will have to wait for this patch. Just curious. Hope this patch gets implemented into 1.10!

[rsc]

Moving to Go 1.11. I hope we'll have a more comprehensive story around company repos then.

[lpar]

I just tried

env GOPATH=/home/meta/tmp go get -v github.mycompany.com/meta/mypackage

and it worked, fetched my package and its dependencies into ~/tmp. Is it possible that this has been fixed on the server side?

[jijojv]

@lpar is private mode not enabled on your GHE ? - i.e. can you clone WITHOUT any auth - that's why it might work.

[lpar]

Aha, further investigation reveals that a long time ago I put

[url "[email protected]:"]
insteadOf = https://github.mycompany.com/

in my ~/.gitconfig and forgot about it, and that's why it worked.

[dmitris]

we had to use b) or use a redirect server that attach go-import meta-tags. for years. I'd love to see a way for go get to work with GHE "directly" but not sure how the thorny issue of GHE authentication can be worked out: go or vgo would need to be able to pick the "ambient" credentials such as from the ssh-agent, would be very interested in any suggestions or experiences. (I'll subscribe to the issue, would be happy to test if this comes to any changes or PoC).

[cjs]

With the GitHub Enterprise 2.13 release, on-premises GitHub Enterprise instances will now send the <meta> tags expected by go get for private instances and all repositories.

For instances that are in private mode, you will still need to manage authentication for the git pull phase of go get, but there are easy workarounds such as putting an authentication token in your .netrc or the insteadOf technique to force SSH as mentioned above.

[rsc]

Calling this fixed. Thanks so much @cjs.

[matope]

@cjs Our team confirmed that it works. Thank you very much!