feature request:replace golang.org lib with github mirror,like glide #1837

bensonfx · 2018-04-28T02:45:03Z

What version of Go (go version) and dep (git describe --tags) are you using??

go version: go version go1.10.2 linux/amd64
dep version: v0.4.1-157-gf9cb443

What dep command did you run?

dep init -v

What did you expect to see?

 init success

What did you see instead?

init failed: unable to solve the dependency graph: Solving failure: 
(1) failed to list versions for https://go.googlesource.com/sys: fatal

description:

for the reason fo GFW，I can go get golang.org libs
So exec "dep init" failed     


I searched the issue,and doc,  and it said using [[constraint]]
but you have to create Gopkg.toml,can dep sure seems do not search the project dependencies

so can add a flag to replace golang.org/x with github?
    like this : dep init -stdlib-mirror.
    if ProjectIdentifier.ProjectRoot is start with golang.org/x,
    replace with github.com/golang, and the new string to ProjectIdentifier.Source.

    so we can download the goalng.org package with alternative url

further more add dep mirror command, to add/remove/list replace rules
    like is : dep mirror -add golang.org/x/${name}:github.com/golang/${name}
    ${name} is placeholder to replace all the golang.org/x projects

The text was updated successfully, but these errors were encountered:

bensonfx · 2018-04-28T08:32:20Z

how about this way
1.add [[mirror]] in Gopkg.toml
2.add flag mirror in dep init,like this
dep init --mirror golang.org/x:github/golang
3.replace all the golang.org package with a the mirror rule
4.generate constraint.source

chentaosheng · 2018-04-29T07:37:22Z

see #1721

chentaosheng · 2018-04-29T07:56:08Z

see #1322
they said: The issue is closed because the source option is not supposed to help you
work around your local machine's network issues.

But I think this is a very bad design. Some company prevent source code leaks. The computer is forbidden to access the Internet.
And some company git server address is 192.168.0.1，but package name is a.com/b/c

bensonfx · 2018-04-29T13:32:19Z

But It is not a "local machine's networkd issues".
The tool's purpose is to resolve the package issue. some package can not download.
and a package with mirrors is very common.so I think it is a problecom.

for the command design.
dep init ---->
why init command do downloading ?? for example. git init and repo init. do not download any code.
move action of downloading packages to dep ensure ??

bensonfx · 2018-05-12T01:07:25Z

@chentaosheng
#1857 PR try to fix this issue

co-kevin · 2018-05-24T03:55:30Z

git clone https://github.com/bensonfx/dep -b golang-package-github-mirror
cd dep
make install

I tested PR. It didn't work.

Solving failure: unable to deduce repository and source type for "golang.org/x/net": unable to read metadata: unable to fetch raw metadata: failed HTTP request to URL "http://golang.org/x/net?go-get=1": Get http://golang.org/x/net?go-get=1: dial tcp 216.239.37.1:80: i/o timeout

Finally use proxy fix it.

$ HTTP_PROXY=<your proxy> dep

Test network below, if you can see same result, so dep should also have access to golang.org.

$ HTTP_PROXY=<your proxy> curl golang.org/x/exp\?go-get\=1   
<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
<meta name="go-import" content="golang.org/x/exp git https://go.googlesource.com/exp">
<meta name="go-source" content="golang.org/x/exp https://github.com/golang/exp/ https://github.com/golang/exp/tree/master{/dir} https://github.com/golang/exp/blob/master{/dir}/{file}#L{line}">
<meta http-equiv="refresh" content="0; url=https://godoc.org/golang.org/x/exp">
</head>
<body>
Nothing to see here; <a href="https://godoc.org/golang.org/x/exp">move along</a>.
</body>
</html>

bensonfx · 2018-05-24T14:46:14Z

@hookszhang use branch golang-package-github-mirror
git clone https://github.com/bensonfx/dep -b golang-package-github-mirror
I'll merge the patch to master

co-kevin · 2018-05-25T03:01:08Z

@bensonfx I forgot to mention that I was using the branch golang-package-github-mirror.

Is it good to work with you?

I noticed that this variable was never used.

// dep/gps/deduce.go line:78
stdlibRegex  = regexp.MustCompile(`^(?P<root>golang\.org/x(/[A-Za-z0-9_.\-]+))((?:/[A-Za-z0-9_.\-]+)*)$`)

bensonfx · 2018-05-25T05:03:59Z

@hookszhang
first, it will use http to get meta from golang.org. if failed.it will add the rule for golang.org.
and then, it will use return githubc mirror.

sometimes, golang.org can be conenected. but go.googlesource.com can not be connected.
in that case. the golang mirror rule will not be added.

in your case, golang.org mirror rule should be added.
add a log the in below . and try again.

source_manage.go

co-kevin · 2018-05-25T06:42:24Z

The logs have been added as below.

// gps/source_manager.go line:543
func (sm *SourceMgr) DeduceProjectRoot(ip string) (ProjectRoot, error) {
	if atomic.LoadInt32(&sm.releasing) == 1 {
		return "", ErrSourceManagerIsReleased
	}

	// TODO(sdboyer) refactor deduceRootPath() so that this validation can move
	// back down below a cache point, rather than executing on every call.
	if !pathvld.MatchString(ip) {
		return "", errors.Errorf("%q is not a valid import path", ip)
	}

	pd, err := sm.deduceCoord.deduceRootPath(context.TODO(), ip)

	if err != nil {
		log.Printf("[debug] ip: %s, err: %s", ip, err.Error())
	}

	// as we can not deduce root path from golang.org,
	// try github mirror and add golang.org/x rules to Trie tree
	if err != nil && strings.HasPrefix(ip, stdlibPrefix) {
		log.Printf("[debug] ip: %s, has prefix: %s", ip, stdlibPrefix)
		sm.deduceCoord.deducext.Insert(stdlibPrefix, stdlibDeducer{regexp: stdlibRegex})
		pd, err = sm.deduceCoord.deduceKnownPaths(ip)
		if err != nil {
			log.Printf("[debug] err: %s", err.Error())
		}
	}
	return ProjectRoot(pd.root), err
}

Retry, i got this:

$ dep ensure -add golang.org/x/image
Fetching sources...
2018/05/25 14:34:34 [debug] ip: golang.org/x/image, err: unable to deduce repository and source type for "golang.org/x/image": unable to read metadata: unable to fetch raw metadata: failed HTTP request to URL "http://golang.org/x/image?go-get=1": Get http://golang.org/x/image?go-get=1: dial tcp 216.239.37.1:80: i/o timeout
2018/05/25 14:34:34 [debug] ip: golang.org/x/image, has prefix: golang.org/x

Failed to add the dependencies:

  ✗ failed to fetch source for golang.org/x/image: unable to deduce repository and source type for "golang.org/x/image": unable to read metadata: unable to fetch raw metadata: failed HTTP request to URL "http://golang.org/x/image?go-get=1": Get http://golang.og/x/image?go-get=1: dial tcp 216.239.37.1:80: i/o timeout

adding dependencies failed

One more question, dep ensure -v throws error and no print debug log

(19/22) Failed to write golang.org/x/sys@master
(20/22) Failed to write golang.org/x/net@master
(21/22) Failed to write golang.org/x/sync@master
(22/22) Failed to write golang.org/x/crypto@master
grouped write of manifest, lock and vendor: error while writing out vendor tree: failed to write dep tree: failed to export golang.org/x/sys: unable to deduce repository and source type for "golang.org/x/sys": unable to read metadata: unable to fetch raw metadata: failed HTTP request to URL "http://golang.org/x/sys?go-get=1": Get http://golang.org/x/sys?go-get=1: dial tcp 216.239.37.1:80: i/o timeout

I think we should use the proxy, by the way GTMD GFW!

bensonfx · 2018-05-25T07:34:54Z

weird
maybe the problem is the flag add.
let me check.maybe should add the flag.

the way using proxy is not convenient.

tedli · 2018-08-22T09:57:28Z

Hi, I forked dep my fork, and add a -proxyurl argument what can let dep fetch dependencies through the specified porxy.

And for now on, my fork just support a global proxy, all network traffic (only go-get=1 and git) forwarded by using the proxy.

Futher, I think I can add a config file, something like a ~/.depconfig for global config, and extra fields in Gopkg.toml for project config.

bensonfx closed this as completed Apr 28, 2018

bensonfx reopened this Apr 28, 2018

bensonfx mentioned this issue May 12, 2018

use github mirror for golang.org packages when getting golang.org http meta failed #1857

Closed

bensonfx closed this as completed Feb 21, 2020

prietyc123 mentioned this issue Mar 19, 2020

making package testingutil and ui work for unit test build on windows redhat-developer/odo#2630

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feature request:replace golang.org lib with github mirror,like glide #1837

feature request:replace golang.org lib with github mirror,like glide #1837

bensonfx commented Apr 28, 2018 •

edited

Loading

bensonfx commented Apr 28, 2018

chentaosheng commented Apr 29, 2018

chentaosheng commented Apr 29, 2018

bensonfx commented Apr 29, 2018

bensonfx commented May 12, 2018 •

edited

Loading

co-kevin commented May 24, 2018 •

edited

Loading

bensonfx commented May 24, 2018 •

edited

Loading

co-kevin commented May 25, 2018

bensonfx commented May 25, 2018 •

edited

Loading

co-kevin commented May 25, 2018 •

edited

Loading

bensonfx commented May 25, 2018 •

edited

Loading

tedli commented Aug 22, 2018

feature request:replace golang.org lib with github mirror,like glide #1837

feature request:replace golang.org lib with github mirror,like glide #1837

Comments

bensonfx commented Apr 28, 2018 • edited Loading

What version of Go (go version) and dep (git describe --tags) are you using??

What dep command did you run?

What did you expect to see?

What did you see instead?

description:

bensonfx commented Apr 28, 2018

chentaosheng commented Apr 29, 2018

chentaosheng commented Apr 29, 2018

bensonfx commented Apr 29, 2018

bensonfx commented May 12, 2018 • edited Loading

co-kevin commented May 24, 2018 • edited Loading

bensonfx commented May 24, 2018 • edited Loading

co-kevin commented May 25, 2018

bensonfx commented May 25, 2018 • edited Loading

co-kevin commented May 25, 2018 • edited Loading

bensonfx commented May 25, 2018 • edited Loading

tedli commented Aug 22, 2018

bensonfx commented Apr 28, 2018 •

edited

Loading

bensonfx commented May 12, 2018 •

edited

Loading

co-kevin commented May 24, 2018 •

edited

Loading

bensonfx commented May 24, 2018 •

edited

Loading

bensonfx commented May 25, 2018 •

edited

Loading

co-kevin commented May 25, 2018 •

edited

Loading

bensonfx commented May 25, 2018 •

edited

Loading