FAQ recommendation for testing dependencies doesn't work

[jsternberg]

This is a general comment about the FAQ. The FAQ has a section about testing a dependency and this recommendation doesn't work:

https://golang.github.io/dep/docs/FAQ.html#how-do-i-test-changes-to-a-dependency

I have two projects. One project depends on another project. Both projects utilize the same dependency. This is pretty common. If you use something like a custom logging type and the logging type is in a third package, both dependencies will likely use it.

For clarity, I'm going to give my specific example using influxdb since that's the primary project I work on. We have a library for the language, influxql. influxdb is locked to require a specific revision of influxql and the enterprise version is intended to inherit the version from influxdb so they depend on the same version without too much of a fuss.

If I want to modify influxdb to test it in the enterprise version before submitting a pull request, the FAQ recommends that I delete vendor/github.com/influxdata/influxdb from the enterprise directory and modify the dependency in the GOPATH instead. When I try this, because both projects use dep, I get a type mismatch because the vendored dependencies of influxdb have now become the arguments that need to be passed into the functions. This is the same reason why dep will strip vendor directories when it is vendoring projects because Go doesn't consider these structs equivalent.

To get to this point, here are the commands I have run:

[influxdb] $ dep ensure
[influxdb-enterprise] $ dep ensure
[influxdb-enterprise] $ rm -r vendor/github.com/influxdata/influxdb
[influxdb-enterprise] $ go install ./cmd/...

So I try to delete the vendor directory from the project on the GOPATH so that it is not using its own vendored dependencies since those would need to be stripped out.

[influxdb] $ rm -r vendor/
[influxdb-enterprise] $ go install ./cmd/...

This still fails because now influxdb isn't using any of the vendored dependencies and it tries to use an old version of influxql that happens to be on my path. This is because the dependencies vendored for enterprise aren't globally used as transitive dependencies for the direct dependencies since the vendor folder only applies to code compiled within the enterprise directory. Since the dependency is no longer vendored inside of enterprise, it can't find the locked versions.

Is there any advice for this type of situation? I imagine that this situation will be common and it seems the only way to deal with it is to manually go to each project and do git checkout on the needed revision since you can't use the vendor folder.

[jsternberg]

As an aside, this workaround seems to be functional. I'm not sure if there are any problems with it since I just tried to compile code with it and didn't do an exhaustive check.

[influxdb-enterprise] $ rm -r vendor/github.com/influxdata/influxdb
[influxdb-enterprise] $ ln -s $GOPATH/src/github.com/influxdata/influxdb vendor/github.com/influxdata/influxdb

[sdboyer]

Ugh. Yeah, that FAQ section is incorrect - that hack only works if the dependency you're moving back to GOPATH has no external dependencies. Sorry about that.

We generally refer to this as the multi-project workflow, and unfortunately, it's not an easy thing in dep's current design. virtualgo might help.

With some of the paths that vgo has laid out, i'm now more comfortable with the idea of adding file:// support in Gopkg.toml source fields (or some equivalent). That would effectively allow you to do this. Not a quick thing to accomplish, though 😢

[jsternberg]

I think there are a few things that dep could do to aid the multi-project workflow. I'm not a big fan of using virtualgo. The project is very similar to virtualenv, which is pretty annoying to use. While you can improve it by using bindfs, that requires a FUSE filesystem and on Mac OS X it requires OSX Fuse. So if we can find some low effort methods of improving this functionality, it would be very helpful.

As mentioned above, you can get this functionality by using a symlink. There are a few problems I have found with using a symlink though.

1. dep ensure will erase it. Luckily, it removes the symlink rather than overwriting the directory it points to which avoids accidentally clobbering a working directory.
2. When manually doing symlinks, accidentally deleting your working directory is common because of the commands used. To add the symlink, I do this as mentioned above:

$ rm -r vendor/github.com/influxdata/influxdb
$ ln -s $GOPATH/src/github.com/influxdata/influxdb vendor/github.com/influxdata/influxdb

If you do this a second time when there is already a symlink there, you accidentally run rm -r on your working directory.

While there's no way to avoid that, we could add a utility command to dep for this common workflow that's more aware than the naive method. We could make it so dep ensure will preserve symlinks and print a warning message and add a -f option to have it "force" to erase symlinks. We could then add a dep link or dep local that looks for the root of a project and creates a symlink in the vendor directory. If the symlink has already been correctly created, then it would do nothing. Then you can add a dep unlink or use dep ensure -f to remove the symlink later so you avoid having the user manually invoke rm -r.

I think these would be relatively simple implementations that could help give a suitable GOPATH-only workflow for doing multi-project workflows.

[cben]

AFAICT, rm -r symlink merely removes the link without recursing inside. But rm -r symlink/, and easy mistake to make esp. with tab completion, does recurse :(. rm (GNU coreutils) 8.30.
Anyway thanks for the warning 🙇‍♂️

[mvdan]

Dep was officially deprecated earlier this year, and the proposal to archive this repository was accepted. As such, I'm closing outstanding issues before archiving the repository. For any further comments, please use the proposal thread on the Go issue tracker. Thanks!