Pkg.add race condition

[jiahao]

Somewhat perverse, but if you Pkg.add in one Julia instance and while that is going, start another Julia instance and Pkg.add something else, it deletes all the previously pulled packages from the first instance.

Example:

julia> Pkg.add("GSL") #Instance 1
INFO: Cloning cache of BinDeps from git://github.com/JuliaLang/BinDeps.jl.git
INFO: Cloning cache of GSL from git://github.com/jiahao/GSL.jl.git
INFO: Cloning cache of Homebrew from git://github.com/JuliaLang/Homebrew.jl.git
INFO: Cloning cache of URIParser from git://github.com/loladiro/URIParser.jl.git
INFO: Installing BinDeps v0.2.12
INFO: Installing GSL v0.1.1
INFO: Installing Homebrew v0.0.4
INFO: Installing URIParser v0.0.1
INFO: Building Homebrew
INFO: Cloning brew from https://github.com/staticfloat/homebrew.git
... (builds)

julia> Pkg.add("Example") #Instance 2
INFO: Cloning cache of Example from git://github.com/JuliaLang/Example.jl.git
INFO: Installing Example v0.0.0
INFO: Removing BinDeps v0.2.12
INFO: Removing GSL v0.1.1
INFO: Removing Homebrew v0.0.4
INFO: Removing URIParser v0.0.1
INFO: Package database updated

(GSL build in Instance 1 dies horribly a short while later)

[JeffBezanson]

I don't think we intended to guard against this.

[mlubin]

Given that Pkg commands are not quite instantaneous, I think this is a relatively common issue that users could run into.

[StefanKarpinski]

We should probably use file locking to avoid this sort of race condition. Until now, I've just let it go, but obviously it would be better to handle it correctly. @loladiro, @vtjnash – what's the status of file locking APIs that libuv gives us? Do we have a single API that will work across platforms or do we need to implement that ourselves?

[staticfloat]

Doesn't git already solve this for us? If I try to git commit something while I'm already git commit'ing something, it fails, giving me this error:

$ git commit -a
fatal: Unable to create '/tmp/repo/.git/index.lock': File exists.

Would it not be simpler to roll this into Pkg once we're using libgit2 directly?

[jakebolewski]

@staticfloat you are right. libgit2 does implement a cross platform locking mechanism
(see https://github.com/libgit2/libgit2/blob/79194bcdc956406979cd27ac99198826860d3f20/src/fileops.c#L66) which uses atomic symlinks. I'll update LibGit2.jl to incoporate this so we can ultimately leverage this in Pkg.

[malmaud]

I'm speculatively removing the parallel tag since this doesn't seem to concern parallel computation.