Use yarn instead of npm

[epixa]

Facebook just released yarn, which offers some key benefits over npm that aim to solve problems we're currently struggling with. Specifically:

• It's deterministic
• It uses diffable lock files to fix dependencies instead of shrinkwrap
• It has first-class support for cached dependencies, so we don't have to rely on the npm registry itself for builds
• It verifies checksums, so we know whether the installed dependencies are exactly what they're suppose to be

We can/would continue to use npm as our download registry, so if we ever had issues with yarn, switching back to npm would be a minor task.

[w33ble]

$ time npm install
...
real 1m0.739s
user 0m58.002s
sys 0m19.500s

$ du -sh node_modules
141M node_modules

---

$ time yarn
...
real 0m29.598s
user 0m23.574s
sys 0m14.129s

$ du -sh node_modules
129M node_modules

That time drops to about 11s once the cache is populated. Not bad!

[spalger]

We talked about this collectively and decided that yarn makes sense for installs, but there are a few issues that will need to be resovled:

• Installing directories deletes scoped dependencies - Installing directories deletes scoped dependencies yarnpkg/yarn#991
• @bigfunger/decompress-zip is a scoped module with a bin, which breaks install - Yarn install fails on scoped dependency with bin attr in package.json yarnpkg/yarn#759
• optional - yarn run doesn't work - yarn run fails when options specified yarnpkg/yarn#1156
• saved exact default - Respect save-exact option from .npmrc in project directory yarnpkg/yarn#1088 (talked with @tylersmalley and we agreed that the yarn.lock file is a better way of accomplishing what we tried to accomplish with exact versions)
• yarn is painfully slow on windows - Need a big optimization for Windows yarnpkg/yarn#990

[tylersmalley]

yarn run fixed in v0.16

[tylersmalley]

We should also look into useing --cache-folder to persist the tar'd packages to the repository. This should alleviate some of the pains of committing node_modules to the repository with the benefits of actually doing so.

[w33ble]

We should also look into useing --cache-folder to persist the tar'd packages to the repository

I thought the lock file was supposed to be enough to keep things consistent. What does checking in the tars give us, besides not having to download all that stuff every time?

[tylersmalley]

By lock file do you mean the shrinkwrap? We're currently pinning packages to specific versions in the package.json which I don't necessarily feel is the correct approach.

By committing the packages, it would make it much easier and faster to switch branches. We could also go back to using package version ranges.

[w33ble]

I meant the yarn.lock file, since this is about using yarn. I'm guessing we're going to commit that file as well, otherwise there's little point. As I understand it, that lock file would effectively allow us to go back to version ranges, but would effectively make yarn pin the versions for us based on when then committed lock file was created.

By committing the packages, it would make it much easier and faster to switch branches. We could also go back to using package version ranges.

The lock files gives us the later.

Would users still need to run yarn [install] to get package changes? I think the answer is yes, but I'm pretty sure you're right that they'd get all the package updates much, much faster since there's nothing to actually download, yarn would just re-link things. Is that the main perk then, faster switching between module versions in various branches?

[spalger]

Is that the main perk then, faster switching between module versions in various branches?

It will trim 1+ minutes off of every CI build

[w33ble]

It will trim 1+ minutes off of every CI build

Compared to npm3, or compared to yarn without the checked in cache-folder?

[spalger]

npm3, might not be a whole lot quicker than a warmed up yarn cache

[epixa]

The one thing I can think of that it gives us is guaranteed consistency of the dependency source code. No pulled dependencies (e.g. leftpad), for example.

In any case, I think we should probably treat that as a separate issue to pulling yarn in for the sake of locking. It's an additional feature we'd have at our disposal at least.

[w33ble]

The one thing I can think of that it gives us is guaranteed consistency of the dependency source code.

This would protect us from packages disappearing from their source, npm or otherwise. As for consistency, I thought that the lock file gave us that as well. Isn't that what all the checksumming stuff does?

Note that I'm not trying to be contrary here, I don't really have an issue checking in the cache folder. I'm genuinely curious about this stuff.

[epixa]

@w33ble Yeah, you're right about consistency there. Yarn's lock file would catch a change in source code for a given version, so it can guarantee consistency in that way, but at best it can really just error when something is wrong. The cache would make us more resilient to that sort of problem by relying on the original source when we first get it set up.

I'm not 100% sold on checking in the deps for what it's worth, so we're both just trying to make the situation clear.

[tylersmalley]

@spalger, we no longer use @bigfunger/decompress-zip so that can be checked off.

[Mpdreamz]

yarn on windows should be faster too now: yarnpkg/yarn#3234

[epixa]

I reverted this PR, so I'm reopening this. See #11637 (comment)