Deduplicate the lab and retro assets

[jtpio]

Problem

Tracking the following comment from https://github.com/jtpio/jupyterlite/issues/16#issuecomment-821980005 in its own issue:

Further while talking about release assets... Right now, the release is
about 14mb, 64 uncompressed. A huge proportion of that is the duplication
of large assets in lab/build and classic/build. It's probably worth
exploring how to get as much as possible to end up in ../static, relative
to each of them.

Proposed Solution

• Put all assets in a common /static folder
• Rename index.js, style.js and bundle.js to index-lab.js, index-retro.js ... so they don't collide
• Update the references to ./build, bundle.js and others

Additional context

[bollwyvl]

Thanks for starting this. Some additional wrinkles on this:

• might block MVP
  • when this gets tackled, we gotta get cache-busting in there
    • the top-level can be handled by index.js?v=<date>
    • but subsequent requests below it need to "just work" on redeployment update, or there's gonna be a lot of really confused users
• should not block MVP:
  • re Integrate into existing Jupyter-based workflows #41, it would be ideal if there was a solid way to extract which static are needed for which app, such that jupyter lite build --app @jupyterlite/retro-lite ./my-site only ships the assets needed, while adding another --app would only increase the size by the delta of missing packages, without having to know index-this from index-that
  • have a path (if "fork it and..." in the docs, initially, so be it) to shipping (only) that novel third application (e.g. CustomLabLite). This is a case where, unless we unbundle everything into federated modules, someone will still need all the nodejs machinery, no real way around it at present

[bollwyvl]

So i got a bit hung up on how we would refactor the various scripts to not make use of data... and the interplay between bootstrap.js and the sharing piece, as they will both actually be on the page at the same time, while multiple apps will not be on the page at the same time... but might share worker code, etc. So I'll probably take a look at some other stuff before re-engaging on this, in case someone else has a more concrete plan forward 🙏 !

As we think about #104, I guess there's a latent refactor in the baseline of @jupyterlab/builder (though maybe we land it here first)... when it gets to the good stuff, might we have a pattern that supported

serverliteExtensions -> package.json#/jupyterlite/???

or, more generally

whateverExtensions => package.json#/???/???

[bollwyvl]

Thinking about this more: what if we just made everything into federated extensions?

Pro:

• site owners
  • improve debuggability
  • allow swapping out specific packages
• maintainers
  • allow more parallel builds, and rebuild less on each incremental change
  • even if not parallel, webpack time/memory/open files increase faster-than-linearly

Con:

• end users
  • more initial downloads (but not that many more)... hard to assess vs HTTP/2 etc.
• maintainers
  • might need to get more clever with our understanding of inter-package dependencies for the CLI
  • most of these core features haven't been tested as federated extensions, and may reveal extra gotchas

[jtpio]

what if we just made everything into federated extensions?

That sounds very tempting 👍

Probably up until now there was some assumptions on what the base apps like JupyterLab would offer. But in the end they too are also just a set of extensions and probably should not have better treatment than the others.

[jtpio]

Also somehow related: it would be nice if we could improve the way the apps are built so we don't duplicate efforts. For example when we make a change to RetroLab we wouldn't need to update lite too if the change is compatible (for instance when adding a new plugin).

Example PRs for reference: #287 and jupyterlab/retrolab#187

Ideally, we would mostly work at the plugin level, with disabledExtensions and a list of extra plugins. And hopefully most of the boilerplate could be hidden somewhere. Although this might require quite a bit of refactoring.

[bollwyvl]

With #274, we are now using pyodide 0.18, which also ships an officially-maintaiend npm package.

Semi-luckily, this ~100kb package is not a full distribution of all the WASM packages, which still needs to be handled out-of-band and match the package exactly, but does provide full typescript typings... and further includes typings from emscripten.

However, as this package is not exactly small, it will exacerbate our on-going OoM/too-many-files issues, but is probably pretty important to move forward.

Since we'll have better metadata about what version of pyodide we're using, it's would probably be time to take the first step towards #45... i'm imagining we'll start a pyolite section in jupyter_lite_config.json and offer something like use_local_pydodide: True (or a path to an equivalent tarball) and deploy it someplace predictable during the build. It will, however, be entirely impractical to load it n times for multiple apps.