API experiments system

[jasonkuhrt]

What

• A way to have unstable APIs in stable releases and have developers consuming this know the difference, with zero or almost zero additional cognitive load

Why

• We want to explore topics like Server System #295 without going through long-lived PRs
• We want to practice agile which means we need to incrementally ship
• But incrementally shipping can go against great DX because results over time can be jagged, not seamless, inconsistent concepts or API style, etc.
• The solution is a way to express the difference between experimental and non-experimental APIs

Sketches

• We need a way for users to opt into experiments

   // app.ts
   import { app } from 'graphql-santa'
   
   app.settings.experiments.enable('foo', 'bar', 'qux')
   // or
   app.enableExperiment('foo')
   app.enableExperiment('foo')
   app.enableExperiment('qux')
   // or
   import { experiments } from 'graphql-santa'
   experiments.enable('...')
   // or
   // ...

• There should be experiment support for worktime too, not just runtime... could defer this for another issue, e.g. make this one about "runtime experiments" and then another issue for "worktime experiments".

• We need a way to mark functions as experimental:

   const experimentsLogger = logger.child('experiments')
  
   function experiment<F extends Function>(name: string, f: F): F {
   	return function (...args) {
   		if (project.settings.experiments[name].enabled) {
   			return f(...args)
   		} else {
   			if (project.settings.stage === 'production') {
   				throw new Error('...')
   			} else {
   				experimentLogger
   					.warning('not_enrolled', { experiment: name })
   					.pretty(`This is an experimental API. You are not allowed to use it until you explicitly opt-in. If you do not then your app will be force-exited in production. Please either stop using this API or opt-in. To opt-in add this to your settings:\n\n{ "experiments": { "${name}": true } }`)
   				f(...args)
   			}
   		}
   	}
   }

• In the above:

  • somehow getting a reference to the user's project settings
  • warnings in dev, fatal in prod
  • nice logger integration

• What about if experiments did not even show up in the API until being opted into?

  • dynamically set things to undefined
  • run typegen (this is the harder part)
  • is this simpler for the user?
  • By doing this we can add lots of experiments and not pollute the autocomplete for everyone
  • By doing this it makes trying experiments a little more annoying. Trying anything requires first opting in. With a warn-based approach user can try at will, deferring the configurations until go-to-prod time. In other words a warn approach is more gradual.
  • We could mitigate the criticism of autocomplete pollution by having a flag to enable experiments at all or not, if totally off, then typegen is used to not pollute autocomplete.

[Weakky]

I really, really like the approach where we mark experimental features and warn in dev. Again, I'd stay away from typegen as much as possible.

I also like your top-level experiments component. Makes me wonder whether experiments should live in the settings or live there. TBD.