Support switching of METAFLOW_PROFILE on the fly in the client

[tuulos]

currently, Metaflow client loads configuration based on the METAFLOW_PROFILE env var. It is not straightforward to switch between different profiles e.g. in a notebook, as it requires a hard kernel restart to reload a config with a new profile

one can work around the issue with a snippet like this

import os
from multiprocessing import Pool

def latest_run(flow_name):
    from metaflow import Flow
    return Flow(flow_name).latest_run.pathspec

def call(op, args, profile=None):
    os.environ['METAFLOW_PROFILE'] = profile
    with Pool(processes=1) as pool:
        return pool.apply(op, args)

if __name__ == '__main__':
    print(call(latest_run, ('HelloFlow',), profile='local'))
    print(call(latest_run, ('HelloFlow',), profile='mozart'))

but it'd be useful to have support for multiple profiles natively

discussion thread