Easy to use Progress with multiprocessing module

[jdoiro3]

First off, I wanted to thank everyone involved with developing and maintaining rich. I love it.

I've recently been toying with the idea of extending rich.progress.Progress to to make it easy to report progress for things happening in subprocesses without relying on the fork method or needing the user to create and pass Queues/other objects to a progress_bar callable. I ended up writing multiprogress. I'm not sure of the implementation details but like the interface. Here's a simple example:

import time
import random
from concurrent.futures import ProcessPoolExecutor
from multiprogress import MultiProgress, progress_bar


def do_work(n: int) -> int:
    sleep_for = random.randint(0, 2)
    for _ in progress_bar(
        range(1, n + 2), desc=f"Sleeping for {sleep_for} secs for each {n} iterations."
    ):
        time.sleep(sleep_for)
    return sleep_for


def demo():
    with ProcessPoolExecutor() as p, MultiProgress():
        print(list(p.map(do_work, range(10))))

The idea is do_work could be defined in another module and progress_bar doesn't need to know about MultiProgress, have anything passed to it (e.g., a Queue) or a special subprocess initialization step. You just use it in your function(s) that are called in subprocesses. You just need to make sure before creating child processes you enter a MultiProgress context. The README contains a more complex example where the progress_bar needs to be passed a key that's used when the main process initializes multiple MultiProgress instances. The key lets progress_bar know which MultiProgress it's reporting to.

My question for the team is: does it make sense for something like this to be part of rich or kept as a separate project? I'm leaning toward keeping it separate but wanted to ask.

[willmcgugan]

I'm not accepting new features in Rich now. Maintenance only. But I would be happy to recommend your project as this comes up quite often!