Skip to content

support multiple quorums on the same LighthouseServer #173

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
d4l3k opened this issue Apr 25, 2025 · 0 comments
Open

support multiple quorums on the same LighthouseServer #173

d4l3k opened this issue Apr 25, 2025 · 0 comments
Labels
enhancement New feature or request lighthouse Lighthouse and quorum related

Comments

@d4l3k
Copy link
Member

d4l3k commented Apr 25, 2025

Having the ability to support multiple quorums on the same lighthouse server would make it much easier to deploy torchft in certain scenarios.

With this feature you could deploy a single lighthouse server and then use it for all jobs running in that cluster by using the job ID. This simplifies discovery and would work with most batch job schedulers.

Two possible designs:

1. room_id outside of GRPC

We likely want to make it so you can create a new Lighthouse client with a certain key and it'll automatically isolate the requests to that namespace.

Implementing this cleanly on the server side may be a bit tricky -- we may need to do some manipulation under the hood to instantiate one LighthouseServer instance per incoming request to make this cleaner and avoid polluting the API with "room" ids.

https://github.com/pytorch/torchft/blob/main/src/lighthouse.rs#L601

1. room_id as field on heartbeat + quorum methods

This may be simpler in some ways to implement as we can just add a room_id field to all the lighthouse requests and then internally route as necessary. No magic with GRPC services is required.

@d4l3k d4l3k added enhancement New feature or request lighthouse Lighthouse and quorum related labels Apr 25, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request lighthouse Lighthouse and quorum related
Projects
None yet
Development

No branches or pull requests

1 participant