Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow partial processing of read datasets #63

Closed
johnlees opened this issue Jun 22, 2021 · 1 comment · Fixed by #64
Closed

Allow partial processing of read datasets #63

johnlees opened this issue Jun 22, 2021 · 1 comment · Fixed by #64
Assignees
Labels
enhancement New feature or request

Comments

@johnlees
Copy link
Member

For large read datasets, their size (especially plus a large countmin table) may exceed device memory. However they can happily be loaded (asynchronously) in blocks, as each read is independent.

This change will require:

  1. Remove the read interleaving, and loading into shared. A sync block load can be used to load all the reads for a warp into shared directly and more simply.
  2. Pin the host memory in the DeviceReads class
  3. Add a loadBlock method to DeviceReads which loads up to a specified size using memcpy async on a non-default stream
  4. Launch the kernel on a second non-default stream.
  5. Iterate through loadBlock, waiting on kernel completion.

The block size will be (device memory - size of countmin table) / 2 - epsilon

@johnlees johnlees added the enhancement New feature or request label Jun 22, 2021
@johnlees johnlees self-assigned this Jun 22, 2021
@johnlees
Copy link
Member Author

Iterating over k-mers is a bit irritating when this is used. If the buffer loading is only to be done once, then signs for all k-mers need to be maintained on the device (fine) as well as the countmin table (not fine). To start with, let's just try keeping the iteration as-is, and loading the buffer on and off for every k-mer length and see how that copes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant