Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ENH: Serialize+parallelize 4D apply() into 3D+t and add 'low memory' loading #215

Merged
merged 24 commits into from
Aug 10, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
b922fa5
wip: initiate implementation
oesteban Jul 23, 2024
6064b8c
enh: draft implementation of serialize 4d
Jul 24, 2024
e47a476
fix: passes more tests, more suggestions in progress
Jul 25, 2024
1616a35
fix: pass tests
Jul 25, 2024
6292daf
fix: pass tests, serialization implemented
Jul 25, 2024
86b3d11
wip: initiate implementation
oesteban Jul 23, 2024
23daabb
Merge remote-tracking branch 'jmarabotto/patch/oesteban-pr' into enh/…
oesteban Jul 29, 2024
79e5cad
enh: integrating @jmarabotto's code
oesteban Jul 30, 2024
e0bde09
fix: ensure output dtype when resampling
oesteban Jul 30, 2024
fbb0451
fix: resolve some failing tests
oesteban Jul 30, 2024
0153472
fix: ensure ``__len__`` is defined for all transforms``
oesteban Jul 31, 2024
85d03b4
fix: clarify what is a 3D transform chain and a 4D transform
oesteban Jul 31, 2024
06a1c01
maint: reorganize tests around the spun-off apply
oesteban Jul 31, 2024
8dd883d
sty: format changed files
oesteban Jul 31, 2024
9f91e2f
Merge remote-tracking branch 'upstream/master' into enh/reenable-para…
oesteban Jul 31, 2024
4c06174
enh: expand test coverage
oesteban Aug 1, 2024
754785f
enh: prepare code for easy parallelization with a process pool executor
oesteban Jul 31, 2024
38bb388
enh: create process pool
oesteban Jul 31, 2024
8ba34c9
Merge pull request #220 from nipy/enh/reenable-parallelization-apply-…
oesteban Aug 1, 2024
026a10a
enh: expand test coverage
oesteban Aug 1, 2024
7dcc78d
sty: add type annotations
oesteban Aug 1, 2024
79305a9
enh: implement a memory limitation mechanism in loading data
oesteban Aug 2, 2024
7c7608f
Merge pull request #221 from nipy/enh/reenable-parallelization-apply-…
oesteban Aug 2, 2024
063e1f0
enh: port from process pool into asyncio concurrent
oesteban Aug 6, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 23 additions & 8 deletions nitransforms/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
#
### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
"""Common interface for transforms."""

from pathlib import Path
import numpy as np
import h5py
Expand Down Expand Up @@ -146,13 +147,13 @@ def from_arrays(cls, coordinates, triangles):
darrays = [
nb.gifti.GiftiDataArray(
coordinates.astype(np.float32),
intent=nb.nifti1.intent_codes['NIFTI_INTENT_POINTSET'],
datatype=nb.nifti1.data_type_codes['NIFTI_TYPE_FLOAT32'],
intent=nb.nifti1.intent_codes["NIFTI_INTENT_POINTSET"],
datatype=nb.nifti1.data_type_codes["NIFTI_TYPE_FLOAT32"],
),
nb.gifti.GiftiDataArray(
triangles.astype(np.int32),
intent=nb.nifti1.intent_codes['NIFTI_INTENT_TRIANGLE'],
datatype=nb.nifti1.data_type_codes['NIFTI_TYPE_INT32'],
intent=nb.nifti1.intent_codes["NIFTI_INTENT_TRIANGLE"],
datatype=nb.nifti1.data_type_codes["NIFTI_TYPE_INT32"],
),
]
gii = nb.gifti.GiftiImage(darrays=darrays)
Expand Down Expand Up @@ -279,6 +280,22 @@ def __add__(self, b):

return TransformChain(transforms=[self, b])

def __len__(self):
"""
Enable ``len()``.

By default, all transforms are of length one.
This must be overriden by transforms arrays and chains.

Example
-------
>>> T1 = TransformBase()
>>> len(T1)
1

"""
return 1

@property
def reference(self):
"""Access a reference space where data will be resampled onto."""
Expand Down Expand Up @@ -335,10 +352,8 @@ def apply(self, *args, **kwargs):

Deprecated. Please use ``nitransforms.resampling.apply`` instead.
"""
message = (
"The `apply` method is deprecated. Please use `nitransforms.resampling.apply` instead."
)
warnings.warn(message, DeprecationWarning, stacklevel=2)
_msg = "This method is deprecated. Please use `nitransforms.resampling.apply` instead."
warnings.warn(_msg, DeprecationWarning, stacklevel=2)
from .resampling import apply

return apply(self, *args, **kwargs)
Expand Down
Loading
Loading