The DArray, or "distributed array", is an abstraction layer on top of Dagger
that allows loading array-like structures into a distributed environment. The
DArray partitions a larger array into smaller "blocks" or "chunks", and those
blocks may be located on any worker in the cluster. The DArray uses a
Parallel Global Address Space (aka "PGAS") model for storing partitions, which
means that a DArray instance contains a reference to every partition in the
greater array; this provides great flexibility in allowing Dagger to choose the
most efficient way to distribute the array's blocks and operate on them in a
heterogeneous manner.
Aside: an alternative model, here termed the "MPI" model, is not yet supported,
but would allow storing only a single partition of the array on each MPI rank
in an MPI cluster. DArray support for this model is planned in the near
future.
This should not be confused with the DistributedArrays.jl package.
A DArray can be created in two ways: through an API similar to the usual
rand, ones, etc. calls, or by distributing an existing array with
DArray, DVector, DMatrix, or distribute.
As an example, one can allocate a random DArray by calling rand with a
Blocks object as the first argument - Blocks specifies the size of
partitions to be constructed, and must be the same number of dimensions as the
array being allocated.
# Add some Julia workers
julia> using Distributed; addprocs(6)
6-element Vector{Int64}:
2
3
4
5
6
7
julia> @everywhere using Dagger
julia> DX = rand(Blocks(50, 50), 100, 100)
DMatrix{Float64}(100, 100) with 2x2 partitions of size 50x50:
0.610404 0.0475367 0.809016 0.311305 0.0306211 0.689645 … 0.220267 0.678548 0.892062 0.0559988
0.680815 0.788349 0.758755 0.0594709 0.640167 0.652266 0.331429 0.798848 0.732432 0.579534
0.306898 0.0805607 0.498372 0.887971 0.244104 0.148825 0.340429 0.029274 0.140624 0.292354
0.0537622 0.844509 0.509145 0.561629 0.566584 0.498554 0.427503 0.835242 0.699405 0.0705192
0.587364 0.59933 0.0624318 0.3795 0.430398 0.0853735 0.379947 0.677105 0.0305861 0.748001
0.14129 0.635562 0.218739 0.0629501 0.373841 0.439933 … 0.308294 0.0966736 0.783333 0.00763648
0.14539 0.331767 0.912498 0.0649541 0.527064 0.249595 0.826705 0.826868 0.41398 0.80321
0.13926 0.353158 0.330615 0.438247 0.284794 0.238837 0.791249 0.415801 0.729545 0.88308
0.769242 0.136001 0.950214 0.171962 0.183646 0.78294 0.570442 0.321894 0.293101 0.911913
0.786168 0.513057 0.781712 0.0191752 0.512821 0.621239 0.50503 0.0472064 0.0368674 0.75981
0.493378 0.129937 0.758052 0.169508 0.0564534 0.846092 … 0.873186 0.396222 0.284 0.0242124
0.12689 0.194842 0.263186 0.213071 0.535613 0.246888 0.579931 0.699231 0.441449 0.882772
0.916144 0.21305 0.629293 0.329303 0.299889 0.127453 0.644012 0.311241 0.713782 0.0554386
⋮ ⋮ ⋱
0.430369 0.597251 0.552528 0.795223 0.46431 0.777119 0.189266 0.499178 0.715808 0.797629
0.235668 0.902973 0.786537 0.951402 0.768312 0.633666 0.724196 0.866373 0.0679498 0.255039
0.605097 0.301349 0.758283 0.681568 0.677913 0.51507 … 0.654614 0.37841 0.86399 0.583924
0.824216 0.62188 0.369671 0.725758 0.735141 0.183666 0.0401394 0.522191 0.849429 0.839651
0.578047 0.775035 0.704695 0.203515 0.00267523 0.869083 0.0975535 0.824887 0.00787017 0.920944
0.805897 0.0275489 0.175715 0.135956 0.389958 0.856349 0.974141 0.586308 0.59695 0.906727
0.212875 0.509612 0.85531 0.266659 0.0695836 0.0551129 0.788085 0.401581 0.948216 0.00242077
0.512997 0.134833 0.895968 0.996953 0.422192 0.991526 … 0.838781 0.141053 0.747722 0.84489
0.283221 0.995152 0.61636 0.75955 0.072718 0.691665 0.151339 0.295759 0.795476 0.203072
0.0946639 0.496832 0.551496 0.848571 0.151074 0.625696 0.673817 0.273958 0.177998 0.563221
0.0900806 0.127274 0.394169 0.140403 0.232985 0.460306 0.536441 0.200297 0.970311 0.0292218
0.0698985 0.463532 0.934776 0.448393 0.606287 0.552196 0.883694 0.212222 0.888415 0.941097The rand(Blocks(50, 50), 100, 100) call specifies that a DMatrix (a DArray matrix)
should be allocated which is in total 100 x 100, split into 4 blocks of size 50
x 50, and initialized with random Float64s. Many other functions, like
randn, sprand, ones, and zeros can be called in this same way.
Alternatively, instead of manually specifying the block size, one can call
rand with an AutoBlocks object to have Dagger automatically choose a block
size:
julia> DX = rand(AutoBlocks(), 100, 100)
DMatrix{Float64}(100, 100) with 1x7 partitions of size 100x15:
0.610404 0.0475367 0.809016 0.311305 0.0306211 0.689645 … 0.220267 0.678548 0.892062 0.0559988
0.680815 0.788349 0.758755 0.0594709 0.640167 0.652266 0.331429 0.798848 0.732432 0.579534
0.306898 0.0805607 0.498372 0.887971 0.244104 0.148825 0.340429 0.029274 0.140624 0.292354
0.0537622 0.844509 0.509145 0.561629 0.566584 0.498554 0.427503 0.835242 0.699405 0.0705192
0.587364 0.59933 0.0624318 0.3795 0.430398 0.0853735 0.379947 0.677105 0.0305861 0.748001
0.14129 0.635562 0.218739 0.0629501 0.373841 0.439933 … 0.308294 0.0966736 0.783333 0.00763648
0.14539 0.331767 0.912498 0.0649541 0.527064 0.249595 0.826705 0.826868 0.41398 0.80321
0.13926 0.353158 0.330615 0.438247 0.284794 0.238837 0.791249 0.415801 0.729545 0.88308
0.769242 0.136001 0.950214 0.171962 0.183646 0.78294 0.570442 0.321894 0.293101 0.911913
0.786168 0.513057 0.781712 0.0191752 0.512821 0.621239 0.50503 0.0472064 0.0368674 0.75981
0.493378 0.129937 0.758052 0.169508 0.0564534 0.846092 … 0.873186 0.396222 0.284 0.0242124
0.12689 0.194842 0.263186 0.213071 0.535613 0.246888 0.579931 0.699231 0.441449 0.882772
0.916144 0.21305 0.629293 0.329303 0.299889 0.127453 0.644012 0.311241 0.713782 0.0554386
⋮ ⋮ ⋱
0.430369 0.597251 0.552528 0.795223 0.46431 0.777119 0.189266 0.499178 0.715808 0.797629
0.235668 0.902973 0.786537 0.951402 0.768312 0.633666 0.724196 0.866373 0.0679498 0.255039
0.605097 0.301349 0.758283 0.681568 0.677913 0.51507 … 0.654614 0.37841 0.86399 0.583924
0.824216 0.62188 0.369671 0.725758 0.735141 0.183666 0.0401394 0.522191 0.849429 0.839651
0.578047 0.775035 0.704695 0.203515 0.00267523 0.869083 0.0975535 0.824887 0.00787017 0.920944
0.805897 0.0275489 0.175715 0.135956 0.389958 0.856349 0.974141 0.586308 0.59695 0.906727
0.212875 0.509612 0.85531 0.266659 0.0695836 0.0551129 0.788085 0.401581 0.948216 0.00242077
0.512997 0.134833 0.895968 0.996953 0.422192 0.991526 … 0.838781 0.141053 0.747722 0.84489
0.283221 0.995152 0.61636 0.75955 0.072718 0.691665 0.151339 0.295759 0.795476 0.203072
0.0946639 0.496832 0.551496 0.848571 0.151074 0.625696 0.673817 0.273958 0.177998 0.563221
0.0900806 0.127274 0.394169 0.140403 0.232985 0.460306 0.536441 0.200297 0.970311 0.0292218
0.0698985 0.463532 0.934776 0.448393 0.606287 0.552196 0.883694 0.212222 0.888415 0.941097We can see above that the DMatrix was partitioned into 7 partitions, each of
a maximum size of 100 x 15. Dagger will automatically partition DArray
objects into as many partitions as there are Dagger processors, to optimize for
parallelism.
Note that the DArray is an asynchronous object (i.e. operations on it may
execute in the background), so to force it to be materialized, fetch may need
to be called:
julia> fetch(DX)
DMatrix{Float64}(100, 100) with 1x7 partitions of size 100x15:
0.610404 0.0475367 0.809016 0.311305 0.0306211 0.689645 … 0.220267 0.678548 0.892062 0.0559988
0.680815 0.788349 0.758755 0.0594709 0.640167 0.652266 0.331429 0.798848 0.732432 0.579534
0.306898 0.0805607 0.498372 0.887971 0.244104 0.148825 0.340429 0.029274 0.140624 0.292354
0.0537622 0.844509 0.509145 0.561629 0.566584 0.498554 0.427503 0.835242 0.699405 0.0705192
0.587364 0.59933 0.0624318 0.3795 0.430398 0.0853735 0.379947 0.677105 0.0305861 0.748001
0.14129 0.635562 0.218739 0.0629501 0.373841 0.439933 … 0.308294 0.0966736 0.783333 0.00763648
0.14539 0.331767 0.912498 0.0649541 0.527064 0.249595 0.826705 0.826868 0.41398 0.80321
0.13926 0.353158 0.330615 0.438247 0.284794 0.238837 0.791249 0.415801 0.729545 0.88308
0.769242 0.136001 0.950214 0.171962 0.183646 0.78294 0.570442 0.321894 0.293101 0.911913
0.786168 0.513057 0.781712 0.0191752 0.512821 0.621239 0.50503 0.0472064 0.0368674 0.75981
0.493378 0.129937 0.758052 0.169508 0.0564534 0.846092 … 0.873186 0.396222 0.284 0.0242124
0.12689 0.194842 0.263186 0.213071 0.535613 0.246888 0.579931 0.699231 0.441449 0.882772
0.916144 0.21305 0.629293 0.329303 0.299889 0.127453 0.644012 0.311241 0.713782 0.0554386
⋮ ⋮ ⋱
0.430369 0.597251 0.552528 0.795223 0.46431 0.777119 0.189266 0.499178 0.715808 0.797629
0.235668 0.902973 0.786537 0.951402 0.768312 0.633666 0.724196 0.866373 0.0679498 0.255039
0.605097 0.301349 0.758283 0.681568 0.677913 0.51507 … 0.654614 0.37841 0.86399 0.583924
0.824216 0.62188 0.369671 0.725758 0.735141 0.183666 0.0401394 0.522191 0.849429 0.839651
0.578047 0.775035 0.704695 0.203515 0.00267523 0.869083 0.0975535 0.824887 0.00787017 0.920944
0.805897 0.0275489 0.175715 0.135956 0.389958 0.856349 0.974141 0.586308 0.59695 0.906727
0.212875 0.509612 0.85531 0.266659 0.0695836 0.0551129 0.788085 0.401581 0.948216 0.00242077
0.512997 0.134833 0.895968 0.996953 0.422192 0.991526 … 0.838781 0.141053 0.747722 0.84489
0.283221 0.995152 0.61636 0.75955 0.072718 0.691665 0.151339 0.295759 0.795476 0.203072
0.0946639 0.496832 0.551496 0.848571 0.151074 0.625696 0.673817 0.273958 0.177998 0.563221
0.0900806 0.127274 0.394169 0.140403 0.232985 0.460306 0.536441 0.200297 0.970311 0.0292218
0.0698985 0.463532 0.934776 0.448393 0.606287 0.552196 0.883694 0.212222 0.888415 0.941097This doesn't change the type or values of the DArray, but it does make sure
that any pending operations have completed. When shown in the REPL, Dagger will
show all of the values of the DArray that have finished being computed, and
otherwise shows a ... for values which are still be computed.
To convert a DArray back into an Array, collect can be used to gather the
data from all the Julia workers that they're on and combine them into a single
Array on the worker calling collect:
julia> collect(DX)
100×100 Matrix{Float64}:
0.610404 0.0475367 0.809016 0.311305 0.0306211 0.689645 … 0.220267 0.678548 0.892062 0.0559988
0.680815 0.788349 0.758755 0.0594709 0.640167 0.652266 0.331429 0.798848 0.732432 0.579534
0.306898 0.0805607 0.498372 0.887971 0.244104 0.148825 0.340429 0.029274 0.140624 0.292354
0.0537622 0.844509 0.509145 0.561629 0.566584 0.498554 0.427503 0.835242 0.699405 0.0705192
0.587364 0.59933 0.0624318 0.3795 0.430398 0.0853735 0.379947 0.677105 0.0305861 0.748001
0.14129 0.635562 0.218739 0.0629501 0.373841 0.439933 … 0.308294 0.0966736 0.783333 0.00763648
0.14539 0.331767 0.912498 0.0649541 0.527064 0.249595 0.826705 0.826868 0.41398 0.80321
0.13926 0.353158 0.330615 0.438247 0.284794 0.238837 0.791249 0.415801 0.729545 0.88308
0.769242 0.136001 0.950214 0.171962 0.183646 0.78294 0.570442 0.321894 0.293101 0.911913
0.786168 0.513057 0.781712 0.0191752 0.512821 0.621239 0.50503 0.0472064 0.0368674 0.75981
0.493378 0.129937 0.758052 0.169508 0.0564534 0.846092 … 0.873186 0.396222 0.284 0.0242124
0.12689 0.194842 0.263186 0.213071 0.535613 0.246888 0.579931 0.699231 0.441449 0.882772
0.916144 0.21305 0.629293 0.329303 0.299889 0.127453 0.644012 0.311241 0.713782 0.0554386
⋮ ⋮ ⋱
0.430369 0.597251 0.552528 0.795223 0.46431 0.777119 0.189266 0.499178 0.715808 0.797629
0.235668 0.902973 0.786537 0.951402 0.768312 0.633666 0.724196 0.866373 0.0679498 0.255039
0.605097 0.301349 0.758283 0.681568 0.677913 0.51507 … 0.654614 0.37841 0.86399 0.583924
0.824216 0.62188 0.369671 0.725758 0.735141 0.183666 0.0401394 0.522191 0.849429 0.839651
0.578047 0.775035 0.704695 0.203515 0.00267523 0.869083 0.0975535 0.824887 0.00787017 0.920944
0.805897 0.0275489 0.175715 0.135956 0.389958 0.856349 0.974141 0.586308 0.59695 0.906727
0.212875 0.509612 0.85531 0.266659 0.0695836 0.0551129 0.788085 0.401581 0.948216 0.00242077
0.512997 0.134833 0.895968 0.996953 0.422192 0.991526 … 0.838781 0.141053 0.747722 0.84489
0.283221 0.995152 0.61636 0.75955 0.072718 0.691665 0.151339 0.295759 0.795476 0.203072
0.0946639 0.496832 0.551496 0.848571 0.151074 0.625696 0.673817 0.273958 0.177998 0.563221
0.0900806 0.127274 0.394169 0.140403 0.232985 0.460306 0.536441 0.200297 0.970311 0.0292218
0.0698985 0.463532 0.934776 0.448393 0.606287 0.552196 0.883694 0.212222 0.888415 0.941097Now let's look at constructing a DArray from an existing array object; we can
do this by calling the DArray constructor or distribute:
julia> Z = zeros(100, 500);
julia> Dzeros = DArray(Z, Blocks(10, 50))
DMatrix{Float64}(100, 500) with 10x10 partitions of size 10x50:
...This will distribute the array partitions (in chunks of 10 x 50 matrices)
across the workers in the Julia cluster in a relatively even distribution;
future operations on a DArray may produce a different distribution from the
one chosen by previous calls.
As the DArray is a subtype of AbstractArray and generally satisfies Julia's
array interface, a variety of common operations (such as broadcast) work as
expected:
julia> DX = rand(Blocks(50,50), 100, 100)
DMatrix{Float64}(100, 100) with 2x2 partitions of size 50x50:
0.498392 0.286688 0.526038 … 0.0679859 0.246031 0.662384 0.415873
0.470772 0.118921 0.338746 0.368685 0.601165 0.43923 0.838116
0.114096 0.214045 0.973305 0.739328 0.476762 0.880491 0.923226
0.950628 0.937549 0.255425 0.800531 0.686832 0.554949 0.95652
0.887815 0.149639 0.381778 0.511954 0.567506 0.599481 0.31642
0.492811 0.517651 0.452395 … 0.048365 0.282697 0.117261 0.0695919
0.96531 0.694923 0.319353 0.0269875 0.725317 0.38704 0.889079
0.642189 0.139344 0.811443 0.713439 0.82764 0.0817175 0.649828
0.470414 0.310536 0.614132 0.91453 0.38133 0.109497 0.678592
0.681798 0.540348 0.898996 0.666149 0.818365 0.608407 0.959402
0.192863 0.319655 0.340089 … 0.339894 0.879239 0.0198826 0.576009
0.70397 0.789439 0.640622 0.863039 0.380762 0.830201 0.273082
0.859905 0.660245 0.170967 0.827866 0.456064 0.158056 0.39331
0.917375 0.564129 0.409167 0.0608749 0.967919 0.358908 0.313862
0.37067 0.619176 0.913832 0.574299 0.366162 0.209266 0.755402
0.272124 0.609023 0.367749 … 0.702147 0.393283 0.947087 0.642886
0.731806 0.246858 0.952142 0.617165 0.667969 0.955148 0.0721093
0.360135 0.776176 0.835084 0.183326 0.714036 0.370287 0.133747
0.767541 0.663163 0.244765 0.825391 0.870428 0.710432 0.936085
0.364802 0.161725 0.416545 0.685533 0.0313213 0.550103 0.622557
⋮ ⋱
0.219181 0.305549 0.137981 0.133313 0.537143 0.613063 0.583891
0.176936 0.930333 0.737141 0.288332 0.525941 0.33041 0.653449
0.49888 0.644244 0.774862 0.757912 0.411029 0.0304365 0.569458
0.462656 0.186863 0.946858 0.784609 0.269699 0.968227 0.409438
0.969422 0.167368 0.205654 … 0.033398 0.759695 0.222605 0.159356
0.0875248 0.498971 0.620837 0.112562 0.597004 0.208103 0.00320475
0.908971 0.208706 0.676567 0.5081 0.118424 0.0320135 0.897443
0.991279 0.835444 0.14738 0.365196 0.426543 0.987013 0.0339898
0.331385 0.46114 0.718353 0.210474 0.21223 0.245349 0.211097
0.88416 0.790778 0.352482 … 0.364377 0.0734304 0.610556 0.986503
0.0325297 0.649128 0.996022 0.842136 0.26821 0.598355 0.923314
0.793668 0.0111804 0.972974 0.401435 0.10282 0.176944 0.312946
0.175388 0.414811 0.930609 0.0303789 0.794293 0.664361 0.509174
0.056124 0.962519 0.51812 0.914509 0.972889 0.909924 0.831407
0.186426 0.17904 0.712901 … 0.661726 0.937605 0.70563 0.434793
0.182262 0.890191 0.123335 0.0570102 0.188695 0.534232 0.864526
0.949261 0.520407 0.0579928 0.473342 0.90016 0.525208 0.224062
0.817864 0.92868 0.513427 0.619016 0.0461629 0.844613 0.734735
0.792413 0.00791863 0.76343 0.890141 0.183165 0.530084 0.521841
julia> DY = DX .+ DX
DMatrix{Float64}(100, 100) with 2x2 partitions of size 50x50:
0.996784 0.573376 1.05208 1.52853 … 0.135972 0.492062 1.32477 0.831746
0.941543 0.237841 0.677493 0.819019 0.737371 1.20233 0.878459 1.67623
0.228193 0.428089 1.94661 1.97741 1.47866 0.953524 1.76098 1.84645
1.90126 1.8751 0.51085 1.20145 1.60106 1.37366 1.1099 1.91304
1.77563 0.299277 0.763556 0.800454 1.02391 1.13501 1.19896 0.63284
0.985622 1.0353 0.904789 0.132049 … 0.09673 0.565395 0.234523 0.139184
1.93062 1.38985 0.638706 0.677675 0.0539751 1.45063 0.774081 1.77816
1.28438 0.278689 1.62289 1.39015 1.42688 1.65528 0.163435 1.29966
0.940828 0.621072 1.22826 0.262374 1.82906 0.76266 0.218993 1.35718
1.3636 1.0807 1.79799 1.30764 1.3323 1.63673 1.21681 1.9188
0.385725 0.639309 0.680178 1.15371 … 0.679787 1.75848 0.0397651 1.15202
1.40794 1.57888 1.28124 0.740523 1.72608 0.761524 1.6604 0.546163
1.71981 1.32049 0.341934 0.0577456 1.65573 0.912128 0.316112 0.78662
1.83475 1.12826 0.818334 1.13474 0.12175 1.93584 0.717816 0.627725
0.741341 1.23835 1.82766 0.868958 1.1486 0.732324 0.418533 1.5108
0.544247 1.21805 0.735498 1.03821 … 1.40429 0.786565 1.89417 1.28577
1.46361 0.493715 1.90428 1.80758 1.23433 1.33594 1.9103 0.144219
0.720269 1.55235 1.67017 1.25524 0.366652 1.42807 0.740574 0.267495
1.53508 1.32633 0.48953 1.90929 1.65078 1.74086 1.42086 1.87217
0.729603 0.32345 0.833089 1.88305 1.37107 0.0626427 1.10021 1.24511
⋮ ⋱
0.438362 0.611098 0.275962 1.59538 0.266626 1.07429 1.22613 1.16778
0.353873 1.86067 1.47428 1.59328 0.576663 1.05188 0.66082 1.3069
0.997761 1.28849 1.54972 0.625172 1.51582 0.822057 0.060873 1.13892
0.925313 0.373726 1.89372 1.97415 1.56922 0.539397 1.93645 0.818876
1.93884 0.334736 0.411308 0.0129113 … 0.0667961 1.51939 0.44521 0.318712
0.17505 0.997942 1.24167 0.190925 0.225124 1.19401 0.416207 0.00640949
1.81794 0.417412 1.35313 1.16716 1.0162 0.236847 0.0640269 1.79489
1.98256 1.67089 0.29476 1.68775 0.730392 0.853086 1.97403 0.0679796
0.662771 0.922279 1.43671 1.56052 0.420949 0.424459 0.490698 0.422194
1.76832 1.58156 0.704965 1.34981 … 0.728755 0.146861 1.22111 1.97301
0.0650594 1.29826 1.99204 1.82428 1.68427 0.53642 1.19671 1.84663
1.58734 0.0223607 1.94595 1.45301 0.80287 0.205641 0.353888 0.625892
0.350777 0.829621 1.86122 1.52899 0.0607578 1.58859 1.32872 1.01835
0.112248 1.92504 1.03624 1.45978 1.82902 1.94578 1.81985 1.66281
0.372851 0.358081 1.4258 1.49133 … 1.32345 1.87521 1.41126 0.869586
0.364524 1.78038 0.24667 0.072136 0.11402 0.37739 1.06846 1.72905
1.89852 1.04081 0.115986 0.227947 0.946684 1.80032 1.05042 0.448124
1.63573 1.85736 1.02685 1.80253 1.23803 0.0923258 1.68923 1.46947
1.58483 0.0158373 1.52686 0.0511455 1.78028 0.36633 1.06017 1.04368
julia> DZ = DY .* 3
DMatrix{Float64}(100, 100) with 2x2 partitions of size 50x50:
2.99035 1.72013 3.15623 4.58558 … 0.407915 1.47619 3.9743 2.49524
2.82463 0.713524 2.03248 2.45706 2.21211 3.60699 2.63538 5.0287
0.684579 1.28427 5.83983 5.93224 4.43597 2.86057 5.28295 5.53936
5.70377 5.62529 1.53255 3.60434 4.80319 4.12099 3.32969 5.73912
5.32689 0.897831 2.29067 2.40136 3.07172 3.40504 3.59689 1.89852
2.95686 3.10591 2.71437 0.396147 … 0.29019 1.69618 0.703568 0.417551
5.79186 4.16954 1.91612 2.03302 0.161925 4.3519 2.32224 5.33448
3.85314 0.836066 4.86866 4.17046 4.28063 4.96584 0.490305 3.89897
2.82249 1.86322 3.68479 0.787122 5.48718 2.28798 0.65698 4.07155
4.09079 3.24209 5.39397 3.92293 3.99689 4.91019 3.65044 5.75641
1.15718 1.91793 2.04053 3.46113 … 2.03936 5.27544 0.119295 3.45606
4.22382 4.73663 3.84373 2.22157 5.17824 2.28457 4.98121 1.63849
5.15943 3.96147 1.0258 0.173237 4.9672 2.73639 0.948335 2.35986
5.50425 3.38477 2.455 3.40421 0.365249 5.80751 2.15345 1.88317
2.22402 3.71505 5.48299 2.60687 3.44579 2.19697 1.2556 4.53241
1.63274 3.65414 2.20649 3.11464 … 4.21288 2.3597 5.68252 3.85731
4.39084 1.48115 5.71285 5.42273 3.70299 4.00781 5.73089 0.432656
2.16081 4.65706 5.0105 3.76573 1.09996 4.28422 2.22172 0.802485
4.60525 3.97898 1.46859 5.72788 4.95234 5.22257 4.26259 5.61651
2.18881 0.970351 2.49927 5.64915 4.1132 0.187928 3.30062 3.73534
⋮ ⋱
1.31509 1.83329 0.827885 4.78613 0.799879 3.22286 3.67838 3.50334
1.06162 5.582 4.42284 4.77983 1.72999 3.15565 1.98246 3.92069
2.99328 3.86546 4.64917 1.87552 4.54747 2.46617 0.182619 3.41675
2.77594 1.12118 5.68115 5.92246 4.70765 1.61819 5.80936 2.45663
5.81653 1.00421 1.23392 0.0387339 … 0.200388 4.55817 1.33563 0.956135
0.525149 2.99383 3.72502 0.572775 0.675371 3.58202 1.24862 0.0192285
5.45382 1.25224 4.0594 3.50149 3.0486 0.710542 0.192081 5.38466
5.94767 5.01267 0.88428 5.06324 2.19118 2.55926 5.92208 0.203939
1.98831 2.76684 4.31012 4.68156 1.26285 1.27338 1.47209 1.26658
5.30496 4.74467 2.11489 4.04942 … 2.18626 0.440583 3.66334 5.91902
0.195178 3.89477 5.97613 5.47285 5.05282 1.60926 3.59013 5.53988
4.76201 0.0670822 5.83784 4.35903 2.40861 0.616922 1.06166 1.87768
1.05233 2.48886 5.58365 4.58697 0.182273 4.76576 3.98617 3.05505
0.336744 5.77511 3.10872 4.37935 5.48705 5.83733 5.45954 4.98844
1.11855 1.07424 4.27741 4.47399 … 3.97035 5.62563 4.23378 2.60876
1.09357 5.34114 0.74001 0.216408 0.342061 1.13217 3.20539 5.18716
5.69556 3.12244 0.347957 0.683841 2.84005 5.40096 3.15125 1.34437
4.90718 5.57208 3.08056 5.40758 3.71409 0.276977 5.06768 4.40841
4.75448 0.0475118 4.58058 0.153437 5.34085 1.09899 3.18051 3.13105Now, DZ will contain the result of computing (DX .+ DX) .* 3.
julia> Dagger.chunks(DZ)
2×2 Matrix{Any}:
DTask (finished) DTask (finished)
DTask (finished) DTask (finished)
julia> Dagger.chunks(fetch(DZ))
2×2 Matrix{Union{DTask, Dagger.Chunk}}:
Chunk{Matrix{Float64}, DRef, ThreadProc, AnyScope}(Matrix{Float64}, ArrayDomain{2}((1:50, 1:50)), DRef(4, 8, 0x0000000000004e20), ThreadProc(4, 1), AnyScope(), true) … Chunk{Matrix{Float64}, DRef, ThreadProc, AnyScope}(Matrix{Float64}, ArrayDomain{2}((1:50, 1:50)), DRef(2, 5, 0x0000000000004e20), ThreadProc(2, 1), AnyScope(), true)
Chunk{Matrix{Float64}, DRef, ThreadProc, AnyScope}(Matrix{Float64}, ArrayDomain{2}((1:50, 1:50)), DRef(5, 5, 0x0000000000004e20), ThreadProc(5, 1), AnyScope(), true) Chunk{Matrix{Float64}, DRef, ThreadProc, AnyScope}(Matrix{Float64}, ArrayDomain{2}((1:50, 1:50)), DRef(3, 3, 0x0000000000004e20), ThreadProc(3, 1), AnyScope(), true)
Here we can see the DArray's internal representation of the partitions, which
are stored as either DTask objects (representing an ongoing or completed
computation) or Chunk objects (which reference data which exist locally or on
other Julia workers). Of course, one doesn't typically need to worry about
these internal details unless implementing low-level operations on DArrays.
Finally, it's easy to see the results of this combination of broadcast
operations; just use collect to get an Array:
julia> collect(DZ)
100×100 Matrix{Float64}:
2.99035 1.72013 3.15623 4.58558 … 0.407915 1.47619 3.9743 2.49524
2.82463 0.713524 2.03248 2.45706 2.21211 3.60699 2.63538 5.0287
0.684579 1.28427 5.83983 5.93224 4.43597 2.86057 5.28295 5.53936
5.70377 5.62529 1.53255 3.60434 4.80319 4.12099 3.32969 5.73912
5.32689 0.897831 2.29067 2.40136 3.07172 3.40504 3.59689 1.89852
2.95686 3.10591 2.71437 0.396147 … 0.29019 1.69618 0.703568 0.417551
5.79186 4.16954 1.91612 2.03302 0.161925 4.3519 2.32224 5.33448
3.85314 0.836066 4.86866 4.17046 4.28063 4.96584 0.490305 3.89897
2.82249 1.86322 3.68479 0.787122 5.48718 2.28798 0.65698 4.07155
4.09079 3.24209 5.39397 3.92293 3.99689 4.91019 3.65044 5.75641
1.15718 1.91793 2.04053 3.46113 … 2.03936 5.27544 0.119295 3.45606
4.22382 4.73663 3.84373 2.22157 5.17824 2.28457 4.98121 1.63849
5.15943 3.96147 1.0258 0.173237 4.9672 2.73639 0.948335 2.35986
5.50425 3.38477 2.455 3.40421 0.365249 5.80751 2.15345 1.88317
2.22402 3.71505 5.48299 2.60687 3.44579 2.19697 1.2556 4.53241
1.63274 3.65414 2.20649 3.11464 … 4.21288 2.3597 5.68252 3.85731
4.39084 1.48115 5.71285 5.42273 3.70299 4.00781 5.73089 0.432656
2.16081 4.65706 5.0105 3.76573 1.09996 4.28422 2.22172 0.802485
4.60525 3.97898 1.46859 5.72788 4.95234 5.22257 4.26259 5.61651
2.18881 0.970351 2.49927 5.64915 4.1132 0.187928 3.30062 3.73534
⋮ ⋱
1.31509 1.83329 0.827885 4.78613 0.799879 3.22286 3.67838 3.50334
1.06162 5.582 4.42284 4.77983 1.72999 3.15565 1.98246 3.92069
2.99328 3.86546 4.64917 1.87552 4.54747 2.46617 0.182619 3.41675
2.77594 1.12118 5.68115 5.92246 4.70765 1.61819 5.80936 2.45663
5.81653 1.00421 1.23392 0.0387339 … 0.200388 4.55817 1.33563 0.956135
0.525149 2.99383 3.72502 0.572775 0.675371 3.58202 1.24862 0.0192285
5.45382 1.25224 4.0594 3.50149 3.0486 0.710542 0.192081 5.38466
5.94767 5.01267 0.88428 5.06324 2.19118 2.55926 5.92208 0.203939
1.98831 2.76684 4.31012 4.68156 1.26285 1.27338 1.47209 1.26658
5.30496 4.74467 2.11489 4.04942 … 2.18626 0.440583 3.66334 5.91902
0.195178 3.89477 5.97613 5.47285 5.05282 1.60926 3.59013 5.53988
4.76201 0.0670822 5.83784 4.35903 2.40861 0.616922 1.06166 1.87768
1.05233 2.48886 5.58365 4.58697 0.182273 4.76576 3.98617 3.05505
0.336744 5.77511 3.10872 4.37935 5.48705 5.83733 5.45954 4.98844
1.11855 1.07424 4.27741 4.47399 … 3.97035 5.62563 4.23378 2.60876
1.09357 5.34114 0.74001 0.216408 0.342061 1.13217 3.20539 5.18716
5.69556 3.12244 0.347957 0.683841 2.84005 5.40096 3.15125 1.34437
4.90718 5.57208 3.08056 5.40758 3.71409 0.276977 5.06768 4.40841
4.75448 0.0475118 4.58058 0.153437 5.34085 1.09899 3.18051 3.13105
A variety of other operations exist on the DArray, and it should generally
behave otherwise similar to any other AbstractArray type. If you find that
it's missing an operation that you need, please file an issue!
This list is not exhaustive, but documents operations which are known to work well with the DArray:
From Base:
getindex/setindex!- Broadcasting
similar/copy/copyto!map/reduce/mapreducesum/prodminimum/maximum/extrema
From Statistics:
meanvarstd
From LinearAlgebra:
transpose/adjoint(Out-of-place transpose)*(Out-of-place Matrix-(Matrix/Vector) multiply)mul!(In-place Matrix-Matrix multiply)cholesky/cholesky!(In-place/Out-of-place Cholesky factorization)