Skip to content

Commit edc762c

Browse files
committedJan 2, 2016
Document localindexes and SharedArray(filename, ...).
1 parent 6d7a50b commit edc762c

File tree

3 files changed

+82
-17
lines changed

3 files changed

+82
-17
lines changed
 

‎base/docs/helpdb/Base.jl

-16
Original file line numberDiff line numberDiff line change
@@ -8743,22 +8743,6 @@ Equivalent to `stat(file).mtime`.
87438743
"""
87448744
mtime
87458745

8746-
"""
8747-
SharedArray(T::Type, dims::NTuple; init=false, pids=Int[])
8748-
8749-
Construct a `SharedArray` of a bitstype `T` and size `dims` across the processes specified
8750-
by `pids` - all of which have to be on the same host.
8751-
8752-
If `pids` is left unspecified, the shared array will be mapped across all processes on the
8753-
current host, including the master. But, `localindexes` and `indexpids` will only refer to
8754-
worker processes. This facilitates work distribution code to use workers for actual
8755-
computation with the master process acting as a driver.
8756-
8757-
If an `init` function of the type `initfn(S::SharedArray)` is specified, it is called on all
8758-
the participating workers.
8759-
"""
8760-
SharedArray
8761-
87628746
"""
87638747
logspace(start, stop, n=50)
87648748

‎base/sharedarray.jl

+58-1
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,20 @@ call{T}(::Type{SharedArray{T}}, m::Integer, n::Integer; kwargs...) =
3535
call{T}(::Type{SharedArray{T}}, m::Integer, n::Integer, o::Integer; kwargs...) =
3636
SharedArray(T, m, n, o; kwargs...)
3737

38+
"""
39+
SharedArray(T::Type, dims::NTuple; init=false, pids=Int[])
40+
41+
Construct a `SharedArray` of a bitstype `T` and size `dims` across the processes specified
42+
by `pids` - all of which have to be on the same host.
43+
44+
If `pids` is left unspecified, the shared array will be mapped across all processes on the
45+
current host, including the master. But, `localindexes` and `indexpids` will only refer to
46+
worker processes. This facilitates work distribution code to use workers for actual
47+
computation with the master process acting as a driver.
48+
49+
If an `init` function of the type `initfn(S::SharedArray)` is specified, it is called on all
50+
the participating workers.
51+
"""
3852
function SharedArray(T::Type, dims::NTuple; init=false, pids=Int[])
3953
N = length(dims)
4054

@@ -96,7 +110,36 @@ end
96110

97111
SharedArray(T, I::Int...; kwargs...) = SharedArray(T, I; kwargs...)
98112

99-
# Create a SharedArray from a disk file
113+
"""
114+
SharedArray(filename::AbstractString, T::Type, dims::NTuple, [offset=0]; mode=nothing, init=false, pids=Int[])
115+
116+
Construct a `SharedArray` backed by the file `filename`, with element
117+
type `T` (must be a `bitstype`) and size `dims`, across the processes
118+
specified by `pids` - all of which have to be on the same host. This
119+
file is mmapped into the host memory, with the following consequences:
120+
121+
- The array data must be represented in binary format (e.g., an ASCII
122+
format like CSV cannot be supported)
123+
124+
- Any changes you make to the array values (e.g., `A[3] = 0`) will
125+
also change the values on disk
126+
127+
If `pids` is left unspecified, the shared array will be mapped across
128+
all processes on the current host, including the master. But,
129+
`localindexes` and `indexpids` will only refer to worker
130+
processes. This facilitates work distribution code to use workers for
131+
actual computation with the master process acting as a driver.
132+
133+
`mode` must be one of `"r"`, `"r+"`, `"w+"`, or `"a+"`, and defaults
134+
to `"r+"` if the file specified by `filename` already exists, or
135+
`"w+"` if not. If an `init` function of the type
136+
`initfn(S::SharedArray)` is specified, it is called on all the
137+
participating workers. You cannot specify an `init` function if the
138+
file is not writable.
139+
140+
`offset` allows you to skip the specified number of bytes at the
141+
beginning of the file.
142+
"""
100143
function SharedArray{T,N}(filename::AbstractString, ::Type{T}, dims::NTuple{N,Int}, offset::Integer=0; mode=nothing, init=false, pids::Vector{Int}=Int[])
101144
isabspath(filename) || throw(ArgumentError("$filename is not an absolute path; try abspath(filename)?"))
102145
isbits(T) || throw(ArgumentError("type of SharedArray elements must be bits types, got $(T)"))
@@ -211,6 +254,20 @@ indexpids(S::SharedArray) = S.pidx
211254
sdata(S::SharedArray) = S.s
212255
sdata(A::AbstractArray) = A
213256

257+
"""
258+
localindexes(S::SharedArray)
259+
260+
Returns a range describing the "default" indexes to be handled by the
261+
current process. This range should be interpreted in the sense of
262+
linear indexing, i.e., as a sub-range of `1:length(S)`. In
263+
multi-process contexts, returns an empty range in the parent process
264+
(or any process for which `indexpids` returns 0).
265+
266+
It's worth emphasizing that `localindexes` exists purely as a
267+
convenience, and you can partition work on the array among workers any
268+
way you wish. For a SharedArray, all indexes should be equally fast
269+
for each worker process.
270+
"""
214271
localindexes(S::SharedArray) = S.pidx > 0 ? range_1dim(S, S.pidx) : 1:0
215272

216273
unsafe_convert{T}(::Type{Ptr{T}}, S::SharedArray) = unsafe_convert(Ptr{T}, sdata(S))

‎doc/stdlib/parallel.rst

+24
Original file line numberDiff line numberDiff line change
@@ -491,6 +491,22 @@ Shared Arrays
491491

492492
If an ``init`` function of the type ``initfn(S::SharedArray)`` is specified, it is called on all the participating workers.
493493

494+
.. function:: SharedArray(filename::AbstractString, T::Type, dims::NTuple, [offset=0]; mode=nothing, init=false, pids=Int[])
495+
496+
.. Docstring generated from Julia source
497+
498+
Construct a ``SharedArray`` backed by the file ``filename``\ , with element type ``T`` (must be a ``bitstype``\ ) and size ``dims``\ , across the processes specified by ``pids`` - all of which have to be on the same host. This file is mmapped into the host memory, with the following consequences:
499+
500+
* The array data must be represented in binary format (e.g., an ASCII format like CSV cannot be supported)
501+
502+
* Any changes you make to the array values (e.g., ``A[3] = 0``\ ) will also change the values on disk
503+
504+
If ``pids`` is left unspecified, the shared array will be mapped across all processes on the current host, including the master. But, ``localindexes`` and ``indexpids`` will only refer to worker processes. This facilitates work distribution code to use workers for actual computation with the master process acting as a driver.
505+
506+
``mode`` must be one of ``"r"``\ , ``"r+"``\ , ``"w+"``\ , or ``"a+"``\ , and defaults to ``"r+"`` if the file specified by ``filename`` already exists, or ``"w+"`` if not. If an ``init`` function of the type ``initfn(S::SharedArray)`` is specified, it is called on all the participating workers. You cannot specify an ``init`` function if the file is not writable.
507+
508+
``offset`` allows you to skip the specified number of bytes at the beginning of the file.
509+
494510
.. function:: procs(S::SharedArray)
495511

496512
.. Docstring generated from Julia source
@@ -509,6 +525,14 @@ Shared Arrays
509525
510526
Returns the index of the current worker into the ``pids`` vector, i.e., the list of workers mapping the SharedArray
511527

528+
.. function:: localindexes(S::SharedArray)
529+
530+
.. Docstring generated from Julia source
531+
532+
Returns a range describing the "default" indexes to be handled by the current process. This range should be interpreted in the sense of linear indexing, i.e., as a sub-range of ``1:length(S)``\ . In multi-process contexts, returns an empty range in the parent process (or any process for which ``indexpids`` returns 0).
533+
534+
It's worth emphasizing that ``localindexes`` exists purely as a convenience, and you can partition work on the array among workers any way you wish. For a SharedArray, all indexes should be equally fast for each worker process.
535+
512536
Cluster Manager Interface
513537
-------------------------
514538

0 commit comments

Comments
 (0)