@parallel ERROR #2805

diegozea · 2013-04-09T20:22:24Z

I was trying to make a parallel version of this:

iden = zeros(Int,length(SEQ),length(SEQ))
for i in 1:length(SEQ)
  for j in (i+1):length(SEQ)
    iden[j,i] = identidad(SEQ[i],SEQ[j])
  end
end

I try with:

identidad = @parallel [ identidad(SEQ[i],SEQ[j]) for i=1:length(SEQ),j=1:length(SEQ) ]

But I get a lot of this ERRORs:

        From worker 2:   in getindex at darray2.jl:208
        From worker 2:   in anonymous at multi.jl:416
#undef          From worker 6:  exception on 6: ERROR: no method getindex(ErrorException,Int64,Int64)
        From worker 6:   in getindex at darray2.jl:208
        From worker 6:   in anonymous at multi.jl:416
#undef          From worker 6:  exception on 6: ERROR: no method getindex(ErrorException,Int64,Int64)
        From worker 6:   in getindex at darray2.jl:208
        From worker 6:   in anonymous at multi.jl:416

The same line without @parallel works fine.

The text was updated successfully, but these errors were encountered:

diegozea · 2013-04-10T01:49:28Z

Can be related to #209

JeffBezanson · 2013-04-10T01:57:13Z

Is identidad available on every process?

diegozea · 2013-04-10T02:21:39Z

Looks Iike I'm doing something wrong... But I didn't know what thing is bad here.

julia> function identidad(seq1,seq2)
         len = length(seq1)
         count = 0
         colgap = 0
         for i in 1:len
             if seq1[i]==seq2[i]
         count += 1
         colgap += seq1[i]=='-'
             end
         end
       (count-colgap)/(len-colgap)
       end
# methods for generic function identidad
identidad(seq1,seq2) at none:2

julia> @everywhere identidad
        From worker 3:  exception on 3: ERROR: identidad not defined
        From worker 3:   in eval at /home/dzea/bin/julia-master/base/sysimg.jl:4
        From worker 3:   in anonymous at multi.jl:1182
        From worker 3:   in anonymous at multi.jl:416
        From worker 4:  exception on 4: ERROR: identidad not defined
        From worker 4:   in eval at /home/dzea/bin/julia-master/base/sysimg.jl:4
        From worker 4:   in anonymous at multi.jl:1182
        From worker 4:   in anonymous at multi.jl:416
        From worker 2:  exception on 2: ERROR: identidad not defined
        From worker 2:   in eval at /home/dzea/bin/julia-master/base/sysimg.jl:4
        From worker 2:   in anonymous at multi.jl:1182
        From worker 2:   in anonymous at multi.jl:416
        From worker 5:  exception on 5: ERROR: identidad not defined
        From worker 6:  exception on 6: ERROR: identidad not defined
        From worker 6:   in eval at /home/dzea/bin/julia-master/base/sysimg.jl:4
        From worker 5:   in eval at /home/dzea/bin/julia-master/base/sysimg.jl:4
        From worker 5:   in anonymous at multi.jl:1182
        From worker 6:   in anonymous at multi.jl:1182
        From worker 5:   in anonymous at multi.jl:416
        From worker 6:   in anonymous at multi.jl:416

julia> iden = @parallel [ identidad(SEQ[i],SEQ[j]) for i=1:length(SEQ),j=1:length(SEQ)]
        From worker 3:  exception on 3: ERROR: in anonymous: SEQ not defined
        From worker 3:   in anonymous at no file:1308
        From worker 3:   in anonymous at multi.jl:416
        From worker 3:  ERROR: interrupt
        From worker 5:  exception on 5: ERROR: in anonymous: SEQ not defined
        From worker 5:   in anonymous at no file:1308
        From worker 5:   in anonymous at multi.jl:416
        From worker 5:  ERROR: interrupt
        From worker 4:  exception on 4: ERROR: in anonymous: SEQ not defined
        From worker 4:   in anonymous at no file:1308
        From worker 4:   in anonymous at multi.jl:416
        From worker 4:  ERROR: interrupt

JeffBezanson · 2013-04-10T02:25:52Z

Put @everywhere before function identidad(...

diegozea · 2013-04-10T02:27:58Z

I try to stop this using ^C but, I can push Crtl + C a lot of times and not return to Julia prompt:

^CERROR: interrupt
 in process_events at stream.jl:312InterruptException()                                                           
 #undef                                                                                 From worker 4:   in process_events at stream.jl:312ERROR: interrupt
        From worker 3:   in process_events at stream.jl:312ERROR: interrupt
        From worker 6:   in process_events at stream.jl:312ERROR: interrupt
        From worker 5:   in process_events at stream.jl:312ERROR: interrupt
        From worker 2:   in process_events at stream.jl:312ERROR: interrupt
        From worker 5:   in process_events at stream.jl:312ERROR: access to undefined reference
        From worker 5:   in tasknotify at stream.jl:101
        From worker 5:   in _uv_hook_readcb at stream.jl:210
^CERROR: interrupt
 in process_events at stream.jl:312     From worker 4:   in process_events at stream.jl:312ERROR: interrupt
        From worker 5:   in process_events at stream.jl:312ERROR: interrupt
        From worker 6:   in process_events at stream.jl:312ERROR: interrupt
        From worker 2:   in process_events at stream.jl:312ERROR: interrupt
        From worker 3:   in process_events at stream.jl:312ERROR: interrupt
        From worker 5:   in process_events at stream.jl:312ERROR: access to undefined reference
        From worker 5:   in tasknotify at stream.jl:101
        From worker 5:   in _uv_hook_readcb at stream.jl:210
^CERROR: interrupt
 in process_events at stream.jl:312InterruptException()                                                           
 #undef                                                                                 From worker 4:   in process_events at stream.jl:312ERROR: interrupt
        From worker 5:   in process_events at stream.jl:312ERROR: interrupt
        From worker 6:   in process_events at stream.jl:312ERROR: interrupt
        From worker 3:   in process_events at stream.jl:312ERROR: interrupt
        From worker 2:   in process_events at stream.jl:312ERROR: interrupt
        From worker 5:   in process_events at stream.jl:312ERROR: access to undefined reference
        From worker 5:   in tasknotify at stream.jl:101
        From worker 5:   in _uv_hook_readcb at stream.jl:210

diegozea · 2013-04-10T02:32:48Z

This is my code:

using FastaRead
using BioSeq

# Lee Secuencias de un Alineamiento Fasta
function read_fasta(filename::String)
    fr = FastaReader(filename)
    sequences = Array{AminoAcid,1}[]
    for (name,seq) in fr
      push!(sequences,aminoacid(seq))
    end
  sequences
end

const SEQ = read_fasta("PF00025.fasta")


@everywhere function identidad(seq1,seq2)
  len = length(seq1)
  count = 0
  colgap = 0
  for i in 1:len
      if seq1[i]==seq2[i]
    count += 1
    colgap += seq1[i]=='-'
      end
  end
(count-colgap)/(len-colgap)
end

iden = @parallel [ identidad(SEQ[i],SEQ[j]) for i=1:length(SEQ),j=1:length(SEQ)]

I'm still getting this error:

        From worker 2:  exception on 2: ERROR: no method getindex(ErrorException,Int64,Int64)
        From worker 2:   in getindex at darray2.jl:208
        From worker 2:   in anonymous at multi.jl:416
        From worker 2:  exception on 2: ERROR: no method getindex(ErrorException,Int64,Int64)
        From worker 2:   in getindex at darray2.jl:208
        From worker 2:   in anonymous at multi.jl:416
 1.0       0.780347  0.992647  0.992647  0.992647  0.992647  …          From worker 2:  exception on 2: ERROR: no method getindex(ErrorException,Int64,Int64)
        From worker 2:   in getindex at darray2.jl:208
        From worker 2:   in anonymous at multi.jl:416
#undef          From worker 2:  exception on 2: ERROR: no method getindex(ErrorException,Int64,Int64)
        From worker 2:   in getindex at darray2.jl:208
        From worker 2:   in anonymous at multi.jl:416

JeffBezanson · 2013-04-10T02:41:02Z

Does the same code work with just 1 processor?

diegozea · 2013-04-10T02:45:07Z

Yes, works fine the same code with only 1 processor:

dzea@deepthought:~/DNA2Seq_dev$ julia-m -p 1
               _
   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: http://docs.julialang.org
   _ _   _| |_  __ _   |  Type "help()" to list help topics
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.2.0
 _/ |\__'_|_|_|\__'_|  |  Commit 1601355dad 2013-04-07 10:40:45
|__/                   |

julia> using FastaRead

julia> using BioSeq

julia> 

julia> # Lee Secuencias de un Alineamiento Fasta

julia> function read_fasta(filename::String)
           fr = FastaReader(filename)
           sequences = Array{AminoAcid,1}[]
           for (name,seq) in fr
             push!(sequences,aminoacid(seq))
           end
         sequences
       end
# methods for generic function read_fasta
read_fasta(filename::String) at none:2

julia> const SEQ = read_fasta("PF00025.fasta");

julia> 

julia> @everywhere function identidad(seq1,seq2)
         len = length(seq1)
         count = 0
         colgap = 0
         for i in 1:len
             if seq1[i]==seq2[i]
         count += 1
         colgap += seq1[i]=='-'
             end
         end
       (count-colgap)/(len-colgap)
       end

julia> 

julia> iden = @parallel [ identidad(SEQ[i],SEQ[j]) for i=1:length(SEQ),j=1:length(SEQ)]
3553x3553 Float64 DArray:
 1.0       0.780347  0.992647  0.992647  0.992647  0.992647  …  0.985294  0.529032  0.977941  0.985294  0.517241  0.446667
 0.780347  1.0       0.786127  0.786127  0.786127  0.786127     0.780347  0.467391  0.774566  0.780347  0.653409  0.370968
 0.992647  0.786127  1.0       1.0       1.0       1.0          0.992647  0.535484  0.985294  0.992647  0.522989  0.453333
 0.992647  0.786127  1.0       1.0       1.0       1.0          0.992647  0.535484  0.985294  0.992647  0.522989  0.453333
 0.992647  0.786127  1.0       1.0       1.0       1.0          0.992647  0.535484  0.985294  0.992647  0.522989  0.453333
 0.992647  0.786127  1.0       1.0       1.0       1.0       …  0.992647  0.535484  0.985294  0.992647  0.522989  0.453333
 0.992647  0.786127  1.0       1.0       1.0       1.0          0.992647  0.535484  0.985294  0.992647  0.522989  0.453333
 0.992647  0.786127  1.0       1.0       1.0       1.0          0.992647  0.535484  0.985294  0.992647  0.522989  0.453333
.
.
.

i7 with 4 cores and 8 threads, running Ubuntu 12.04:

julia> versioninfo()
Julia Version 0.2.0
Commit 1601355dad 2013-04-07 10:40:45
Platform Info:
  OS_NAME: Linux
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT)
  LAPACK: libopenblas
  LIBM: libopenlibm

JeffBezanson · 2013-04-10T02:47:57Z

Put the setup (the usings and the 2 functions) in file.jl and do @everywhere include("file.jl"). That might do it.

diegozea · 2013-04-10T02:57:11Z

Looks like @everywhere include(... and using have some problems:

dzea@deepthought:~/DNA2Seq_dev$ julia-m -p 2
               _
   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: http://docs.julialang.org
   _ _   _| |_  __ _   |  Type "help()" to list help topics
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.2.0
 _/ |\__'_|_|_|\__'_|  |  Commit 1601355dad 2013-04-07 10:40:45
|__/                   |

julia> @everywhere include("file.jl")
        From worker 2:  exception on 2: ERROR: in module path: FastaRead not defined
        From worker 2:   in include_string at loading.jl:65
        From worker 2:   in include_from_node1 at loading.jl:94
        From worker 2:   in eval at /home/dzea/bin/julia-master/base/sysimg.jl:4
        From worker 2:   in anonymous at multi.jl:1182
        From worker 2:   in anonymous at multi.jl:416
        From worker 2:  at /home/dzea/DNA2Seq_dev/file.jl:1

julia> const SEQ = read_fasta("PF00025.fasta");

julia> iden = @parallel [ identidad(SEQ[i],SEQ[j]) for i=1:length(SEQ),j=1:length(SEQ)];
        From worker 2:  exception on 2: ERROR: in anonymous: SEQ not defined
        From worker 2:   in anonymous at no file:1308
        From worker 2:   in anonymous at multi.jl:416

Without @everywhere works fine:

julia> include("file.jl")

julia>

JeffBezanson · 2013-04-10T04:34:41Z

Aha, that is a bug, now fixed! @everywhere include("file.jl") should work now.

diegozea · 2013-04-10T04:39:09Z

Thanks Jeff!!!

diegozea · 2013-04-10T04:57:45Z

@JeffBezanson the @everywhere include("file.jl") is taken too much time:

dzea@deepthought:~/DNA2Seq_dev$ julia-m -p 6
               _
   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: http://docs.julialang.org
   _ _   _| |_  __ _   |  Type "help()" to list help topics
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.2.0
 _/ |\__'_|_|_|\__'_|  |  Commit a8f9da3299 2013-04-10 01:35:27
|__/                   |

julia> tic()
0x000034a0246bdff7

julia> @everywhere include("file.jl")
^CERROR: interrupt
 in process_events at stream.jl:312     From worker 5:  ERROR: interrupt
        From worker 4:  ERROR: interrupt
        From worker 2:  ERROR: interrupt
        From worker 3:  ERROR: interrupt
        From worker 6:  ERROR: interrupt


^CERROR: interrupt
 in process_events at stream.jl:312     From worker 3:   in process_events at stream.jl:312ERROR: interrupt
        From worker 2:   in process_events at stream.jl:312ERROR: interrupt
        From worker 6:   in process_events at stream.jl:312ERROR: interrupt
        From worker 5:   in process_events at stream.jl:312ERROR: interrupt
        From worker 4:   in process_events at stream.jl:312ERROR: interrupt
^CERROR: interrupt
 in process_events at stream.jl:312
julia> toc()
elapsed time: 573.9454197 seconds
573.9454197

dzea@deepthought:~/DNA2Seq_dev$ julia-m -p 6
               _
   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: http://docs.julialang.org
   _ _   _| |_  __ _   |  Type "help()" to list help topics
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.2.0
 _/ |\__'_|_|_|\__'_|  |  Commit a8f9da3299 2013-04-10 01:35:27
|__/                   |

julia> tic()
0x0000353f129b410b

julia> include("file.jl")

julia> toc()
elapsed time: 3.602123848 seconds
3.602123848

julia>

JeffBezanson · 2013-04-10T05:05:38Z

Make sure there is no @everywhere inside the file. That could be causing a deadlock.

diegozea · 2013-04-10T05:12:31Z

Thanks @JeffBezanson!!! Yes, that was the problem. I forgot @everywhere inside the file before a function declaration.

Other thing... Would be possible for
iden = [ identidad(SEQ[i],SEQ[j]) for i=1:length(SEQ)-1,j= ( i +1):length(SEQ) ]
or
iden = @parallel [ identidad(SEQ[i],SEQ[j]) for i=1:length(SEQ)-1,j= ( i +1):length(SEQ) ]
give a Triangular or Symmetric matrix instead of a error? [ now advise about i doesn't defined ]

JeffBezanson closed this as completed in 57704ee Apr 10, 2013

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

@parallel ERROR #2805

@parallel ERROR #2805

diegozea commented Apr 9, 2013

diegozea commented Apr 10, 2013

JeffBezanson commented Apr 10, 2013

diegozea commented Apr 10, 2013

JeffBezanson commented Apr 10, 2013

diegozea commented Apr 10, 2013

diegozea commented Apr 10, 2013

JeffBezanson commented Apr 10, 2013

diegozea commented Apr 10, 2013

JeffBezanson commented Apr 10, 2013

diegozea commented Apr 10, 2013

JeffBezanson commented Apr 10, 2013

diegozea commented Apr 10, 2013

diegozea commented Apr 10, 2013

JeffBezanson commented Apr 10, 2013

diegozea commented Apr 10, 2013

@parallel ERROR #2805

@parallel ERROR #2805

Comments

diegozea commented Apr 9, 2013

diegozea commented Apr 10, 2013

JeffBezanson commented Apr 10, 2013

diegozea commented Apr 10, 2013

JeffBezanson commented Apr 10, 2013

diegozea commented Apr 10, 2013

diegozea commented Apr 10, 2013

JeffBezanson commented Apr 10, 2013

diegozea commented Apr 10, 2013

JeffBezanson commented Apr 10, 2013

diegozea commented Apr 10, 2013

JeffBezanson commented Apr 10, 2013

diegozea commented Apr 10, 2013

diegozea commented Apr 10, 2013

JeffBezanson commented Apr 10, 2013

diegozea commented Apr 10, 2013