lein repl crashes whenever shut down

[hypirion]

Should be fairly obvious to reproduce. As a result, e.g. attempts of doing lein do repl, repl will start the first repl, but crash whenever lein tries to start the second repl.

Stacktrace:

DEBUG='y' lein do repl, repl
Leiningen's classpath: :/home/jeannikl/.lein/self-installs/leiningen-2.3.0-standalone.jar
Applying task do to (repl)
Applying task repl to ()
nREPL server started on port 48752 on host 127.0.0.1
REPL-y 0.2.1
Clojure 1.5.1
    Docs: (doc function-name-here)
          (find-doc "part-of-name-here")
  Source: (source function-name-here)
 Javadoc: (javadoc java-object-or-class-here)
    Exit: Control+D or (exit) or (quit)

user=> (quit)
java.net.SocketException: Socket closed
                                   (Unknown Source) java.net.SocketInputStream.socketRead0
                         SocketInputStream.java:146 java.net.SocketInputStream.read
                       BufferedInputStream.java:235 java.io.BufferedInputStream.fill
                       BufferedInputStream.java:254 java.io.BufferedInputStream.read
                          FilterInputStream.java:83 java.io.FilterInputStream.read
                       PushbackInputStream.java:139 java.io.PushbackInputStream.read
                                     bencode.clj:84 clojure.tools.nrepl.bencode/read-byte
                                    bencode.clj:236 clojure.tools.nrepl.bencode/read-token
                                    bencode.clj:254 clojure.tools.nrepl.bencode/read-bencode
                                   transport.clj:95 clojure.tools.nrepl.transport/bencode[fn]
                                   transport.clj:95 clojure.tools.nrepl.transport/bencode[fn]
                                   transport.clj:42 clojure.tools.nrepl.transport/fn-transport[fn]
                                      core.clj:1836 clojure.core/binding-conveyor-fn[fn]
                                        AFn.java:18 clojure.lang.AFn.call
                                FutureTask.java:334 java.util.concurrent.FutureTask$Sync.innerRun
                                FutureTask.java:166 java.util.concurrent.FutureTask.run
                       ThreadPoolExecutor.java:1146 java.util.concurrent.ThreadPoolExecutor.runWorker
                        ThreadPoolExecutor.java:615 java.util.concurrent.ThreadPoolExecutor$Worker.run
                                    Thread.java:679 java.lang.Thread.run
Bye for now!Applying task repl to ()
java.util.concurrent.RejectedExecutionException
    at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1992)
    at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:822)
    at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1373)
    at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:120)
    at clojure.core$future_call.invoke(core.clj:6305)
    at clojure.tools.nrepl.ack$wait_for_ack.invoke(ack.clj:29)
    at leiningen.repl$server.invoke(repl.clj:185)
    at leiningen.repl$repl.doInvoke(repl.clj:236)
    at clojure.lang.RestFn.invoke(RestFn.java:425)
    at leiningen.repl$repl.invoke(repl.clj:225)
    at clojure.lang.Var.invoke(Var.java:415)
    at clojure.lang.AFn.applyToHelper(AFn.java:161)
    at clojure.lang.Var.applyTo(Var.java:532)
    at clojure.core$apply.invoke(core.clj:619)
    at leiningen.core.main$resolve_task$fn__1289.doInvoke(main.clj:151)
    at clojure.lang.RestFn.invoke(RestFn.java:410)
    at clojure.lang.AFn.applyToHelper(AFn.java:161)
    at clojure.lang.RestFn.applyTo(RestFn.java:132)
    at clojure.lang.AFunction$1.doInvoke(AFunction.java:29)
    at clojure.lang.RestFn.applyTo(RestFn.java:137)
    at clojure.core$apply.invoke(core.clj:619)
    at leiningen.core.main$apply_task.invoke(main.clj:192)
    at leiningen.core.main$resolve_and_apply.invoke(main.clj:196)
    at leiningen.do$do.doInvoke(do.clj:25)
    at clojure.lang.RestFn.invoke(RestFn.java:439)
    at clojure.lang.Var.invoke(Var.java:423)
    at clojure.lang.AFn.applyToHelper(AFn.java:167)
    at clojure.lang.Var.applyTo(Var.java:532)
    at clojure.core$apply.invoke(core.clj:619)
    at leiningen.core.main$resolve_task$fn__1289.doInvoke(main.clj:151)
    at clojure.lang.RestFn.applyTo(RestFn.java:139)
    at clojure.lang.AFunction$1.doInvoke(AFunction.java:29)
    at clojure.lang.RestFn.applyTo(RestFn.java:137)
    at clojure.core$apply.invoke(core.clj:619)
    at leiningen.core.main$apply_task.invoke(main.clj:192)
    at leiningen.core.main$resolve_and_apply.invoke(main.clj:196)
    at leiningen.core.main$_main$fn__1352.invoke(main.clj:265)
    at leiningen.core.main$_main.doInvoke(main.clj:252)
    at clojure.lang.RestFn.invoke(RestFn.java:436)
    at clojure.lang.Var.invoke(Var.java:423)
    at clojure.lang.AFn.applyToHelper(AFn.java:167)
    at clojure.lang.Var.applyTo(Var.java:532)
    at clojure.core$apply.invoke(core.clj:617)
    at clojure.main$main_opt.invoke(main.clj:335)
    at clojure.main$main.doInvoke(main.clj:440)
    at clojure.lang.RestFn.invoke(RestFn.java:482)
    at clojure.lang.Var.invoke(Var.java:431)
    at clojure.lang.AFn.applyToHelper(AFn.java:178)
    at clojure.lang.Var.applyTo(Var.java:532)
    at clojure.main.main(main.java:37)

[hypirion]

It seems like the problem is actually two (potentially independent) problems. The first issue is an upstream problem: trptcolin/reply#123. The second problem seems more like an issue with receiving or sending an ack to tools.nrepl, which may (or may not) be caused by the other problem.

[trptcolin]

I expect the second part is due to shutdown-agents being used. https://github.com/trptcolin/reply/blob/28c5bf9ac203c5c73f8d88373c258b0ccabd6838/src/clj/reply/main.clj#L51-L76

Without shutdown-agents, the repl hangs on exit waiting for threads to finish, due to Agents not using daemon threads (see http://dev.clojure.org/jira/browse/CLJ-124 & http://tech.puredanger.com/2010/06/08/clojure-agent-thread-pools/). I'm actually not using agents or futures directly, but other things are. nREPL uses futures, for example.

Me: "Hey, at least I'm not using (System/exit) anymore."

Can you shed some light on the use case here? Just running lein do repl, repl doesn't seem compelling, but I expect the real use case is more involved. I don't really have any objections to another entry point (or flag) to REPLy that avoids shutdown-agents, to allow spinning the repl up & down.

[hypirion]

Alright, thanks for the heads up. I'll have a closer look later on.

There's currently no use case for this, but ensuring that tasks in general can be started and stopped correctly is a concern for Grenchman whenever you have a persistent Leiningen process running and you're using eval-in :nrepl. It may turn out to be a use case where this fails in the future, although that's pure speculation.

In general I find these bugs/errors whenever I play around and do weird things with Leiningen, many which doesn't make any sense or have any use case.

[hypirion]

Yeah, as you mention, the issue with shutdown-agents is that it kills agents and futures. I think it would make more sense to make Leiningen control the shutdown of agents. Even though it seems unlikely that people use lein do repl, foo, it may be some weird use case, and the foo task may use agents and/or futures.

The reason lein do repl, repl is killed at the specific point we're looking at is because nrepl's wait-for-ack spins up a future trying which attempts to deref a promise.

[trptcolin]

I think this is now safe to close. @hypirion when you have time I'd still like to get confirmation that trptcolin/reply#123 is (or is not) an issue, but that only affects DEBUG="y" anyway. I definitely can't reproduce that part anymore, but that could be due to a number of things (including an OS upgrade).

[hypirion]

@trptcolin Yeah, I can't reproduce this on master anymore after the upgrade.

Apparently, the socket still crashes spectacularly (Only performs a PST when DEBUG=y), but it's from a blocking read, so I know how dealing with it is like. In practise, that doesn't have any effect on how Leiningen works.