@@ -5,126 +5,31 @@ actions it does, such as moving data between workers or executing thunks, and
55tracks how much time and memory allocations these operations consume, among
66other things. It does it through the ` TimespanLogging.jl ` package (which used
77to be directly integrated into Dagger). Saving this information somewhere
8- accessible is disabled by default, but it's quite easy to turn it on, by
9- setting a "log sink" in the ` Context ` being used, as ` ctx.log_sink ` . A variety
10- of log sinks are built-in to TimespanLogging; the ` NoOpLog ` is the default log
11- sink when one isn't explicitly specified, and disables logging entirely (to
12- minimize overhead). There are currently two other log sinks of interest; the
13- first and newer of the two is the ` MultiEventLog ` , which generates multiple
14- independent log streams, one per "consumer" (details in the next section). The
15- second and older sink is the ` LocalEventLog ` , which is explained later in this
16- document. Most users are recommended to use the ` MultiEventLog ` since it's far
17- more flexible and extensible, and is more performant in general.
18-
19- ## MultiEventLog
20-
21- The ` MultiEventLog ` is intended to be configurable to exclude unnecessary
22- information, and to include any built-in or user-defined metrics. It stores a
23- set of "sub-log" streams internally, appending a single element to each of them
24- when an event is generated. This element can be called a "sub-event" (to
25- distinguish it from the higher-level "event" that Dagger creates), and is
26- created by a "consumer". A consumer is a function or callable struct that, when
27- called with the ` Event ` object generated by TimespanLogging, returns a sub-event
28- characterizing whatever information the consumer represents. For example, the
29- ` Dagger.Events.BytesAllocd ` consumer calculates the total bytes allocated and
30- live at any given time within Dagger, and returns the current value when
31- called. Let's construct one:
32-
33- ``` julia
34- ctx = Context ()
35- ml = TimespanLogging. MultiEventLog ()
36-
37- # Add the BytesAllocd consumer to the log as `:bytes`
38- ml[:bytes ] = Dagger. Events. BytesAllocd ()
39-
40- ctx. log_sink = ml
41- ```
42-
43- As we can see above, each consumer gets a unique name as a ` Symbol ` that
44- identifies it. Now that the log sink is attached with a consumer, we can
45- execute some Dagger tasks, and then collect the sub-events generated by
46- ` BytesAllocd ` :
47-
48- ``` julia
49- # Using the lazy API, for explanatory purposes
50- collect (ctx, delayed (+ )(1 , delayed (* )(3 , 4 ))) # Allocates 8 bytes
51- log = TimspanLogging. get_logs! (ctx)[1 ] # Get the logs for worker 1
52- @show log[:bytes ]
53- ```
54-
55- You'll then see that 8 bytes are allocated and then freed during the process of
56- executing and completing those tasks.
57-
58- Note that the ` MultiEventLog ` can also be used perfectly well when using
59- Dagger's eager API:
60-
61- ``` julia
62- ctx = Dagger. Sch. eager_context ()
63- ctx. log_sink = ml
64-
65- a = Dagger. @spawn 3 * 4
66- Dagger. @spawn 1 + a
67- ```
68-
69- There are a variety of other consumers built-in to TimespanLogging and Dagger,
70- under the ` TimespanLogging.Events ` and ` Dagger.Events ` modules, respectively;
71- see [ Dagger Types] ( @ref ) and [ TimespanLogging Types] ( @ref ) for details.
72-
73- The ` MultiEventLog ` also has a mechanism to call a set of functions, called
74- "aggregators", after all consumers have been executed, and are passed the full
75- set of log streams as a ` Dict{Symbol,Vector{Any}} ` . The only one currently
76- shipped with TimespanLogging directly is the ` LogWindow ` , and DaggerWebDash.jl
77- has the ` TableStorage ` which integrates with it; see
78- [ DaggerWebDash Types] ( @ref ) for details.
79-
80- ## LocalEventLog
81-
82- The ` LocalEventLog ` is generally only useful when you want combined events
83- (event start and finish combined as a single unit), and only care about a few
84- simple built-in generated events. Let's attach one to our context:
85-
86- ``` julia
87- ctx = Context ()
88- log = TimespanLogging. LocalEventLog ()
89- ctx. log_sink = log
90- ```
91-
92- Now anytime ` ctx ` is used as the context for a scheduler, the scheduler will
93- log events into ` log ` .
94-
95- Once sufficient data has been accumulated into a ` LocalEventLog ` , it can be
96- gathered to a single host via ` TimespanLogging.get_logs!(log) ` . The result is a
97- ` Vector ` of ` TimespanLogging.Timespan ` objects, which describe some metadata
98- about an operation that occured and the scheduler logged. These events may be
99- introspected directly, or may also be rendered to a DOT-format string:
100-
101- ``` julia
102- logs = TimespanLogging. get_logs! (log)
103- str = Dagger. show_plan (logs)
104- ```
105-
106- ` Dagger.show_plan ` can also be called as ` Dagger.show_plan(io::IO, logs) ` to
107- write the graph to a file or other ` IO ` object. The string generated by this
108- function may be passed to an external tool like ` Graphviz ` for rendering. Note
109- that this method doesn't display input arguments to the DAG (non-` Thunk ` s);
110- you can call ` Dagger.show_plan(logs, thunk) ` , where ` thunk ` is the output
111- ` Thunk ` of the DAG, to render argument nodes.
112-
113- !!! note
114- ` TimespanLogging.get_logs! ` clears out the event logs, so that old events
115- don't mix with new ones from future DAGs.
116-
117- As a convenience, it's possible to set ` ctx.log_file ` to the path to an output
118- file, and then calls to ` compute(ctx, ...) ` /` collect(ctx, ...) ` will
119- automatically write the graph in DOT format to that path. There is also a
120- benefit to this approach over manual calls to ` get_logs! ` and ` show_plan ` : DAGs
121- which aren't ` Thunk ` s (such as operations on the ` Dagger.DArray ` ) will be
122- properly rendered with input arguments (which normally aren't rendered because
123- a ` Thunk ` is dynamically generated from such operations by Dagger before
124- scheduling).
125-
126- ## FilterLog
127-
128- The ` FilterLog ` exists to allow writing events to a user-defined location (such
129- as a database, file, or network socket). It is not currently tested or
130- documented.
8+ accessible is disabled by default, but it's quite easy to turn it on, through
9+ two mechanisms.
10+
11+ The first is ` Dagger.enable_logging! ` , which provides an easy-to-use interface
12+ to both enable and configure logging. The defaults are usually sufficient for
13+ most users, but can be tweaked with keyword arguments.
14+
15+ The second is done by setting a "log sink" in the Dagger ` Context ` being used,
16+ as ` ctx.log_sink ` . These log sinks drive how Dagger's logging behaves, and are
17+ configurable by the user, without the need to tweak any of Dagger's internal
18+ code.
19+
20+ A variety of log sinks are built-in to TimespanLogging; the ` NoOpLog ` is the
21+ default log sink when one isn't explicitly specified, and disables logging
22+ entirely (to minimize overhead). There are currently two other log sinks of
23+ interest; the first and newer of the two is the ` MultiEventLog ` , which
24+ generates multiple independent log streams, one per "consumer" (details in the
25+ next section). This is the log sink that ` enable_logging! ` uses, as it's easily
26+ the most flexible. The second and older sink is the ` LocalEventLog ` , which is
27+ explained later in this document. Most users are recommended to use the
28+ ` MultiEventLog ` (ideally via ` enable_logging! ` ) since it's far more flexible
29+ and extensible, and is more performant in general.
30+
31+ Log sinks are explained in detail in [ Logging: Advanced] ( @ref ) ; however, if
32+ using ` enable_logging! ` , everything is already configured for you. Then, all
33+ you need to do is call ` Dagger.fetch_logs!() ` to get the logs for all workers
34+ as a ` Dict ` . A variety of tools can operate on these logs, including
35+ visualization through ` show_logs ` and ` render_logs ` .
0 commit comments