Skip to content
This repository was archived by the owner on Jan 30, 2023. It is now read-only.

Commit 168ceae

Browse files
committed
Added architecture picture for map_reduce
1 parent 7a33037 commit 168ceae

File tree

4 files changed

+308
-14
lines changed

4 files changed

+308
-14
lines changed
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,277 @@
1+
#FIG 3.2 Produced by xfig version 3.2.5c
2+
Landscape
3+
Center
4+
Metric
5+
A4
6+
100.00
7+
Single
8+
-2
9+
1200 2
10+
6 6028 1350 6210 2430
11+
5 1 0 1 0 7 50 -1 -1 0.000 0 0 0 0 6119.000 1448.500 6029 1411 6119 1351 6209 1411
12+
5 1 0 1 0 7 50 -1 -1 0.000 0 0 0 0 6119.000 2406.500 6029 2369 6119 2309 6209 2369
13+
5 1 0 1 0 7 50 -1 -1 0.000 0 1 0 0 6119.000 2331.500 6029 2369 6119 2429 6209 2369
14+
2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2
15+
6029 1411 6029 2369
16+
2 1 0 1 0 7 50 -1 20 0.000 0 0 -1 0 0 2
17+
6209 1411 6209 2369
18+
-6
19+
6 -630 3510 3855 6480
20+
6 -630 6030 270 6300
21+
5 1 0 1 0 7 50 -1 -1 0.000 0 0 0 0 160.330 6207.000 223 6117 270 6207 223 6297
22+
5 1 0 1 0 7 50 -1 -1 0.000 0 0 0 0 -598.670 6207.000 -536 6117 -489 6207 -536 6297
23+
5 1 0 1 0 7 50 -1 -1 0.000 0 1 0 0 -473.330 6207.000 -536 6117 -583 6207 -536 6297
24+
2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2
25+
223 6117 -536 6117
26+
2 1 0 1 0 7 50 -1 20 0.000 0 0 -1 0 0 2
27+
223 6297 -536 6297
28+
-6
29+
6 2880 5580 3780 5850
30+
5 1 0 1 0 7 50 -1 -1 0.000 0 1 0 0 2989.670 5757.000 2927 5667 2880 5757 2927 5847
31+
5 1 0 1 0 7 50 -1 -1 0.000 0 1 0 0 3748.670 5757.000 3686 5667 3639 5757 3686 5847
32+
5 1 0 1 0 7 50 -1 -1 0.000 0 0 0 0 3623.330 5757.000 3686 5667 3733 5757 3686 5847
33+
2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2
34+
2927 5667 3686 5667
35+
2 1 0 1 0 7 50 -1 20 0.000 0 0 -1 0 0 2
36+
2927 5847 3686 5847
37+
-6
38+
2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2
39+
450 5940 900 5940
40+
2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2
41+
450 5580 900 5580
42+
2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2
43+
450 5040 900 5040
44+
2 2 0 1 0 7 50 -1 45 0.000 0 0 -1 0 0 5
45+
450 4050 900 4050 900 5040 450 5040 450 4050
46+
2 2 0 3 0 7 50 -1 -1 0.000 0 0 -1 0 0 5
47+
450 4050 900 4050 900 6300 450 6300 450 4050
48+
2 1 0 3 0 7 50 -1 -1 0.000 0 0 -1 1 0 2
49+
0 0 1.00 90.00 120.00
50+
1800 4140 2340 4140
51+
2 1 0 3 0 7 50 -1 -1 0.000 0 0 -1 1 0 2
52+
0 0 1.00 90.00 120.00
53+
2610 4680 1800 4680
54+
2 1 0 3 0 7 50 -1 -1 0.000 0 0 7 1 0 2
55+
0 0 3.00 90.00 120.00
56+
900 6210 1800 6210
57+
2 1 0 3 1 7 50 -1 -1 0.000 0 0 7 1 1 2
58+
0 0 3.00 90.00 120.00
59+
0 0 3.00 90.00 120.00
60+
891 5123 1440 4590
61+
2 1 0 3 0 7 50 -1 -1 0.000 0 0 7 1 0 2
62+
0 0 3.00 90.00 120.00
63+
0 6210 450 6210
64+
2 1 0 3 0 7 50 -1 -1 0.000 0 0 7 1 0 2
65+
0 0 3.00 90.00 120.00
66+
3150 5760 2700 5760
67+
2 2 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 5
68+
0 3510 3150 3510 3150 6480 0 6480 0 3510
69+
2 2 0 3 0 7 50 -1 -1 0.000 0 0 -1 0 0 5
70+
1800 5490 2700 5490 2700 6300 1800 6300 1800 5490
71+
2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2
72+
450 6120 900 6120
73+
2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2
74+
450 5760 900 5760
75+
2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2
76+
450 5220 900 5220
77+
2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2
78+
450 5400 900 5400
79+
2 4 0 3 1 7 50 -1 -1 0.000 0 0 7 0 0 5
80+
2790 4680 1440 4680 1440 4140 2790 4140 2790 4680
81+
4 0 0 50 -1 2 20 0.0000 0 225 1380 990 3870 Worker 0\001
82+
4 0 0 50 -1 2 14 0.0000 0 210 1005 1620 4500 main loop\001
83+
4 0 0 50 -1 2 14 0.0000 0 165 630 1260 5040 Nodes\001
84+
4 0 0 50 -1 2 14 0.0000 0 165 540 1980 5850 Thief\001
85+
4 0 0 50 -1 2 14 0.0000 0 165 765 1890 6120 Thread\001
86+
4 0 0 50 -1 2 14 0.0000 0 165 465 1080 6030 theft\001
87+
4 0 0 50 -1 2 14 0.0000 0 165 960 -630 6030 Steal Ans\001
88+
4 0 0 50 -1 2 14 0.0000 0 210 975 2880 5490 Steal Req\001
89+
4 1 0 50 -1 2 14 0.0000 0 165 135 675 4950 S\001
90+
4 1 0 50 -1 2 14 0.0000 0 165 165 675 5235 T\001
91+
4 1 0 50 -1 2 14 0.0000 0 165 180 675 5520 A\001
92+
4 1 0 50 -1 2 14 0.0000 0 165 180 675 5805 C\001
93+
4 1 0 50 -1 2 14 0.0000 0 165 195 675 6090 K\001
94+
-6
95+
6 3600 3510 8085 6480
96+
6 3600 6030 4500 6300
97+
5 1 0 1 0 7 50 -1 -1 0.000 0 0 0 0 4390.330 6207.000 4453 6117 4500 6207 4453 6297
98+
5 1 0 1 0 7 50 -1 -1 0.000 0 0 0 0 3631.330 6207.000 3694 6117 3741 6207 3694 6297
99+
5 1 0 1 0 7 50 -1 -1 0.000 0 1 0 0 3756.670 6207.000 3694 6117 3647 6207 3694 6297
100+
2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2
101+
4453 6117 3694 6117
102+
2 1 0 1 0 7 50 -1 20 0.000 0 0 -1 0 0 2
103+
4453 6297 3694 6297
104+
-6
105+
6 7110 5580 8010 5850
106+
5 1 0 1 0 7 50 -1 -1 0.000 0 1 0 0 7219.670 5757.000 7157 5667 7110 5757 7157 5847
107+
5 1 0 1 0 7 50 -1 -1 0.000 0 1 0 0 7978.670 5757.000 7916 5667 7869 5757 7916 5847
108+
5 1 0 1 0 7 50 -1 -1 0.000 0 0 0 0 7853.330 5757.000 7916 5667 7963 5757 7916 5847
109+
2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2
110+
7157 5667 7916 5667
111+
2 1 0 1 0 7 50 -1 20 0.000 0 0 -1 0 0 2
112+
7157 5847 7916 5847
113+
-6
114+
2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2
115+
4680 5940 5130 5940
116+
2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2
117+
4680 5580 5130 5580
118+
2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2
119+
4680 5040 5130 5040
120+
2 2 0 1 0 7 50 -1 45 0.000 0 0 -1 0 0 5
121+
4680 4050 5130 4050 5130 5040 4680 5040 4680 4050
122+
2 2 0 3 0 7 50 -1 -1 0.000 0 0 -1 0 0 5
123+
4680 4050 5130 4050 5130 6300 4680 6300 4680 4050
124+
2 4 0 3 0 7 50 -1 -1 0.000 0 0 7 0 0 5
125+
7020 4680 7020 4140 5670 4140 5670 4680 7020 4680
126+
2 1 0 3 0 7 50 -1 -1 0.000 0 0 -1 1 0 2
127+
0 0 1.00 90.00 120.00
128+
6030 4140 6570 4140
129+
2 1 0 3 0 7 50 -1 -1 0.000 0 0 -1 1 0 2
130+
0 0 1.00 90.00 120.00
131+
6840 4680 6030 4680
132+
2 1 0 3 0 7 50 -1 -1 0.000 0 0 7 1 0 2
133+
0 0 3.00 90.00 120.00
134+
5130 6210 6030 6210
135+
2 1 0 3 0 7 50 -1 -1 0.000 0 0 7 1 1 2
136+
0 0 3.00 90.00 120.00
137+
0 0 3.00 90.00 120.00
138+
5121 5123 5670 4590
139+
2 1 0 3 0 7 50 -1 -1 0.000 0 0 7 1 0 2
140+
0 0 3.00 90.00 120.00
141+
4230 6210 4680 6210
142+
2 1 0 3 0 7 50 -1 -1 0.000 0 0 7 1 0 2
143+
0 0 3.00 90.00 120.00
144+
7380 5760 6930 5760
145+
2 2 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 5
146+
4230 3510 7380 3510 7380 6480 4230 6480 4230 3510
147+
2 2 0 3 0 7 50 -1 -1 0.000 0 0 -1 0 0 5
148+
6030 5490 6930 5490 6930 6300 6030 6300 6030 5490
149+
2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2
150+
4680 6120 5130 6120
151+
2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2
152+
4680 5760 5130 5760
153+
2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2
154+
4680 5220 5130 5220
155+
2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2
156+
4680 5400 5130 5400
157+
4 0 0 50 -1 2 14 0.0000 0 210 1005 5850 4500 main loop\001
158+
4 0 0 50 -1 2 14 0.0000 0 165 630 5490 5040 Nodes\001
159+
4 0 0 50 -1 2 14 0.0000 0 165 540 6210 5850 Thief\001
160+
4 0 0 50 -1 2 14 0.0000 0 165 765 6120 6120 Thread\001
161+
4 0 0 50 -1 2 14 0.0000 0 165 465 5310 6030 theft\001
162+
4 0 0 50 -1 2 14 0.0000 0 165 960 3600 6030 Steal Ans\001
163+
4 0 0 50 -1 2 14 0.0000 0 210 975 7110 5490 Steal Req\001
164+
4 1 0 50 -1 2 14 0.0000 0 165 135 4905 4950 S\001
165+
4 1 0 50 -1 2 14 0.0000 0 165 165 4905 5235 T\001
166+
4 1 0 50 -1 2 14 0.0000 0 165 180 4905 5520 A\001
167+
4 1 0 50 -1 2 14 0.0000 0 165 180 4905 5805 C\001
168+
4 1 0 50 -1 2 14 0.0000 0 165 195 4905 6090 K\001
169+
4 0 0 50 -1 2 20 0.0000 0 225 1380 5220 3870 Worker 1\001
170+
-6
171+
6 7920 4860 8550 5040
172+
1 3 0 3 0 0 50 -1 20 0.000 1 0.0000 8010 4950 45 45 8010 4950 8055 4950
173+
1 3 0 3 0 0 50 -1 20 0.000 1 0.0000 8235 4950 45 45 8235 4950 8280 4950
174+
1 3 0 3 0 0 50 -1 20 0.000 1 0.0000 8460 4950 45 45 8460 4950 8505 4950
175+
-6
176+
6 8550 3510 13035 6480
177+
6 8550 6030 9450 6300
178+
5 1 0 1 0 7 50 -1 -1 0.000 0 0 0 0 9340.330 6207.000 9403 6117 9450 6207 9403 6297
179+
5 1 0 1 0 7 50 -1 -1 0.000 0 0 0 0 8581.330 6207.000 8644 6117 8691 6207 8644 6297
180+
5 1 0 1 0 7 50 -1 -1 0.000 0 1 0 0 8706.670 6207.000 8644 6117 8597 6207 8644 6297
181+
2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2
182+
9403 6117 8644 6117
183+
2 1 0 1 0 7 50 -1 20 0.000 0 0 -1 0 0 2
184+
9403 6297 8644 6297
185+
-6
186+
6 12060 5580 12960 5850
187+
5 1 0 1 0 7 50 -1 -1 0.000 0 1 0 0 12169.670 5757.000 12107 5667 12060 5757 12107 5847
188+
5 1 0 1 0 7 50 -1 -1 0.000 0 1 0 0 12928.670 5757.000 12866 5667 12819 5757 12866 5847
189+
5 1 0 1 0 7 50 -1 -1 0.000 0 0 0 0 12803.330 5757.000 12866 5667 12913 5757 12866 5847
190+
2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2
191+
12107 5667 12866 5667
192+
2 1 0 1 0 7 50 -1 20 0.000 0 0 -1 0 0 2
193+
12107 5847 12866 5847
194+
-6
195+
2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2
196+
9630 5940 10080 5940
197+
2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2
198+
9630 5580 10080 5580
199+
2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2
200+
9630 5040 10080 5040
201+
2 2 0 1 0 7 50 -1 45 0.000 0 0 -1 0 0 5
202+
9630 4050 10080 4050 10080 5040 9630 5040 9630 4050
203+
2 2 0 3 0 7 50 -1 -1 0.000 0 0 -1 0 0 5
204+
9630 4050 10080 4050 10080 6300 9630 6300 9630 4050
205+
2 4 0 3 0 7 50 -1 -1 0.000 0 0 7 0 0 5
206+
11970 4680 11970 4140 10620 4140 10620 4680 11970 4680
207+
2 1 0 3 0 7 50 -1 -1 0.000 0 0 -1 1 0 2
208+
0 0 1.00 90.00 120.00
209+
10980 4140 11520 4140
210+
2 1 0 3 0 7 50 -1 -1 0.000 0 0 -1 1 0 2
211+
0 0 1.00 90.00 120.00
212+
11790 4680 10980 4680
213+
2 1 0 3 0 7 50 -1 -1 0.000 0 0 7 1 0 2
214+
0 0 3.00 90.00 120.00
215+
10080 6210 10980 6210
216+
2 1 0 3 0 7 50 -1 -1 0.000 0 0 7 1 1 2
217+
0 0 3.00 90.00 120.00
218+
0 0 3.00 90.00 120.00
219+
10071 5123 10620 4590
220+
2 1 0 3 0 7 50 -1 -1 0.000 0 0 7 1 0 2
221+
0 0 3.00 90.00 120.00
222+
9180 6210 9630 6210
223+
2 1 0 3 0 7 50 -1 -1 0.000 0 0 7 1 0 2
224+
0 0 3.00 90.00 120.00
225+
12330 5760 11880 5760
226+
2 2 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 5
227+
9180 3510 12330 3510 12330 6480 9180 6480 9180 3510
228+
2 2 0 3 0 7 50 -1 -1 0.000 0 0 -1 0 0 5
229+
10980 5490 11880 5490 11880 6300 10980 6300 10980 5490
230+
2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2
231+
9630 6120 10080 6120
232+
2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2
233+
9630 5760 10080 5760
234+
2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2
235+
9630 5220 10080 5220
236+
2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2
237+
9630 5400 10080 5400
238+
4 0 0 50 -1 2 14 0.0000 0 210 1005 10800 4500 main loop\001
239+
4 0 0 50 -1 2 14 0.0000 0 165 630 10440 5040 Nodes\001
240+
4 0 0 50 -1 2 14 0.0000 0 165 540 11160 5850 Thief\001
241+
4 0 0 50 -1 2 14 0.0000 0 165 765 11070 6120 Thread\001
242+
4 0 0 50 -1 2 14 0.0000 0 165 465 10260 6030 theft\001
243+
4 0 0 50 -1 2 14 0.0000 0 165 960 8550 6030 Steal Ans\001
244+
4 0 0 50 -1 2 14 0.0000 0 210 975 12060 5490 Steal Req\001
245+
4 1 0 50 -1 2 14 0.0000 0 165 135 9855 4950 S\001
246+
4 1 0 50 -1 2 14 0.0000 0 165 165 9855 5235 T\001
247+
4 1 0 50 -1 2 14 0.0000 0 165 180 9855 5520 A\001
248+
4 1 0 50 -1 2 14 0.0000 0 165 180 9855 5805 C\001
249+
4 1 0 50 -1 2 14 0.0000 0 165 195 9855 6090 K\001
250+
4 0 0 50 -1 2 20 0.0000 0 225 1455 10170 3870 Worker N\001
251+
-6
252+
2 2 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 5
253+
2700 450 7200 450 7200 2250 2700 2250 2700 450
254+
2 2 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 5
255+
3600 1350 4050 1350 4050 1800 3600 1800 3600 1350
256+
2 1 0 3 0 0 50 -1 20 0.000 0 0 7 1 0 2
257+
0 0 3.00 90.00 120.00
258+
6120 2655 6120 1350
259+
2 1 0 3 0 0 50 -1 20 0.000 0 0 -1 0 0 2
260+
2700 3510 6120 2655
261+
2 1 0 3 0 0 50 -1 20 0.000 0 0 -1 0 0 2
262+
5850 3510 6120 2655
263+
2 1 0 3 0 0 50 -1 20 0.000 0 0 -1 0 0 2
264+
9630 3510 6120 2655
265+
2 1 0 3 4 -1 50 -1 -1 0.000 0 0 -1 1 0 3
266+
3 1 3.00 90.00 120.00
267+
4230 4950 3780 5760 3150 5760
268+
2 1 0 3 4 -1 50 -1 -1 0.000 0 0 7 1 0 2
269+
3 1 3.00 90.00 120.00
270+
945 6210 1755 6210
271+
2 1 0 3 4 -1 50 -1 -1 0.000 0 0 -1 1 0 2
272+
3 1 3.00 90.00 120.00
273+
2700 6210 4230 6210
274+
4 0 0 50 -1 2 20 0.0000 0 225 1020 4500 900 Master\001
275+
4 0 0 50 -1 2 16 0.0000 0 180 1485 3150 1260 Active Tasks\001
276+
4 0 0 50 -1 2 16 0.0000 0 180 840 5760 1260 Results\001
277+
4 1 4 50 -1 2 16 0.0000 0 180 135 3825 1665 4\001
Binary file not shown.
Loading

src/sage/combinat/map_reduce.py

+31-14
Original file line numberDiff line numberDiff line change
@@ -207,6 +207,13 @@
207207
- ``worker._thief`` -- a :class:`Thread` which is in charge of stealing from
208208
``worker._todo``.
209209
210+
Here is a schematic of the architecture:
211+
212+
.. _figure-map_reduce_arch:
213+
214+
.. figure:: ../../media/map_reduce_arch.png
215+
216+
210217
How thefts are performed
211218
------------------------
212219
@@ -223,7 +230,7 @@
223230
- ``W`` signals to the master that it is idle :meth:`master._signal_task_done`;
224231
- ``W`` chose a victim ``V`` at random;
225232
- ``W`` sends a request to ``V`` : it puts its identifier into ``V._request``;
226-
- ``W`` tries to read a node from ``W._read_task``. Then three things can happen:
233+
- ``W`` tries to read a node from ``W._read_task``. Then three things may happen:
227234
228235
+ a proper node is read. Then the theft was a success and ``W`` starts
229236
working locally on the received node.
@@ -235,13 +242,24 @@
235242
shutdown.
236243
237244
We now describe the protocol on the victims side. Each worker process contains
238-
a :class:`Thread` which acts like some kinds of Troyan horse during theft. It
239-
is normally blocked waiting for a steal request.
240-
241-
From the point of view of ``V` here is what happens:
245+
a :class:`Thread` which we call ``T`` for thief which acts like some kinds of
246+
Troyan horse during theft. It is normally blocked waiting for a steal request.
242247
248+
From the point of view of ``V`` and ``T``, here is what happens:
243249
250+
- during normal time ``T`` is blocked waiting on ``V._request``;
251+
- upon steal request, ``T`` wakes up receiving the identification of ``W``;
252+
- ``T`` signal to the master that a new task is starting by
253+
:meth:`master._signal_task_start`;
254+
- Two things may happen depending if the queue ``V._todo`` is empty or not.
255+
Remark that due to the GIL, there is no parallel execution between the
256+
victim ``V`` and its thief tread ``T``.
244257
258+
+ If ``V._todo`` is empty, then ``None`` is answered on
259+
``W._write_task``. The task is immediately signaled to end the the master
260+
through :meth:`master._signal_task_done`.
261+
+ Otherwise, a node is removed from the bottom of ``V._todo``. The node is
262+
sent to ``W`` on ``W._write_task``. The task will be ended by ``W``.
245263
246264
Are there examples of classes ?
247265
-------------------------------
@@ -304,7 +322,7 @@ def proc_number(max_proc = None):
304322
305323
INPUT:
306324
307-
- `max_proc` -- the maximum number of process used
325+
- ``max_proc`` -- the maximum number of process used
308326
309327
EXAMPLE::
310328
@@ -346,14 +364,13 @@ class RESetMapReduce(object):
346364
Description of the set:
347365
348366
- either ``forest=f`` -- where ``f`` is a :class:`RecursivelyEnumeratedSet`
349-
- ``roots`` -- = None,
350-
children = None,
351-
post_process = None,
352-
map_function = None,
353-
reduce_function = None,
354-
reduce_init = None,
355-
forest = None
356-
367+
- ``roots`` -- (default to ``None``),
368+
- ``children` -- (default to ``None``),
369+
- ``post_process`` -- (default to ``None``),
370+
- ``map_function`` -- (default to ``None``),
371+
- ``reduce_function`` -- (default to ``None``),
372+
- ``reduce_init`` -- (default to ``None``),
373+
- ``forest`` -- (default to ``None``)
357374
"""
358375
def __init__(self, roots = None,
359376
children = None,

0 commit comments

Comments
 (0)