-
Notifications
You must be signed in to change notification settings - Fork 189
/
Copy pathdataflow_tasks.html
422 lines (409 loc) · 29.7 KB
/
dataflow_tasks.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
<!DOCTYPE html>
<html lang="en" data-content_root="./">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<title>Using the dataflow Firetasks — FireWorks 2.0.4 documentation</title>
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=03e43079" />
<link rel="stylesheet" type="text/css" href="_static/nature.css?v=b0beeceb" />
<script src="_static/documentation_options.js?v=adc66a14"></script>
<script src="_static/doctools.js?v=9bcbadda"></script>
<script src="_static/sphinx_highlight.js?v=dc90522c"></script>
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
<link rel="next" title="Controlling the directory and Worker of execution" href="controlworker.html" />
<link rel="prev" title="Using the built-in FilePad Tasks" href="filepad_tasks.html" />
<link href='https://fonts.googleapis.com/css?family=Lato:400,700' rel='stylesheet' type='text/css'>
</head><body>
<div class="related" role="navigation" aria-label="Related">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="genindex.html" title="General Index"
accesskey="I">index</a></li>
<li class="right" >
<a href="py-modindex.html" title="Python Module Index"
>modules</a> |</li>
<li class="right" >
<a href="controlworker.html" title="Controlling the directory and Worker of execution"
accesskey="N">next</a> |</li>
<li class="right" >
<a href="filepad_tasks.html" title="Using the built-in FilePad Tasks"
accesskey="P">previous</a> |</li>
<li class="nav-item nav-item-0"><a href="index.html">FireWorks 2.0.4 documentation</a> »</li>
<li class="nav-item nav-item-this"><a href="">Using the dataflow Firetasks</a></li>
</ul>
</div>
<div class="document">
<div class="documentwrapper">
<div class="bodywrapper">
<div class="body" role="main">
<section id="using-the-dataflow-firetasks">
<h1>Using the dataflow Firetasks<a class="headerlink" href="#using-the-dataflow-firetasks" title="Link to this heading">¶</a></h1>
<p>This group includes custom Firetasks to manage dataflow between Fireworks. The
input data and output data are stored in the Firework <strong>spec</strong> and passed to the
subsequent Firetasks and Fireworks via <em>FWAction</em> objects. The module includes:</p>
<ul class="simple">
<li><p>CommandLineTask</p></li>
<li><p>ForeachTask</p></li>
<li><p>JoinDictTask</p></li>
<li><p>JoinListTask</p></li>
<li><p>ImportDataTask</p></li>
</ul>
<p>To use a Python function in a dataflow context, see e.g. <a class="reference internal" href="pytask.html"><span class="doc">PyTask</span></a>.</p>
<section id="commandlinetask">
<h2>CommandLineTask<a class="headerlink" href="#commandlinetask" title="Link to this heading">¶</a></h2>
<p>The CommandLineTask provides methods to handle commands in a shell with
command line options, manage the inputs and outputs of commands and receive
file metadata from parent Fireworks and pass file metadata to child Fireworks.</p>
<section id="required-parameters">
<h3>Required parameters<a class="headerlink" href="#required-parameters" title="Link to this heading">¶</a></h3>
<ul>
<li><p><strong>command_spec</strong> <em>(dict)</em>: a dictionary specification of the command.
It has the following structure:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">command_spec</span> <span class="o">=</span> <span class="p">{</span>
<span class="s1">'command'</span><span class="p">:</span> <span class="p">(</span><span class="nb">list</span><span class="p">),</span> <span class="c1"># mandatory, list of strings</span>
<span class="n">inputs</span><span class="p">[</span><span class="mi">0</span><span class="p">]:</span> <span class="p">(</span><span class="nb">dict</span><span class="p">),</span> <span class="c1"># optional</span>
<span class="n">inputs</span><span class="p">[</span><span class="mi">1</span><span class="p">]:</span> <span class="p">(</span><span class="nb">dict</span><span class="p">),</span> <span class="c1"># optional</span>
<span class="c1"># ...</span>
<span class="n">outputs</span><span class="p">[</span><span class="mi">0</span><span class="p">]:</span> <span class="p">(</span><span class="nb">dict</span><span class="p">),</span> <span class="c1"># optional</span>
<span class="n">outputs</span><span class="p">[</span><span class="mi">1</span><span class="p">]:</span> <span class="p">(</span><span class="nb">dict</span><span class="p">),</span> <span class="c1"># optional</span>
<span class="c1"># ...</span>
<span class="p">}</span>
</pre></div>
</div>
</li>
</ul>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>When a <code class="docutils literal notranslate"><span class="pre">str</span></code> is found instead of <code class="docutils literal notranslate"><span class="pre">dict</span></code> for some input or output key, for example <code class="docutils literal notranslate"><span class="pre">inputs[1]:</span> <span class="pre">'string'</span></code>, then <code class="docutils literal notranslate"><span class="pre">'string'</span></code> is automatically replaced with <code class="docutils literal notranslate"><span class="pre">{spec['string']}</span></code>.</p>
</div>
<p>The <code class="docutils literal notranslate"><span class="pre">command</span></code> key is a representation of the command as to be used with the
<em>Subprocess</em> package. The optional keys <code class="docutils literal notranslate"><span class="pre">inputs[0]</span></code>, <code class="docutils literal notranslate"><span class="pre">inputs[1]</span></code>, …,
<code class="docutils literal notranslate"><span class="pre">outputs[0]</span></code>, <code class="docutils literal notranslate"><span class="pre">outputs[0]</span></code>, …, are
the actual keys specified in <code class="docutils literal notranslate"><span class="pre">inputs</span></code> and <code class="docutils literal notranslate"><span class="pre">outputs</span></code>.
The dictionaries <code class="docutils literal notranslate"><span class="pre">dict1</span></code>, <code class="docutils literal notranslate"><span class="pre">dict2</span></code>, etc. have the following schema:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="p">{</span>
<span class="s1">'binding'</span><span class="p">:</span> <span class="p">{</span>
<span class="n">prefix</span><span class="p">:</span> <span class="nb">str</span> <span class="ow">or</span> <span class="kc">None</span><span class="p">,</span>
<span class="n">separator</span><span class="p">:</span> <span class="nb">str</span> <span class="ow">or</span> <span class="kc">None</span>
<span class="p">},</span>
<span class="s1">'source'</span><span class="p">:</span> <span class="p">{</span>
<span class="s1">'type'</span><span class="p">:</span> <span class="s1">'path'</span> <span class="ow">or</span> <span class="s1">'data'</span> <span class="ow">or</span> <span class="s1">'identifier'</span>
<span class="ow">or</span> <span class="s1">'stdin'</span> <span class="ow">or</span> <span class="s1">'stdout'</span> <span class="ow">or</span> <span class="s1">'stderr'</span> <span class="ow">or</span> <span class="kc">None</span><span class="p">,</span>
<span class="s1">'value'</span><span class="p">:</span> <span class="nb">str</span> <span class="ow">or</span> <span class="nb">int</span> <span class="ow">or</span> <span class="nb">float</span>
<span class="p">},</span>
<span class="s1">'target'</span><span class="p">:</span> <span class="p">{</span>
<span class="s1">'type'</span><span class="p">:</span> <span class="s1">'path'</span> <span class="ow">or</span> <span class="s1">'data'</span> <span class="ow">or</span> <span class="s1">'identifier'</span>
<span class="ow">or</span> <span class="s1">'stdin'</span> <span class="ow">or</span> <span class="s1">'stdout'</span> <span class="ow">or</span> <span class="s1">'stderr'</span> <span class="ow">or</span> <span class="kc">None</span><span class="p">,</span>
<span class="s1">'value'</span><span class="p">:</span> <span class="nb">str</span>
<span class="p">}</span>
<span class="p">}</span>
</pre></div>
</div>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>If the <code class="docutils literal notranslate"><span class="pre">type</span></code> in the <code class="docutils literal notranslate"><span class="pre">source</span></code> field is <code class="docutils literal notranslate"><span class="pre">data</span></code> then <code class="docutils literal notranslate"><span class="pre">value</span></code> can be of types <code class="docutils literal notranslate"><span class="pre">str</span></code>, <code class="docutils literal notranslate"><span class="pre">int</span></code> and <code class="docutils literal notranslate"><span class="pre">float</span></code>.</p>
</div>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>When a <code class="docutils literal notranslate"><span class="pre">str</span></code> is found instead of <code class="docutils literal notranslate"><span class="pre">dict</span></code> for some <code class="docutils literal notranslate"><span class="pre">source</span></code>, for example <code class="docutils literal notranslate"><span class="pre">{'source':</span> <span class="pre">'string'}</span></code>, then <code class="docutils literal notranslate"><span class="pre">string</span></code> is replaced with <code class="docutils literal notranslate"><span class="pre">spec['string']</span></code>.</p>
</div>
</section>
<section id="optional-parameters">
<h3>Optional parameters<a class="headerlink" href="#optional-parameters" title="Link to this heading">¶</a></h3>
<ul class="simple">
<li><p><strong>inputs</strong> <em>(list)</em>: a list of keys, one for each input argument;
default is empty.</p></li>
<li><p><strong>outputs</strong> <em>(list)</em>: a list of keys, one for each output argument;
default is empty.</p></li>
<li><p><strong>chunk_number</strong> <em>(int)</em>: the serial number of the Firetask when it is part
of a parallel group generated by a ForeachTask; default is <code class="docutils literal notranslate"><span class="pre">None</span></code>.</p></li>
</ul>
</section>
<section id="example">
<h3>Example<a class="headerlink" href="#example" title="Link to this heading">¶</a></h3>
<p>The following workflow executes the command <code class="docutils literal notranslate"><span class="pre">echo</span> <span class="pre">-n</span> <span class="pre">Hello</span> <span class="pre">world!</span></code> in the
command line:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>fws:
- fw_id: 1
name: Run a command and store the result
spec:
_tasks:
- _fw_name: CommandLineTask
command_spec:
command: [echo, -n]
input string: {source: input string}
output file:
source: {type: stdout}
target: {type: path, value: /tmp}
inputs: [input string]
outputs: [output file]
input string: {type: data, value: Hello world!}
links: {}
metadata: {}
</pre></div>
</div>
<p>The STDOUT output is collected and stored in a new file under <code class="docutils literal notranslate"><span class="pre">/tmp</span></code>. The
full path of the file is stored in <strong>spec</strong> of the current Firework and all
child Fireworks with key <code class="docutils literal notranslate"><span class="pre">output</span> <span class="pre">file</span></code>.</p>
</section>
</section>
<section id="foreachtask">
<h2>ForeachTask<a class="headerlink" href="#foreachtask" title="Link to this heading">¶</a></h2>
<p>The purpose of ForeachTask is to dynamically branch the workflow between
this Firework and its children by inserting a parallel section of child
Fireworks. The number of the spawned parallel Fireworks is determined by the
length of the list specified by the <code class="docutils literal notranslate"><span class="pre">split</span></code> parameter or the optional
<code class="docutils literal notranslate"><span class="pre">number</span> <span class="pre">of</span> <span class="pre">chunks</span></code> parameter. Each child Firework contains a Firetask (of
classes PyTask, CommandLineTask or any Firetask with <code class="docutils literal notranslate"><span class="pre">inputs</span></code> parameter)
which processes one element (or one chunk) from this list. The output is passed
to the <strong>spec</strong> of the Firework(s) right after the detour using a push method,
i.e. the outputs of all parallel Fireworks are collected in a list specified
in the <code class="docutils literal notranslate"><span class="pre">outputs</span></code> argument.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>the ordering of elements (or chunks) in the resulting <code class="docutils literal notranslate"><span class="pre">outputs</span></code> list can be different from that in the original list depending on the execution order of spawned Fireworks.</p>
</div>
<section id="id1">
<h3>Required parameters<a class="headerlink" href="#id1" title="Link to this heading">¶</a></h3>
<ul class="simple">
<li><p><strong>task</strong> <em>(dict)</em>: a dictionary representation of the Firetask</p></li>
<li><p><strong>split</strong> <em>(str)</em>: a key in <strong>spec</strong> which contains input data to be
distributed over the parallel child Fireworks. This key must also be available
both in the <code class="docutils literal notranslate"><span class="pre">inputs</span></code> parameter of the Firetask and in the <strong>spec</strong>.</p></li>
</ul>
</section>
<section id="id2">
<h3>Optional parameters<a class="headerlink" href="#id2" title="Link to this heading">¶</a></h3>
<ul class="simple">
<li><p><strong>number of chunks</strong> <em>(int)</em>: if provided, the input list, specified with
<code class="docutils literal notranslate"><span class="pre">split</span></code> will be divided into this number of sub-lists (chunks) and each chunk
will be processed by a separate child Firework. This parameter can be used to
reduce the number of parallel Fireworks. Default is the length of the input
data list specified in <code class="docutils literal notranslate"><span class="pre">split</span></code>, i.e. for each data element a child Firework
will be created.</p></li>
</ul>
</section>
<section id="id3">
<h3>Example<a class="headerlink" href="#id3" title="Link to this heading">¶</a></h3>
<p>The following example demonstrates the use of ForeachTask in combination with
PyTask:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">fws</span><span class="p">:</span>
<span class="o">-</span> <span class="n">fw_id</span><span class="p">:</span> <span class="mi">1</span>
<span class="n">name</span><span class="p">:</span> <span class="n">Grind</span> <span class="n">coffee</span>
<span class="n">spec</span><span class="p">:</span>
<span class="n">_tasks</span><span class="p">:</span>
<span class="o">-</span> <span class="n">_fw_name</span><span class="p">:</span> <span class="n">ForeachTask</span>
<span class="n">split</span><span class="p">:</span> <span class="n">coffee</span> <span class="n">beans</span>
<span class="n">task</span><span class="p">:</span>
<span class="n">_fw_name</span><span class="p">:</span> <span class="n">PyTask</span>
<span class="n">func</span><span class="p">:</span> <span class="n">auxiliary</span><span class="o">.</span><span class="n">printurn</span>
<span class="n">inputs</span><span class="p">:</span> <span class="p">[</span><span class="n">coffee</span> <span class="n">beans</span><span class="p">]</span>
<span class="n">outputs</span><span class="p">:</span> <span class="p">[</span><span class="n">coffee</span> <span class="n">powder</span><span class="p">]</span>
<span class="n">coffee</span> <span class="n">beans</span><span class="p">:</span> <span class="p">[</span><span class="n">arabica</span><span class="p">,</span> <span class="n">robusta</span><span class="p">,</span> <span class="n">liberica</span><span class="p">]</span>
<span class="o">-</span> <span class="n">fw_id</span><span class="p">:</span> <span class="mi">2</span>
<span class="n">name</span><span class="p">:</span> <span class="n">Brew</span> <span class="n">coffee</span>
<span class="n">spec</span><span class="p">:</span>
<span class="n">_tasks</span><span class="p">:</span>
<span class="o">-</span> <span class="n">_fw_name</span><span class="p">:</span> <span class="n">ForeachTask</span>
<span class="n">split</span><span class="p">:</span> <span class="n">coffee</span> <span class="n">powder</span>
<span class="n">task</span><span class="p">:</span>
<span class="n">_fw_name</span><span class="p">:</span> <span class="n">PyTask</span>
<span class="n">func</span><span class="p">:</span> <span class="n">auxiliary</span><span class="o">.</span><span class="n">printurn</span>
<span class="n">inputs</span><span class="p">:</span> <span class="p">[</span><span class="n">coffee</span> <span class="n">powder</span><span class="p">,</span> <span class="n">water</span><span class="p">]</span>
<span class="n">outputs</span><span class="p">:</span> <span class="p">[</span><span class="n">pure</span> <span class="n">coffee</span><span class="p">]</span>
<span class="n">water</span><span class="p">:</span> <span class="n">workflowing</span> <span class="n">water</span>
<span class="o">-</span> <span class="n">fw_id</span><span class="p">:</span> <span class="mi">3</span>
<span class="n">name</span><span class="p">:</span> <span class="n">Serve</span> <span class="n">coffee</span>
<span class="n">spec</span><span class="p">:</span>
<span class="n">_tasks</span><span class="p">:</span>
<span class="o">-</span> <span class="n">_fw_name</span><span class="p">:</span> <span class="n">PyTask</span>
<span class="n">func</span><span class="p">:</span> <span class="n">auxiliary</span><span class="o">.</span><span class="n">printurn</span>
<span class="n">inputs</span><span class="p">:</span> <span class="p">[</span><span class="n">pure</span> <span class="n">coffee</span><span class="p">]</span>
<span class="n">links</span><span class="p">:</span>
<span class="s1">'1'</span><span class="p">:</span> <span class="p">[</span><span class="mi">2</span><span class="p">]</span>
<span class="s1">'2'</span><span class="p">:</span> <span class="p">[</span><span class="mi">3</span><span class="p">]</span>
<span class="n">metadata</span><span class="p">:</span> <span class="p">{}</span>
<span class="n">name</span><span class="p">:</span> <span class="n">Workflow</span> <span class="k">for</span> <span class="n">many</span> <span class="n">sorts</span> <span class="n">of</span> <span class="n">coffee</span>
</pre></div>
</div>
<p>In this example the function <code class="docutils literal notranslate"><span class="pre">auxiliary.printurn</span></code> prints and returns all
its arguments:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">def</span><span class="w"> </span><span class="nf">printurn</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">):</span>
<span class="n">result</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">arg</span> <span class="ow">in</span> <span class="n">args</span><span class="p">:</span>
<span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">arg</span><span class="p">,</span> <span class="nb">list</span><span class="p">)</span> <span class="ow">and</span> <span class="nb">len</span><span class="p">(</span><span class="n">arg</span><span class="p">)</span> <span class="o">==</span> <span class="mi">1</span><span class="p">:</span>
<span class="n">result</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">arg</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">result</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">arg</span><span class="p">)</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">result</span><span class="p">)</span> <span class="o">==</span> <span class="mi">1</span><span class="p">:</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">result</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="nb">print</span><span class="p">(</span><span class="n">result</span><span class="p">)</span>
<span class="k">return</span> <span class="n">result</span>
</pre></div>
</div>
<p>The module <code class="docutils literal notranslate"><span class="pre">auxiliary</span></code>, i.e. the file <code class="docutils literal notranslate"><span class="pre">auxiliary.py</span></code> must be in
<code class="docutils literal notranslate"><span class="pre">$PYTHONPATH</span></code>.</p>
</section>
</section>
<section id="joindicttask">
<h2>JoinDictTask<a class="headerlink" href="#joindicttask" title="Link to this heading">¶</a></h2>
<p>This Firetask combines the specified items from <strong>spec</strong> into a new or existing dictionary in <strong>spec</strong>.</p>
<section id="id4">
<h3>Required parameters<a class="headerlink" href="#id4" title="Link to this heading">¶</a></h3>
<ul class="simple">
<li><p><strong>inputs</strong> <em>(list)</em>: a list of <strong>spec</strong> keys</p></li>
<li><p><strong>output</strong> <em>(str)</em>: a <strong>spec</strong> key under which the joined items will be stored</p></li>
</ul>
</section>
<section id="id5">
<h3>Optional parameters<a class="headerlink" href="#id5" title="Link to this heading">¶</a></h3>
<ul class="simple">
<li><p><strong>rename</strong> <em>(dict)</em>: a dictionary with key translations for keys, specified
in <code class="docutils literal notranslate"><span class="pre">inputs</span></code>, e.g. <code class="docutils literal notranslate"><span class="pre">{'old</span> <span class="pre">name</span> <span class="pre">1':</span> <span class="pre">'new</span> <span class="pre">name</span> <span class="pre">1',</span> <span class="pre">'old</span> <span class="pre">name</span> <span class="pre">2':</span> <span class="pre">'new</span> <span class="pre">name</span> <span class="pre">2'}</span></code></p></li>
</ul>
</section>
</section>
<section id="joinlisttask">
<h2>JoinListTask<a class="headerlink" href="#joinlisttask" title="Link to this heading">¶</a></h2>
<p>This Firetask combines the items specified by <strong>spec</strong> keys into a new or existing list in <strong>spec</strong>.</p>
<section id="id6">
<h3>Required parameters<a class="headerlink" href="#id6" title="Link to this heading">¶</a></h3>
<ul class="simple">
<li><p><strong>inputs</strong> <em>(list)</em>: a list of <strong>spec</strong> keys</p></li>
<li><p><strong>output</strong> <em>(str)</em>: a <strong>spec</strong> key under which the joined items will be stored</p></li>
</ul>
</section>
<section id="id7">
<h3>Optional parameters<a class="headerlink" href="#id7" title="Link to this heading">¶</a></h3>
<p>None.</p>
</section>
</section>
<section id="importdatatask">
<h2>ImportDataTask<a class="headerlink" href="#importdatatask" title="Link to this heading">¶</a></h2>
<p>This Firetask updates a dictionary in <strong>spec</strong> with JSON/YAML data from file in a
nested dictionary specified by a map string (see below).</p>
<section id="id8">
<h3>Required parameters<a class="headerlink" href="#id8" title="Link to this heading">¶</a></h3>
<ul class="simple">
<li><p><strong>filename</strong> <em>(str)</em>: a filename from which the data is imported. The name
must end with either <code class="docutils literal notranslate"><span class="pre">.json</span></code> or <code class="docutils literal notranslate"><span class="pre">.yaml</span></code>.</p></li>
<li><p><strong>mapstring</strong> <em>(str)</em>: a map string in the format <code class="docutils literal notranslate"><span class="pre">maplist[0]/maplist[1]/...</span></code>.
At least <code class="docutils literal notranslate"><span class="pre">maplist[0]</span></code> has to be defined because this is the key in <strong>spec</strong>
to be used for the import. Every further nesting can be specified by extending
the mapstring, for example if <code class="docutils literal notranslate"><span class="pre">mapstring</span></code> is <code class="docutils literal notranslate"><span class="pre">maplist[0]/maplist[1]</span></code> then
the data will be imported as <code class="docutils literal notranslate"><span class="pre">spec[maplist[0]][maplist[1]]</span></code>.</p></li>
</ul>
</section>
<section id="id9">
<h3>Optional parameters<a class="headerlink" href="#id9" title="Link to this heading">¶</a></h3>
<p>None.</p>
</section>
</section>
</section>
<div class="clearer"></div>
</div>
</div>
</div>
<div class="sphinxsidebar" role="navigation" aria-label="Main">
<div class="sphinxsidebarwrapper">
<div>
<h3><a href="index.html">Table of Contents</a></h3>
<ul>
<li><a class="reference internal" href="#">Using the dataflow Firetasks</a><ul>
<li><a class="reference internal" href="#commandlinetask">CommandLineTask</a><ul>
<li><a class="reference internal" href="#required-parameters">Required parameters</a></li>
<li><a class="reference internal" href="#optional-parameters">Optional parameters</a></li>
<li><a class="reference internal" href="#example">Example</a></li>
</ul>
</li>
<li><a class="reference internal" href="#foreachtask">ForeachTask</a><ul>
<li><a class="reference internal" href="#id1">Required parameters</a></li>
<li><a class="reference internal" href="#id2">Optional parameters</a></li>
<li><a class="reference internal" href="#id3">Example</a></li>
</ul>
</li>
<li><a class="reference internal" href="#joindicttask">JoinDictTask</a><ul>
<li><a class="reference internal" href="#id4">Required parameters</a></li>
<li><a class="reference internal" href="#id5">Optional parameters</a></li>
</ul>
</li>
<li><a class="reference internal" href="#joinlisttask">JoinListTask</a><ul>
<li><a class="reference internal" href="#id6">Required parameters</a></li>
<li><a class="reference internal" href="#id7">Optional parameters</a></li>
</ul>
</li>
<li><a class="reference internal" href="#importdatatask">ImportDataTask</a><ul>
<li><a class="reference internal" href="#id8">Required parameters</a></li>
<li><a class="reference internal" href="#id9">Optional parameters</a></li>
</ul>
</li>
</ul>
</li>
</ul>
</div>
<div>
<h4>Previous topic</h4>
<p class="topless"><a href="filepad_tasks.html"
title="previous chapter">Using the built-in FilePad Tasks</a></p>
</div>
<div>
<h4>Next topic</h4>
<p class="topless"><a href="controlworker.html"
title="next chapter">Controlling the directory and Worker of execution</a></p>
</div>
<div role="note" aria-label="source link">
<h3>This Page</h3>
<ul class="this-page-menu">
<li><a href="_sources/dataflow_tasks.rst.txt"
rel="nofollow">Show Source</a></li>
</ul>
</div>
<search id="searchbox" style="display: none" role="search">
<h3 id="searchlabel">Quick search</h3>
<div class="searchformwrapper">
<form class="search" action="search.html" method="get">
<input type="text" name="q" aria-labelledby="searchlabel" autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false"/>
<input type="submit" value="Go" />
</form>
</div>
</search>
<script>document.getElementById('searchbox').style.display = "block"</script>
</div>
</div>
<div class="clearer"></div>
</div>
<div class="related" role="navigation" aria-label="Related">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="genindex.html" title="General Index"
>index</a></li>
<li class="right" >
<a href="py-modindex.html" title="Python Module Index"
>modules</a> |</li>
<li class="right" >
<a href="controlworker.html" title="Controlling the directory and Worker of execution"
>next</a> |</li>
<li class="right" >
<a href="filepad_tasks.html" title="Using the built-in FilePad Tasks"
>previous</a> |</li>
<li class="nav-item nav-item-0"><a href="index.html">FireWorks 2.0.4 documentation</a> »</li>
<li class="nav-item nav-item-this"><a href="">Using the dataflow Firetasks</a></li>
</ul>
</div>
<div class="footer" role="contentinfo">
© Copyright 2013, Anubhav Jain.
Created using <a href="https://www.sphinx-doc.org/">Sphinx</a> 8.1.3.
</div>
<script>
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
ga('create', 'UA-53488807-1', 'auto');
ga('send', 'pageview');
</script>
</body>
</html>