-
Notifications
You must be signed in to change notification settings - Fork 189
/
Copy pathtemplatewritertask.html
293 lines (278 loc) · 24.1 KB
/
templatewritertask.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
<!DOCTYPE html>
<html lang="en" data-content_root="./">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<title>Using the TemplateWriterTask to write files — FireWorks 2.0.4 documentation</title>
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=03e43079" />
<link rel="stylesheet" type="text/css" href="_static/nature.css?v=b0beeceb" />
<script src="_static/documentation_options.js?v=adc66a14"></script>
<script src="_static/doctools.js?v=9bcbadda"></script>
<script src="_static/sphinx_highlight.js?v=dc90522c"></script>
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
<link rel="next" title="Using the built-in FileIO Tasks" href="fileiotasks.html" />
<link rel="prev" title="Using the ScriptTask to execute commands" href="scripttask.html" />
<link href='https://fonts.googleapis.com/css?family=Lato:400,700' rel='stylesheet' type='text/css'>
</head><body>
<div class="related" role="navigation" aria-label="Related">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="genindex.html" title="General Index"
accesskey="I">index</a></li>
<li class="right" >
<a href="py-modindex.html" title="Python Module Index"
>modules</a> |</li>
<li class="right" >
<a href="fileiotasks.html" title="Using the built-in FileIO Tasks"
accesskey="N">next</a> |</li>
<li class="right" >
<a href="scripttask.html" title="Using the ScriptTask to execute commands"
accesskey="P">previous</a> |</li>
<li class="nav-item nav-item-0"><a href="index.html">FireWorks 2.0.4 documentation</a> »</li>
<li class="nav-item nav-item-this"><a href="">Using the TemplateWriterTask to write files</a></li>
</ul>
</div>
<div class="document">
<div class="documentwrapper">
<div class="bodywrapper">
<div class="body" role="main">
<section id="using-the-templatewritertask-to-write-files">
<h1>Using the TemplateWriterTask to write files<a class="headerlink" href="#using-the-templatewritertask-to-write-files" title="Link to this heading">¶</a></h1>
<p>A common task in scientific workflows is to write an input file in a format that can be read by a program, and then execute that program. When automating the same program for different inputs (like different molecules or sections of a galaxy), slight modifications to the input file are needed. This tutorial introduces the built-in <em>TemplateWriterTask</em> as a method for writing input files (or any other type of templated file).</p>
<p>We presented an example of using the <em>TemplateWriterTask</em> and subsequently running a program in the <a class="reference internal" href="firetask_tutorial.html"><span class="doc">Firetask tutorial</span></a>. If you didn’t already complete the first part of that tutorial, we suggest you do that first. This tutorial contains more details on how the <em>TemplateWriterTask</em> works.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>The <em>TemplateWriterTask</em> is uses the Jinja2 templating engine, which provides a simple, extensible templating language.</p>
</div>
<section id="a-simple-template-variable-substitutions">
<h2>A simple template - variable substitutions<a class="headerlink" href="#a-simple-template-variable-substitutions" title="Link to this heading">¶</a></h2>
<p>We introduced a simple template in the <a class="reference internal" href="firetask_tutorial.html"><span class="doc">Firetask tutorial</span></a>. Let’s explore this template in more detail.</p>
<ol class="arabic">
<li><p>Navigate to the template writer tutorial directory in your installation directory:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">cd</span> <span class="o"><</span><span class="n">INSTALL_DIR</span><span class="o">>/</span><span class="n">fw_tutorials</span><span class="o">/</span><span class="n">templatewritertask</span>
</pre></div>
</div>
</li>
<li><p>Look inside the file <code class="docutils literal notranslate"><span class="pre">simple_template_copy.txt</span></code>:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">option1</span> <span class="o">=</span> <span class="p">{{</span><span class="n">opt1</span><span class="p">}}</span>
<span class="n">option2</span> <span class="o">=</span> <span class="p">{{</span><span class="n">opt2</span><span class="p">}}</span>
</pre></div>
</div>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>The template file can be any text file, with any extension (or no extension at all). The actual template file is stored within the FireWorks code, so modifying this copy of a file won’t have any effect (more on this later in the tutorial).</p>
</div>
</li>
<li><p>Most of the template file is interpreted literally and is static. However, the text inside double curly braces <code class="docutils literal notranslate"><span class="pre">{{</span></code> and <code class="docutils literal notranslate"><span class="pre">}}</span></code> are <strong>variables</strong> that will be replaced later on by other text.</p></li>
<li><p>We set the variables <strong>opt1</strong> and <strong>opt2</strong> using a <em>Context</em>. This is a dictionary that contains specific values for these parameters. Let’s see how this is defined by looking inside <code class="docutils literal notranslate"><span class="pre">fw_template.yaml</span></code>:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">spec</span><span class="p">:</span>
<span class="n">_tasks</span><span class="p">:</span>
<span class="o">-</span> <span class="n">_fw_name</span><span class="p">:</span> <span class="n">TemplateWriterTask</span>
<span class="n">template_file</span><span class="p">:</span> <span class="n">simple_template</span><span class="o">.</span><span class="n">txt</span>
<span class="n">context</span><span class="p">:</span>
<span class="n">opt1</span><span class="p">:</span> <span class="mf">5.0</span>
<span class="n">opt2</span><span class="p">:</span> <span class="n">fast</span> <span class="n">method</span>
<span class="n">output_file</span><span class="p">:</span> <span class="n">inputs</span><span class="o">.</span><span class="n">txt</span>
</pre></div>
</div>
<p>Note that we have specified a <code class="docutils literal notranslate"><span class="pre">template_file</span></code>, a <code class="docutils literal notranslate"><span class="pre">context</span></code>, and an <code class="docutils literal notranslate"><span class="pre">output_file</span></code>. All three of these parameters are needed to use the <em>TemplateWriterTask</em>.</p>
</li>
<li><p>In the Firework above, we are setting <strong>opt1</strong> to 5.0 and <strong>opt2</strong> to “fast method”. If we wanted to change these parameters, we can create a file like <code class="docutils literal notranslate"><span class="pre">fw_template2.yaml</span></code>:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">spec</span><span class="p">:</span>
<span class="n">_tasks</span><span class="p">:</span>
<span class="o">-</span> <span class="n">_fw_name</span><span class="p">:</span> <span class="n">TemplateWriterTask</span>
<span class="n">template_file</span><span class="p">:</span> <span class="n">simple_template</span><span class="o">.</span><span class="n">txt</span>
<span class="n">context</span><span class="p">:</span>
<span class="n">opt1</span><span class="p">:</span> <span class="mf">10.0</span>
<span class="n">opt2</span><span class="p">:</span> <span class="n">stable</span> <span class="n">method</span>
<span class="n">output_file</span><span class="p">:</span> <span class="n">inputs</span><span class="o">.</span><span class="n">txt</span>
</pre></div>
</div>
</li>
<li><p>This second Firework is identical to the first, except that we have changed the value of variables <strong>opt1</strong> and <strong>opt2</strong>. Thus, we have only changed the parameter values we care about when creating a new Firework. In addition, one could easily perform searches based on <strong>opt1</strong> and <strong>opt2</strong> values using MongoDB’s built-in search capabilities.</p></li>
<li><p>Let’s reset the database, add these FireWorks to the LaunchPad, and then execute them:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">lpad</span> <span class="n">reset</span>
<span class="n">lpad</span> <span class="n">add</span> <span class="n">fw_template</span><span class="o">.</span><span class="n">yaml</span>
<span class="n">lpad</span> <span class="n">add</span> <span class="n">fw_template2</span><span class="o">.</span><span class="n">yaml</span>
<span class="n">rlaunch</span> <span class="o">--</span><span class="n">silencer</span> <span class="n">rapidfire</span>
</pre></div>
</div>
</li>
<li><p>If all went well, you should have two <code class="docutils literal notranslate"><span class="pre">launcher_</span></code> subdirectories. Each directory contains a file called <code class="docutils literal notranslate"><span class="pre">inputs.txt</span></code> that uses the same template file but different Contexts to create unique input files. Recall from the <a class="reference internal" href="firetask_tutorial.html"><span class="doc">Firetask tutorial</span></a> that you could use a multi-task Firework to subsequently run a code that processes the input file to produce useful outputs.</p></li>
</ol>
</section>
<section id="a-more-advanced-template-if-then-and-for">
<h2>A more advanced template - <em>if/then</em> and <em>for</em><a class="headerlink" href="#a-more-advanced-template-if-then-and-for" title="Link to this heading">¶</a></h2>
<p>Template files are not restricted to simple variable substitutions with curly braces. You can also define <em>if/then</em> statements and <em>for loops</em> that process array-like items. This can make your templates more flexible, for example writing an input tag only if a certain variable is present in the Context.</p>
<ol class="arabic">
<li><p>Staying in the template writer tutorial directory, look inside the file <code class="docutils literal notranslate"><span class="pre">advanced_template_copy.txt</span></code>:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">option1</span> <span class="o">=</span> <span class="p">{{</span><span class="n">opt1</span><span class="p">}}</span>
<span class="n">option2</span> <span class="o">=</span> <span class="p">{{</span><span class="n">opt2</span><span class="p">}}</span>
<span class="p">{</span><span class="o">%</span> <span class="k">if</span> <span class="n">optparam</span> <span class="o">%</span><span class="p">}</span><span class="n">OPTIONAL</span> <span class="n">PARAMETER</span>
<span class="p">{{</span> <span class="n">optparam</span> <span class="p">}}{</span><span class="o">%</span> <span class="n">endif</span> <span class="o">%</span><span class="p">}</span>
<span class="n">LOOP</span> <span class="n">PARAMETERS</span>
<span class="p">{</span><span class="o">%</span> <span class="k">for</span> <span class="n">param</span> <span class="ow">in</span> <span class="n">param_list</span> <span class="o">%</span><span class="p">}{{</span> <span class="n">param</span> <span class="p">}}</span>
<span class="p">{</span><span class="o">%</span> <span class="n">endfor</span> <span class="o">%</span><span class="p">}</span>
</pre></div>
</div>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>The actual template file is stored within the FireWorks code, so modifying this copy of a file won’t have any effect (more on this later in the tutorial).</p>
</div>
</li>
<li><p>Note that this template contains some additional tags. In particular, in between <code class="docutils literal notranslate"><span class="pre">{%</span></code> and <code class="docutils literal notranslate"><span class="pre">%}</span></code> we have some code that contains <em>if/then</em> statements and a <em>for</em> loop.</p></li>
<li><p>A Context for this template is in <code class="docutils literal notranslate"><span class="pre">fw_advanced.yaml</span></code>:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">spec</span><span class="p">:</span>
<span class="n">_tasks</span><span class="p">:</span>
<span class="o">-</span> <span class="n">_fw_name</span><span class="p">:</span> <span class="n">TemplateWriterTask</span>
<span class="n">context</span><span class="p">:</span>
<span class="n">opt1</span><span class="p">:</span> <span class="mf">5.0</span>
<span class="n">opt2</span><span class="p">:</span> <span class="n">fast</span> <span class="n">method</span>
<span class="n">optparam</span><span class="p">:</span> <span class="n">true</span>
<span class="n">param_list</span><span class="p">:</span>
<span class="o">-</span> <span class="mi">1</span>
<span class="o">-</span> <span class="mi">2</span>
<span class="o">-</span> <span class="mi">3</span>
<span class="o">-</span> <span class="mi">4</span>
<span class="n">output_file</span><span class="p">:</span> <span class="n">inputs_advanced</span><span class="o">.</span><span class="n">txt</span>
<span class="n">template_file</span><span class="p">:</span> <span class="n">advanced_template</span><span class="o">.</span><span class="n">txt</span>
</pre></div>
</div>
</li>
<li><p>Let’s run this Firework and examine what happens:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">lpad</span> <span class="n">reset</span>
<span class="n">lpad</span> <span class="n">add</span> <span class="n">fw_advanced</span><span class="o">.</span><span class="n">yaml</span>
<span class="n">rlaunch</span> <span class="o">--</span><span class="n">silencer</span> <span class="n">singleshot</span>
</pre></div>
</div>
</li>
<li><p>You’ll notice that we’ve iterated over our loop, and the optional parameter is indeed written to <code class="docutils literal notranslate"><span class="pre">inputs_advanced.txt</span></code>.</p></li>
<li><p>Now, try deleting the line containing the <code class="docutils literal notranslate"><span class="pre">optparam</span></code> and repeating the launch process. You’ll see that the lines pertaining to the <code class="docutils literal notranslate"><span class="pre">OPTIONAL</span> <span class="pre">PARAMETER</span></code> are no longer written!</p></li>
</ol>
<p>Therefore, with Jinja2’s templating language we can write fairly general templates. While variable substitutions, <em>if/then</em> statements, and <em>for loops</em> should cover the majority of cases, you can see even more features in the <a class="reference external" href="http://jinja.pocoo.org">official Jinja2 documentation</a>.</p>
</section>
<section id="writing-your-own-templates">
<h2>Writing your own templates<a class="headerlink" href="#writing-your-own-templates" title="Link to this heading">¶</a></h2>
<p>When writing your own templates, you have a few options on where to store the templates so they can be read by FireWorks. Note that all the worker computers using the templates must have the most recent templates installed locally.</p>
<section id="option-1-the-user-objects-directory-of-the-fireworks-code">
<h3>Option 1: The user_objects directory of the FireWorks code<a class="headerlink" href="#option-1-the-user-objects-directory-of-the-fireworks-code" title="Link to this heading">¶</a></h3>
<p>The default place that FireWorks looks for templates is in the <code class="docutils literal notranslate"><span class="pre">user_objects/firetasks/templates</span></code> directory of your FireWorks installation. Indeed, the <code class="docutils literal notranslate"><span class="pre">simple_template.txt</span></code> and <code class="docutils literal notranslate"><span class="pre">advanced_template.txt</span></code> files used in this tutorial are stored there (that’s why modifying the tutorial files has no effect on the result). Any templates you put in this directory (or its subdirectories) will be read by FireWorks; just put the relative path of your template as the <code class="docutils literal notranslate"><span class="pre">template_file</span></code> parameter.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>If you do not know how to find the correct directory, type <code class="docutils literal notranslate"><span class="pre">lpad</span> <span class="pre">version</span></code>. Then navigate to the install directory, then <code class="docutils literal notranslate"><span class="pre">cd</span> <span class="pre">fireworks/user_objects/firetasks/templates</span></code>.</p>
</div>
</section>
<section id="option-2-set-the-template-directory-in-fwconfig">
<h3>Option 2: Set the template directory in FWConfig<a class="headerlink" href="#option-2-set-the-template-directory-in-fwconfig" title="Link to this heading">¶</a></h3>
<p>If you do not want to store your templates within the FireWorks code, you can set a template directory in the <a class="reference internal" href="config_tutorial.html"><span class="doc">FWConfig</span></a>. Just set the parameter <code class="docutils literal notranslate"><span class="pre">TEMPLATE_DIR</span></code> to point to the location of your templates. Then the <code class="docutils literal notranslate"><span class="pre">template_file</span></code> parameter you pass to your FireWorks will be relative to this path. Remember to do this for all your workers!</p>
</section>
</section>
<section id="additional-options">
<h2>Additional options<a class="headerlink" href="#additional-options" title="Link to this heading">¶</a></h2>
<p>In addition to <code class="docutils literal notranslate"><span class="pre">template_file</span></code>, <code class="docutils literal notranslate"><span class="pre">context</span></code>, and <code class="docutils literal notranslate"><span class="pre">output_file</span></code>, the following options can be passed into <code class="docutils literal notranslate"><span class="pre">TemplateWriterTask</span></code>:</p>
<blockquote>
<div><ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">append</span></code> - append to the output file, rather than overwriting it</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">template_dir</span></code> - this is actually a third option for setting your template dir</p></li>
</ul>
</div></blockquote>
</section>
<section id="the-use-global-spec-option">
<h2>The _use_global_spec option<a class="headerlink" href="#the-use-global-spec-option" title="Link to this heading">¶</a></h2>
<p>By default, the parameters for the TemplateWriterTask should be defined within the <code class="docutils literal notranslate"><span class="pre">_task</span></code> section of the <strong>spec</strong> corresponding to the TemplateWriterTask, not as a root key of the <strong>spec</strong>. If you’d like to instead specify the parameters in the root of the <strong>spec</strong>, you can set <code class="docutils literal notranslate"><span class="pre">_use_global_spec</span></code> to True within the <code class="docutils literal notranslate"><span class="pre">_task</span></code> section. Note that <code class="docutils literal notranslate"><span class="pre">_use_global_spec</span></code> can simplify querying and communication of parameters between FireWorks but can cause problems if you have multiple TemplateWriterTasks within the same Firework.</p>
</section>
<section id="python-example">
<h2>Python example<a class="headerlink" href="#python-example" title="Link to this heading">¶</a></h2>
<p>A runnable Python example illustrating the use of templates was given in the <a class="reference internal" href="firetask_tutorial.html"><span class="doc">Firetask tutorial</span></a>.</p>
</section>
</section>
<div class="clearer"></div>
</div>
</div>
</div>
<div class="sphinxsidebar" role="navigation" aria-label="Main">
<div class="sphinxsidebarwrapper">
<div>
<h3><a href="index.html">Table of Contents</a></h3>
<ul>
<li><a class="reference internal" href="#">Using the TemplateWriterTask to write files</a><ul>
<li><a class="reference internal" href="#a-simple-template-variable-substitutions">A simple template - variable substitutions</a></li>
<li><a class="reference internal" href="#a-more-advanced-template-if-then-and-for">A more advanced template - <em>if/then</em> and <em>for</em></a></li>
<li><a class="reference internal" href="#writing-your-own-templates">Writing your own templates</a><ul>
<li><a class="reference internal" href="#option-1-the-user-objects-directory-of-the-fireworks-code">Option 1: The user_objects directory of the FireWorks code</a></li>
<li><a class="reference internal" href="#option-2-set-the-template-directory-in-fwconfig">Option 2: Set the template directory in FWConfig</a></li>
</ul>
</li>
<li><a class="reference internal" href="#additional-options">Additional options</a></li>
<li><a class="reference internal" href="#the-use-global-spec-option">The _use_global_spec option</a></li>
<li><a class="reference internal" href="#python-example">Python example</a></li>
</ul>
</li>
</ul>
</div>
<div>
<h4>Previous topic</h4>
<p class="topless"><a href="scripttask.html"
title="previous chapter">Using the ScriptTask to execute commands</a></p>
</div>
<div>
<h4>Next topic</h4>
<p class="topless"><a href="fileiotasks.html"
title="next chapter">Using the built-in FileIO Tasks</a></p>
</div>
<div role="note" aria-label="source link">
<h3>This Page</h3>
<ul class="this-page-menu">
<li><a href="_sources/templatewritertask.rst.txt"
rel="nofollow">Show Source</a></li>
</ul>
</div>
<search id="searchbox" style="display: none" role="search">
<h3 id="searchlabel">Quick search</h3>
<div class="searchformwrapper">
<form class="search" action="search.html" method="get">
<input type="text" name="q" aria-labelledby="searchlabel" autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false"/>
<input type="submit" value="Go" />
</form>
</div>
</search>
<script>document.getElementById('searchbox').style.display = "block"</script>
</div>
</div>
<div class="clearer"></div>
</div>
<div class="related" role="navigation" aria-label="Related">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="genindex.html" title="General Index"
>index</a></li>
<li class="right" >
<a href="py-modindex.html" title="Python Module Index"
>modules</a> |</li>
<li class="right" >
<a href="fileiotasks.html" title="Using the built-in FileIO Tasks"
>next</a> |</li>
<li class="right" >
<a href="scripttask.html" title="Using the ScriptTask to execute commands"
>previous</a> |</li>
<li class="nav-item nav-item-0"><a href="index.html">FireWorks 2.0.4 documentation</a> »</li>
<li class="nav-item nav-item-this"><a href="">Using the TemplateWriterTask to write files</a></li>
</ul>
</div>
<div class="footer" role="contentinfo">
© Copyright 2013, Anubhav Jain.
Created using <a href="https://www.sphinx-doc.org/">Sphinx</a> 8.1.3.
</div>
<script>
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
ga('create', 'UA-53488807-1', 'auto');
ga('send', 'pageview');
</script>
</body>
</html>