-
Notifications
You must be signed in to change notification settings - Fork 189
/
Copy pathqueue_tutorial.html
459 lines (444 loc) · 41.5 KB
/
queue_tutorial.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
<!DOCTYPE html>
<html lang="en" data-content_root="./">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<title>Launch Rockets through a queue — FireWorks 2.0.4 documentation</title>
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=03e43079" />
<link rel="stylesheet" type="text/css" href="_static/nature.css?v=b0beeceb" />
<script src="_static/documentation_options.js?v=adc66a14"></script>
<script src="_static/doctools.js?v=9bcbadda"></script>
<script src="_static/sphinx_highlight.js?v=dc90522c"></script>
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
<link rel="next" title="Advanced queue submission (reservation mode)" href="queue_tutorial_pt2.html" />
<link rel="prev" title="Worker Tutorial" href="worker_tutorial.html" />
<link href='https://fonts.googleapis.com/css?family=Lato:400,700' rel='stylesheet' type='text/css'>
</head><body>
<div class="related" role="navigation" aria-label="Related">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="genindex.html" title="General Index"
accesskey="I">index</a></li>
<li class="right" >
<a href="py-modindex.html" title="Python Module Index"
>modules</a> |</li>
<li class="right" >
<a href="queue_tutorial_pt2.html" title="Advanced queue submission (reservation mode)"
accesskey="N">next</a> |</li>
<li class="right" >
<a href="worker_tutorial.html" title="Worker Tutorial"
accesskey="P">previous</a> |</li>
<li class="nav-item nav-item-0"><a href="index.html">FireWorks 2.0.4 documentation</a> »</li>
<li class="nav-item nav-item-this"><a href="">Launch Rockets through a queue</a></li>
</ul>
</div>
<div class="document">
<div class="documentwrapper">
<div class="bodywrapper">
<div class="body" role="main">
<section id="launch-rockets-through-a-queue">
<h1>Launch Rockets through a queue<a class="headerlink" href="#launch-rockets-through-a-queue" title="Link to this heading">¶</a></h1>
<p>If your FireWorker is a large, shared resource (such as a computing cluster or supercomputing center), you probably won’t be able to launch Rockets directly. Instead, you’ll submit Rockets through an existing queueing system that allocates computer resources.</p>
<p>The simplest way to execute jobs through a queue would be to write a templated queue file and then submit it as a two-task Firework, as in the <a class="reference internal" href="firetask_tutorial.html"><span class="doc">Firetask tutorial</span></a>. However, FireWorks then considers your “job” to only be queue submission, and will consider the job completed after the queue submission is complete. FireWorks will not know when the actual payload starts running, or is finished, or if the job finishes successfully. Thus, many of the useful management and monitoring features of FireWorks will not be available to you.</p>
<p>A more powerful way to execute jobs through a queue is presented in this tutorial. In this method, the queue file runs <code class="docutils literal notranslate"><span class="pre">rlaunch</span></code> instead of running your desired program. This method is just like typing <code class="docutils literal notranslate"><span class="pre">rlaunch</span></code> into a Terminal window like in the core tutorials, except that now we are submitting a queue script that does the typing for us (it’s very low-tech!). In particular, FireWorks is <em>completely unaware</em> that you are running through a queue!</p>
<p>The jobs we will submit to the queue are basically placeholder jobs that are asleep until the job starts running. When the job is actually assigned computer resources and runs, the script “wakes” up and runs the Rocket Launcher, which then figures out what Firework to run.</p>
<p>The advantage of this low-tech system is that it is quite durable; if your queue system goes down or you delete a job from the queue, there are zero repercussions. You don’t have to tell FireWorks to run those jobs somewhere else, because FireWorks never knew about your queue in the first place. In addition, if you are running on multiple machines and the queue becomes backlogged on one of them, it does not matter at all. Your submission job stuck in the queue is not preventing high-priority jobs from running on other machines.</p>
<p>There are also some disadvantages to this simple system for which you might want to tell FireWorks about the queue. We’ll discuss these limitations at the end of the tutorial and direct you on how to overcome them in the next tutorial. For now, we suggest that you get things working simply.</p>
<section id="launch-a-single-job-through-a-queue">
<h2>Launch a single job through a queue<a class="headerlink" href="#launch-a-single-job-through-a-queue" title="Link to this heading">¶</a></h2>
<p>To get warmed up, let’s set up a <em>Queue Launcher</em> to run a single Firework through a queueing system.</p>
<section id="configure-the-queue-launcher">
<h3>Configure the Queue Launcher<a class="headerlink" href="#configure-the-queue-launcher" title="Link to this heading">¶</a></h3>
<p>The Queue Launcher needs to write and submit a queue script that contains an executable (in our case, a Rocket Launcher). This is achieved through a QueueAdapter file.</p>
<ol class="arabic">
<li><p>Move to the <code class="docutils literal notranslate"><span class="pre">queue</span></code> tutorial directory on your FireWorker:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">cd</span> <span class="o"><</span><span class="n">INSTALL_DIR</span><span class="o">>/</span><span class="n">fw_tutorials</span><span class="o">/</span><span class="n">queue</span>
</pre></div>
</div>
</li>
<li><p>Locate an appropriate QueueAdapter file. The files are usually named <code class="docutils literal notranslate"><span class="pre">qadapter_<QUEUE>.yaml</span></code> where <code class="docutils literal notranslate"><span class="pre"><QUEUE></span></code> is the supported queue system.</p></li>
</ol>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>If you cannot find a working QueueAdapter file for your specific queuing system, or the queue script needs modification, you can <a class="reference internal" href="qadapter_programming.html"><span class="doc">write/modify your own QueueAdapter</span></a>. Or, simply contact us for help (see <a class="reference internal" href="index.html#contributing-label"><span class="std std-ref">Contributing / Contact / Bug Reports</span></a>) and we will help create one for your system.</p>
</div>
<ol class="arabic">
<li><p>Copy your chosen QueueAdapter file to a new name:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">cp</span> <span class="n">qadapter_</span><span class="o"><</span><span class="n">QUEUE</span><span class="o">>.</span><span class="n">yaml</span> <span class="n">my_qadapter</span><span class="o">.</span><span class="n">yaml</span>
</pre></div>
</div>
</li>
<li><p>Navigate to clean working directory on the FireWorker. For example:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">mkdir</span> <span class="o">~/</span><span class="n">queue_tests</span>
<span class="n">cd</span> <span class="o">~/</span><span class="n">queue_tests</span>
</pre></div>
</div>
</li>
<li><p>Copy over your queue file and the test FW to this directory:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">cp</span> <span class="o"><</span><span class="n">INSTALL_DIR</span><span class="o">>/</span><span class="n">fw_tutorials</span><span class="o">/</span><span class="n">queue</span><span class="o">/</span><span class="n">my_qadapter</span><span class="o">.</span><span class="n">yaml</span> <span class="o">.</span>
<span class="n">cp</span> <span class="o"><</span><span class="n">INSTALL_DIR</span><span class="o">>/</span><span class="n">fw_tutorials</span><span class="o">/</span><span class="n">queue</span><span class="o">/</span><span class="n">fw_test</span><span class="o">.</span><span class="n">yaml</span> <span class="o">.</span>
</pre></div>
</div>
</li>
<li><p>Copy over your LaunchPad and FireWorker files from the worker tutorial:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">cp</span> <span class="o"><</span><span class="n">INSTALL_DIR</span><span class="o">>/</span><span class="n">fw_tutorials</span><span class="o">/</span><span class="n">worker</span><span class="o">/</span><span class="n">my_fworker</span><span class="o">.</span><span class="n">yaml</span> <span class="o">.</span>
<span class="n">cp</span> <span class="o"><</span><span class="n">INSTALL_DIR</span><span class="o">>/</span><span class="n">fw_tutorials</span><span class="o">/</span><span class="n">worker</span><span class="o">/</span><span class="n">my_launchpad</span><span class="o">.</span><span class="n">yaml</span> <span class="o">.</span>
</pre></div>
</div>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>If you do not have these files, please go back and regenerate them according to the instructions <a class="reference internal" href="worker_tutorial.html"><span class="doc">here</span></a>.</p>
</div>
</li>
<li><p>Open <code class="docutils literal notranslate"><span class="pre">my_qadapter.yaml</span></code> and modify it as follows:</p>
<ol class="loweralpha simple">
<li><p>In the part that specifies running <code class="docutils literal notranslate"><span class="pre">rlaunch</span></code>, modify the <code class="docutils literal notranslate"><span class="pre">path/to/my_fworker.yaml</span></code> to contain the <strong>absolute path</strong> of the <code class="docutils literal notranslate"><span class="pre">my_fworker.yaml</span></code> file on your machine.</p></li>
<li><p>On the same line, modify the <code class="docutils literal notranslate"><span class="pre">path/to/my_launchpad.yaml</span></code> to contain the <strong>absolute path</strong> of the <code class="docutils literal notranslate"><span class="pre">my_launchpad.yaml</span></code> file on your machine.</p></li>
<li><p>For the logdir parameter, modify the <code class="docutils literal notranslate"><span class="pre">path/to/logging</span></code> text to contain the <strong>absolute path</strong> of where you would like FireWorks logs to go. For example, you might create a <code class="docutils literal notranslate"><span class="pre">fw_logs</span></code> directory inside your home directory, and point the logdir parameter there.</p></li>
</ol>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Be sure to indicate the full, absolute path name; do not use BASH shortcuts like ‘.’, ‘..’, or ‘~’, and do not indicate a relative path.</p>
</div>
</li>
</ol>
<p>You are now ready to begin!</p>
</section>
<section id="add-some-fireworks">
<h3>Add some FireWorks<a class="headerlink" href="#add-some-fireworks" title="Link to this heading">¶</a></h3>
<p>Staying in your testing directory, let’s reset our database and add a new Firework, all from our FireWorker:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">lpad</span> <span class="n">reset</span>
<span class="n">lpad</span> <span class="n">add</span> <span class="n">fw_test</span><span class="o">.</span><span class="n">yaml</span>
</pre></div>
</div>
</section>
<section id="submit-a-job">
<h3>Submit a job<a class="headerlink" href="#submit-a-job" title="Link to this heading">¶</a></h3>
<ol class="arabic">
<li><p>Try submitting a job using the command:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">qlaunch</span> <span class="n">singleshot</span>
</pre></div>
</div>
</li>
</ol>
<blockquote>
<div><div class="admonition tip">
<p class="admonition-title">Tip</p>
<p>Similar to the Rocket Launcher, if you use the names <code class="docutils literal notranslate"><span class="pre">my_launchpad.yaml</span></code>, <code class="docutils literal notranslate"><span class="pre">my_fworker.yaml</span></code>, and <code class="docutils literal notranslate"><span class="pre">my_qadapter.yaml</span></code>, then you don’t need to specify the <code class="docutils literal notranslate"><span class="pre">-l</span></code>, <code class="docutils literal notranslate"><span class="pre">-w</span></code>, and <code class="docutils literal notranslate"><span class="pre">-q</span></code> options explicitly. FireWorks will automatically search for these files in the current directory, or in a configuration directory that you specify with a single <code class="docutils literal notranslate"><span class="pre">-c</span></code> parameter, or in the directories specified by your <a class="reference internal" href="config_tutorial.html"><span class="doc">FWConfig file</span></a>.</p>
</div>
</div></blockquote>
<ol class="arabic">
<li><p>This should have submitted a job to the queue in the current directory. You can read the log files in the logging directory, and/or check the status of your queue to ensure your job appeared.</p></li>
<li><p>After your queue manager runs your job, you should see the file <code class="docutils literal notranslate"><span class="pre">howdy.txt</span></code> in the current directory.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Here, it is possible you have errors in connecting to your database. The first thing to double-check is that you are correctly referencing the correct <code class="docutils literal notranslate"><span class="pre">my_launchpad.yaml</span></code> file in the <code class="docutils literal notranslate"><span class="pre">rocket_launch</span></code> command of your <code class="docutils literal notranslate"><span class="pre">my_qadapter.yaml</span></code> and that this file is properly set up to connect to the correct host/port with the right credentials. The second thing to verify is whether your MongoDB server can accept inbound connections. This is typically not a problem if you use a cloud service like <a class="reference external" href="http://mlab.com/">mLab</a> or <code class="docutils literal notranslate"><span class="pre">MongoDB</span> <span class="pre">Atlas</span></code>, but can be tricky if you are hosting your own MongoDB server. You should try connecting to your server from outside computers and see if it works using the credentials/information in your <code class="docutils literal notranslate"><span class="pre">my_launchpad.yaml</span></code>. Third, in some cases, firewall issues on shared resources prevent your compute node from accessing the LaunchPad hosted on your FireServer - meaning the outgoing connection is blocked. You should confirm that your compute nodes can access external database servers. You might try to submit an <em>interactive job</em> to your queue that allows you to type shell commands inside a running job. Once on the compute node, you can try accessing your LaunchPad: <code class="docutils literal notranslate"><span class="pre">lpad</span> <span class="pre">-l</span> <span class="pre">my_launchpad.yaml</span> <span class="pre">get_fw</span> <span class="pre">1</span></code>. If you cannot access the LaunchPad from your compute node, the first thing to do is contact a system administrator for assistance. If you are convinced that there is no way for the compute nodes to access a network, you might try <a class="reference internal" href="offline_tutorial.html"><span class="doc">running FireWorks in offline mode</span></a>.</p>
</div>
</li>
</ol>
<p>If everything ran successfully, congratulations! You just executed a Firework through a queue!</p>
</section>
</section>
<section id="adding-more-power-using-rapid-fire-mode">
<h2>Adding more power: using rapid-fire mode<a class="headerlink" href="#adding-more-power-using-rapid-fire-mode" title="Link to this heading">¶</a></h2>
<p>While launching a single job to a queue is nice, a more powerful use case is to submit a large number of jobs at once, or to maintain a certain number of jobs in the queue. Like the Rocket Launcher, the Queue Launcher can be run in a “rapid-fire” mode that provides these features.</p>
<ol class="arabic">
<li><p>Clean your working directory of everything but four files: <code class="docutils literal notranslate"><span class="pre">fw_test.yaml</span></code>, <code class="docutils literal notranslate"><span class="pre">my_qadapter.yaml</span></code>, <code class="docutils literal notranslate"><span class="pre">my_fworker.yaml</span></code>, and <code class="docutils literal notranslate"><span class="pre">my_launchpad.yaml</span></code></p></li>
<li><p>Let’s reset our database and add three new FireWorks, all from our FireWorker:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">lpad</span> <span class="n">reset</span>
<span class="n">lpad</span> <span class="n">add</span> <span class="n">fw_test</span><span class="o">.</span><span class="n">yaml</span>
<span class="n">lpad</span> <span class="n">add</span> <span class="n">fw_test</span><span class="o">.</span><span class="n">yaml</span>
<span class="n">lpad</span> <span class="n">add</span> <span class="n">fw_test</span><span class="o">.</span><span class="n">yaml</span>
</pre></div>
</div>
</li>
<li><p>Submit several jobs with a single command:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">qlaunch</span> <span class="n">rapidfire</span> <span class="o">-</span><span class="n">m</span> <span class="mi">3</span>
</pre></div>
</div>
<div class="admonition important">
<p class="admonition-title">Important</p>
<p>The Queue Launcher sleeps between each job submission to give time for the queue manager to ‘breathe’. It might take a few minutes to submit all the jobs.</p>
</div>
<div class="admonition important">
<p class="admonition-title">Important</p>
<p>The command above submits jobs until you have at most 3 jobs in the queue under your username. If you had some jobs existing in the queue before running this command, you might need to increase the <code class="docutils literal notranslate"><span class="pre">-m</span></code> parameter.</p>
</div>
</li>
<li><p>The rapid-fire command should have created a directory beginning with the tag <code class="docutils literal notranslate"><span class="pre">block_</span></code>. Navigate inside this directory, and confirm that three directories starting with the tag <code class="docutils literal notranslate"><span class="pre">launch</span></code> were created. The <code class="docutils literal notranslate"><span class="pre">launch</span></code> directories contain your individual jobs.</p></li>
</ol>
<p>You’ve now launched multiple Rockets with a single command, all through a queueing system!</p>
</section>
<section id="continually-submit-jobs-to-the-queue">
<h2>Continually submit jobs to the queue<a class="headerlink" href="#continually-submit-jobs-to-the-queue" title="Link to this heading">¶</a></h2>
<p>You might want to set up your worker so that it maintains a certain number of jobs in the queue indefinitely. That way, it will continuously pull FireWorks from the LaunchPad. Let’s set this up.</p>
<ol class="arabic">
<li><p>Clean your working directory of everything but four files: <code class="docutils literal notranslate"><span class="pre">fw_test.yaml</span></code>, <code class="docutils literal notranslate"><span class="pre">my_qadapter.yaml</span></code>, <code class="docutils literal notranslate"><span class="pre">my_fworker.yaml</span></code>, and <code class="docutils literal notranslate"><span class="pre">my_launchpad.yaml</span></code>.</p></li>
<li><p>Let’s reset our database and add four new FireWorks this time:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">lpad</span> <span class="n">reset</span>
<span class="n">lpad</span> <span class="n">add</span> <span class="n">fw_test</span><span class="o">.</span><span class="n">yaml</span>
<span class="n">lpad</span> <span class="n">add</span> <span class="n">fw_test</span><span class="o">.</span><span class="n">yaml</span>
<span class="n">lpad</span> <span class="n">add</span> <span class="n">fw_test</span><span class="o">.</span><span class="n">yaml</span>
<span class="n">lpad</span> <span class="n">add</span> <span class="n">fw_test</span><span class="o">.</span><span class="n">yaml</span>
</pre></div>
</div>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>We have omitted the <code class="docutils literal notranslate"><span class="pre">-l</span></code> parameter. You can use this shortcut when using the standard file name (<code class="docutils literal notranslate"><span class="pre">my_launchpad.yaml</span></code>) for the LaunchPad.</p>
</div>
</li>
<li><p>Run the queue launcher in <strong>infinite</strong> mode:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">qlaunch</span> <span class="n">rapidfire</span> <span class="o">-</span><span class="n">m</span> <span class="mi">2</span> <span class="o">--</span><span class="n">nlaunches</span> <span class="n">infinite</span>
</pre></div>
</div>
</li>
<li><p>This command will always maintain 2 jobs in the queue. When a job finishes, another will be submitted to take its place!</p></li>
</ol>
</section>
<section id="running-multiple-rockets-per-queue-job">
<h2>Running multiple Rockets per queue job<a class="headerlink" href="#running-multiple-rockets-per-queue-job" title="Link to this heading">¶</a></h2>
<p>So far, each queue script we submitted has only one job. We can also submit multiple jobs per queue script by running the <code class="docutils literal notranslate"><span class="pre">rapidfire</span></code> option of the <em>Rocket Launcher</em> inside the Queue Launcher. Then, a single queue script will run multiple Rockets.</p>
<ol class="arabic">
<li><p>Clean your working directory of everything but four files: <code class="docutils literal notranslate"><span class="pre">fw_test.yaml</span></code>, <code class="docutils literal notranslate"><span class="pre">my_qadapter.yaml</span></code>, <code class="docutils literal notranslate"><span class="pre">my_fworker.yaml</span></code>, and <code class="docutils literal notranslate"><span class="pre">my_launchpad.yaml</span></code>.</p></li>
<li><p>Copy your QueueAdapter file to <code class="docutils literal notranslate"><span class="pre">my_qp_multi.yaml</span></code>:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">cp</span> <span class="n">my_qadapter</span><span class="o">.</span><span class="n">yaml</span> <span class="n">my_qp_multi</span><span class="o">.</span><span class="n">yaml</span>
</pre></div>
</div>
</li>
<li><p>Edit <code class="docutils literal notranslate"><span class="pre">my_qp_multi.yaml</span></code> as follows:</p>
<blockquote>
<div><ol class="loweralpha simple">
<li><p>In the part that specifies running <code class="docutils literal notranslate"><span class="pre">rlaunch</span></code>, modify the <code class="docutils literal notranslate"><span class="pre">singleshot</span></code> text to read <code class="docutils literal notranslate"><span class="pre">rapidfire</span></code>.</p></li>
</ol>
</div></blockquote>
</li>
<li><p>Let’s add three FireWorks to the LaunchPad and submit a <em>single</em> queue script:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">lpad</span> <span class="n">reset</span>
<span class="n">lpad</span> <span class="n">add</span> <span class="n">fw_test</span><span class="o">.</span><span class="n">yaml</span>
<span class="n">lpad</span> <span class="n">add</span> <span class="n">fw_test</span><span class="o">.</span><span class="n">yaml</span>
<span class="n">lpad</span> <span class="n">add</span> <span class="n">fw_test</span><span class="o">.</span><span class="n">yaml</span>
<span class="n">qlaunch</span> <span class="o">-</span><span class="n">q</span> <span class="n">my_qp_multi</span><span class="o">.</span><span class="n">yaml</span> <span class="n">singleshot</span>
</pre></div>
</div>
</li>
<li><p>You should confirm that only a single job got submitted to the queue. However, when the job starts running, you’ll see that all three of your jobs completed in separate <code class="docutils literal notranslate"><span class="pre">launcher_</span></code> directories!</p></li>
</ol>
<div class="admonition warning">
<p class="admonition-title">Warning</p>
<p>Note that when running in rapidfire mode, there is an increased likelihood that a Firework will be killed by the job walltime. To mitigate this, you can either limit the number of jobs executed by rapidfire (using either the <code class="docutils literal notranslate"><span class="pre">nlaunches</span></code> or <code class="docutils literal notranslate"><span class="pre">timeout</span></code> parameters), or you can let the Firework be killed and use the error recovery features (see docs on that topic) to rerun the killed Firework.</p>
</div>
</section>
<section id="remote-qlaunch">
<h2>Remote qlaunch<a class="headerlink" href="#remote-qlaunch" title="Link to this heading">¶</a></h2>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Remote qlaunch requires the <a class="reference external" href="http://docs.fabfile.org/">Fabric</a> package
to be installed.</p>
</div>
<p>The qlaunch command also comes with options to do simple remote queue
administration. This remote capability is extremely useful if you need to
maintain jobs in queues across a number of computing resources from a single
location (as opposed to ssh into each resource and doing qlaunch).</p>
<p>A few recommendations:</p>
<ol class="arabic simple">
<li><p>It is helpful if you configure all your fireworks in all the resources
you want to use similarly. For example, you can use the default
$HOME/.fireworks location, or setup every resource in a similar
location.</p></li>
<li><p>Passwordless ssh should ideally be configured for all clusters from the
machine you want to run qlaunch from. While qlaunch provides a “-rp”
option to specify the password, it is less secure and less powerful (e.g.,
you can’t manage lots of resources with a single command).</p></li>
</ol>
<section id="sample-usage">
<h3>Sample usage<a class="headerlink" href="#sample-usage" title="Link to this heading">¶</a></h3>
<p>All remote options start with “-r” or “–remote”.</p>
<ol class="arabic">
<li><p>Running qlaunch rapidfire on one server:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">qlaunch</span> <span class="o">-</span><span class="n">rh</span> <span class="n">compute</span><span class="o">.</span><span class="n">host</span><span class="o">.</span><span class="n">gov</span> <span class="o">-</span><span class="n">ru</span> <span class="n">user</span> <span class="n">rapidfire</span> <span class="o">-</span><span class="n">m</span> <span class="mi">50</span>
</pre></div>
</div>
<p>Note that rapidfire options such as “-m 50” are automatically transmitted
to the resource.</p>
</li>
<li><p>Running qlaunch rapidfire on a host with multiple queue configurations.
This is useful when you have multiple FireWorks configurations (e.g.
different queue or FireWorker configurations) for a single resource. A single
command runs qlaunch rapidfire on all configurations:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">qlaunch</span> <span class="o">-</span><span class="n">rh</span> <span class="n">compute</span><span class="o">.</span><span class="n">host</span><span class="o">.</span><span class="n">gov</span> <span class="o">-</span><span class="n">rc</span> <span class="o">/</span><span class="n">path</span><span class="o">/</span><span class="n">to</span><span class="o">/</span><span class="n">config1</span> <span class="o">/</span><span class="n">path</span><span class="o">/</span><span class="n">to</span><span class="o">/</span><span class="n">config2</span> <span class="o">-</span><span class="n">ru</span> <span class="n">user</span> <span class="n">rapidfire</span>
</pre></div>
</div>
</li>
<li><p>Running qlaunch rapidfire on multiple hosts with the same username.
Without the rc option, it is assumed that $HOME/.fireworks is where the
fireworks configuration is located on all hosts:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">qlaunch</span> <span class="o">-</span><span class="n">rh</span> <span class="n">compute</span><span class="o">.</span><span class="n">host1</span><span class="o">.</span><span class="n">gov</span> <span class="n">compute</span><span class="o">.</span><span class="n">host2</span><span class="o">.</span><span class="n">gov</span> <span class="o">-</span><span class="n">ru</span> <span class="n">user</span> <span class="n">rapidfire</span>
</pre></div>
</div>
</li>
</ol>
</section>
<section id="limitations">
<h3>Limitations<a class="headerlink" href="#limitations" title="Link to this heading">¶</a></h3>
<p>To keep the code simple, qlaunch default remote options are limited to
similar configurations across multiple resources. If you have more
complicated setups, e.g., different users, different queue configurations
across different computing resources, remote qlaunch will not be able to
handle these. However, you can always write your own fabfile.py and use
Fabric’s far more sophisticated execution model.</p>
<p>A simple fabfile.py for different users on different hosts is given below. If
you require Fabric’s sophistication, I encourage you to read <a class="reference external" href="http://docs.fabfile.org/">Fabric’s
official documentation</a>:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span><span class="w"> </span><span class="nn">fabric.api</span><span class="w"> </span><span class="kn">import</span> <span class="n">run</span><span class="p">,</span> <span class="n">env</span>
<span class="n">env</span><span class="o">.</span><span class="n">hosts</span> <span class="o">=</span> <span class="p">[</span><span class="s2">"user1@compute.host1.gov"</span><span class="p">,</span> <span class="s2">"user2@compute.host2.gov"</span><span class="p">]</span>
<span class="k">def</span><span class="w"> </span><span class="nf">qlaunch</span><span class="p">():</span>
<span class="n">run</span><span class="p">(</span><span class="s1">'qlaunch -c $HOME/.fireworks rapidfire'</span><span class="p">)</span>
</pre></div>
</div>
</section>
</section>
<section id="more-information">
<h2>More information<a class="headerlink" href="#more-information" title="Link to this heading">¶</a></h2>
<ol class="arabic">
<li><p>As with all FireWorks scripts, you can run the built-in help for more information:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">qlaunch</span> <span class="o">-</span><span class="n">h</span>
<span class="n">qlaunch</span> <span class="n">singleshot</span> <span class="o">-</span><span class="n">h</span>
<span class="n">qlaunch</span> <span class="n">rapidfire</span> <span class="o">-</span><span class="n">h</span>
</pre></div>
</div>
</li>
</ol>
</section>
<section id="submitting-to-the-queue-with-python">
<h2>Submitting to the queue with Python<a class="headerlink" href="#submitting-to-the-queue-with-python" title="Link to this heading">¶</a></h2>
<p>Although most people will submit jobs to the queue with the command line, you can also submit jobs via Python. The example is similar to previous tutorials for running the <code class="docutils literal notranslate"><span class="pre">rapidfire</span></code> command, however you want to replace <code class="docutils literal notranslate"><span class="pre">fireworks.core.rocket_launcher.rapidfire</span></code> with <code class="docutils literal notranslate"><span class="pre">fireworks.core.queue_launcher.rapidfire</span></code> and adjust the parameters accordingly.</p>
</section>
<section id="limitations-and-next-steps">
<h2>Limitations and Next Steps<a class="headerlink" href="#limitations-and-next-steps" title="Link to this heading">¶</a></h2>
<p>The information in this tutorial might be all you need to automate your application. However, as we noted previously, there are some limitations to running under a model in which FireWorks is completely unaware of the existence of queues. In the simple queue execution model, limitations include:</p>
<ol class="arabic simple">
<li><p><strong>You can’t track how many of your jobs are queued</strong></p></li>
</ol>
<p>Since FireWorks is unaware of your queue, there’s no way to track how many of your jobs are queued up on various machines. You’ll have to wait until they start running before their presence is reported to FireWorks.</p>
<ol class="arabic simple" start="2">
<li><p><strong>You might submit too many jobs to the queue</strong></p></li>
</ol>
<p>It’s possible to submit more queue scripts than exist jobs in the database. Before submitting a queue script, the Queue Launcher checks that at least one unstarted job exists in the database. However, let’s take an example where you have one Firework in the database that’s ready to run. Nothing in the current system prevents you from using the Queue Launcher to rapid-fire 20 jobs to the queue. You won’t be prevented from submitting queue scripts until that Firework has actually started running.</p>
<p>If the number of jobs in your database is kept much higher than the number of jobs you keep in your queues, then you shouldn’t run into this problem at all; all your submitted queue scripts will always find a job to run. Even if this is not the case, the additional queue scripts should pose only a minor penalty. Any extra queue scripts will wake up, find nothing to do, and exit without wasting more than few seconds of computer time. If you are using rapid-fire mode, you’ll also end up with an additional <code class="docutils literal notranslate"><span class="pre">launcher_</span></code> directory. You can look at the REMOVE_USELESS_DIRS option of the <a class="reference internal" href="config_tutorial.html"><span class="doc">FW config</span></a> as a solution to this.</p>
<ol class="arabic simple" start="3">
<li><p><strong>You can’t easily tailor queue parameters (e.g. walltime) individually for each the job</strong></p></li>
</ol>
<p>Perhaps the most severe limitation is that the Queue Launcher submits queue scripts with identical queue parameters (e.g., all jobs will have the same walltime, use the same number of cores, etc.)</p>
<p>If you have just two or three sets of queue parameters for your different job types, you can work around this limitation. First, recall that you can use the FireWorker file to restrict which jobs get run (see tutorial). If you have two types of jobs, you can run <em>two</em> Queue Launchers. Each of these Queue Launchers use different queue parameters, corresponding to the two types of jobs you’d like to run. In addition, each Queue Launcher should be run with a corresponding FireWorker that restricts that jobs for that launcher to the desired job type.</p>
<p>While this solution works for a few different job types, it is not practical if you have many job types. In addition, it requires some coordination between Firework categories, FireWorkers, and Queue Launchers. Therefore, if setting multiple sets of queue parameters is needed for your application, we suggest that you read on for a solution.</p>
<p><strong>To solve these problems, you must *reserve* FireWorks in advance</strong>.</p>
<section id="next-step-reserving-fireworks-to-overcome-limitations">
<h3>Next step: reserving FireWorks to overcome limitations<a class="headerlink" href="#next-step-reserving-fireworks-to-overcome-limitations" title="Link to this heading">¶</a></h3>
<p>If you feel these limitations impact your workflow, you should forge on to the next tutorial: <a class="reference internal" href="queue_tutorial_pt2.html"><span class="doc">Reserving FireWorks upon queue submission</span></a>. We’ll explain how <em>reserving</em> FireWorks upon queue submission can solve the limitations of simple queue submission, at the expense of added complexity and introducing some new limitations and considerations.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>If you are planning to complete the next tutorial, you should save your working directory with the files: <code class="docutils literal notranslate"><span class="pre">fw_test.yaml</span></code>, <code class="docutils literal notranslate"><span class="pre">my_qadapter.yaml</span></code>, <code class="docutils literal notranslate"><span class="pre">my_fworker.yaml</span></code>, and <code class="docutils literal notranslate"><span class="pre">my_launchpad.yaml</span></code>. We’ll use it in the next tutorial.</p>
</div>
</section>
</section>
</section>
<div class="clearer"></div>
</div>
</div>
</div>
<div class="sphinxsidebar" role="navigation" aria-label="Main">
<div class="sphinxsidebarwrapper">
<div>
<h3><a href="index.html">Table of Contents</a></h3>
<ul>
<li><a class="reference internal" href="#">Launch Rockets through a queue</a><ul>
<li><a class="reference internal" href="#launch-a-single-job-through-a-queue">Launch a single job through a queue</a><ul>
<li><a class="reference internal" href="#configure-the-queue-launcher">Configure the Queue Launcher</a></li>
<li><a class="reference internal" href="#add-some-fireworks">Add some FireWorks</a></li>
<li><a class="reference internal" href="#submit-a-job">Submit a job</a></li>
</ul>
</li>
<li><a class="reference internal" href="#adding-more-power-using-rapid-fire-mode">Adding more power: using rapid-fire mode</a></li>
<li><a class="reference internal" href="#continually-submit-jobs-to-the-queue">Continually submit jobs to the queue</a></li>
<li><a class="reference internal" href="#running-multiple-rockets-per-queue-job">Running multiple Rockets per queue job</a></li>
<li><a class="reference internal" href="#remote-qlaunch">Remote qlaunch</a><ul>
<li><a class="reference internal" href="#sample-usage">Sample usage</a></li>
<li><a class="reference internal" href="#limitations">Limitations</a></li>
</ul>
</li>
<li><a class="reference internal" href="#more-information">More information</a></li>
<li><a class="reference internal" href="#submitting-to-the-queue-with-python">Submitting to the queue with Python</a></li>
<li><a class="reference internal" href="#limitations-and-next-steps">Limitations and Next Steps</a><ul>
<li><a class="reference internal" href="#next-step-reserving-fireworks-to-overcome-limitations">Next step: reserving FireWorks to overcome limitations</a></li>
</ul>
</li>
</ul>
</li>
</ul>
</div>
<div>
<h4>Previous topic</h4>
<p class="topless"><a href="worker_tutorial.html"
title="previous chapter">Worker Tutorial</a></p>
</div>
<div>
<h4>Next topic</h4>
<p class="topless"><a href="queue_tutorial_pt2.html"
title="next chapter">Advanced queue submission (reservation mode)</a></p>
</div>
<div role="note" aria-label="source link">
<h3>This Page</h3>
<ul class="this-page-menu">
<li><a href="_sources/queue_tutorial.rst.txt"
rel="nofollow">Show Source</a></li>
</ul>
</div>
<search id="searchbox" style="display: none" role="search">
<h3 id="searchlabel">Quick search</h3>
<div class="searchformwrapper">
<form class="search" action="search.html" method="get">
<input type="text" name="q" aria-labelledby="searchlabel" autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false"/>
<input type="submit" value="Go" />
</form>
</div>
</search>
<script>document.getElementById('searchbox').style.display = "block"</script>
</div>
</div>
<div class="clearer"></div>
</div>
<div class="related" role="navigation" aria-label="Related">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="genindex.html" title="General Index"
>index</a></li>
<li class="right" >
<a href="py-modindex.html" title="Python Module Index"
>modules</a> |</li>
<li class="right" >
<a href="queue_tutorial_pt2.html" title="Advanced queue submission (reservation mode)"
>next</a> |</li>
<li class="right" >
<a href="worker_tutorial.html" title="Worker Tutorial"
>previous</a> |</li>
<li class="nav-item nav-item-0"><a href="index.html">FireWorks 2.0.4 documentation</a> »</li>
<li class="nav-item nav-item-this"><a href="">Launch Rockets through a queue</a></li>
</ul>
</div>
<div class="footer" role="contentinfo">
© Copyright 2013, Anubhav Jain.
Created using <a href="https://www.sphinx-doc.org/">Sphinx</a> 8.1.3.
</div>
<script>
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
ga('create', 'UA-53488807-1', 'auto');
ga('send', 'pageview');
</script>
</body>
</html>