-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathatom.xml
1275 lines (1196 loc) · 337 KB
/
atom.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
<title><![CDATA[Promise John]]></title>
<subtitle><![CDATA[My Blog]]></subtitle>
<link href="/atom.xml" rel="self"/>
<link href="http://promisejohn.github.io//"/>
<updated>2015-06-14T07:49:56.278Z</updated>
<id>http://promisejohn.github.io//</id>
<author>
<name><![CDATA[promise john]]></name>
</author>
<generator uri="http://hexo.io/">Hexo</generator>
<entry>
<title><![CDATA[NodeJS开发环境配置]]></title>
<link href="http://promisejohn.github.io/2015/06/13/NodeJSdevEnvSetting/"/>
<id>http://promisejohn.github.io/2015/06/13/NodeJSdevEnvSetting/</id>
<published>2015-06-13T07:09:27.000Z</published>
<updated>2015-06-14T07:49:56.278Z</updated>
<content type="html"><![CDATA[<h1 id="NodeJS开发环境配置">NodeJS开发环境配置</h1><h2 id="安装nvm、node和npm">安装nvm、node和npm</h2><p>nvm是nodejs的多版本管理利器，node是nodejs的解释器，npm是nodejs的包管理工具。</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># ubuntu安装之后会自动添加配置到~/.profile，可以直接cut到自己喜欢的比如~/.bashrc</span></span><br><span class="line">$ wget -qO- https://raw.githubusercontent.com/creationix/nvm/v0.<span class="number">25.4</span>/install.sh | bash</span><br><span class="line">$ nvm install <span class="number">0.12</span>.<span class="number">4</span></span><br><span class="line">$ nvm use <span class="number">0.12</span>.<span class="number">4</span></span><br><span class="line">$ nvm <span class="built_in">alias</span> default <span class="number">0.12</span>.<span class="number">4</span></span><br><span class="line">$ node --version</span><br></pre></td></tr></table></figure>
<p>由于网速问题，国内可以使用<a href="https://npm.taobao.org/" target="_blank" rel="external">taobao的npm镜像</a>：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">$ npm install -g cnpm --registry=https://registry.npm.taobao.org</span><br><span class="line">$ cnpm install [package name]</span><br></pre></td></tr></table></figure>
<h2 id="Build工具：grunt/gulp,_bower,_yeoman">Build工具：grunt/gulp, bower, yeoman</h2><p>自动编译、依赖管理、自动化测试、打包发布、项目模板工具、文档自动生成…… ，这些基本上属于每个项目（不论语言差异）构建的标配，javascript目前也配齐了，于是乎开发效率直线提升:)</p>
<ul>
<li>grunt是javascript的自动化构建工具，类似于Java的gradle/maven/ant，Python的tox，C/C++的make和scons等。</li>
<li>gulp也是构建工具，它采用类似jQuery的流式配置简化了任务的编写。</li>
<li>bower是一个依赖管理的工具，可以自动化安装bootstrap、angulajs、jquery等包，解决他们之间的依赖关系。</li>
<li>yeoman是一个生成项目框架scaffolding工具，遵循Convention over Configuration，用过RoR或者Django的应该都知道快速开发形式.d可以配合generator-webapp、generator-angular等模板快速生成项目结构。</li>
</ul>
<h3 id="安装上述工具到本地：">安装上述工具到本地：</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">$ cnpm install -g yo bower grunt-cli gulp</span><br><span class="line"><span class="comment"># 安装yo项目模板</span></span><br><span class="line">$ cnpm install -g generator-angular</span><br></pre></td></tr></table></figure>
<h3 id="生成一个HelloWorld">生成一个HelloWorld</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">$ mkdir -p ~/dev/nodejs/helloworld && <span class="built_in">cd</span> <span class="variable">$_</span></span><br><span class="line">$ yo angular</span><br><span class="line">$ grunt serve</span><br><span class="line">$ grunt <span class="built_in">test</span></span><br><span class="line">$ grunt</span><br></pre></td></tr></table></figure>
<p>这样就把一个前段项目的框架生成完毕了，可以进入开发阶段。</p>
<h2 id="编辑器：Atom">编辑器：Atom</h2><p>使用chrome和nodejs开发的Atom，几个月前看还是离sublimetext挺远，现在看几乎快要完全超越！——<strong>ubuntu下的中文算是个麻烦事，自定义以后还凑合</strong>。</p>
<ul>
<li>安装可以直接从<a href="https://atom.io/" target="_blank" rel="external">官网</a>开始。</li>
<li>开源字体从<a href="http://wenq.org/" target="_blank" rel="external">文泉驿</a>开始。</li>
</ul>
<p>参考配置：</p>
<figure class="highlight"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br></pre></td><td class="code"><pre><span class="line">@font-family: "DejaVu Sans Mono", "WenQuanYi Zen Hei";</span><br><span class="line"></span><br><span class="line">.tree-view, .title, .current-path, .tooltip {</span><br><span class="line"> font-family: @font-family;</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line">.terminal {</span><br><span class="line"> font-family: @font-family !important;</span><br><span class="line"> div {</span><br><span class="line"> white-space: nowrap;</span><br><span class="line"> }</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line">// style the background and foreground colors on the atom-text-editor-element</span><br><span class="line">// itself</span><br><span class="line">atom-text-editor {</span><br><span class="line"> font-family: @font-family;</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line">// To style other content in the text editor's shadow DOM, use the ::shadow</span><br><span class="line">// expression</span><br><span class="line">atom-text-editor::shadow .cursor {</span><br><span class="line"> font-family: @font-family;</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line">.markdown-preview {</span><br><span class="line"> font-family: @font-family;</span><br><span class="line"> atom-text-editor {</span><br><span class="line"> font-family: @font-family;</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<p>Atom也有大量的插件可以使用，比如把hexo集成进来，可以少开个term了：）</p>
<ul>
<li>VIM开启了编辑器的多模式状态，让敲键盘更尽兴；</li>
<li>SublimeText方便了编辑器的DIY，性能也很好；</li>
<li>Atom让Web开发更彻底，它本身就是个基于浏览器内核的工具；此外，它来自于Github:)</li>
</ul>
<h2 id="参考">参考</h2><ol>
<li><a href="https://atom.io/" title="Atom Offical site" target="_blank" rel="external">Atom Offical</a></li>
<li><a href="http://wenq.org/" title="文泉驿官网" target="_blank" rel="external">文泉驿官网</a></li>
<li><a href="https://npm.taobao.org/" title="Taobao NPM" target="_blank" rel="external">Taobao NPM</a></li>
<li><a href="http://gruntjs.com/" title="Grunt Offical" target="_blank" rel="external">Grunt Offical</a></li>
<li><a href="http://bower.io/" title="Bower Offical" target="_blank" rel="external">Bower Offical</a></li>
<li><a href="http://yeoman.io/" title="Yeoman Offical" target="_blank" rel="external">Yeoman Offical</a></li>
</ol>
]]></content>
<summary type="html">
<![CDATA[<h1 id="NodeJS开发环境配置">NodeJS开发环境配置</h1><h2 id="安装nvm、node和npm">安装nvm、node和npm</h2><p>nvm是nodejs的多版本管理利器，node是nodejs的解释器，npm是nodejs的包管理工具。</p]]>
</summary>
<category term="dev" scheme="http://promisejohn.github.io/tags/dev/"/>
<category term="nodejs" scheme="http://promisejohn.github.io/tags/nodejs/"/>
<category term="Tech" scheme="http://promisejohn.github.io/categories/Tech/"/>
</entry>
<entry>
<title><![CDATA[Hello Hexo]]></title>
<link href="http://promisejohn.github.io/2015/06/13/HelloHexo/"/>
<id>http://promisejohn.github.io/2015/06/13/HelloHexo/</id>
<published>2015-06-13T07:06:01.239Z</published>
<updated>2015-06-13T07:06:01.239Z</updated>
<content type="html"><![CDATA[<p>Welcome to <a href="http://hexo.io/" target="_blank" rel="external">Hexo</a>! This is your very first post. Check <a href="http://hexo.io/docs/" target="_blank" rel="external">documentation</a> for more info. If you get any problems when using Hexo, you can find the answer in <a href="http://hexo.io/docs/troubleshooting.html" target="_blank" rel="external">troubleshooting</a> or you can ask me on <a href="https://github.com/hexojs/hexo/issues" target="_blank" rel="external">GitHub</a>.</p>
<h2 id="Quick_Start">Quick Start</h2><h3 id="Create_a_new_post">Create a new post</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">$ hexo new <span class="string">"My New Post"</span></span><br></pre></td></tr></table></figure>
<p>More info: <a href="http://hexo.io/docs/writing.html" target="_blank" rel="external">Writing</a></p>
<h3 id="Run_server">Run server</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">$ hexo server</span><br></pre></td></tr></table></figure>
<p>More info: <a href="http://hexo.io/docs/server.html" target="_blank" rel="external">Server</a></p>
<h3 id="Generate_static_files">Generate static files</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">$ hexo generate <span class="comment"># 'hexo g' for short</span></span><br></pre></td></tr></table></figure>
<p>More info: <a href="http://hexo.io/docs/generating.html" target="_blank" rel="external">Generating</a></p>
<h3 id="Deploy_to_remote_sites">Deploy to remote sites</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">$ hexo deploy <span class="comment"># 'hexo d' for short</span></span><br></pre></td></tr></table></figure>
<p>More info: <a href="http://hexo.io/docs/deployment.html" target="_blank" rel="external">Deployment</a></p>
<h3 id="Useful_hexo_plugins">Useful hexo plugins</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">$ npm install hexo-generator-feed --save</span><br></pre></td></tr></table></figure>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">$ npm install hexo-generator-sitemap --save</span><br></pre></td></tr></table></figure>
<h3 id="Put_the_whole_source_folder_into_git">Put the whole source folder into git</h3><figure class="highlight"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">... # Dependency directory node_modules # hexo generated files, deployed to github.io pages public .deploy_git ...</span><br></pre></td></tr></table></figure>
<h3 id="Some_notes_about_doc_syntax">Some notes about doc syntax</h3><ol>
<li>use <code><!-- more --></code> to hide blog details below.</li>
<li>use <code>toc: false</code> to hide the topic menu about the blog.</li>
<li>use <code>mathjax: true</code> to enable LaTex math prents.</li>
</ol>
<h3 id="Awsome_markdown_editors:">Awsome markdown editors:</h3><ul>
<li>vim, sublimetext, …</li>
<li><a href="http://25.io/mou/" target="_blank" rel="external"><strong>Mou</strong></a></li>
<li><a href="http://macdown.uranusjr.com/" target="_blank" rel="external"><strong>MacDown</strong></a></li>
</ul>
]]></content>
<summary type="html">
<![CDATA[<p>Welcome to <a href="http://hexo.io/" target="_blank" rel="external">Hexo</a>! This is your very first post. Check <a href="http://hexo.io]]>
</summary>
<category term="hexo" scheme="http://promisejohn.github.io/tags/hexo/"/>
<category term="Tech" scheme="http://promisejohn.github.io/categories/Tech/"/>
</entry>
<entry>
<title><![CDATA[HelloHadoop]]></title>
<link href="http://promisejohn.github.io/2015/06/04/HelloHadoop/"/>
<id>http://promisejohn.github.io/2015/06/04/HelloHadoop/</id>
<published>2015-06-04T02:54:16.000Z</published>
<updated>2015-06-14T13:56:14.266Z</updated>
<content type="html"><![CDATA[<h1 id="Hadoop系列">Hadoop系列</h1><p>大数据依旧在热炒，hadoop虽然不是唯一的代表，却也是各家必谈之资本，玩玩大数据。</p>
<h2 id="通过ambari部署hadoop集群">通过ambari部署hadoop集群</h2><p>在cloudlab119-123上部署集群，其中119作为ambari server控制端。</p>
<p>安装部署：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">$ <span class="built_in">cd</span> /etc/yum.repo.d</span><br><span class="line">$ wget http://public-repo-<span class="number">1</span>.hortonworks.com/ambari/centos6/<span class="number">2</span>.x/updates/<span class="number">2.0</span>.<span class="number">0</span>/ambari.repo</span><br><span class="line">$ yum install -y ambari-server</span><br><span class="line">$ ambari-server setup <span class="comment"># 按指示操作即可</span></span><br><span class="line">$ ambari-server start <span class="comment"># *:8080端口</span></span><br></pre></td></tr></table></figure>
<p>配置，打开<a href="http://cloudlab119:8080，默认密码admin:admin。在cloudlab119上对所有节点做免密码认证：" target="_blank" rel="external">http://cloudlab119:8080，默认密码admin:admin。在cloudlab119上对所有节点做免密码认证：</a></p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">$ vi /etc/hosts</span><br><span class="line">$ ssh-keygen -t rsa -P <span class="string">''</span> <span class="operator">-f</span> ~/.ssh/hadoop119</span><br><span class="line">$ <span class="keyword">for</span> i <span class="keyword">in</span> {<span class="number">119</span>,<span class="number">120</span>,<span class="number">121</span>,<span class="number">122</span>,<span class="number">123</span>}; <span class="keyword">do</span> ssh-copy-id -i ~/.ssh/hadoop119.pub root@cloudlab<span class="variable">$i</span>; <span class="keyword">done</span>;</span><br></pre></td></tr></table></figure>
<p>按提示建议操作，如ntpd、iptables、关闭THP：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">$ chkconfig ntpd on</span><br><span class="line">$ service ntpd start</span><br><span class="line">$ service iptables stop <span class="comment"># 生产环境中参照文档把对应端口打开</span></span><br><span class="line">$ chkconfig iptables off</span><br><span class="line">$ <span class="built_in">echo</span> never > /sys/kernel/mm/redhat_transparent_hugepage/enabled <span class="comment"># 关闭THP，如果提示文件系统readonly，重启机器再执行</span></span><br></pre></td></tr></table></figure>
<p>按提示一步步操作即可，但中间如果出现下载包失败，则会导致整个安装失败，所以最好提前把官方源同步到本地，做镜像后安装。</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line">$ yum install -y yum-utils createrepo</span><br><span class="line">$ mkdir -p /var/www/html/ambari/centos6 && <span class="built_in">cd</span> <span class="variable">$_</span></span><br><span class="line">$ reposync -r Updates-ambari-<span class="number">2.0</span>.<span class="number">0</span></span><br><span class="line">$ createrepo Updates-ambari-<span class="number">2.0</span>.<span class="number">0</span></span><br><span class="line">$ mkdir -p /var/www/html/hdp/centos6 && <span class="built_in">cd</span> <span class="variable">$_</span></span><br><span class="line">$ reposync -r HDP-<span class="number">2.2</span></span><br><span class="line">$ reposync -r HDP-UTILS-<span class="number">1.1</span>.<span class="number">0.20</span></span><br><span class="line">$ createrepo HDP-<span class="number">2.2</span></span><br><span class="line">$ createrepo HDP-UTILS-<span class="number">1.1</span>.<span class="number">0.20</span></span><br><span class="line">$ vim /etc/yum.repo.d/ambari.repo <span class="comment"># 修改baseurl指向本地镜像</span></span><br></pre></td></tr></table></figure>
<p>安装之后，发现某几台服务器内存占用率非常高，可以通过Ambari增加Host节点，然后迁移部分的组件到新的机器。</p>
<h2 id="通过Docker部署集群">通过Docker部署集群</h2><p>可以自己做docker image，简单起见可以先用sequenceiq的image。</p>
<p>单节点测试：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">$ docker pull sequenceiq/hadoop-docker:<span class="number">2.7</span>.<span class="number">0</span></span><br><span class="line">$ docker run -it sequenceiq/hadoop-docker:<span class="number">2.7</span>.<span class="number">0</span> /etc/bootstrap.sh -bash</span><br><span class="line">$ <span class="built_in">cd</span> <span class="variable">$HADOOP_PREFIX</span> <span class="comment"># /usr/local/hadoop</span></span><br><span class="line">$ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-<span class="number">2.7</span>.<span class="number">0</span>.jar grep input output <span class="string">'dfs[a-z.]+'</span> <span class="comment"># 统计key出现次数</span></span><br><span class="line">$ bin/hdfs dfs -cat output/* <span class="comment"># 在用户Home下的output下</span></span><br></pre></td></tr></table></figure>
<p>多节点部署，可以用ambari镜像，准备好blueprint.json：</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">{ "host_groups" : [ { "name" : "host_group_1", "components" : [ { "name" : "ZOOKEEPER_SERVER" }, { "name" : "ZOOKEEPER_CLIENT" }, { "name" : "AMBARI_SERVER" }, { "name" : "HDFS_CLIENT" }, { "name" : "NODEMANAGER" }, { "name" : "MAPREDUCE2_CLIENT" }, { "name" : "APP_TIMELINE_SERVER" }, { "name" : "DATANODE" }, { "name" : "YARN_CLIENT" }, { "name" : "RESOURCEMANAGER" } ], "cardinality" : "1" }, { "name" : "host_group_2", "components" : [ { "name" : "ZOOKEEPER_SERVER" }, { "name" : "ZOOKEEPER_CLIENT" }, { "name" : "SECONDARY_NAMENODE" }, { "name" : "NODEMANAGER" }, { "name" : "YARN_CLIENT" }, { "name" : "DATANODE" }], "cardinality" : "1" }, { "name" : "host_group_3", "components" : [ { "name" : "ZOOKEEPER_SERVER" }, { "name" : "ZOOKEEPER_CLIENT" }, { "name" : "NAMENODE" }, { "name" : "NODEMANAGER" }, { "name" : "YARN_CLIENT" }, { "name" : "DATANODE" }], "cardinality" : "1" } ], "Blueprints" : { "blueprint_name" : "blueprint-c1", "stack_name" : "HDP", "stack_version" : "2.2" } }</span><br></pre></td></tr></table></figure>
<p>可以在ambari server上查找：<code>http://172.17.0.13:8080/api/v1/blueprints</code>。</p>
<p>创建集群：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line">$ docker pull sequenceiq/ambari:<span class="number">1.7</span>.<span class="number">0</span></span><br><span class="line">$ curl -Lo .amb https://github.com/sequenceiq/docker-ambari/raw/master/ambari-functions && . .amb</span><br><span class="line">$ amb-start-cluster <span class="number">3</span> <span class="comment"># 注意默认使用的是sequenceiq/ambari:1.7.0-warmup，可以修改 .amb</span></span><br><span class="line">$ amb-shell <span class="comment"># 又启了个container</span></span><br><span class="line">ambshell > host list</span><br><span class="line">ambshell > blueprint add --url http://<span class="number">172.17</span>.<span class="number">42.1</span>/bp.json <span class="comment"># 准备好json保存到某个能访问的http服务器</span></span><br><span class="line">ambshell > cluster build --blueprint blueprint-c1</span><br><span class="line">ambshell > cluster assign --hostGroup host_group_1 --host amb0.mycorp.kom</span><br><span class="line">ambshell > cluster assign --hostGroup host_group_2 --host amb1.mycorp.kom</span><br><span class="line">ambshell > cluster assign --hostGroup host_group_3 --host amb2.mycorp.kom</span><br><span class="line">ambshell > blueprint show --id blueprint-c1</span><br><span class="line">ambshell > cluster preview</span><br><span class="line">ambshell > cluster create</span><br></pre></td></tr></table></figure>
<p>中间安装过程如果出现失败，可以到ambari-server上看详细情况，当然也可以直接向之前方式一样，通过GUI安装部署；或者开着浏览器看amb-shell执行过程。<br>如果实在VM上远程使用，可以在docker所在机器上做NAT映射直接访问：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">$ iptables -t nat -A PREROUTING <span class="operator">-d</span> <span class="number">10.101</span>.<span class="number">29.26</span> -p tcp --dport <span class="number">8000</span> -j DNAT --to-destination <span class="number">172.17</span>.<span class="number">0.20</span>:<span class="number">8080</span> <span class="comment"># 可以用端口转发直接远程浏览器访问</span></span><br></pre></td></tr></table></figure>
<p>做个简单的测试：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">$ docker <span class="built_in">exec</span> -it <span class="number">9572938</span>bb253 /bin/bash <span class="comment"># 进入到容器内</span></span><br><span class="line">bash <span class="comment"># su hdfs # 切换到HDFS的超级用户</span></span><br><span class="line">bash <span class="comment"># hdfs dfsadmin -report</span></span><br><span class="line">bash <span class="comment"># hdfs dfs -ls /</span></span><br></pre></td></tr></table></figure>
<h2 id="HDFS">HDFS</h2><p>HDFS由NameNode、SNameNode和DataNode组成，HA的时候还有另一个NameNode（StandBy）。SNameNode专门从NameNode获取FSImage和Edits，为NameNode合并生成新的FSImage。</p>
<p>HDFS有个超级用户，就是启动NameNode的那个linux账号。</p>
<h3 id="基本Shell操作">基本Shell操作</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><span class="line">$ ps -elf | grep NameNode <span class="comment">#看看哪个账号启动了NameNode，那个账号就是超级用户</span></span><br><span class="line">$ su <span class="operator">-s</span> /bin/sh -c <span class="string">'hadoop fs -ls /'</span> hdfs <span class="comment">#使用超级账号执行命令</span></span><br><span class="line">$ su hdfs <span class="comment">#切换到超级账号</span></span><br><span class="line">$ useradd -G hdfs promise</span><br><span class="line">$ hdfs dfs -mkdir /user/promise</span><br><span class="line">$ hdfs dfs -chown promise:hdfs /user/promise</span><br><span class="line">$ hdfs dfs -ls /user/</span><br><span class="line">$ hdfs dfs -mkdir -p /user/promise/dev/hello</span><br><span class="line">$ hdfs dfs -chmod -R <span class="number">777</span> /user/promise/dev</span><br><span class="line">$ hdfs dfs -ls /user/promise/dev/</span><br><span class="line">$ <span class="built_in">echo</span> “helloworld” > hello.txt</span><br><span class="line">$ hdfs dfs -put hello.txt /user/promise/helloworld.txt</span><br><span class="line">$ hdfs dfs -put - hdfs://<span class="number">192.168</span>.<span class="number">182.119</span>/user/promise/test.txt <span class="comment"># 从stdin输入，Ctrl-D结束</span></span><br><span class="line">$ hdfs dfs -cat hdfs://<span class="number">192.168</span>.<span class="number">182.119</span>/user/promise/test.txt</span><br><span class="line">$ hdfs dfs -cat /user/promise/helloworld.txt</span><br><span class="line">$ hdfs dfs -get /user/promise/helloworld.txt hello2.txt</span><br><span class="line">$ hdfs dfs -tail <span class="operator">-f</span> /user/promise/helloworld.txt <span class="comment"># 查看追加的文件写入</span></span><br><span class="line">$ hdfs dfs -appendToFile - /user/promise/helloworld.txt</span><br></pre></td></tr></table></figure>
<h3 id="webhdfs操作">webhdfs操作</h3><p>启用了webhdfs之后（<code>hdfs-site</code>中<code>dfs.webhdfs.enabled=true</code>），可以通过HTTP方式访问HDFS：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">$ curl -i <span class="string">"http://192.168.182.119:50070/webhdfs/v1/user/promise/?op=LISTSTATUS"</span></span><br><span class="line">$ curl -i -L <span class="string">"http://192.168.182.119:50070/webhdfs/v1/user/promise/helloworld.txt?op=OPEN"</span> <span class="comment"># 通过重定向到DataNode获取文件</span></span><br><span class="line">$ curl -i -X PUT <span class="string">"http://192.168.182.119:50070/webhdfs/v1/user/promise/hello?user.name=promise&op=MKDIRS"</span></span><br></pre></td></tr></table></figure>
<p>通过HTTP方式创建和追加文件都需要通过2阶段实现：先在NameNode上创建，获得DataNode URI后再向URI上传文件。</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">$ curl -i -X PUT <span class="string">"http://192.168.182.119:50070/webhdfs/v1/user/promise/hello/hi.txt?op=CREATE"</span></span><br><span class="line">$ curl -i -X PUT -T hi.txt <span class="string">"http://192.168.182.119:50075/webhdfs/v1/user/promise/hello/hi.txt?user.name=promise&op=CREATE&namenoderpcaddress=192.168.182.119:8020&overwrite=false"</span> <span class="comment"># 需要声明账号</span></span><br></pre></td></tr></table></figure>
<h3 id="查看离线的FSImage和Edits：">查看离线的FSImage和Edits：</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">$ hdfs oiv -p XML -i /hadoop/hdfs/namenode/current/fsimage_0000000000000018887 -o fsimage.xml <span class="comment"># 生成XML格式</span></span><br><span class="line">$ hdfs oiv -i /hadoop/hdfs/namenode/current/fsimage_0000000000000018887 <span class="comment"># 在线查看形式</span></span><br><span class="line">$ hdfs dfs -ls webhdfs://<span class="number">127.0</span>.<span class="number">0.1</span>:<span class="number">5978</span>/</span><br><span class="line">$ hdfs oev -i /hadoop/hdfs/namenode/current/edits_0000000000000000001-<span class="number">0000000000000005150</span> -o edits.xml <span class="comment"># 查看edits</span></span><br></pre></td></tr></table></figure>
<h3 id="使用Java_API">使用Java API</h3><p>Java在大型软件系统开发中有利于更清晰的架构设计和团队分工，而且有大量的第三方优质框架可用；可惜在命令里写HelloWorld很啰嗦，方便起见，直接用maven生成基本文件结构。</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">$ <span class="built_in">export</span> PATH=<span class="variable">$PATH</span>:/opt/apache-maven-<span class="number">3.3</span>.<span class="number">3</span>/bin/</span><br><span class="line">$ <span class="built_in">export</span> JAVA_HOME=/usr/jdk64/jdk1.<span class="number">7.0</span>_67/</span><br><span class="line">$ <span class="built_in">export</span> MAVEN_OPTS=<span class="string">"-Xms256m -Xmx512m"</span></span><br><span class="line">$ mkdir ~/dev && <span class="built_in">cd</span> <span class="variable">$_</span></span><br><span class="line">$ mvn archetype:generate -DgroupId=org.tecstack -DartifactId=hellohdfs -DarchetypeArtifactId=maven-archetype-quickstart -DinteractiveMode=<span class="literal">false</span></span><br><span class="line">$ <span class="built_in">cd</span> hellohdfs && mvn package</span><br><span class="line">$ vim ./src/main/java/org/tecstack/App.java</span><br></pre></td></tr></table></figure>
<figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">package</span> org.tecstack;</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> org.apache.hadoop.conf.Configuration;</span><br><span class="line"><span class="keyword">import</span> org.apache.hadoop.fs.FileStatus;</span><br><span class="line"><span class="keyword">import</span> org.apache.hadoop.fs.FileSystem;</span><br><span class="line"><span class="keyword">import</span> org.apache.hadoop.fs.Path;</span><br><span class="line"></span><br><span class="line"><span class="comment">/**</span><br><span class="line"> * Hello HDFS</span><br><span class="line"> *</span><br><span class="line"> */</span></span><br><span class="line"><span class="keyword">public</span> <span class="class"><span class="keyword">class</span> <span class="title">App</span></span><br><span class="line"></span>{</span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">void</span> <span class="title">main</span><span class="params">( String[] args )</span></span><br><span class="line"> </span>{</span><br><span class="line"> <span class="keyword">try</span> {</span><br><span class="line"> Configuration conf=<span class="keyword">new</span> Configuration();</span><br><span class="line"> FileSystem hdfs=FileSystem.get(conf);</span><br><span class="line"> Path dst =<span class="keyword">new</span> Path(<span class="string">"/user/promise/helloworld.txt"</span>);</span><br><span class="line"> FileStatus files[]=hdfs.listStatus(dst);</span><br><span class="line"> <span class="keyword">for</span>(FileStatus file:files) {</span><br><span class="line"> System.out.println(file.getPath());</span><br><span class="line"> }</span><br><span class="line"> } <span class="keyword">catch</span> (Exception e) {</span><br><span class="line"> e.printStackTrace();</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<p>在<code>pom.xml</code>增加hadoop依赖包:</p>
<figure class="highlight xml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br></pre></td><td class="code"><pre><span class="line"><span class="tag"><<span class="title">project</span> <span class="attribute">xmlns</span>=<span class="value">"http://maven.apache.org/POM/4.0.0"</span> <span class="attribute">xmlns:xsi</span>=<span class="value">"http://www.w3.org/2001/XMLSchema-instance"</span></span><br><span class="line"> <span class="attribute">xsi:schemaLocation</span>=<span class="value">"http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd"</span>></span></span><br><span class="line"> <span class="tag"><<span class="title">modelVersion</span>></span>4.0.0<span class="tag"></<span class="title">modelVersion</span>></span></span><br><span class="line"> <span class="tag"><<span class="title">groupId</span>></span>org.tecstack<span class="tag"></<span class="title">groupId</span>></span></span><br><span class="line"> <span class="tag"><<span class="title">artifactId</span>></span>hellohdfs<span class="tag"></<span class="title">artifactId</span>></span></span><br><span class="line"> <span class="tag"><<span class="title">packaging</span>></span>jar<span class="tag"></<span class="title">packaging</span>></span></span><br><span class="line"> <span class="tag"><<span class="title">version</span>></span>1.0-SNAPSHOT<span class="tag"></<span class="title">version</span>></span></span><br><span class="line"> <span class="tag"><<span class="title">name</span>></span>hellohdfs<span class="tag"></<span class="title">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="title">url</span>></span>http://maven.apache.org<span class="tag"></<span class="title">url</span>></span></span><br><span class="line"> <span class="tag"><<span class="title">dependencies</span>></span></span><br><span class="line"> <span class="tag"><<span class="title">dependency</span>></span></span><br><span class="line"> <span class="tag"><<span class="title">groupId</span>></span>org.apache.hadoop<span class="tag"></<span class="title">groupId</span>></span></span><br><span class="line"> <span class="tag"><<span class="title">artifactId</span>></span>hadoop-common<span class="tag"></<span class="title">artifactId</span>></span></span><br><span class="line"> <span class="tag"><<span class="title">version</span>></span>2.6.0<span class="tag"></<span class="title">version</span>></span></span><br><span class="line"> <span class="tag"></<span class="title">dependency</span>></span></span><br><span class="line"> <span class="tag"><<span class="title">dependency</span>></span></span><br><span class="line"> <span class="tag"><<span class="title">groupId</span>></span>org.apache.hadoop<span class="tag"></<span class="title">groupId</span>></span></span><br><span class="line"> <span class="tag"><<span class="title">artifactId</span>></span>hadoop-hdfs<span class="tag"></<span class="title">artifactId</span>></span></span><br><span class="line"> <span class="tag"><<span class="title">version</span>></span>2.6.0<span class="tag"></<span class="title">version</span>></span></span><br><span class="line"> <span class="tag"></<span class="title">dependency</span>></span></span><br><span class="line"> <span class="tag"><<span class="title">dependency</span>></span></span><br><span class="line"> <span class="tag"><<span class="title">groupId</span>></span>org.apache.hadoop<span class="tag"></<span class="title">groupId</span>></span></span><br><span class="line"> <span class="tag"><<span class="title">artifactId</span>></span>hadoop-client<span class="tag"></<span class="title">artifactId</span>></span></span><br><span class="line"> <span class="tag"><<span class="title">version</span>></span>2.6.0<span class="tag"></<span class="title">version</span>></span></span><br><span class="line"> <span class="tag"></<span class="title">dependency</span>></span></span><br><span class="line"> <span class="tag"><<span class="title">dependency</span>></span></span><br><span class="line"> <span class="tag"><<span class="title">groupId</span>></span>junit<span class="tag"></<span class="title">groupId</span>></span></span><br><span class="line"> <span class="tag"><<span class="title">artifactId</span>></span>junit<span class="tag"></<span class="title">artifactId</span>></span></span><br><span class="line"> <span class="tag"><<span class="title">version</span>></span>3.8.1<span class="tag"></<span class="title">version</span>></span></span><br><span class="line"> <span class="tag"><<span class="title">scope</span>></span>test<span class="tag"></<span class="title">scope</span>></span></span><br><span class="line"> <span class="tag"></<span class="title">dependency</span>></span></span><br><span class="line"> <span class="tag"></<span class="title">dependencies</span>></span></span><br><span class="line"><span class="tag"></<span class="title">project</span>></span></span><br></pre></td></tr></table></figure>
<p>打包运行：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">$ mvn package</span><br><span class="line">$ mkdir -p src/main/resources</span><br><span class="line">$ scp cloudlab119:/etc/hadoop/conf/core-site.xml src/main/resources</span><br><span class="line">$ mvn <span class="built_in">exec</span>:java -Dexec.mainClass=<span class="string">"org.tecstack.App"</span> -Dexec.cleanupDaemonThreads=<span class="literal">false</span></span><br><span class="line">$ mvn dependency:copy-dependencies <span class="comment"># 导出依赖的包</span></span><br><span class="line">$ mvn resources:resources <span class="comment"># 导出资源</span></span><br><span class="line">$ mvn eclipse:eclipse <span class="comment"># 生成eclipse工程文件，.project, .classpath</span></span><br></pre></td></tr></table></figure>
<p>关于maven的exec插件，也可以通过配置<code>pom.xml</code>的plugin的参数简化执行：</p>
<figure class="highlight xml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line"><span class="tag"><<span class="title">build</span>></span></span><br><span class="line"> <span class="tag"><<span class="title">plugins</span>></span></span><br><span class="line"> <span class="tag"><<span class="title">plugin</span>></span></span><br><span class="line"> <span class="tag"><<span class="title">groupId</span>></span>org.codehaus.mojo<span class="tag"></<span class="title">groupId</span>></span></span><br><span class="line"> <span class="tag"><<span class="title">artifactId</span>></span>exec-maven-plugin<span class="tag"></<span class="title">artifactId</span>></span></span><br><span class="line"> <span class="tag"><<span class="title">version</span>></span>1.4.0<span class="tag"></<span class="title">version</span>></span></span><br><span class="line"> <span class="tag"><<span class="title">executions</span>></span></span><br><span class="line"> <span class="tag"><<span class="title">execution</span>></span></span><br><span class="line"> <span class="tag"><<span class="title">goals</span>></span></span><br><span class="line"> <span class="tag"><<span class="title">goal</span>></span>java<span class="tag"></<span class="title">goal</span>></span></span><br><span class="line"> <span class="tag"></<span class="title">goals</span>></span></span><br><span class="line"> <span class="tag"></<span class="title">execution</span>></span></span><br><span class="line"> <span class="tag"></<span class="title">executions</span>></span></span><br><span class="line"> <span class="tag"><<span class="title">configuration</span>></span></span><br><span class="line"> <span class="tag"><<span class="title">mainClass</span>></span>org.tecstack.App<span class="tag"></<span class="title">mainClass</span>></span></span><br><span class="line"> <span class="tag"><<span class="title">cleanupDaemonThreads</span>></span>false<span class="tag"></<span class="title">cleanupDaemonThreads</span>></span></span><br><span class="line"> <span class="tag"></<span class="title">configuration</span>></span></span><br><span class="line"> <span class="tag"></<span class="title">plugin</span>></span></span><br><span class="line"> <span class="tag"></<span class="title">plugins</span>></span></span><br><span class="line"><span class="tag"></<span class="title">build</span>></span></span><br></pre></td></tr></table></figure>
<p>之后运行只需要<code>mvn exec:java</code>，如果需要定制更多参数，比如JVM内存，单独启动进程执行等，可以使用<code>exec:exec</code>插件，具体看[codehaus官网][<a href="http://mojo.codehaus.org/exec-maven-plugin/usage.html]。" target="_blank" rel="external">http://mojo.codehaus.org/exec-maven-plugin/usage.html]。</a></p>
<h2 id="HUE交互界面">HUE交互界面</h2><p>Hue是一个基于Django开发的webapp，通过WebHDFS或HttpFS的一种访问HDFS的数据，在HDFS HA部署方式中只能使用HttpFS。<br>如通过WebHDFS方式访问，需要修改<code>hdfs-site.xml</code>：</p>
<figure class="highlight xml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="tag"><<span class="title">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="title">name</span>></span>dfs.webhdfs.enabled<span class="tag"></<span class="title">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="title">value</span>></span>true<span class="tag"></<span class="title">value</span>></span></span><br><span class="line"><span class="tag"></<span class="title">property</span>></span></span><br></pre></td></tr></table></figure>
<p>修改<code>core-site.xml</code>使Hue账号可以代理其他用户：</p>
<figure class="highlight xml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="tag"><<span class="title">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="title">name</span>></span>hadoop.proxyuser.hue.hosts<span class="tag"></<span class="title">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="title">value</span>></span>*<span class="tag"></<span class="title">value</span>></span></span><br><span class="line"><span class="tag"></<span class="title">property</span>></span></span><br><span class="line"><span class="tag"><<span class="title">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="title">name</span>></span>hadoop.proxyuser.hue.groups<span class="tag"></<span class="title">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="title">value</span>></span>*<span class="tag"></<span class="title">value</span>></span></span><br><span class="line"><span class="tag"></<span class="title">property</span>></span></span><br></pre></td></tr></table></figure>
<p>修改完参数需要重新启动HDFS，通过Ambari操作很方便。</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">$ yum install -y hue hue-server</span><br><span class="line">$ vim /etc/hue/conf/hue.ini <span class="comment"># 修改各个服务的URI，可以通过Ambari看到服务所在的服务器位置；修改默认监听端口不与其它冲突</span></span><br><span class="line">$ service hue start</span><br></pre></td></tr></table></figure>
<p>默认登陆账号为<code>admin:admin</code>，可以通过hdfs为该账号创建一个专用的文件夹<code>/user/hue</code>进行测试，正式环境中可以考虑与其他身份系统对接。</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">$ su hdfs</span><br><span class="line">$ hdfs dfs -mkdir /user/hue</span><br><span class="line">$ hdfs dfs -chown -R admin:hadoop /user/hue</span><br><span class="line">$ hdfs dfs -ls /user/</span><br></pre></td></tr></table></figure>
<h2 id="导入数据到HBase">导入数据到HBase</h2><p>通过hcat建立数据库表结构，通过pig导入数据。</p>
<p>建立数据文件<code>data.tsv</code>:</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">row1 c1 c2 row2 c1 c2 row3 c1 c2 row4 c1 c2 row5 c1 c2 row6 c1 c2 row7 c1 c2 row8 c1 c2 row9 c1 c2 row10 c1 c2</span><br></pre></td></tr></table></figure>
<p>准备建表DDL，<code>simple.ddl</code>：</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">CREATE TABLE simple_hcat_load_table (id STRING, c1 STRING, c2 STRING) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ( 'hbase.columns.mapping' = 'd:c1,d:c2' ) TBLPROPERTIES ( 'hbase.table.name' = 'simple_hcat_load_table' );</span><br></pre></td></tr></table></figure>
<p>准备导入脚本，<code>simple.bulkload.pig</code>：</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">A = LOAD 'hdfs:///tmp/data.tsv' USING PigStorage('\t') AS (id:chararray, c1:chararray, c2:chararray); -- DUMP A; STORE A INTO 'hbase://simple_hcat_load_table' USING org.apache.hive.hcatalog.pig.HCatStorer();</span><br></pre></td></tr></table></figure>
<p>把文件放到HDFS，建表，导入：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">$ su - hdfs</span><br><span class="line">$ hdfs dfs -put data.tsv /tmp/</span><br><span class="line">$ hcat <span class="operator">-f</span> simple.ddl</span><br><span class="line">$ pig -useHCatalog simple.bulkload.pig</span><br></pre></td></tr></table></figure>
<p>通过Hue的HCat可以查看数据库和表数据，也可以通过HBase Shell：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">$ su - hdfs</span><br><span class="line">$ hbase shell</span><br><span class="line">hbase > list <span class="comment"># 查询所有表</span></span><br><span class="line">hbase > scan <span class="string">'simple_hcat_load_table'</span> <span class="comment"># 查询所有数据</span></span><br><span class="line">hbase > describe <span class="string">'simple_hcat_load_table'</span></span><br></pre></td></tr></table></figure>
<p>也可以通过Hive Shell：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">$ su - hdfs</span><br><span class="line">$ hive</span><br><span class="line">hive > show tables;</span><br><span class="line">hive > select count(*) from simple_hcat_load_table;</span><br><span class="line">hive > desc simple_hcat_load_table;</span><br></pre></td></tr></table></figure>
<p>如果建表过程中出现类似<code>java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/HBaseConfiguration</code>如下的找不到类情况，检查hcat的配置文件是否有HBase目录：<code>vim /usr/bin/hcat</code>。</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">...</span><br><span class="line"><span class="built_in">export</span> HBASE_HOME=/usr/hdp/<span class="number">2.2</span>.<span class="number">4.2</span>-<span class="number">2</span>/hbase</span><br><span class="line">...</span><br></pre></td></tr></table></figure>
<p>此外，由于Hue集成的HCat调用的是Hive下的hcat，需要在<code>/etc/hive-hcatalog/conf/hcat-env.sh</code>中指定<code>HBASE_HOME</code>:</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">...</span><br><span class="line">HBASE_HOME=<span class="variable">${HBASE_HOME:-/usr/hdp/current/hbase-client}</span></span><br><span class="line">...</span><br></pre></td></tr></table></figure>
<p>pig在使用MapReduce模式执行时，可以根据log打开MR Job跟踪，如果也出现<code>java.lang.NoClassDefFoundError</code>类错误，说明mapreduce服务的classpath有缺漏，可以通过Ambari修改MapReduce服务<code>Advanced mapred-site</code>配置中的<code>mapreduce.application.classpath</code>，比如出现HBase相关类找不到，则添加HBase相关的库<code>/usr/hdp/2.2.4.2-2/hbase/lib/*</code>，然后重启MapReduce服务即可。</p>
<p>如果出现执行权限错误，需要检查如下配置是否存在：<br><code>hive-site.xml</code>:</p>
<figure class="highlight xml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line"><span class="tag"><<span class="title">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="title">name</span>></span>hive.security.metastore.authorization.manager<span class="tag"></<span class="title">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="title">value</span>></span>org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider</span><br><span class="line"><span class="tag"></<span class="title">value</span>></span></span><br><span class="line"><span class="tag"></<span class="title">property</span>></span></span><br><span class="line"><span class="tag"><<span class="title">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="title">name</span>></span>hive.security.metastore.authenticator.manager<span class="tag"></<span class="title">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="title">value</span>></span>org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator</span><br><span class="line"><span class="tag"></<span class="title">value</span>></span></span><br><span class="line"><span class="tag"></<span class="title">property</span>></span></span><br><span class="line"><span class="tag"><<span class="title">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="title">name</span>></span>hive.metastore.pre.event.listeners<span class="tag"></<span class="title">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="title">value</span>></span>org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener</span><br><span class="line"><span class="tag"></<span class="title">value</span>></span></span><br><span class="line"><span class="tag"></<span class="title">property</span>></span></span><br><span class="line"><span class="tag"><<span class="title">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="title">name</span>></span>hive.metastore.execute.setugi<span class="tag"></<span class="title">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="title">value</span>></span>true<span class="tag"></<span class="title">value</span>></span></span><br><span class="line"><span class="tag"></<span class="title">property</span>></span></span><br></pre></td></tr></table></figure>
<p><code>webhcat-site.xml</code>:<br>正式环境中可以明确具体使用的账号名称，最小化权限。</p>
<figure class="highlight xml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="tag"><<span class="title">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="title">name</span>></span>webhcat.proxyuser.hue.hosts<span class="tag"></<span class="title">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="title">value</span>></span>*<span class="tag"></<span class="title">value</span>></span></span><br><span class="line"><span class="tag"></<span class="title">property</span>></span></span><br><span class="line"><span class="tag"><<span class="title">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="title">name</span>></span>webhcat.proxyuser.hue.groups<span class="tag"></<span class="title">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="title">value</span>></span>*<span class="tag"></<span class="title">value</span>></span></span><br><span class="line"><span class="tag"></<span class="title">property</span>></span></span><br></pre></td></tr></table></figure>
<p><code>core-site.xml</code>:</p>
<figure class="highlight xml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="tag"><<span class="title">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="title">name</span>></span>hadoop.proxyuser.hcat.group<span class="tag"></<span class="title">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="title">value</span>></span>*<span class="tag"></<span class="title">value</span>></span></span><br><span class="line"><span class="tag"></<span class="title">property</span>></span></span><br><span class="line"><span class="tag"><<span class="title">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="title">name</span>></span>hadoop.proxyuser.hcat.hosts<span class="tag"></<span class="title">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="title">value</span>></span>*<span class="tag"></<span class="title">value</span>></span></span><br><span class="line"><span class="tag"></<span class="title">property</span>></span></span><br></pre></td></tr></table></figure>
<h2 id="使用小结">使用小结</h2><p>Hadoop系架构经过引入YARN，真正实现了数据和应用的分离，在YARN架构上可以实现出了原来的MapReduce之外的更多App，比如流式处理、实时处理、图计算等，如此可以真正实现“把应用挪到数据旁边高效执行”，构建大数据平台。此外，做数据分析时可以发现周边工具很丰富，比如hue统一的综合管理界面、从数据文件识别结构并管理数据的hcat、数据处理高级语言pig、支持SQL的Hive，使得在Hadoop平台上做数据分析非常方便（可以参考<a href="http://zh.hortonworks.com/hadoop-tutorial/hello-world-an-introduction-to-hadoop-hcatalog-hive-and-pig/" title="hcat & hive & pig" target="_blank" rel="external">这里</a>）。<br>另外，对比YARN利用OS进程隔离分配资源之外，<a href="http://mesos.apache.org/" title="Apache Mesos" target="_blank" rel="external">Mesos</a>结合了Container技术实现容器隔离分配资源，以此实现更通用的框架（用各种语言写的各种计算框架）。当然YARN也进入了<a href="http://hadoop.apache.org/docs/r2.7.0/hadoop-yarn/hadoop-yarn-site/DockerContainerExecutor.html" title="YARN DCE" target="_blank" rel="external">DCE</a>，通过docker实现容器隔离。不过正式应用还是有待Linux kernel本身功能的成熟，以及docker之类管理工具的完善。<br>最后，数据处理优选python scikit-learn系工具，当大到一定程度，或需要多人同时工作时，hadoop系平台是个不错的选择。</p>
<h2 id="参考">参考</h2><ol>
<li><a href="http://docs.hortonworks.com/HDPDocuments/Ambari-2.0.0.0/Ambari_Doc_Suite/ADS_v200.html" title="Ambari Official Docs" target="_blank" rel="external">Ambari Official Docs</a></li>
<li><a href="http://hadoop.apache.org/docs/stable2/hadoop-project-dist/hadoop-common/FileSystemShell.html" title="HDFS Official Doc" target="_blank" rel="external">HDFS Official Doc</a></li>
<li><a href="https://github.com/sequenceiq/hadoop-docker" title="hadoop-docker ffrom sequenceiq" target="_blank" rel="external">Hadoop-docker from sequenceiq</a></li>
<li><a href="https://cwiki.apache.org/confluence/display/AMBARI/Blueprints#Blueprints-Step1:CreateBlueprint" title="Ambari docs on apache" target="_blank" rel="external">Ambari Docs on Apache</a></li>
<li><a href="https://blog.codecentric.de/en/2014/05/lambda-cluster-provisioning/" title="Sample Blueprint" target="_blank" rel="external">Sample BluePrint</a></li>
<li><a href="http://ju.outofmemory.cn/entry/128881" title="安装配置Hue" target="_blank" rel="external">安装配置Hue</a></li>
<li><a href="http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.2.4/HDP_Man_Install_v224/index.html" title="手动安装HDP" target="_blank" rel="external">手动安装HDP</a></li>
<li><a href="https://cwiki.apache.org/confluence/display/AMBARI/Blueprints" title="Ambari Blueprint" target="_blank" rel="external">Ambari Blueprint</a></li>
<li><a href="http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.2.4/Importing_Data_HBase_v224/index.html" title="导入数据到HBase" target="_blank" rel="external">导入数据到HBase</a></li>
<li><a href="http://mesos.apache.org/" title="Apache Mesos" target="_blank" rel="external">Apache Mesos</a></li>
<li><a href="http://hadoop.apache.org/docs/r2.7.0/hadoop-yarn/hadoop-yarn-site/DockerContainerExecutor.html" title="YARN DCE" target="_blank" rel="external">YARN DCE</a></li>
</ol>
]]></content>
<summary type="html">
<![CDATA[<h1 id="Hadoop系列">Hadoop系列</h1><p>大数据依旧在热炒，hadoop虽然不是唯一的代表，却也是各家必谈之资本，玩玩大数据。</p>
<h2 id="通过ambari部署hadoop集群">通过ambari部署hadoop集群</h2><p>在clou]]>
</summary>
<category term="ambari" scheme="http://promisejohn.github.io/tags/ambari/"/>
<category term="hadoop" scheme="http://promisejohn.github.io/tags/hadoop/"/>
<category term="hbase" scheme="http://promisejohn.github.io/tags/hbase/"/>
<category term="hdfs" scheme="http://promisejohn.github.io/tags/hdfs/"/>
<category term="hive" scheme="http://promisejohn.github.io/tags/hive/"/>
<category term="Tech" scheme="http://promisejohn.github.io/categories/Tech/"/>
</entry>
<entry>
<title><![CDATA[使用Linux的tgtd提供iscsi服务]]></title>
<link href="http://promisejohn.github.io/2015/06/04/iSCSIwithTgtd/"/>
<id>http://promisejohn.github.io/2015/06/04/iSCSIwithTgtd/</id>
<published>2015-06-04T02:54:10.000Z</published>
<updated>2015-06-14T13:57:14.086Z</updated>
<content type="html"><![CDATA[<h2 id="安装部署">安装部署</h2><table>
<thead>
<tr>
<th>IP</th>
<th>角色</th>
</tr>
</thead>
<tbody>
<tr>
<td>192.168.182.156</td>
<td>tgtd</td>
</tr>
<tr>
<td>192.168.182.157</td>
<td>client1</td>
</tr>
<tr>
<td>192.168.182.158</td>
<td>client2</td>
</tr>
</tbody>
</table>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">$ yum install -y scsi-target-utils</span><br><span class="line">$ chkconfig tgtd on</span><br><span class="line">$ service tgtd start</span><br></pre></td></tr></table></figure>
<p>配置方法2种：</p>
<ol>
<li>tgtadm，在线修改</li>
<li>conf配置文件</li>
</ol>
<h2 id="在服务端增加一个Target">在服务端增加一个Target</h2><p>主要流程：</p>
<ol>
<li>建立target</li>
<li>为target增加backstorage</li>
<li>配置客户端访问target的控制策略</li>
</ol>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><span class="line">$ tgtadm --lld iscsi --op new --mode target --tid <span class="number">1</span> --targetname iqn.<span class="number">2015</span>-<span class="number">05</span>-<span class="number">04</span>.org.tecstack.storage.tg1 <span class="comment"># add target</span></span><br><span class="line"><span class="comment"># 用losetup映射块设备作为backstorage</span></span><br><span class="line">$ mkdir /opt/tgtstorage</span><br><span class="line">$ dd <span class="keyword">if</span>=/dev/zero of=/opt/tgtstorage/disk0.img bs=<span class="number">1</span>M count=<span class="number">5120</span></span><br><span class="line">$ losetup <span class="operator">-f</span> /opt/tgtstorage/disk0.img <span class="comment"># 映射为设备</span></span><br><span class="line">$ losetup <span class="operator">-a</span> <span class="comment"># /dev/loop0</span></span><br><span class="line">$ tgtadm --lld iscsi --op new --mode logicalunit --tid <span class="number">1</span> --lun <span class="number">1</span> --backing-store /dev/loop0</span><br><span class="line"><span class="comment"># 直接用块文件作为backstorage添加第二个块</span></span><br><span class="line">$ dd <span class="keyword">if</span>=/dev/zero of=/opt/tgtstorage/disk1.img bs=<span class="number">1</span>M count=<span class="number">5120</span></span><br><span class="line">$ tgtadm --lld iscsi --op new --mode logicalunit --tid <span class="number">1</span> --lun <span class="number">2</span> --backing-store /opt/tgtstorage/disk1.img</span><br><span class="line"><span class="comment"># 绑定客户端ip</span></span><br><span class="line">$ tgtadm --lld iscsi --mode target --op <span class="built_in">bind</span> --tid <span class="number">1</span> --initiator-address=<span class="number">192.168</span>.<span class="number">182.157</span></span><br><span class="line">$ tgtadm --lld iscsi --mode target --op <span class="built_in">bind</span> --tid <span class="number">1</span> --initiator-address=<span class="number">192.168</span>.<span class="number">182.158</span></span><br><span class="line">$ tgt-admin --dump |grep -v default-driver > /etc/tgt/conf.d/my-targets.conf <span class="comment"># 通过tgt-admin保存为配置文件，注意与tgtadm的区别！</span></span><br><span class="line">$ tgtadm --lld iscsi --mode target --op show</span><br><span class="line">$ tgt-admin --show <span class="comment"># 和上一个命令一样，tgt-admin是tgtadm的perl封装</span></span><br></pre></td></tr></table></figure>
<p>在服务端禁用某个客户端：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">$ tgtadm --lld iscsi --mode target --op unbind --tid <span class="number">1</span> --initiator-address=<span class="number">192.168</span>.<span class="number">182.158</span> <span class="comment"># unbind，使不可发现</span></span><br></pre></td></tr></table></figure>
<h2 id="在客户端使用target">在客户端使用target</h2><p>iscsi存储使用的主要流程：</p>
<ol>
<li>发现target</li>
<li>login到target</li>
<li>使用存储管理工具使用块设备</li>
</ol>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line">$ iscsiadm --mode discovery --type sendtargets --portal <span class="number">192.168</span>.<span class="number">182.156</span></span><br><span class="line">$ ls -lh /var/lib/iscsi/nodes/ <span class="comment"># 可以查看到对应的target和卷资料</span></span><br><span class="line">$ iscsiadm -m node <span class="comment"># 查看当前机器上所有target</span></span><br><span class="line">$ iscsiadm -m node -T iqn.<span class="number">2015</span>-<span class="number">05</span>-<span class="number">04</span>.org.tecstack.storage.tg1 --login <span class="comment"># 登陆target</span></span><br><span class="line">$ fdisk <span class="operator">-l</span> <span class="comment"># 登陆后就可以看到块设备</span></span><br><span class="line">$ iscsiadm -m node -T iqn.<span class="number">2015</span>-<span class="number">05</span>-<span class="number">04</span>.org.tecstack.storage.tg1 --logout <span class="comment">#登出target</span></span><br><span class="line">$ fdisk <span class="operator">-l</span> <span class="comment"># 登出后设备移除</span></span><br><span class="line">$ iscsiadm -m node -o delete -T iqn.<span class="number">2015</span>-<span class="number">05</span>-<span class="number">04</span>.org.tecstack.storage.tg1 <span class="comment"># 删除target</span></span><br><span class="line">$ ls -lh /var/lib/iscsi/nodes/</span><br><span class="line">$ iscsiadm -m node -T iqn.<span class="number">2015</span>-<span class="number">05</span>-<span class="number">04</span>.org.tecstack.storage.tg1 -p <span class="number">192.168</span>.<span class="number">182.156</span> -- op update -n node.startup -v automatic <span class="comment"># 自动login</span></span><br></pre></td></tr></table></figure>
<p>当login到target之后，就可以使用，比如通过LVM：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line">$ fdisk <span class="operator">-l</span> <span class="comment"># 增加了一个/dev/sdb块设备</span></span><br><span class="line">$ pvcreate /dev/sdb</span><br><span class="line">$ pvdisplay</span><br><span class="line">$ vgcreate myiscsi /dev/sdb</span><br><span class="line">$ vgdisplay</span><br><span class="line">$ lvcreate <span class="operator">-l</span> <span class="number">1024</span> -n vdisk0 myiscsi</span><br><span class="line">$ lvdisplay</span><br><span class="line">$ ls -lh /dev/myiscsi/vdisk0 <span class="comment"># 生成的块设备位置</span></span><br><span class="line">$ mkfs.ext4 /dev/myiscsi/vdisk0</span><br><span class="line">$ mkdir -p /opt/myiscsidata</span><br><span class="line">$ mount /dev/myiscsi/vdisk0 /opt/myiscsidata/</span><br><span class="line">$ df -h</span><br></pre></td></tr></table></figure>
<p>运行过程中为target添加后端存储，客户端需要logout后重新login才能看到。但是如果重新login后块设备名会变化，比如变成<code>/dev/sdc</code>。可以通过文件系统的UUID来识别设备并挂载：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">$ tune2fs <span class="operator">-l</span> /dev/sdc</span><br></pre></td></tr></table></figure>
<h3 id="GFS2测试">GFS2测试</h3><h4 id="创建和挂载iscsi块存储">创建和挂载iscsi块存储</h4><p>在target端创建LUN</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">$ dd <span class="keyword">if</span>=/dev/zero of=/opt/tgtstorage/disk_gfs.img bs=<span class="number">1</span>M count=<span class="number">5120</span></span><br><span class="line">$ tgtadm --lld iscsi --op new --mode logicalunit --tid <span class="number">1</span> --lun <span class="number">3</span> --backing-store /opt/tgtstorage/disk_gfs.img</span><br><span class="line">$ tgtadm --lld iscsi --mode target --op <span class="built_in">bind</span> --tid <span class="number">1</span> --initiator-address=<span class="number">192.168</span>.<span class="number">182.157</span></span><br><span class="line">$ tgtadm --lld iscsi --mode target --op <span class="built_in">bind</span> --tid <span class="number">1</span> --initiator-address=<span class="number">192.168</span>.<span class="number">182.158</span></span><br><span class="line">$ tgt-admin --show</span><br></pre></td></tr></table></figure>
<p>在client1和client2上挂载块设备：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">$ iscsiadm --mode discovery --type sendtargets --portal <span class="number">192.168</span>.<span class="number">182.156</span></span><br><span class="line">$ iscsiadm -m node</span><br><span class="line">$ iscsiadm -m node -T iqn.<span class="number">2015</span>-<span class="number">05</span>-<span class="number">04</span>.org.tecstack.storage.tg1 --login</span><br><span class="line">$ fdisk <span class="operator">-l</span></span><br></pre></td></tr></table></figure>
<h4 id="使用luci配置集群：">使用luci配置集群：</h4><p>在target端配置：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">$ yum -y install luci</span><br><span class="line">$ service luci start</span><br></pre></td></tr></table></figure>
<p>在client1和client2上配置：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">$ yum install -y ricci</span><br><span class="line">$ service ricci start</span><br><span class="line">$ passwd ricci <span class="comment"># 123456，节点密码</span></span><br></pre></td></tr></table></figure>
<p>通过访问target的https（默认端口8084，账号同Linux本地root账号），创建集群，新增节点，选择下载包，勾选启用共享文件系统。管理工具会自动帮助安装：<code>cman rgmanager lvm2-cluster sg3_utils gfs2-utils</code>，并启动相关服务。</p>
<h4 id="在一台集群节点上创建LVM逻辑卷，格式化为GFS文件系统">在一台集群节点上创建LVM逻辑卷，格式化为GFS文件系统</h4><p>识别scsi和块设备的对应关系：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">$ ls <span class="operator">-l</span> /dev/disk/by-path/*tecstack*</span><br><span class="line">$ scsi_id -gu /dev/sdb</span><br></pre></td></tr></table></figure>
<p>在client1上执行：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">$ pvcreate /dev/sdb</span><br><span class="line">$ vgcreate gfstest /dev/sdb</span><br><span class="line">$ lvcreate <span class="operator">-l</span> <span class="number">1024</span> -n gfsdisk0 gfstest</span><br><span class="line">$ mkfs.gfs2 -j2 -p lock_dlm -t gfstest:gfs2 /dev/gfstest/gfsdisk0</span><br></pre></td></tr></table></figure>
<h4 id="在两台集群节点上同时挂载GFS文件系统">在两台集群节点上同时挂载GFS文件系统</h4><p>在client1和client2上执行：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">$ mkdir /mnt/gfstest</span><br><span class="line">$ mount /dev/gfstest/gfsdisk0 /mnt/gfstest</span><br></pre></td></tr></table></figure>
<p>参考：</p>
<ol>
<li><a href="http://blog.delouw.ch/2013/07/07/creating-and-managing-iscsi-targets/" title="creating and managing iscsi targets" target="_blank" rel="external">creating and managing iscsi targets</a></li>
<li><a href="http://stgt.sourceforge.net/manpages/tgtadm.8.html" title="tgtadm official man page" target="_blank" rel="external">tgtadm man page</a></li>
<li><a href="http://linux.vbird.org/linux_server/0460iscsi.php" title="iscsi使用案例" target="_blank" rel="external">iscsi使用案例</a></li>
<li><a href="https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/5/html/Global_File_System/s1-config-tasks.html" title="GFS配置" target="_blank" rel="external">GFS配置</a></li>
</ol>
]]></content>
<summary type="html">
<![CDATA[<h2 id="安装部署">安装部署</h2><table>
<thead>
<tr>
<th>IP</th>
<th>角色</th>
</tr>
</thead>
<tbody>
<tr>
<td>192.168.182.156</td>
<td>tgtd</td>
</tr>]]>
</summary>
<category term="iscsi" scheme="http://promisejohn.github.io/tags/iscsi/"/>
<category term="linux" scheme="http://promisejohn.github.io/tags/linux/"/>
<category term="tgtd" scheme="http://promisejohn.github.io/tags/tgtd/"/>
<category term="Tech" scheme="http://promisejohn.github.io/categories/Tech/"/>
</entry>
<entry>
<title><![CDATA[HelloDocker]]></title>
<link href="http://promisejohn.github.io/2015/05/13/HelloDocker/"/>
<id>http://promisejohn.github.io/2015/05/13/HelloDocker/</id>
<published>2015-05-13T07:56:19.000Z</published>
<updated>2015-06-13T07:06:01.239Z</updated>
<content type="html"><![CDATA[<h1 id="Docker概述">Docker概述</h1><p>Docker是基于Go语言开发的容器管理工具，而且抽象级别比lxc这些管理工具高，目前官方默认通过libcontainer管理容器。lxc是一些kernel patch（namespaces）和userspace tool（cgroup）的集合，通过cgroup这个用户态管理工具实现对隔离资源的管理。Docker相比LXC增加了镜像服务（通过UnionFS和DeviceMapper），同时简化了操作复杂度，可以很方便实现应用分发部署，甚至是扩容。目前也有一些周边管理工具（Compose、Flocker、weaver）在创新，在docker基础上实现更多实用功能，如迁移，流程编排，网络管理等。跟openstack相比，docker更年轻，也更轻量级，但两者在某些场景下又可以很好结合起来，比如通过docker将openstack的管理节点实现高可用、可扩展。</p>
<p>docker的部署在发行版上比较简单，但对内核版本有不同要求，比如在centos6.5上要求内核版本至少2.6.32-431。</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">$ yum install -y docker-io</span><br><span class="line">$ chkconfig docker on</span><br><span class="line">$ service docker start</span><br><span class="line">$ docker run -it ubuntu /bin/bash</span><br><span class="line">$ docker info <span class="comment">#查看docker信息</span></span><br></pre></td></tr></table></figure>
<h2 id="Docker基本用法">Docker基本用法</h2><h3 id="基本命令使用：">基本命令使用：</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br></pre></td><td class="code"><pre><span class="line">$ docker run ubuntu:<span class="number">14.04</span> /bin/<span class="built_in">echo</span> <span class="string">'Hello world'</span></span><br><span class="line">$ docker run <span class="operator">-d</span> ubuntu:<span class="number">14.04</span> /bin/sh -c <span class="string">"while true; do echo hello world; sleep 1; done"</span> <span class="comment"># 后台运行</span></span><br><span class="line">$ docker ps <span class="operator">-a</span></span><br><span class="line">$ docker logs instance_name</span><br><span class="line">$ docker logs <span class="operator">-f</span> instance_name <span class="comment"># 持续输出</span></span><br><span class="line">$ docker stop/start/restart instance_name <span class="comment"># stop容器后还能启动，但网络信息会变</span></span><br><span class="line">$ docker ps -aq | xargs docker rm <span class="comment"># 删除所有容器，运行中的会保留</span></span><br><span class="line">$ docker images ＃ -tree参数可以看层级关系，最大<span class="number">128</span>层</span><br><span class="line">$ docker save -o ubuntu.tar ubuntu:<span class="number">14.04</span> <span class="comment"># 把镜像保存到本地</span></span><br><span class="line">$ tar -tf ubuntu.tar <span class="comment"># 可以看到image的结构，包含了多层FS</span></span><br><span class="line">$ docker rmi ubuntu</span><br><span class="line">$ docker run <span class="operator">-d</span> -P training/webapp python app.py <span class="comment"># 映射所有端口</span></span><br><span class="line">$ docker port instance_name <span class="number">5000</span> <span class="comment">#查看被映射的端口</span></span><br><span class="line">$ docker run <span class="operator">-d</span> -p <span class="number">5000</span>:<span class="number">5000</span> training/webapp python app.py <span class="comment"># 映射5000端口</span></span><br><span class="line">$ docker run <span class="operator">-d</span> -p <span class="number">127.0</span>.<span class="number">0.1</span>:<span class="number">5000</span>:<span class="number">5000</span>/udp training/webapp python app.py <span class="comment"># 映射本机UDP端口</span></span><br><span class="line">$ docker top instance_name</span><br><span class="line">$ docker inspect instance_name <span class="comment"># 查看容器配置</span></span><br><span class="line">$ docker inspect <span class="operator">-f</span> <span class="string">'{{ .NetworkSettings.IPAddress }}'</span> instance_name <span class="comment"># 查看具体的配置信息，如IP</span></span><br><span class="line">$ docker pull centos</span><br><span class="line">$ docker commit -m <span class="string">"some modify hints"</span> <span class="operator">-a</span> <span class="string">"author info"</span> </span><br><span class="line">contain_id repo_name/image_name:tags</span><br><span class="line">$ docker tag container_id repo_name/image_name:tags</span><br><span class="line">$ docker <span class="built_in">exec</span> -it CONTAINER_NAME /bin/bash <span class="comment"># 1.3开始支持在运行的容器内执行命令</span></span><br></pre></td></tr></table></figure>
<h3 id="使用Dockerfile创建image">使用Dockerfile创建image</h3><p><code>mkdir myimage && cd $_ && vim Dockerfile</code>:</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"># This is a comment FROM ubuntu:14.04 MAINTAINER Promise John <promise.jon@gmail.com> RUN apt-get update && apt-get install -y nginx</span><br></pre></td></tr></table></figure>
<h3 id="使用Linking">使用Linking</h3><p>Docker主要通过环境变量和<code>/etc/hosts</code>文件来提供访问途径：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">$ docker run <span class="operator">-d</span> --name db training/postgres</span><br><span class="line">$ docker run <span class="operator">-d</span> -P --name web --link db:db training/webapp python app.py</span><br><span class="line">$ docker inspect <span class="operator">-f</span> <span class="string">"{{ .Name }}"</span> instance_id <span class="comment"># 查看容器名</span></span><br><span class="line">$ docker inspect <span class="operator">-f</span> <span class="string">"{{ .HostConfig.Links }}"</span> web</span><br><span class="line">$ docker run --rm --name web2 --link db:db training/webapp env</span><br><span class="line">$ docker run --rm --name web2 --link db:db training/webapp cat /etc/hosts</span><br></pre></td></tr></table></figure>
<h3 id="Storage">Storage</h3><p>主要包括独立的卷和共享本地文件系统：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">$ docker run <span class="operator">-d</span> -P --name web -v /webapp training/webapp python app.py <span class="comment"># volumn不会随容器消失，作为数据独立存在</span></span><br><span class="line">$ docker inspect web <span class="comment"># 查看/var/lib/docker/volumes和vfs</span></span><br><span class="line">$ docker run <span class="operator">-d</span> -P --name web -v /src/webapp:/opt/webapp training/webapp python app.py <span class="comment"># 挂载</span></span><br><span class="line">$ docker run <span class="operator">-d</span> -P --name web -v /src/webapp:/opt/webapp:ro training/webapp python app.py <span class="comment"># 只读挂载</span></span><br><span class="line">$ docker run --rm -it -v ~/.bash_<span class="built_in">history</span>:/.bash_<span class="built_in">history</span> ubuntu /bin/bash <span class="comment"># 只挂载某个文件</span></span><br></pre></td></tr></table></figure>
<p>Container间共享数据：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">$ docker create -v /dbdata --name dbdata training/postgres /bin/<span class="literal">true</span></span><br><span class="line">$ docker run <span class="operator">-d</span> --volumes-from dbdata --name db1 training/postgres</span><br><span class="line">$ docker run <span class="operator">-d</span> --volumes-from dbdata --name db2 training/postgres</span><br><span class="line">$ docker run --volumes-from dbdata -v $(<span class="built_in">pwd</span>):/backup ubuntu tar cvf /backup/backup.tar /dbdata <span class="comment"># 备份容器数据</span></span><br><span class="line">$ docker run --volumes-from dbdata2 -v $(<span class="built_in">pwd</span>):/backup busybox tar xvf /backup/backup.tar <span class="comment"># 恢复数据</span></span><br></pre></td></tr></table></figure>
<h2 id="Docker本地仓库">Docker本地仓库</h2><p>Docker Hub基本用法：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">$ docker search ubuntu</span><br><span class="line">$ docker pull ubuntu:<span class="number">14.04</span></span><br><span class="line">$ docker login <span class="comment"># 配置文件在~/.dockercfg</span></span><br><span class="line">$ docker push yourname/newimage</span><br></pre></td></tr></table></figure>
<p>配置本地的registry：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">$ docker run <span class="operator">-d</span> -p <span class="number">5000</span>:<span class="number">5000</span> registry</span><br><span class="line">$ docker tag ubuntu:<span class="number">14.04</span> localhost:<span class="number">5000</span>/ubuntu:<span class="number">14.04</span></span><br><span class="line">$ docker push localhost:<span class="number">5000</span>/ubuntu:<span class="number">14.04</span></span><br><span class="line">$ curl -v http://localhost:<span class="number">5000</span>/v1/repositories/ubuntu/tags/<span class="number">14.04</span> <span class="comment"># 查看image id</span></span><br><span class="line">$ docker run <span class="operator">-d</span> -p <span class="number">5000</span>:<span class="number">5000</span> \</span><br><span class="line"> <span class="operator">-e</span> STANDALONE=<span class="literal">false</span> \</span><br><span class="line"> <span class="operator">-e</span> MIRROR_SOURCE=https://registry-<span class="number">1</span>.docker.io \</span><br><span class="line"> <span class="operator">-e</span> MIRROR_SOURCE_INDEX=https://index.docker.io \</span><br><span class="line"> registry <span class="comment"># 作为官网镜像启动，代理模式</span></span><br></pre></td></tr></table></figure>
<p>使用本地的registry：</p>
<p>可以使用<code>--registry-mirror</code>参数启动：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">$ docker --registry-mirror=http://<span class="number">10.101</span>.<span class="number">29.26</span> <span class="operator">-d</span></span><br></pre></td></tr></table></figure>
<h2 id="Docker_Compose">Docker Compose</h2><p>Docker Compose可以方便地实现应用的组合。</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">$ pip install docker-compose</span><br><span class="line">$ mkdir compose && <span class="built_in">cd</span> <span class="variable">$_</span></span><br></pre></td></tr></table></figure>
<p>增加<code>app.py</code>：</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> flask <span class="keyword">import</span> Flask</span><br><span class="line"><span class="keyword">from</span> redis <span class="keyword">import</span> Redis</span><br><span class="line"><span class="keyword">import</span> os</span><br><span class="line">app = Flask(__name__)</span><br><span class="line">redis = Redis(host=<span class="string">'redis'</span>, port=<span class="number">6379</span>)</span><br><span class="line"></span><br><span class="line"><span class="decorator">@app.route('/')</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">hello</span><span class="params">()</span>:</span></span><br><span class="line"> redis.incr(<span class="string">'hits'</span>)</span><br><span class="line"> <span class="keyword">return</span> <span class="string">'Hello World! I have been seen %s times.'</span> % redis.get(<span class="string">'hits'</span>)</span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">"__main__"</span>:</span><br><span class="line"> app.run(host=<span class="string">"0.0.0.0"</span>, debug=<span class="keyword">True</span>)</span><br></pre></td></tr></table></figure>
<p>增加<code>requirements.txt</code>：</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">flask redis</span><br></pre></td></tr></table></figure>
<p>增加<code>Dockerfile</code>：</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">FROM python:2.7 ADD . /code WORKDIR /code RUN pip install -r requirements.txt</span><br></pre></td></tr></table></figure>
<p>增加<code>docker-compose.yml</code>：</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">web: build: . command: python app.py ports: - "5000:5000" volumes: - .:/code links: - redis redis: image: redis</span><br></pre></td></tr></table></figure>
<p>启动并管理服务：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">$ docker-compose up <span class="operator">-d</span></span><br><span class="line">$ docker-compose ps</span><br><span class="line">$ docker-compose run web env</span><br><span class="line">$ docker-compose stop</span><br></pre></td></tr></table></figure>
<h2 id="小结">小结</h2><p>Docker在快速应用分发、扩展等方面有很高的自动化能力，但是跟主机虚拟化一样，网络也是需要重点考虑的难点。默认情况下都是通过本地的一个bridge提供容器之间的通信，通过NAT为应用提供地址，一般的应用部署足以应付，但如果要实现更复杂的网络场景，如多主机上container组成大规模的L3负载均衡集群、应用无缝迁移等，还是需要大量的定制，目前这方面的周边产品开发比较活跃。</p>
<h2 id="参考">参考</h2><ol>
<li><a href="https://docs.docker.com" title="Official Docs" target="_blank" rel="external">Official Docs</a></li>
<li><a href="http://www.infoq.com/cn/articles/analysis-of-docker-file-system-aufs-and-devicemapper" title="Docker fs with aufs and devicemapper" target="_blank" rel="external">Docker with aufs and devicemapper</a></li>
<li><a href="http://www.ibm.com/developerworks/cn/linux/l-lxc-containers/" title="LXC TOOL" target="_blank" rel="external">LXC Tools</a></li>
<li><a href="https://docs.docker.com/compose" title="Docker Compose" target="_blank" rel="external">Docker Compose</a></li>
</ol>
]]></content>
<summary type="html">
<![CDATA[<h1 id="Docker概述">Docker概述</h1><p>Docker是基于Go语言开发的容器管理工具，而且抽象级别比lxc这些管理工具高，目前官方默认通过libcontainer管理容器。lxc是一些kernel patch（namespaces）和userspace]]>
</summary>
<category term="docker" scheme="http://promisejohn.github.io/tags/docker/"/>
<category term="Tech" scheme="http://promisejohn.github.io/categories/Tech/"/>
</entry>
<entry>
<title><![CDATA[BuildKVMImages]]></title>
<link href="http://promisejohn.github.io/2015/05/11/BuildKVMImages/"/>
<id>http://promisejohn.github.io/2015/05/11/BuildKVMImages/</id>
<published>2015-05-11T07:33:37.000Z</published>
<updated>2015-06-13T07:06:01.239Z</updated>
<content type="html"><![CDATA[<h2 id="手动创建KVM虚拟机镜像">手动创建KVM虚拟机镜像</h2><p>通过libvirt系的本地命令行管理工具，也可以方便地创建虚拟机。</p>
<ul>
<li>qemu-img：生成虚拟机磁盘文件</li>
<li>virsh：命令行虚拟机管理工具</li>
</ul>
<h3 id="生成Domain_XML文件">生成Domain XML文件</h3><p>如果要用X11远程到物理机使用GUI工具，需要配置sshd，安装<code>xauth</code>：</p>
<p><code>/etc/ssh/sshd_config</code>:</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">... X11Forwarding yes X11UseLocalhost no ...</span><br></pre></td></tr></table></figure>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">$ /etc/init.d/sshd reload</span><br><span class="line">$ yum install -y xauth</span><br><span class="line">$ ssh -X user@host</span><br><span class="line">$ virt-manager&</span><br></pre></td></tr></table></figure>
<p>也可以直接用官方的example修改配置：</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><domain type = 'kvm'> <name>centos66</name> <memory>1048576</memory> <vcpu>1</vcpu> <os> <type arch = 'x86_64'machine = 'pc'>hvm</type> <boot dev = 'cdrom'/> <boot dev = 'hd'/> </os> <features> <acpi/> <apic/> <pae/> </features> <clock offset = 'utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/libexec/qemu-kvm</emulator> <disk type = 'file' device = 'disk'> <driver name = 'qemu' type = 'raw' cache='none'/> <source file = '/opt/vmdisks/centos66.raw'/> <target dev='vda' bus='virtio'/> </disk> <disk type = 'file' device = 'cdrom'> <source file = '/opt/CentOS-6.6-x86_64-bin-DVD1.iso'/> <target dev = 'hdb' bus = 'ide'/> </disk> "centos66.xml" 48L, 1243C written 8,20-34 Top <target dev = 'hdb' bus = 'ide'/> </disk> <interface type='network'> <source network='default'/> <model type='virtio'/> </interface> <input type='tablet' bus='usb'/> <input type='mouse' bus='ps2'/> <graphics type='vnc' port='-1' autoport='yes' listen='0.0.0.0'> <listen type='address' address='0.0.0.0'/> </graphics> </devices> </domain></span><br></pre></td></tr></table></figure>
<h3 id="创建虚拟机">创建虚拟机</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">$ mkdir -p /opt/vmdisks</span><br><span class="line">$ qemu-img create <span class="operator">-f</span> raw /opt/vmdisks/centos66.raw <span class="number">8</span>G</span><br><span class="line">$ virsh create centos66.xml</span><br><span class="line">$ ip addr</span><br><span class="line">$ virsh vncdisplay centos66 <span class="comment">#获取ip和vnc端口登陆</span></span><br></pre></td></tr></table></figure>
<p>接下来就进入安装操作系统的界面，一路按指印即可。</p>
<h3 id="远程管理libvirt主机">远程管理libvirt主机</h3><p>可以用qemu+ssh，节点间互信后迁移；也可以用qemu+tcp，修改<code>/etc/libvirt/libvirtd.conf</code>:</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">listen_tls = 0 listen_tcp = 1 tcp_port = "16509" listen_addr = "0.0.0.0" auth_tcp = "none" # 生产环境建议加上授权验证</span><br></pre></td></tr></table></figure>
<p>修改<code>/etc/init.d/libvirtd</code>:</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">LIBVIRTD_CONFIG=/etc/libvirt/libvirtd.conf LIBVIRTD_ARGS="--listen"</span><br></pre></td></tr></table></figure>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">$ service libvirtd restart</span><br><span class="line">$ ss -ln | grep <span class="number">16509</span></span><br></pre></td></tr></table></figure>
<p>在其他节点就可以连接（注意iptables）：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">$ virsh -c qemu+tcp://<span class="number">192.168</span>.<span class="number">182.156</span>/system</span><br></pre></td></tr></table></figure>
<p>克隆虚拟机：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">$ virsh define centos66.xml</span><br><span class="line">$ virt-clone --original=centos66 --name=centos66_01 <span class="operator">-f</span> centos66_01.raw</span><br><span class="line">$ virsh list --all</span><br></pre></td></tr></table></figure>
<h3 id="参考">参考</h3><ol>
<li><a href="http://libvirt.org/formatdomain.html" title="libvirt官方虚拟机规格XML格式说明" target="_blank" rel="external">libvirt官方虚拟机规格XML格式说明</a></li>
</ol>
]]></content>
<summary type="html">
<![CDATA[<h2 id="手动创建KVM虚拟机镜像">手动创建KVM虚拟机镜像</h2><p>通过libvirt系的本地命令行管理工具，也可以方便地创建虚拟机。</p>
<ul>
<li>qemu-img：生成虚拟机磁盘文件</li>
<li>virsh：命令行虚拟机管理工具</li>
<]]>
</summary>
<category term="kvm" scheme="http://promisejohn.github.io/tags/kvm/"/>
<category term="libvirt" scheme="http://promisejohn.github.io/tags/libvirt/"/>
<category term="Tech" scheme="http://promisejohn.github.io/categories/Tech/"/>
</entry>
<entry>
<title><![CDATA[Openstack安装部署]]></title>
<link href="http://promisejohn.github.io/2015/05/07/HelloOpenstack/"/>
<id>http://promisejohn.github.io/2015/05/07/HelloOpenstack/</id>
<published>2015-05-07T07:12:56.000Z</published>
<updated>2015-06-13T07:06:01.243Z</updated>
<content type="html"><![CDATA[<h2 id="参考架构及部署规划">参考架构及部署规划</h2><p><img src="http://docs.openstack.org/icehouse/install-guide/install/yum/content/figures/1/a/common/figures/openstack_havana_conceptual_arch.png" alt="参考架构"></p>
<ul>
<li>操作系统：CentOS6.5</li>
<li>第三方yum源：epel, rdo</li>
</ul>
<p>节点部署角色：</p>
<table>
<thead>
<tr>
<th>节点名</th>
<th>internal ip</th>
<th>public ip</th>
<th>Role</th>
</tr>
</thead>
<tbody>
<tr>
<td>oscontroller</td>
<td>10.0.100.145</td>
<td>192.168.182.150</td>
<td>nova, glance, cinder, image, neutron, dashboard, heat</td>
</tr>
<tr>
<td>osnetwork</td>
<td>10.0.100.146</td>
<td>192.168.182.151</td>
<td>ML2, OVS, L2 Agent, L3 Agent, DHCP Agent</td>
</tr>
<tr>
<td>oscompute1</td>
<td>10.0.100.147</td>
<td>192.168.182.152</td>
<td>nova-compute</td>
</tr>
<tr>
<td>oscompute2</td>
<td>10.0.100.148</td>
<td>192.168.182.153</td>
<td>nova-compute</td>
</tr>
<tr>
<td>oskeystone</td>
<td>10.0.100.149</td>
<td>192.168.182.154</td>
<td>qpid/rabbitmq, keystone, mysql， memcached</td>
</tr>
<tr>
<td>osmeter</td>
<td>10.0.100.150</td>
<td>192.168.182.155</td>
<td>ceilometer, mongodb</td>
</tr>
<tr>
<td>osswift0</td>
<td>10.0.100.139</td>
<td>192.168.182.144</td>
<td>swift0, swift-proxy-server</td>
</tr>
<tr>
<td>osswift1</td>
<td>10.0.100.140</td>
<td>192.168.182.145</td>
<td>swift1</td>
</tr>
<tr>
<td>osswift2</td>
<td>10.0.100.141</td>
<td>192.168.182.146</td>
<td>swift2</td>
</tr>
<tr>
<td>osceph0</td>
<td>10.0.100.142</td>
<td>192.168.182.147</td>
<td>ceph0</td>
</tr>
<tr>
<td>osceph1</td>
<td>10.0.100.143</td>
<td>192.168.182.148</td>
<td>ceph1 # 暂时不用</td>
</tr>
<tr>
<td>osceph2</td>
<td>10.0.100.144</td>
<td>192.168.182.149</td>
<td>ceph2 # 暂时不用</td>
</tr>
</tbody>
</table>
<p><img src="http://docs.openstack.org/icehouse/install-guide/install/yum/content/figures/1/figures/installguide_arch-neutron.png" alt="参考部署架构"></p>
<a id="more"></a>
<h2 id="基础环境配置">基础环境配置</h2><h3 id="Network_staff">Network staff</h3><h4 id="配置远程登录，使用~/-ssh/config来简化远程登录操作（免密码验证）：">配置远程登录，使用~/.ssh/config来简化远程登录操作（免密码验证）：</h4><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">$ ssh-keygen -t rsa -P <span class="string">''</span> <span class="operator">-f</span> ~/.ssh/cloudlab150.key</span><br><span class="line">$ ssh-copy-id -i ~/.ssh/cloudlab150.key.pub root@<span class="number">192.168</span>.<span class="number">182.150</span> <span class="comment"># 这里需要输入密码一次</span></span><br></pre></td></tr></table></figure>
<p>编辑<code>~/.ssh/config</code>:</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">Host oscontroller 		HostName 192.168.182.150 		User root 		PreferredAuthentications publickey 		IdentityFile ~/.ssh/cloudlab150.key</span><br></pre></td></tr></table></figure>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">$ ssh oscontroller</span><br></pre></td></tr></table></figure>
<p>通过脚本自动生成Key并添加到~/.ssh/config：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">$ <span class="keyword">for</span> i <span class="keyword">in</span> {<span class="number">144</span>..<span class="number">154</span>}; <span class="keyword">do</span> ssh-keygen -t rsa -P <span class="string">''</span> <span class="operator">-f</span> ~/.ssh/cloudlab<span class="variable">$i</span>.key; ssh-copy-id -i ~/.ssh/cloudlab<span class="variable">$i</span>.key.pub root@<span class="number">192.168</span>.<span class="number">182</span>.<span class="variable">$i</span> ;<span class="keyword">done</span>;</span><br><span class="line">$ <span class="keyword">for</span> i <span class="keyword">in</span> {<span class="number">144</span>..<span class="number">154</span>}; <span class="keyword">do</span> <span class="built_in">echo</span> <span class="operator">-e</span> <span class="string">"Host cloudlab<span class="variable">$i</span>\n HostName 192.168.182.<span class="variable">$i</span>\n User root\n PreferredAuthentications publickey\n IdentityFile ~/.ssh/cloudlab<span class="variable">$i</span>.key"</span> >> ~/.ssh/config;<span class="keyword">done</span>;</span><br></pre></td></tr></table></figure>
<p>编辑<code>~/.ssh/config</code>: 根据部署规划可以添加Host别名，方便记忆</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">... Host cloudlab154 oskeystone 		HostName 192.168.182.154 ...</span><br></pre></td></tr></table></figure>
<h3 id="配置主机HostName及/etc/hosts">配置主机HostName及/etc/hosts</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">$ vim /etc/sysconfig/network</span><br><span class="line">$ hostname oskeystone <span class="comment"># 一台台配置oscontroller, osnetwork...</span></span><br></pre></td></tr></table></figure>
<p>编辑<code>/etc/hosts</code>: 可以统一配置然后通过scp等工具分发到所有节点</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">192.168.182.144 osswift0 192.168.182.145 osswift1 192.168.182.146 osswift2 192.168.182.147 osceph0 192.168.182.148 osceph1 192.168.182.149 osceph2 192.168.182.150 oscontroller 192.168.182.151 osnetwork 192.168.182.152 oscompute1 192.168.182.153 oscompute2 192.168.182.154 oskeystone</span><br></pre></td></tr></table></figure>
<h3 id="配置时区，启用NTP服务">配置时区，启用NTP服务</h3><p>在oskeystone上部署NTP服务器，其他节点从oskeystone上同步时间：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">$ yum install ntp <span class="comment"># 每个节点都需要ntp client</span></span><br><span class="line">$ chkconfig ntpd on <span class="comment"># oskeystone自动启动ntpd</span></span><br></pre></td></tr></table></figure>
<p>编辑<code>/etc/ntpd.conf</code>:<br>服务器端，即oskeystone:</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">... server	202.112.10.60 # s1a.time.edu.cn server	127.127.1.0 # local clock fudge	127.127.1.0 stratum 10 restrict 10.0.100.0 mask 255.255.255.0 nomodify ...</span><br></pre></td></tr></table></figure>
<p>客户端：</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">server 10.0.100.149 restrict 10.0.100.149 nomodify notrap noquery</span><br></pre></td></tr></table></figure>
<p>启动ntpd服务：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">$ iptables -F <span class="comment"># 服务器端需要开放UDP 123端口，这里简单测试直接清空</span></span><br><span class="line">$ ntpdate <span class="number">10.0</span>.<span class="number">100.149</span> <span class="comment"># 手动同步一次</span></span><br><span class="line">$ service ntpd start</span><br><span class="line">$ ntpq -p <span class="comment"># 检查时钟源</span></span><br><span class="line">$ ntpstat <span class="comment"># 检查时间同步状态</span></span><br></pre></td></tr></table></figure>
<p>调整时区：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">$ tzselect</span><br><span class="line">$ TZ=<span class="string">'Asia/Shanghai'</span>; <span class="built_in">export</span> TZ</span><br></pre></td></tr></table></figure>
<h3 id="密码准备">密码准备</h3><p>提前准备好安装过程中用到的密码：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">for</span> i <span class="keyword">in</span> KEYSTONE GLANCE NOVA CINDER NEUTRON HEAT CEILOMETER; <span class="keyword">do</span> <span class="built_in">echo</span> <span class="variable">$i</span><span class="string">"_DBPASS"</span> `openssl rand -hex <span class="number">10</span>`; <span class="built_in">echo</span> <span class="variable">$i</span><span class="string">"_PASS"</span> `openssl rand -hex <span class="number">10</span>`; <span class="keyword">done</span></span><br><span class="line"><span class="keyword">for</span> i <span class="keyword">in</span> MYSQL_PASS QPID_PASS DEMO_PASS ADMIN_PASS DASH_DBPASS METADATA_SECRET SWIFT_PASS; <span class="keyword">do</span> <span class="built_in">echo</span> <span class="variable">$i</span> `openssl rand -hex <span class="number">10</span>`;<span class="keyword">done</span>;</span><br></pre></td></tr></table></figure>
<p>记录好生成的输出：</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">KEYSTONE_DBPASS 2e37d19cb04e42cf83ae KEYSTONE_PASS 86be667d6555105f9eb6 #不需要 GLANCE_DBPASS 9fb6d0fddec4a6dcfa10 GLANCE_PASS 7cb6a024c60dd85c0b71 NOVA_DBPASS e5a43fc52ce0aaa1c7b4 NOVA_PASS fd53a063286a4f22ab7b CINDER_DBPASS 791ce55fa6888c065bf3 CINDER_PASS 89edec5c2869dce44c61 NEUTRON_DBPASS fd535f9e2441a28a252d NEUTRON_PASS 033d164bcc70e1244be7 HEAT_DBPASS 51eb27f53983633f3337 HEAT_PASS 3817bfface1b24918d4b CEILOMETER_DBPASS 072aac393486f9b29235 CEILOMETER_PASS eb8ed86ef2178168c458 MYSQL_PASS a904019ba8cc0b14bef2 QPID_PASS 232a62982a3bcdc86cba DEMO_PASS c3f7871eaa28ca146d09 ADMIN_PASS b517d18b5663e8e9c2ae DASH_DBPASS 32ec0f799695e5ff6fe0 METADATA_SECRET 69e2f4db01fe4100bd32 SWIFT_PASS d8e42c6a28bf6eb08b76</span><br></pre></td></tr></table></figure>
<h3 id="数据库安装配置">数据库安装配置</h3><p>选择在oskeystone上安装mysql作为公共的数据库服务器：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">$ yum install mysql mysql-server MySQL-python</span><br><span class="line">$ yum install MySQL-python <span class="comment"># 需要访问mysql的客户端需要安装python驱动，主要是oscontroller</span></span><br></pre></td></tr></table></figure>
<p>编辑<code>/etc/my.cnf</code>：</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[mysqld] ... bind-address = 10.0.100.149 default-storage-engine = innodb innodb_file_per_table collation-server = utf8_general_ci init-connect = 'SET NAMES utf8' character-set-server = utf8 ...</span><br></pre></td></tr></table></figure>
<p>启动mysql数据库：</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">$ service mysqld start $ chkconfig mysqld on $ mysql_install_db $ mysql_secure_installation</span><br></pre></td></tr></table></figure>
<h3 id="消息服务安装配置">消息服务安装配置</h3><p>在oskeystone安装Qpid消息服务器：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">$ yum install qpid-cpp-server</span><br></pre></td></tr></table></figure>
<p>暂时不做验证，编辑<code>/etc/qpidd.conf</code>:</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">auth=no</span><br></pre></td></tr></table></figure>
<p>启动qpid服务端：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">$ service qpidd start</span><br><span class="line">$ chkconfig qpidd on</span><br></pre></td></tr></table></figure>
<h3 id="yum第三方源添加">yum第三方源添加</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">$ yum install -y yum-plugin-priorities</span><br><span class="line">$ yum install -y http://repos.fedorapeople.org/repos/openstack/openstack-icehouse/rdo-release-icehouse-<span class="number">3</span>.noarch.rpm</span><br><span class="line">$ yum install -y http://dl.fedoraproject.org/pub/epel/<span class="number">6</span>/x86_64/epel-release-<span class="number">6</span>-<span class="number">8</span>.noarch.rpm</span><br></pre></td></tr></table></figure>
<p>验证第三方源：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">$ yum install openstack-utils <span class="comment"># 基于crudini，对于多个值的option，需要手动修改配置</span></span><br><span class="line">$ yum install openstack-selinux</span><br></pre></td></tr></table></figure>
<p>更新组件：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">$ yum upgrade</span><br><span class="line">$ reboot <span class="comment"># 如果更新了kernel，需要重启</span></span><br></pre></td></tr></table></figure>
<h2 id="ID_Service:_Keystone">ID Service: Keystone</h2><p><img src="http://docs.openstack.org/icehouse/install-guide/install/yum/content/figures/2/figures/SCH_5002_V00_NUAC-Keystone.png" alt="keystone原理"><br>最关键的几个概念中，<strong>tenant</strong>最关键，它是一个资源容器，里面包含了用户、服务列表、组织、账户等资源，是用来聚集和隔离资源的单元。从第2、3步中可以看到，用户是先确定了tenant再获取对应的服务列表。而Keystone在整个过程中，对每一个步骤都进行了鉴权操作。</p>
<h3 id="在oskeystone上安装keystone服务：">在oskeystone上安装keystone服务：</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br></pre></td><td class="code"><pre><span class="line">$ yum install openstack-keystone python-keystoneclient</span><br><span class="line">$ openstack-config --set /etc/keystone/keystone.conf \</span><br><span class="line"> database connection mysql://keystone:<span class="number">2</span>e37d19cb04e42cf83ae@<span class="number">10.0</span>.<span class="number">100.149</span>/keystone</span><br><span class="line">$ mysql -u root -p</span><br><span class="line">mysql> CREATE DATABASE keystone;</span><br><span class="line">mysql> GRANT ALL PRIVILEGES ON keystone.* TO <span class="string">'keystone'</span>@<span class="string">'localhost'</span> \</span><br><span class="line"> IDENTIFIED BY <span class="string">'KEYSTONE_DBPASS'</span>;</span><br><span class="line">mysql> GRANT ALL PRIVILEGES ON keystone.* TO <span class="string">'keystone'</span>@<span class="string">'%'</span> \</span><br><span class="line"> IDENTIFIED BY <span class="string">'KEYSTONE_DBPASS'</span>;</span><br><span class="line">mysql> <span class="built_in">exit</span></span><br><span class="line">$ chown -R keystone:keystone /var/<span class="built_in">log</span>/keystone</span><br><span class="line">$ chown -R keystone:keystone /etc/keystone/</span><br><span class="line">$ su <span class="operator">-s</span> /bin/sh -c <span class="string">"keystone-manage db_sync"</span> keystone</span><br><span class="line">$ ADMIN_TOKEN=$(openssl rand -hex <span class="number">10</span>) <span class="comment"># keystone和其他service之间的token</span></span><br><span class="line">$ <span class="built_in">echo</span> <span class="variable">$ADMIN_TOKEN</span></span><br><span class="line">$ openstack-config --set /etc/keystone/keystone.conf DEFAULT admin_token <span class="variable">$ADMIN_TOKEN</span></span><br><span class="line">$ keystone-manage pki_setup --keystone-user keystone --keystone-group keystone <span class="comment"># 使用 PKI token，效率会低，但是安全。</span></span><br><span class="line">$ chown -R keystone:keystone /etc/keystone/ssl</span><br><span class="line">$ chmod -R o-rwx /etc/keystone/ssl</span><br><span class="line">$ service openstack-keystone start</span><br><span class="line">$ chkconfig openstack-keystone on</span><br></pre></td></tr></table></figure>
<p>把过期的token从数据库删除，在log中记录，防止mysql性能恶化：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">$ (crontab <span class="operator">-l</span> -u keystone <span class="number">2</span>>&<span class="number">1</span> | grep -q token_flush) || \</span><br><span class="line"><span class="built_in">echo</span> <span class="string">'@hourly /usr/bin/keystone-manage token_flush >/var/log/keystone/keystone-tokenflush.log 2>&1'</span> >> /var/spool/cron/keystone</span><br></pre></td></tr></table></figure>
<h3 id="定义User,_tenant,_role">定义User, tenant, role</h3><p>用ADMIN_TOKEN创建管理员和普通DEMO用户账号：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line">$ <span class="built_in">export</span> OS_SERVICE_TOKEN=b412e43d938a38122d84 <span class="comment"># 之前创建的ADMIN_TOKEN</span></span><br><span class="line">$ <span class="built_in">export</span> OS_SERVICE_ENDPOINT=http://oskeystone:<span class="number">35357</span>/v2.<span class="number">0</span></span><br><span class="line">$ keystone user-create --name=admin --pass=b517d18b5663e8e9c2ae --email=admin@tecstack.org <span class="comment"># 密码开始时已批量生成</span></span><br><span class="line">$ keystone role-create --name=admin</span><br><span class="line">$ keystone tenant-create --name=admin --description=<span class="string">"Admin Tenant"</span></span><br><span class="line">$ keystone user-role-add --user=admin --tenant=admin --role=admin</span><br><span class="line">$ keystone user-role-add --user=admin --role=_member_ --tenant=admin</span><br><span class="line">$ keystone user-create --name=demo --pass=c3f7871eaa28ca146d09 --email=demo@tecstack.org</span><br><span class="line">$ keystone tenant-create --name=demo --description=<span class="string">"Demo Tenant"</span></span><br><span class="line">$ keystone user-role-add --user=demo --role=_member_ --tenant=demo</span><br></pre></td></tr></table></figure>
<p>创建service之间互访的tenant：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">$ keystone tenant-create --name=service --description=<span class="string">"Service Tenant"</span></span><br></pre></td></tr></table></figure>
<p>在keystone上注册服务：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">$ keystone service-create --name=keystone --type=identity \</span><br><span class="line"> --description=<span class="string">"OpenStack Identity"</span></span><br><span class="line">$ keystone endpoint-create \</span><br><span class="line"> --service-id=$(keystone service-list | awk <span class="string">'/ identity / {print $2}'</span>) \</span><br><span class="line"> --publicurl=http://<span class="number">192.168</span>.<span class="number">182.154</span>:<span class="number">5000</span>/v2.<span class="number">0</span> \</span><br><span class="line"> --internalurl=http://<span class="number">10.0</span>.<span class="number">100.149</span>:<span class="number">5000</span>/v2.<span class="number">0</span> \</span><br><span class="line"> --adminurl=http://<span class="number">10.0</span>.<span class="number">100.149</span>:<span class="number">35357</span>/v2.<span class="number">0</span></span><br></pre></td></tr></table></figure>
<p>验证keystone服务是否注册成功：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">$ <span class="built_in">unset</span> OS_SERVICE_TOKEN OS_SERVICE_ENDPOINT</span><br><span class="line">$ keystone --os-username=admin --os-password=b517d18b5663e8e9c2ae \</span><br><span class="line"> --os-auth-url=http://<span class="number">10.0</span>.<span class="number">100.149</span>:<span class="number">35357</span>/v2.<span class="number">0</span> token-get</span><br><span class="line">$ keystone --os-username=admin --os-password=b517d18b5663e8e9c2ae \</span><br><span class="line"> --os-tenant-name=admin --os-auth-url=http://<span class="number">10.0</span>.<span class="number">100.149</span>:<span class="number">35357</span>/v2.<span class="number">0</span> \</span><br><span class="line"> token-get <span class="comment"># 带tenant权限</span></span><br></pre></td></tr></table></figure>
<p>新增一个环境变量设置文件<code>~/adminrc</code>简化客户端命令执行参数：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="built_in">export</span> OS_USERNAME=admin</span><br><span class="line"><span class="built_in">export</span> OS_PASSWORD=b517d18b5663e8e9c2ae</span><br><span class="line"><span class="built_in">export</span> OS_TENANT_NAME=admin</span><br><span class="line"><span class="built_in">export</span> OS_AUTH_URL=http://<span class="number">10.0</span>.<span class="number">100.149</span>:<span class="number">35357</span>/v2.<span class="number">0</span></span><br><span class="line"><span class="built_in">export</span> PS1=<span class="string">'[\u@\h \W(keystone_admin)]$ '</span> <span class="comment"># 识别是否处于管理员状态</span></span><br></pre></td></tr></table></figure>
<p>验证管理员权限：</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">$ source ~/adminrc $ keystone user-list</span><br></pre></td></tr></table></figure>
<p>同样也建立一个Demo普通用户的<code>~/demorc</code>：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="built_in">export</span> OS_USERNAME=demo</span><br><span class="line"><span class="built_in">export</span> OS_PASSWORD=c3f7871eaa28ca146d09</span><br><span class="line"><span class="built_in">export</span> OS_TENANT_NAME=demo</span><br><span class="line"><span class="built_in">export</span> OS_AUTH_URL=http://<span class="number">10.0</span>.<span class="number">100.149</span>:<span class="number">35357</span>/v2.<span class="number">0</span></span><br><span class="line"><span class="built_in">export</span> PS1=<span class="string">'[\u@\h \W(keystone_demo)]$ '</span></span><br></pre></td></tr></table></figure>
<h2 id="Openstack_Service_Clients">Openstack Service Clients</h2><p>Openstack的服务端都提供RESTful API，客户端都是通过curl方式进行交互，且都是基于python2.x实现。<br>可以通过yum安装，也可以通过pip安装，也可以通过pyenv、virtualenv等创建隔离环境后安装以辅助开发多个版本，具体可以参考<a href="http://promisejohn.github.io/2015/04/15/PythonDevEnvSetting/" title="Python开发环境搭建">Python开发环境搭建</a>。这里直接安装到系统版本的python下：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">$ yum groupinstall -y <span class="string">"Development Tools"</span> <span class="comment"># 有些依赖需要gcc编译</span></span><br><span class="line">$ yum install -y python-devel</span><br><span class="line">$ <span class="built_in">alias</span> easy_install=<span class="string">'easy_install -i http://pypi.douban.com/simple'</span> <span class="comment"># 用douban源加速</span></span><br><span class="line">$ easy_install pip</span><br></pre></td></tr></table></figure>
<p>修改pip源，<code>~/.pip/pip.conf</code>:</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[global] index-url = http://pypi.douban.com/simple</span><br></pre></td></tr></table></figure>
<p>批量安装Clients：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">$ <span class="keyword">for</span> i <span class="keyword">in</span> cinder nova keystone glance neutron swift heat ceilometer; <span class="keyword">do</span> pip install <span class="string">"python-"</span><span class="variable">$i</span><span class="string">"client"</span>;<span class="keyword">done</span>;</span><br></pre></td></tr></table></figure>
<h2 id="Images_Service:_Glance">Images Service: Glance</h2><p>默认采用本地文件系统存储镜像：<code>/var/lib/glance/images/</code><br>glance-api提供API，glance-registry负责image的metadata存取查询。metadata保存在mysql，image文件本身可以存在文件系统或对象存储等。</p>
<h3 id="安装Glance_Service">安装Glance Service</h3><p>在oscontroller上部署Glance Service：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">$ yum install -y openstack-glance python-glanceclient</span><br><span class="line">$ openstack-config --set /etc/glance/glance-api.conf database \</span><br><span class="line"> connection mysql://glance:<span class="number">9</span>fb6d0fddec4a6dcfa10@<span class="number">10.0</span>.<span class="number">100.149</span>/glance</span><br><span class="line">$ openstack-config --set /etc/glance/glance-registry.conf database \</span><br><span class="line"> connection mysql://glance:<span class="number">9</span>fb6d0fddec4a6dcfa10@<span class="number">10.0</span>.<span class="number">100.149</span>/glance</span><br></pre></td></tr></table></figure>
<p>在oskeystone上增加glance数据库：</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">$ mysql -u root -pa904019ba8cc0b14bef2 mysql> CREATE DATABASE glance; mysql> GRANT ALL PRIVILEGES ON glance.* TO 'glance'@'localhost' \ IDENTIFIED BY '9fb6d0fddec4a6dcfa10'; mysql> GRANT ALL PRIVILEGES ON glance.* TO 'glance'@'%' \ IDENTIFIED BY '9fb6d0fddec4a6dcfa10';</span><br></pre></td></tr></table></figure>
<p>在oscontroller上同步数据库：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">$ chown -R glance:glance /etc/glance</span><br><span class="line">$ chown -R glance:glance /var/<span class="built_in">log</span>/glance</span><br><span class="line">$ chown -R glance:glance /var/lib/glance</span><br><span class="line">$ chown -R glance:glance /var/run/glance</span><br><span class="line">$ su <span class="operator">-s</span> /bin/sh -c <span class="string">"glance-manage db_sync"</span> glance <span class="comment"># 使用glance账户执行命令</span></span><br></pre></td></tr></table></figure>
<p>Ops，出现Warning和Error：</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">...PowmInsecureWarning: Not using mpz_powm_sec. You should rebuild using libgmp >= 5 to avoid timing attack vulnerability. _warn("Not using mpz_powm_sec. You should rebuild using libgmp >= 5 to avoid timing attack vulnerability.", PowmInsecureWarning)...</span><br></pre></td></tr></table></figure>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">$ tailf /var/log/glance/api.log ... glance AttributeError: 'NoneType' object has no attribute 'replace' ...</span><br></pre></td></tr></table></figure>
<p>Warning可以在<a href="http://techglimpse.com/openstack-installation-errors-solutions/" title="openstack安装错误方案" target="_blank" rel="external">这个网址</a>查到解决方案：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line">$ wget https://gmplib.org/download/gmp/gmp-<span class="number">6.0</span>.<span class="number">0</span>a.tar.bz2</span><br><span class="line">$ tar xvf gmp-<span class="number">6.0</span>.<span class="number">0</span>a.tar.bz2</span><br><span class="line">$ <span class="built_in">cd</span> gmp-<span class="number">6.0</span>.<span class="number">0</span></span><br><span class="line">$ ./configure</span><br><span class="line">$ make</span><br><span class="line">$ make check</span><br><span class="line">$ make install</span><br><span class="line">$ wget https://ftp.dlitz.net/pub/dlitz/crypto/pycrypto/pycrypto-<span class="number">2.6</span>.<span class="number">1</span>.tar.gz <span class="comment"># 从源码安装，而不是从pip，否则会有ImportError: .../_AES.so: undefined symbol: rpl_malloc错误</span></span><br><span class="line">$ tar xvf pycrypto-<span class="number">2.6</span>.<span class="number">1</span>.tar.gz</span><br><span class="line">$ <span class="built_in">cd</span> pycrypto-<span class="number">2.6</span>.<span class="number">1</span></span><br><span class="line">$ <span class="built_in">export</span> ac_cv_func_malloc_0_nonnull=yes</span><br><span class="line">$ ./configure</span><br><span class="line">$ python setup.py build</span><br><span class="line">$ python setup.py install</span><br></pre></td></tr></table></figure>
<p>出现的错误很奇怪，网上也没什么案例，只能一步步看日志trace：</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">File "/usr/lib/python2.6/site-packages/glance/openstack/common/db/sqlalchemy/migration.py", line 236, in db_version 2015-04-22 02:43:28.196 27939 TRACE glance meta.reflect(bind=engine) 2015-04-22 02:43:28.196 27939 TRACE glance File "/usr/lib64/python2.6/site-packages/sqlalchemy/schema.py", line 2776, in reflect 2015-04-22 02:43:28.196 27939 TRACE glance connection=conn)) 2015-04-22 02:43:28.196 27939 TRACE glance File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 1677, in table_names 2015-04-22 02:43:28.196 27939 TRACE glance return self.dialect.get_table_names(conn, schema)</span><br></pre></td></tr></table></figure>
<p>可以关注的是3处关键词：<strong>bind=engine</strong>, <strong>connection=conn</strong>, <strong>get_table_names(conn, schema)</strong>，很大程度上表明数据库连接建立失败，而且是在获取schema的时候，重新检查一下<code>/etc/glance/glance-api.conf</code>，发现mysql连接最后少加了schema，即误写成了<code>mysql://glance:9fb6d0fddec4a6dcfa10@10.0.100.149/</code>，少了<code>glance</code>，改正后再执行db_sync成功。</p>
<h3 id="创建Glance服务权限：">创建Glance服务权限：</h3><p>创建Glance Service与Keystone验证的用户：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">$ <span class="built_in">source</span> ~/adminrc <span class="comment"># 加载admin环境变量</span></span><br><span class="line">$ keystone user-create --name=glance --pass=<span class="number">7</span>cb6a024c60dd85c0b71 \</span><br><span class="line"> --email=glance@tecstack.org</span><br><span class="line">$ keystone user-role-add --user=glance --tenant=service --role=admin</span><br></pre></td></tr></table></figure>
<p>配置Glance Service通过Keystone进行鉴权：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br></pre></td><td class="code"><pre><span class="line">$ openstack-config --set /etc/glance/glance-api.conf keystone_authtoken \</span><br><span class="line"> auth_uri http://<span class="number">10.0</span>.<span class="number">100.149</span>:<span class="number">5000</span></span><br><span class="line">$ openstack-config --set /etc/glance/glance-api.conf keystone_authtoken \</span><br><span class="line"> auth_host <span class="number">10.0</span>.<span class="number">100.149</span></span><br><span class="line">$ openstack-config --set /etc/glance/glance-api.conf keystone_authtoken \</span><br><span class="line"> auth_port <span class="number">35357</span></span><br><span class="line">$ openstack-config --set /etc/glance/glance-api.conf keystone_authtoken \</span><br><span class="line"> auth_protocol http</span><br><span class="line">$ openstack-config --set /etc/glance/glance-api.conf keystone_authtoken \</span><br><span class="line"> admin_tenant_name service</span><br><span class="line">$ openstack-config --set /etc/glance/glance-api.conf keystone_authtoken \</span><br><span class="line"> admin_user glance</span><br><span class="line">$ openstack-config --set /etc/glance/glance-api.conf keystone_authtoken \</span><br><span class="line"> admin_password <span class="number">7</span>cb6a024c60dd85c0b71</span><br><span class="line">$ openstack-config --set /etc/glance/glance-api.conf paste_deploy \</span><br><span class="line"> flavor keystone</span><br><span class="line">$ openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken \</span><br><span class="line"> auth_uri http://<span class="number">10.0</span>.<span class="number">100.149</span>:<span class="number">5000</span></span><br><span class="line">$ openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken \</span><br><span class="line"> auth_host <span class="number">10.0</span>.<span class="number">100.149</span></span><br><span class="line">$ openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken \</span><br><span class="line"> auth_port <span class="number">35357</span></span><br><span class="line">$ openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken \</span><br><span class="line"> auth_protocol http</span><br><span class="line">$ openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken \</span><br><span class="line"> admin_tenant_name service</span><br><span class="line">$ openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken \</span><br><span class="line"> admin_user glance</span><br><span class="line">$ openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken \</span><br><span class="line"> admin_password <span class="number">7</span>cb6a024c60dd85c0b71</span><br><span class="line">$ openstack-config --set /etc/glance/glance-registry.conf paste_deploy \</span><br><span class="line"> flavor keystone</span><br></pre></td></tr></table></figure>
<p>注册Glance Service服务：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">$ keystone service-create --name=glance --type=image \</span><br><span class="line"> --description=<span class="string">"OpenStack Image Service"</span></span><br><span class="line">$ keystone endpoint-create \</span><br><span class="line"> --service-id=$(keystone service-list | awk <span class="string">'/ image / {print $2}'</span>) \</span><br><span class="line"> --publicurl=http://<span class="number">192.168</span>.<span class="number">182.150</span>:<span class="number">9292</span> \</span><br><span class="line"> --internalurl=http://<span class="number">10.0</span>.<span class="number">100.145</span>:<span class="number">9292</span> \</span><br><span class="line"> --adminurl=http://<span class="number">10.0</span>.<span class="number">100.145</span>:<span class="number">9292</span></span><br></pre></td></tr></table></figure>
<p>启动Glance服务进程：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">$ service openstack-glance-api start</span><br><span class="line">$ service openstack-glance-registry start</span><br><span class="line">$ chkconfig openstack-glance-api on</span><br><span class="line">$ chkconfig openstack-glance-registry on</span><br></pre></td></tr></table></figure>
<h3 id="验证Glance_Service">验证Glance Service</h3><p>用cirros验证镜像服务：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">$ wget http://cdn.download.cirros-cloud.net/<span class="number">0.3</span>.<span class="number">2</span>/cirros-<span class="number">0.3</span>.<span class="number">2</span>-x86_64-disk.img</span><br><span class="line">$ file cirros-<span class="number">0.3</span>.<span class="number">2</span>-x86_64-disk.img</span><br><span class="line">cirros-<span class="number">0.3</span>.<span class="number">2</span>-x86_64-disk.img: Qemu Image, Format: Qcow , Version: <span class="number">2</span></span><br><span class="line">$ <span class="built_in">source</span> ~/adminrc</span><br><span class="line">$ glance image-create --name <span class="string">"cirros-0.3.2-x86_64"</span> --disk-format qcow2 \</span><br><span class="line"> --container-format bare --is-public True --progress < cirros-<span class="number">0.3</span>.<span class="number">2</span>-x86_64-disk.img</span><br><span class="line">$ glance image-list</span><br></pre></td></tr></table></figure>
<p>如果不想下载镜像，可以直接通过<code>--copy-from</code>参数直接导入：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">$ glance image-create --name=<span class="string">"cirros-0.3.2-x86_64"</span> --disk-format=qcow2 \</span><br><span class="line"> --container-format=bare --is-public=<span class="literal">true</span> \</span><br><span class="line"> --copy-from http://cdn.download.cirros-cloud.net/<span class="number">0.3</span>.<span class="number">2</span>/cirros-<span class="number">0.3</span>.<span class="number">2</span>-x86_64-disk.img</span><br></pre></td></tr></table></figure>
<h2 id="Compute_Service:_Nova">Compute Service: Nova</h2><p>Compute是IaaS的核心，<code>nova-api</code>,<code>nova-api-metadata</code>提供API服务；<code>nova-compute</code>调用hypervisor API启停虚拟机，<code>nova-scheduler</code>从请求队列调度分配主机，<code>nova-conductor</code>把<code>nova-compute</code>和数据库解耦，可以横向扩展，不建议跟<code>nova-compute</code>一起部署；<code>nova-network</code>, <code>nova-dhcpbridge</code>，网络部分功能逐步合并到Neutron Service；<code>nova-consoleauth</code>，负责控制台鉴权，<code>nova-novncproxy</code>和<code>nova-xvpnvncproxy</code>分别对应浏览器和java客户端，<code>nova-cert</code>管理x509证书。<code>nova-objectstore</code>和<code>euca2ools</code>是针对EC2场景的工具，实现Image Service和S3的互通；<code>nova</code>,<code>nova-manage</code>是本地客户端。</p>
<h3 id="Nova安装">Nova安装</h3><p>在oscontroller上安装nova管理端：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">$ yum install -y openstack-nova-api openstack-nova-cert openstack-nova-conductor openstack-nova-console openstack-nova-novncproxy openstack-nova-scheduler python-novaclient</span><br><span class="line">$ openstack-config --set /etc/nova/nova.conf \</span><br><span class="line"> database connection mysql://nova:e5a43<span class="built_in">fc</span>52ce0aaa1c7b4@<span class="number">10.0</span>.<span class="number">100.149</span>/nova</span><br><span class="line">$ openstack-config --set /etc/nova/nova.conf \</span><br><span class="line"> DEFAULT rpc_backend qpid</span><br><span class="line">$ openstack-config --set /etc/nova/nova.conf DEFAULT qpid_hostname <span class="number">10.0</span>.<span class="number">100.149</span></span><br><span class="line">$ openstack-config --set /etc/nova/nova.conf DEFAULT my_ip <span class="number">10.0</span>.<span class="number">100.145</span></span><br><span class="line">$ openstack-config --set /etc/nova/nova.conf DEFAULT vncserver_listen <span class="number">10.0</span>.<span class="number">100.145</span></span><br><span class="line">$ openstack-config --set /etc/nova/nova.conf DEFAULT vncserver_proxyclient_address <span class="number">10.0</span>.<span class="number">100.145</span></span><br></pre></td></tr></table></figure>
<p>在oskeystone上创建nova的数据库：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">$ mysql -u root -pa904019ba8cc0b14bef2</span><br><span class="line">mysql> CREATE DATABASE nova;</span><br><span class="line">mysql> GRANT ALL PRIVILEGES ON nova.* TO <span class="string">'nova'</span>@<span class="string">'%'</span> \</span><br><span class="line">IDENTIFIED BY <span class="string">'e5a43fc52ce0aaa1c7b4'</span>;</span><br></pre></td></tr></table></figure>
<p>在oscontroller上同步nova数据库：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">$ chown -R nova:nova /etc/nova/</span><br><span class="line">$ chown -R nova:nova /var/<span class="built_in">log</span>/nova/</span><br><span class="line">$ chown -R nova:nova /var/run/nova/</span><br><span class="line">$ chown -R nova:nova /var/lib/nova/</span><br><span class="line">$ su <span class="operator">-s</span> /bin/sh -c <span class="string">"nova-manage db sync"</span> nova</span><br></pre></td></tr></table></figure>
<p>创建Nova Service在Keystone上的账号权限，配置其使用keystone做鉴权：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line">$ keystone user-create --name=nova --pass=fd53a063286a4f22ab7b --email=nova@tecstack.org</span><br><span class="line">$ keystone user-role-add --user=nova --tenant=service --role=admin</span><br><span class="line">$ openstack-config --set /etc/nova/nova.conf DEFAULT auth_strategy keystone</span><br><span class="line">$ openstack-config --set /etc/nova/nova.conf keystone_authtoken auth_uri http://<span class="number">10.0</span>.<span class="number">100.149</span>:<span class="number">5000</span></span><br><span class="line">$ openstack-config --set /etc/nova/nova.conf keystone_authtoken auth_host <span class="number">10.0</span>.<span class="number">100.149</span></span><br><span class="line">$ openstack-config --set /etc/nova/nova.conf keystone_authtoken auth_protocol http</span><br><span class="line">$ openstack-config --set /etc/nova/nova.conf keystone_authtoken auth_port <span class="number">35357</span></span><br><span class="line">$ openstack-config --set /etc/nova/nova.conf keystone_authtoken admin_user nova</span><br><span class="line">$ openstack-config --set /etc/nova/nova.conf keystone_authtoken admin_tenant_name service</span><br><span class="line">$ openstack-config --set /etc/nova/nova.conf keystone_authtoken admin_password fd53a063286a4f22ab7b</span><br></pre></td></tr></table></figure>
<p>在oskeystone上注册Nova Service：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">$ <span class="built_in">source</span> ~/adminrc</span><br><span class="line">$ keystone service-create --name=nova --type=compute \</span><br><span class="line"> --description=<span class="string">"OpenStack Compute"</span></span><br><span class="line">$ keystone endpoint-create \</span><br><span class="line"> --service-id=$(keystone service-list | awk <span class="string">'/ compute / {print $2}'</span>) \</span><br><span class="line"> --publicurl=http://<span class="number">192.168</span>.<span class="number">182.150</span>:<span class="number">8774</span>/v2/%\(tenant_id\)s \</span><br><span class="line"> --internalurl=http://<span class="number">10.0</span>.<span class="number">100.145</span>:<span class="number">8774</span>/v2/%\(tenant_id\)s \</span><br><span class="line"> --adminurl=http://<span class="number">10.0</span>.<span class="number">100.145</span>:<span class="number">8774</span>/v2/%\(tenant_id\)s</span><br></pre></td></tr></table></figure>
<p>启动Nova服务程序：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">$ <span class="keyword">for</span> svc <span class="keyword">in</span> openstack-nova-{api,cert,consoleauth,scheduler,conductor,novncproxy}; <span class="keyword">do</span> service <span class="variable">$svc</span> start; chkconfig <span class="variable">$svc</span> on; <span class="keyword">done</span>;</span><br><span class="line">$ <span class="keyword">for</span> svc <span class="keyword">in</span> openstack-nova-{api,cert,consoleauth,scheduler,conductor,novncproxy}; <span class="keyword">do</span> service <span class="variable">$svc</span> status; chkconfig | grep <span class="variable">$svc</span>; <span class="keyword">done</span>; <span class="comment"># 检查进程状态及自动启动设置状态</span></span><br></pre></td></tr></table></figure>
<p>检查Nova Service部署</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">$ <span class="built_in">source</span> ~/adminrc</span><br><span class="line">$ nova image-list</span><br></pre></td></tr></table></figure>
<h3 id="部署计算节点：">部署计算节点：</h3><p>在oscompute1和oscompute2上安装：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">$ yum install -y openstack-nova-compute</span><br><span class="line">$ egrep -c <span class="string">'(vmx|svm)'</span> /proc/cpuinfo <span class="comment"># 确定CPU支持虚拟化，如果不支持硬件虚拟化，需要配置nova.conf中的libvirt virt_type为qemu</span></span><br><span class="line"><span class="comment"># openstack-config --set /etc/nova/nova.conf libvirt virt_type qemu</span></span><br></pre></td></tr></table></figure>
<p>编辑<code>/etc/nova/nova.conf</code>：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line">$ openstack-config --set /etc/nova/nova.conf database connection mysql://nova:e5a43<span class="built_in">fc</span>52ce0aaa1c7b4@<span class="number">10.0</span>.<span class="number">100.149</span>/nova</span><br><span class="line">$ openstack-config --set /etc/nova/nova.conf DEFAULT auth_strategy keystone</span><br><span class="line">$ openstack-config --set /etc/nova/nova.conf keystone_authtoken auth_uri http://<span class="number">10.0</span>.<span class="number">100.149</span>:<span class="number">5000</span></span><br><span class="line">$ openstack-config --set /etc/nova/nova.conf keystone_authtoken auth_host <span class="number">10.0</span>.<span class="number">100.149</span></span><br><span class="line">$ openstack-config --set /etc/nova/nova.conf keystone_authtoken auth_protocol http</span><br><span class="line">$ openstack-config --set /etc/nova/nova.conf keystone_authtoken auth_port <span class="number">35357</span></span><br><span class="line">$ openstack-config --set /etc/nova/nova.conf keystone_authtoken admin_user nova</span><br><span class="line">$ openstack-config --set /etc/nova/nova.conf keystone_authtoken admin_tenant_name service</span><br><span class="line">$ openstack-config --set /etc/nova/nova.conf keystone_authtoken admin_password fd53a063286a4f22ab7b</span><br><span class="line">$ openstack-config --set /etc/nova/nova.conf DEFAULT rpc_backend qpid</span><br><span class="line">$ openstack-config --set /etc/nova/nova.conf DEFAULT qpid_hostname <span class="number">10.0</span>.<span class="number">100.149</span></span><br></pre></td></tr></table></figure>
<p>在计算节点启用console：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">$ openstack-config --set /etc/nova/nova.conf DEFAULT my_ip <span class="number">192.168</span>.<span class="number">182.152</span>/<span class="number">153</span></span><br><span class="line">$ openstack-config --set /etc/nova/nova.conf DEFAULT vnc_enabled True</span><br><span class="line">$ openstack-config --set /etc/nova/nova.conf DEFAULT vncserver_listen <span class="number">0.0</span>.<span class="number">0.0</span></span><br><span class="line">$ openstack-config --set /etc/nova/nova.conf DEFAULT vncserver_proxyclient_address <span class="number">192.168</span>.<span class="number">182.152</span>/<span class="number">153</span></span><br><span class="line">$ openstack-config --set /etc/nova/nova.conf \</span><br><span class="line"> DEFAULT novncproxy_base_url http://<span class="number">192.168</span>.<span class="number">182.150</span>:<span class="number">6080</span>/vnc_auto.html</span><br></pre></td></tr></table></figure>
<p>配置Glance：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">$ openstack-config --set /etc/nova/nova.conf DEFAULT glance_host <span class="number">10.0</span>.<span class="number">100.145</span></span><br></pre></td></tr></table></figure>
<p>启动进程服务：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">$ service libvirtd start</span><br><span class="line">$ service messagebus start</span><br><span class="line">$ service openstack-nova-compute start</span><br><span class="line">$ chkconfig libvirtd on</span><br><span class="line">$ chkconfig messagebus on</span><br><span class="line">$ chkconfig openstack-nova-compute on</span><br></pre></td></tr></table></figure>
<h2 id="Network_Service:_Neutron_with_ML2">Network Service: Neutron with ML2</h2><p>网络部分关键概念包括：networks, subnets, routes，至少有1个“外部”网络。网络功能由plugin提供，如core plugin, security group plugin, FW, LB, …</p>
<h3 id="部署Neutron管理端">部署Neutron管理端</h3><h4 id="在oskeystone上准备Neutron数据库：">在oskeystone上准备Neutron数据库：</h4><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">$ mysql -u root -p</span><br><span class="line">mysql> CREATE DATABASE neutron;</span><br><span class="line">mysql> GRANT ALL PRIVILEGES ON neutron.* TO <span class="string">'neutron'</span>@<span class="string">'%'</span> \</span><br><span class="line">IDENTIFIED BY <span class="string">'fd535f9e2441a28a252d'</span>;</span><br></pre></td></tr></table></figure>
<p>创建Neutron的keystone账号，并注册Neutron服务：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">$ <span class="built_in">source</span> ~/adminrc</span><br><span class="line">$ keystone user-create --name neutron --pass <span class="number">033</span>d164bcc70e1244be7 --email neutron@tecstack.org</span><br><span class="line">$ keystone user-role-add --user neutron --tenant service --role admin</span><br><span class="line">$ keystone service-create --name neutron --type network --description <span class="string">"OpenStack Networking"</span></span><br><span class="line">$ keystone endpoint-create \</span><br><span class="line"> --service-id $(keystone service-list | awk <span class="string">'/ network / {print $2}'</span>) \</span><br><span class="line"> --publicurl http://<span class="number">192.168</span>.<span class="number">182.150</span>:<span class="number">9696</span> \</span><br><span class="line"> --adminurl http://<span class="number">10.0</span>.<span class="number">100.145</span>:<span class="number">9696</span> \</span><br><span class="line"> --internalurl http://<span class="number">10.0</span>.<span class="number">100.145</span>:<span class="number">9696</span></span><br></pre></td></tr></table></figure>
<h4 id="在oscontroller上部署配置Neutron服务端：">在oscontroller上部署配置Neutron服务端：</h4><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br></pre></td><td class="code"><pre><span class="line">$ yum install -y openstack-neutron openstack-neutron-ml2 python-neutronclient</span><br><span class="line">$ openstack-config --set /etc/neutron/neutron.conf database connection \</span><br><span class="line"> mysql://neutron:fd535f9e2441a28a252d@<span class="number">10.0</span>.<span class="number">100.149</span>/neutron</span><br><span class="line">$ openstack-config --set /etc/neutron/neutron.conf DEFAULT \</span><br><span class="line"> auth_strategy keystone</span><br><span class="line">$ openstack-config --set /etc/neutron/neutron.conf keystone_authtoken \</span><br><span class="line"> auth_uri http://<span class="number">10.0</span>.<span class="number">100.149</span>:<span class="number">5000</span></span><br><span class="line">$ openstack-config --set /etc/neutron/neutron.conf keystone_authtoken \</span><br><span class="line"> auth_host <span class="number">10.0</span>.<span class="number">100.149</span></span><br><span class="line">$ openstack-config --set /etc/neutron/neutron.conf keystone_authtoken \</span><br><span class="line"> auth_protocol http</span><br><span class="line">$ openstack-config --set /etc/neutron/neutron.conf keystone_authtoken \</span><br><span class="line"> auth_port <span class="number">35357</span></span><br><span class="line">$ openstack-config --set /etc/neutron/neutron.conf keystone_authtoken \</span><br><span class="line"> admin_tenant_name service</span><br><span class="line">$ openstack-config --set /etc/neutron/neutron.conf keystone_authtoken \</span><br><span class="line"> admin_user neutron</span><br><span class="line">$ openstack-config --set /etc/neutron/neutron.conf keystone_authtoken \</span><br><span class="line"> admin_password <span class="number">033</span>d164bcc70e1244be7</span><br><span class="line">$ openstack-config --set /etc/neutron/neutron.conf DEFAULT \</span><br><span class="line"> core_plugin ml2</span><br><span class="line">$ openstack-config --set /etc/neutron/neutron.conf DEFAULT \</span><br><span class="line"> service_plugins router</span><br></pre></td></tr></table></figure>
<p>配置Neutron使用MQ：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">$ openstack-config --set /etc/neutron/neutron.conf DEFAULT \</span><br><span class="line"> rpc_backend neutron.openstack.common.rpc.impl_qpid</span><br><span class="line">$ openstack-config --set /etc/neutron/neutron.conf DEFAULT \</span><br><span class="line"> qpid_hostname <span class="number">10.0</span>.<span class="number">100.149</span></span><br></pre></td></tr></table></figure>
<p>配置Neutron与计算节点的通知：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line">$ openstack-config --set /etc/neutron/neutron.conf DEFAULT \</span><br><span class="line"> notify_nova_on_port_status_changes True</span><br><span class="line">$ openstack-config --set /etc/neutron/neutron.conf DEFAULT \</span><br><span class="line"> notify_nova_on_port_data_changes True</span><br><span class="line">$ openstack-config --set /etc/neutron/neutron.conf DEFAULT \</span><br><span class="line"> nova_url http://<span class="number">10.0</span>.<span class="number">100.145</span>:<span class="number">8774</span>/v2</span><br><span class="line">$ openstack-config --set /etc/neutron/neutron.conf DEFAULT \</span><br><span class="line"> nova_admin_username nova</span><br><span class="line">$ openstack-config --set /etc/neutron/neutron.conf DEFAULT \</span><br><span class="line"> nova_admin_tenant_id $(keystone tenant-list | awk <span class="string">'/ service / { print $2 }'</span>)</span><br><span class="line">$ openstack-config --set /etc/neutron/neutron.conf DEFAULT \</span><br><span class="line"> nova_admin_password fd53a063286a4f22ab7b</span><br><span class="line">$ openstack-config --set /etc/neutron/neutron.conf DEFAULT \</span><br><span class="line"> nova_admin_auth_url http://<span class="number">10.0</span>.<span class="number">100.149</span>:<span class="number">35357</span>/v2.<span class="number">0</span></span><br></pre></td></tr></table></figure>
<p>创建neutron ml2 plugn配置链接：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">$ ln <span class="operator">-s</span> plugins/ml2/ml2_conf.ini /etc/neutron/plugin.ini</span><br></pre></td></tr></table></figure>
<p>配置ML2插件：ML2使用OVS处理网络流量。在oscontroller上配置：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line">$ openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2 \</span><br><span class="line"> <span class="built_in">type</span>_drivers gre</span><br><span class="line">$ openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2 \</span><br><span class="line"> tenant_network_types gre</span><br><span class="line">$ openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2 \</span><br><span class="line"> mechanism_drivers openvswitch</span><br><span class="line">$ openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2_<span class="built_in">type</span>_gre \</span><br><span class="line"> tunnel_id_ranges <span class="number">1</span>:<span class="number">1000</span></span><br><span class="line">$ openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini securitygroup \</span><br><span class="line"> firewall_driver neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver</span><br><span class="line">$ openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini securitygroup \</span><br><span class="line"> <span class="built_in">enable</span>_security_group True</span><br></pre></td></tr></table></figure>
<h4 id="在oscontroller上：配置Nova管理端使用Neutron管理网络：">在oscontroller上：配置Nova管理端使用Neutron管理网络：</h4><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line">$ openstack-config --set /etc/nova/nova.conf DEFAULT \</span><br><span class="line"> network_api_class nova.network.neutronv2.api.API</span><br><span class="line">$ openstack-config --set /etc/nova/nova.conf DEFAULT \</span><br><span class="line"> neutron_url http://<span class="number">10.0</span>.<span class="number">100.145</span>:<span class="number">9696</span></span><br><span class="line">$ openstack-config --set /etc/nova/nova.conf DEFAULT \</span><br><span class="line"> neutron_auth_strategy keystone</span><br><span class="line">$ openstack-config --set /etc/nova/nova.conf DEFAULT \</span><br><span class="line"> neutron_admin_tenant_name service</span><br><span class="line">$ openstack-config --set /etc/nova/nova.conf DEFAULT \</span><br><span class="line"> neutron_admin_username neutron</span><br><span class="line">$ openstack-config --set /etc/nova/nova.conf DEFAULT \</span><br><span class="line"> neutron_admin_password <span class="number">033</span>d164bcc70e1244be7</span><br><span class="line">$ openstack-config --set /etc/nova/nova.conf DEFAULT \</span><br><span class="line"> neutron_admin_auth_url http://<span class="number">10.0</span>.<span class="number">100.149</span>:<span class="number">35357</span>/v2.<span class="number">0</span></span><br><span class="line">$ openstack-config --set /etc/nova/nova.conf DEFAULT \</span><br><span class="line"> linuxnet_interface_driver nova.network.linux_net.LinuxOVSInterfaceDriver</span><br><span class="line">$ openstack-config --set /etc/nova/nova.conf DEFAULT \</span><br><span class="line"> firewall_driver nova.virt.firewall.NoopFirewallDriver <span class="comment"># 禁用nova-compute内置的FW功能。</span></span><br><span class="line">$ openstack-config --set /etc/nova/nova.conf DEFAULT \</span><br><span class="line"> security_group_api neutron</span><br></pre></td></tr></table></figure>
<p>在oscontroller上重启Nova服务：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">$ service openstack-nova-api restart</span><br><span class="line">$ service openstack-nova-scheduler restart</span><br><span class="line">$ service openstack-nova-conductor restart</span><br></pre></td></tr></table></figure>
<h4 id="在oscontroller上启动Neutron服务：">在oscontroller上启动Neutron服务：</h4><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">$ service neutron-server start</span><br><span class="line">$ chkconfig neutron-server on</span><br></pre></td></tr></table></figure>
<p>正常情况下，neutron-server在启动的时候会自动初始化数据库，如果出错，可以在oscontroller上手动建立数据库：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">$ openstack-config --set /etc/neutron/neutron.conf DEFAULT \</span><br><span class="line"> core_plugin neutron.plugins.ml2.plugin.Ml2Plugin</span><br><span class="line">$ openstack-config --set /etc/neutron/neutron.conf DEFAULT \</span><br><span class="line"> service_plugins neutron.services.l3_router.l3_router_plugin.L3RouterPlugin</span><br><span class="line">$ su <span class="operator">-s</span> /bin/sh -c <span class="string">"neutron-db-manage --config-file /etc/neutron/neutron.conf \</span><br><span class="line"> --config-file /etc/neutron/plugin.ini upgrade head"</span> neutron</span><br></pre></td></tr></table></figure>
<h3 id="在osnetwork上配置Linux转发及Neutron_Agent">在osnetwork上配置Linux转发及Neutron Agent</h3><h4 id="启用IP转发">启用IP转发</h4><p>编辑<code>/etc/sysctl.conf</code>：</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">net.ipv4.ip_forward=1 net.ipv4.conf.all.rp_filter=0 net.ipv4.conf.default.rp_filter=0 net.bridge.bridge-nf-call-arptables=1 net.bridge.bridge-nf-call-iptables=1 net.bridge.bridge-nf-call-ip6tables=1</span><br></pre></td></tr></table></figure>
<p>使生效：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">$ sysctl -p</span><br></pre></td></tr></table></figure>
<h4 id="部署配置Neutron及ML2,_OVS,_L3：">部署配置Neutron及ML2, OVS, L3：</h4><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br></pre></td><td class="code"><pre><span class="line">$ yum install -y openstack-neutron openstack-neutron-ml2 \</span><br><span class="line"> openstack-neutron-openvswitch</span><br><span class="line">$ openstack-config --set /etc/neutron/neutron.conf DEFAULT \</span><br><span class="line"> auth_strategy keystone</span><br><span class="line">$ openstack-config --set /etc/neutron/neutron.conf keystone_authtoken \</span><br><span class="line"> auth_uri http://<span class="number">10.0</span>.<span class="number">100.149</span>:<span class="number">5000</span></span><br><span class="line">$ openstack-config --set /etc/neutron/neutron.conf keystone_authtoken \</span><br><span class="line"> auth_host <span class="number">10.0</span>.<span class="number">100.149</span></span><br><span class="line">$ openstack-config --set /etc/neutron/neutron.conf keystone_authtoken \</span><br><span class="line"> auth_protocol http</span><br><span class="line">$ openstack-config --set /etc/neutron/neutron.conf keystone_authtoken \</span><br><span class="line"> auth_port <span class="number">35357</span></span><br><span class="line">$ openstack-config --set /etc/neutron/neutron.conf keystone_authtoken \</span><br><span class="line"> admin_tenant_name service</span><br><span class="line">$ openstack-config --set /etc/neutron/neutron.conf keystone_authtoken \</span><br><span class="line"> admin_user neutron</span><br><span class="line">$ openstack-config --set /etc/neutron/neutron.conf keystone_authtoken \</span><br><span class="line"> admin_password <span class="number">033</span>d164bcc70e1244be7</span><br><span class="line">$ openstack-config --set /etc/neutron/neutron.conf DEFAULT \</span><br><span class="line"> rpc_backend neutron.openstack.common.rpc.impl_qpid</span><br><span class="line">$ openstack-config --set /etc/neutron/neutron.conf DEFAULT \</span><br><span class="line"> qpid_hostname <span class="number">10.0</span>.<span class="number">100.149</span></span><br><span class="line">$ openstack-config --set /etc/neutron/neutron.conf DEFAULT \</span><br><span class="line"> core_plugin ml2</span><br><span class="line">$ openstack-config --set /etc/neutron/neutron.conf DEFAULT \</span><br><span class="line"> service_plugins router</span><br><span class="line">$ openstack-config --set /etc/neutron/l3_agent.ini DEFAULT \</span><br><span class="line"> interface_driver neutron.agent.linux.interface.OVSInterfaceDriver</span><br><span class="line">$ openstack-config --set /etc/neutron/l3_agent.ini DEFAULT \</span><br><span class="line"> use_namespaces True</span><br></pre></td></tr></table></figure>
<p>官方文档对于网络包大小解释的比较清楚，直接引用如下：</p>
<p>><br>Tunneling protocols such as generic routing encapsulation (GRE) include additional packet headers that increase overhead and decrease space available for the payload or user data. Without knowledge of the virtual network infrastructure, instances attempt to send packets using the default Ethernet maximum transmission unit <strong>(MTU) of 1500 bytes</strong>. Internet protocol (IP) networks contain the path MTU discovery (<strong>PMTUD</strong>) mechanism to detect end-to-end MTU and adjust packet size accordingly. However, some operating systems and networks block or otherwise <strong>lack support for PMTUD</strong> causing performance degradation or connectivity failure.<br>><br>Ideally, you can prevent these problems by enabling <strong>jumbo frames on the physical network</strong> that contains your tenant virtual networks. Jumbo frames support MTUs up to approximately <strong>9000 bytes</strong> which negates the impact of GRE overhead on virtual networks. However, many network devices lack support for jumbo frames and OpenStack administrators often lack control of network infrastructure. Given the latter complications, you can also prevent MTU problems by <strong>reducing the instance MTU to account for GRE overhead</strong>. Determining the proper MTU value often takes experimentation, but <strong>1454 bytes</strong> works in most environments. You can configure the DHCP server that assigns IP addresses to your instances to also adjust the MTU.<br>Some cloud images such as CirrOS ignore the DHCP MTU option.</p>
<p>大致意思是默认的MTU为1500字节，当加入了GRE等tunnel包头后会超过1500字节，IP协议理论上通过PMTUD可以自动调节传输路径上的MTU大小，但有些OS和网络设备因缺乏对PMTUD的支持会拦截此类包导致通断或性能问题。Jumbo frames支持最大9000字节的MTU，但是需要硬件网络设备的支持。<br>通常情况下，可以从虚拟机端设计MTU的大小，一般1454字节（减去了GRE的包头大小）在大部分环境下可行，MTU的设置可以通过DHCP服务器配置，但有些OS如CirrOS不支持DHCP配置MTU。</p>
<h4 id="配置osnetwork上的dhcp组件：">配置osnetwork上的dhcp组件：</h4><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">$ openstack-config --set /etc/neutron/dhcp_agent.ini DEFAULT \</span><br><span class="line"> interface_driver neutron.agent.linux.interface.OVSInterfaceDriver</span><br><span class="line">$ openstack-config --set /etc/neutron/dhcp_agent.ini DEFAULT \</span><br><span class="line"> dhcp_driver neutron.agent.linux.dhcp.Dnsmasq</span><br><span class="line">$ openstack-config --set /etc/neutron/dhcp_agent.ini DEFAULT \</span><br><span class="line"> use_namespaces True</span><br><span class="line">$ openstack-config --set /etc/neutron/dhcp_agent.ini DEFAULT \</span><br><span class="line"> dnsmasq_config_file /etc/neutron/dnsmasq-neutron.conf <span class="comment"># DHCP MTU</span></span><br></pre></td></tr></table></figure>
<p>新建编辑<code>/etc/neutron/dnsmasq-neutron.conf</code>：</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">dhcp-option-force=26,1454</span><br></pre></td></tr></table></figure>
<p>停止所有dnsmasq进程：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">killall dnsmasq</span><br></pre></td></tr></table></figure>
<h4 id="配置metadata_agent组件：">配置metadata agent组件：</h4><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line">$ openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT \</span><br><span class="line"> auth_url http://<span class="number">10.0</span>.<span class="number">100.149</span>:<span class="number">5000</span>/v2.<span class="number">0</span></span><br><span class="line">$ openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT \</span><br><span class="line"> auth_region regionOne</span><br><span class="line">$ openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT \</span><br><span class="line"> admin_tenant_name service</span><br><span class="line">$ openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT \</span><br><span class="line"> admin_user neutron</span><br><span class="line">$ openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT \</span><br><span class="line"> admin_password <span class="number">033</span>d164bcc70e1244be7</span><br><span class="line">$ openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT \</span><br><span class="line"> nova_metadata_ip <span class="number">10.0</span>.<span class="number">100.145</span></span><br><span class="line">$ openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT \</span><br><span class="line"> metadata_proxy_shared_secret <span class="number">69</span>e2f4db01fe4100bd32</span><br></pre></td></tr></table></figure>
<h4 id="配置oscontroller上的nova：">配置oscontroller上的nova：</h4><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">$ openstack-config --set /etc/nova/nova.conf DEFAULT \</span><br><span class="line"> service_neutron_metadata_proxy <span class="literal">true</span></span><br><span class="line">$ openstack-config --set /etc/nova/nova.conf DEFAULT \</span><br><span class="line"> neutron_metadata_proxy_shared_secret <span class="number">69</span>e2f4db01fe4100bd32</span><br><span class="line">$ service openstack-nova-api restart</span><br></pre></td></tr></table></figure>
<h4 id="配置osnetwork上的ML2,_OVS插件：">配置osnetwork上的ML2, OVS插件：</h4><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line">$ openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2 \</span><br><span class="line"> <span class="built_in">type</span>_drivers gre</span><br><span class="line">$ openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2 \</span><br><span class="line"> tenant_network_types gre</span><br><span class="line">$ openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2 \</span><br><span class="line"> mechanism_drivers openvswitch</span><br><span class="line">$ openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2_<span class="built_in">type</span>_gre \</span><br><span class="line"> tunnel_id_ranges <span class="number">1</span>:<span class="number">1000</span></span><br><span class="line">$ openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ovs \</span><br><span class="line"> <span class="built_in">local</span>_ip <span class="number">10.0</span>.<span class="number">100.146</span> <span class="comment"># osnetwork的内部ip</span></span><br><span class="line">$ openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ovs \</span><br><span class="line"> tunnel_<span class="built_in">type</span> gre</span><br><span class="line">$ openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ovs \</span><br><span class="line"> <span class="built_in">enable</span>_tunneling True</span><br><span class="line">$ openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini securitygroup \</span><br><span class="line"> firewall_driver neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver</span><br><span class="line">$ openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini securitygroup \</span><br><span class="line"> <span class="built_in">enable</span>_security_group True</span><br><span class="line">$ ln <span class="operator">-s</span> plugins/ml2/ml2_conf.ini /etc/neutron/plugin.ini</span><br></pre></td></tr></table></figure>
<p>修复打包的bug，让neutron-openvswitch-agent使用<code>plugin.ini</code>配置文件：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">$ cp /etc/init.d/neutron-openvswitch-agent /etc/init.d/neutron-openvswitch-agent.orig</span><br><span class="line">$ sed -i <span class="string">'s,plugins/openvswitch/ovs_neutron_plugin.ini,plugin.ini,g'</span> /etc/init.d/neutron-openvswitch-agent</span><br></pre></td></tr></table></figure>
<h4 id="启动OVS，添加br-int,_br-ex为内部和外部bridge：">启动OVS，添加br-int, br-ex为内部和外部bridge：</h4><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">$ service openvswitch start</span><br><span class="line">$ chkconfig openvswitch on</span><br><span class="line">$ ovs-vsctl add-br br-int</span><br><span class="line">$ ovs-vsctl add-br br-ex</span><br><span class="line">$ ovs-vsctl add-port br-ex eth1</span><br></pre></td></tr></table></figure>
<p>官网提示内部虚拟机和外部网络之间的吞吐可以通过关闭GRO提升，临时关闭方法：<code>ethtool -K INTERFACE_NAME gro off</code></p>
<p>Ops. 发现添加port后网络连接断开了。Google下，Linux bridge kernel module一直要求一旦某个interface加入到了在某个bridge下时，不能再单独配置IP被访问，如果要访问host，则可以把这个IP（或其他可访问的IP）分配给所在的bridge。可以参考<a href="http://www.gossamer-threads.com/lists/openstack/operators/34627" title="Problem configuring openvswitch br-ex" target="_blank" rel="external">这个链接</a>。对应的解决方案就是把原来的public ip从eth1删除，配置到br-exs上：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">$ ip addr del <span class="number">192.168</span>.<span class="number">182.151</span>/<span class="number">25</span> dev eth1</span><br><span class="line">$ ip addr add <span class="number">192.168</span>.<span class="number">182.151</span>/<span class="number">25</span> dev br-ex</span><br><span class="line">$ ip route del default via <span class="number">192.168</span>.<span class="number">182.254</span> dev eth1 <span class="comment"># 改变默认路由</span></span><br><span class="line">$ ip route add default via <span class="number">192.168</span>.<span class="number">182.254</span> dev br-ex</span><br></pre></td></tr></table></figure>
<p>这样就能再次访问osnetwork节点，看<a href="https://www.rdoproject.org/Neutron_with_existing_external_network" title="Neutron_with_existing_external_network" target="_blank" rel="external">RDO的文档</a>里也有提及，建议把br-ex写到配置文件并删除eth1配置文件里的IP信息后重启网络：</p>
<p>新增<code>/etc/sysconfig/network-scripts/ifcfg-br-ex</code>:</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">DEVICE=br-ex DEVICETYPE=ovs TYPE=OVSBridge BOOTPROTO=static IPADDR=192.168.182.151 NETMASK=255.255.255.128 # your netmask GATEWAY=192.168.182.254 # your gateway #DNS1=192.168.122.1 # your nameserver ONBOOT=yes</span><br></pre></td></tr></table></figure>
<p>修改<code>/etc/sysconfig/network-scripts/ifcfg-eth1</code>:</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">DEVICE=eth1 HWADDR=00:50:56:b8:dd:e0 # your hwaddr TYPE=OVSPort DEVICETYPE=ovs OVS_BRIDGE=br-ex ONBOOT=yes #NM_CONTROLLED=yes #BOOTPROTO=none #IPADDR=192.168.182.151 #NETMASK=255.255.255.128 #GATEWAY=192.168.182.254 #IPV6INIT=no #USERCTL=no</span><br></pre></td></tr></table></figure>
<p>修改完之后，执行：<code>service network restart</code>，系统会自动把默认路由指向br-ex。</p>
<h4 id="启动Neutron_Agent服务：">启动Neutron Agent服务：</h4><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">$ <span class="keyword">for</span> svc <span class="keyword">in</span> neutron-{openvswitch,l3,dhcp,metadata}-agent; <span class="keyword">do</span> \</span><br><span class="line"> service <span class="variable">$svc</span> start; chkconfig <span class="variable">$svc</span> on;<span class="keyword">done</span>;</span><br><span class="line">$ <span class="keyword">for</span> svc <span class="keyword">in</span> neutron-{openvswitch,l3,dhcp,metadata}-agent; <span class="keyword">do</span> \</span><br><span class="line"> service <span class="variable">$svc</span> status; chkconfig | grep <span class="variable">$svc</span>;<span class="keyword">done</span>;</span><br></pre></td></tr></table></figure>
<h3 id="在oscompute1和oscompute2上配置Linux转发及Neutron_Agent">在oscompute1和oscompute2上配置Linux转发及Neutron Agent</h3><h4 id="启用Linux转发">启用Linux转发</h4><p>编辑<code>/etc/sysctl.conf</code>：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">net.ipv4.conf.all.rp_filter=<span class="number">0</span></span><br><span class="line">net.ipv4.conf.default.rp_filter=<span class="number">0</span></span><br><span class="line">net.bridge.bridge-nf-call-arptables=<span class="number">1</span></span><br><span class="line">net.bridge.bridge-nf-call-iptables=<span class="number">1</span></span><br><span class="line">net.bridge.bridge-nf-call-ip6tables=<span class="number">1</span></span><br></pre></td></tr></table></figure>
<p>使生效：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">$ sysctl -p</span><br></pre></td></tr></table></figure>
<h4 id="安装部署Neutron_ML2和OVS">安装部署Neutron ML2和OVS</h4><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br></pre></td><td class="code"><pre><span class="line">$ yum install -y openstack-neutron-ml2 openstack-neutron-openvswitch</span><br><span class="line">$ openstack-config --set /etc/neutron/neutron.conf DEFAULT \</span><br><span class="line"> auth_strategy keystone</span><br><span class="line">$ openstack-config --set /etc/neutron/neutron.conf keystone_authtoken \</span><br><span class="line"> auth_uri http://<span class="number">10.0</span>.<span class="number">100.149</span>:<span class="number">5000</span></span><br><span class="line">$ openstack-config --set /etc/neutron/neutron.conf keystone_authtoken \</span><br><span class="line"> auth_host <span class="number">10.0</span>.<span class="number">100.149</span></span><br><span class="line">$ openstack-config --set /etc/neutron/neutron.conf keystone_authtoken \</span><br><span class="line"> auth_protocol http</span><br><span class="line">$ openstack-config --set /etc/neutron/neutron.conf keystone_authtoken \</span><br><span class="line"> auth_port <span class="number">35357</span></span><br><span class="line">$ openstack-config --set /etc/neutron/neutron.conf keystone_authtoken \</span><br><span class="line"> admin_tenant_name service</span><br><span class="line">$ openstack-config --set /etc/neutron/neutron.conf keystone_authtoken \</span><br><span class="line"> admin_user neutron</span><br><span class="line">$ openstack-config --set /etc/neutron/neutron.conf keystone_authtoken \</span><br><span class="line"> admin_password <span class="number">033</span>d164bcc70e1244be7</span><br><span class="line">$ openstack-config --set /etc/neutron/neutron.conf DEFAULT \</span><br><span class="line"> rpc_backend neutron.openstack.common.rpc.impl_qpid</span><br><span class="line">$ openstack-config --set /etc/neutron/neutron.conf DEFAULT \</span><br><span class="line"> qpid_hostname <span class="number">10.0</span>.<span class="number">100.149</span></span><br><span class="line">$ openstack-config --set /etc/neutron/neutron.conf DEFAULT \</span><br><span class="line"> core_plugin ml2</span><br><span class="line">$ openstack-config --set /etc/neutron/neutron.conf DEFAULT \</span><br><span class="line"> service_plugins router</span><br></pre></td></tr></table></figure>
<h4 id="配置ML2">配置ML2</h4><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line">$ openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2 \</span><br><span class="line"> <span class="built_in">type</span>_drivers gre</span><br><span class="line">$ openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2 \</span><br><span class="line"> tenant_network_types gre</span><br><span class="line">$ openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2 \</span><br><span class="line"> mechanism_drivers openvswitch</span><br><span class="line">$ openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2_<span class="built_in">type</span>_gre \</span><br><span class="line"> tunnel_id_ranges <span class="number">1</span>:<span class="number">1000</span></span><br><span class="line">$ openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ovs \</span><br><span class="line"> <span class="built_in">local</span>_ip <span class="number">10.0</span>.<span class="number">100.147</span>/<span class="number">148</span></span><br><span class="line">$ openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ovs \</span><br><span class="line"> tunnel_<span class="built_in">type</span> gre</span><br><span class="line">$ openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ovs \</span><br><span class="line"> <span class="built_in">enable</span>_tunneling True</span><br><span class="line">$ openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini securitygroup \</span><br><span class="line"> firewall_driver neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver</span><br><span class="line">$ openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini securitygroup \</span><br><span class="line"> <span class="built_in">enable</span>_security_group True</span><br><span class="line">$ ln <span class="operator">-s</span> plugins/ml2/ml2_conf.ini /etc/neutron/plugin.ini</span><br></pre></td></tr></table></figure>
<h4 id="配置OVS">配置OVS</h4><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">$ service openvswitch start</span><br><span class="line">$ chkconfig openvswitch on</span><br><span class="line">$ ovs-vsctl add-br br-int</span><br></pre></td></tr></table></figure>
<h4 id="配置Nova-Compute">配置Nova-Compute</h4><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line">$ openstack-config --set /etc/nova/nova.conf DEFAULT \</span><br><span class="line"> network_api_class nova.network.neutronv2.api.API</span><br><span class="line">$ openstack-config --set /etc/nova/nova.conf DEFAULT \</span><br><span class="line"> neutron_url http://<span class="number">10.0</span>.<span class="number">100.145</span>:<span class="number">9696</span></span><br><span class="line">$ openstack-config --set /etc/nova/nova.conf DEFAULT \</span><br><span class="line"> neutron_auth_strategy keystone</span><br><span class="line">$ openstack-config --set /etc/nova/nova.conf DEFAULT \</span><br><span class="line"> neutron_admin_tenant_name service</span><br><span class="line">$ openstack-config --set /etc/nova/nova.conf DEFAULT \</span><br><span class="line"> neutron_admin_username neutron</span><br><span class="line">$ openstack-config --set /etc/nova/nova.conf DEFAULT \</span><br><span class="line"> neutron_admin_password <span class="number">033</span>d164bcc70e1244be7</span><br><span class="line">$ openstack-config --set /etc/nova/nova.conf DEFAULT \</span><br><span class="line"> neutron_admin_auth_url http://<span class="number">10.0</span>.<span class="number">100.149</span>:<span class="number">35357</span>/v2.<span class="number">0</span></span><br><span class="line">$ openstack-config --set /etc/nova/nova.conf DEFAULT \</span><br><span class="line"> linuxnet_interface_driver nova.network.linux_net.LinuxOVSInterfaceDriver</span><br><span class="line">$ openstack-config --set /etc/nova/nova.conf DEFAULT \</span><br><span class="line"> firewall_driver nova.virt.firewall.NoopFirewallDriver</span><br><span class="line">$ openstack-config --set /etc/nova/nova.conf DEFAULT \</span><br><span class="line"> security_group_api neutron</span><br></pre></td></tr></table></figure>
<h4 id="修改neutron-openvswitch-agent使用plugin-ini配置：">修改<code>neutron-openvswitch-agent</code>使用plugin.ini配置：</h4><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">$ cp /etc/init.d/neutron-openvswitch-agent /etc/init.d/neutron-openvswitch-agent.orig</span><br><span class="line">$ sed -i <span class="string">'s,plugins/openvswitch/ovs_neutron_plugin.ini,plugin.ini,g'</span> /etc/init.d/neutron-openvswitch-agent</span><br></pre></td></tr></table></figure>
<h4 id="启动nova-computer,neutron-openvswitch-agent进程">启动<code>nova-computer</code>,<code>neutron-openvswitch-agent</code>进程</h4><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">$ service openstack-nova-compute restart</span><br><span class="line">$ service neutron-openvswitch-agent start</span><br><span class="line">$ chkconfig neutron-openvswitch-agent on</span><br></pre></td></tr></table></figure>
<h3 id="初始化虚拟机的网络">初始化虚拟机的网络</h3><p>启动第一个虚拟机实例之前，必须先创建给虚拟机连接的虚拟网络，包括外部网络和租户网络。<br>在oscontroller上，通过admin账户创建外部网络，共享给所有租户；通过demo账户创建租户私有网络，再创建路由实现与外部网络连通。</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><span class="line">$ <span class="built_in">source</span> adminrc</span><br><span class="line">$ neutron net-create ext-net --shared --router:external=True</span><br><span class="line">$ neutron subnet-create ext-net --name ext-subnet \</span><br><span class="line"> --allocation-pool start=<span class="number">192.168</span>.<span class="number">182.200</span>,end=<span class="number">192.168</span>.<span class="number">182.220</span> \</span><br><span class="line"> --disable-dhcp --gateway <span class="number">192.168</span>.<span class="number">182.254</span> <span class="number">192.168</span>.<span class="number">182.128</span>/<span class="number">25</span></span><br><span class="line">$ <span class="built_in">source</span> demorc</span><br><span class="line">$ neutron net-create demo-net</span><br><span class="line">$ neutron subnet-create demo-net --name demo-subnet \</span><br><span class="line"> --gateway <span class="number">192.168</span>.<span class="number">1.1</span> <span class="number">192.168</span>.<span class="number">1.0</span>/<span class="number">24</span></span><br><span class="line">$ neutron router-create demo-router</span><br><span class="line">$ neutron router-interface-add demo-router demo-subnet</span><br><span class="line">$ ip netns list <span class="comment"># 能看到新增了一个`qrouter-4bd0dc16-ae05-4304-9483-41c0f9a67775`的namespace</span></span><br><span class="line">$ ip netns <span class="built_in">exec</span> qrouter-<span class="number">4</span>bd0dc16-ae05-<span class="number">4304</span>-<span class="number">9483</span>-<span class="number">41</span>c0f9a67775 ip addr <span class="comment"># 能看到新增了`qr-af814b8f-c9`这个port。</span></span><br><span class="line">$ ovs-vsctl list-ports br-int <span class="comment"># 能看到增加了`qr-af814b8f-c9`这个port。</span></span><br><span class="line">$ neutron router-gateway-set demo-router ext-net</span><br><span class="line">$ ip netns <span class="built_in">exec</span> qrouter-<span class="number">4</span>bd0dc16-ae05-<span class="number">4304</span>-<span class="number">9483</span>-<span class="number">41</span>c0f9a67775 ip addr <span class="comment"># 能看到新增了`qg-269d911c-25`这个port，且它的ip就是ext-subnet里的floating IP。</span></span><br><span class="line">$ ovs-vsctl list-ports br-ex <span class="comment"># 能看到增加了`qg-269d911c-25`这个port。</span></span><br></pre></td></tr></table></figure>
<p>检验连通性，在osnetwork上ping所创建的router（第一个floating IP地址）：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">$ ping -c <span class="number">4</span> <span class="number">192.168</span>.<span class="number">182.200</span></span><br><span class="line">$ ip netns <span class="built_in">exec</span> qrouter-<span class="number">4</span>bd0dc16-ae05-<span class="number">4304</span>-<span class="number">9483</span>-<span class="number">41</span>c0f9a67775 ping <span class="number">192.168</span>.<span class="number">182.151</span> <span class="comment"># 从router发起ping到外部的网络，首先可以ping通所在物理机osnetwork的端口</span></span><br><span class="line">$ ip netns <span class="built_in">exec</span> qrouter-<span class="number">4</span>bd0dc16-ae05-<span class="number">4304</span>-<span class="number">9483</span>-<span class="number">41</span>c0f9a67775 ping <span class="number">192.168</span>.<span class="number">182.100</span> <span class="comment"># 可以ping通网关和其他子网下的Hosts</span></span><br></pre></td></tr></table></figure>
<p>从这里可以理解，底层是通过Linux kernel的net namespace实现了网络隔离，即多租户能力。其实neutron给router添加interface和设定gateway的的时候，其实是在底层创建了一个net namespace并把租户子网的网关和从外部网络获取的floatingIP添加到了router中作为port，从而实现租户内私有网络和外部公共网络打通。</p>
<p>如果发现从router里可以和osnetwork互相ping通，但是无法和同一子网下的其他主机ping通，很大程度上是二层交换问题，可以通过在osnetwork和router上同时抓包，分析arp请求；如果发现arp无返回，则说明二层osnetwork之外的交换机配置有问题，比如在vmware虚拟化环境下安装openstack，则需要把vmware的虚拟交换机开启<code>混杂模式</code>。用到的调测命令如下，更多可以参考<a href="http://docs.openstack.org/openstack-ops/content/network_troubleshooting.html" title="Network troubleshooting" target="_blank" rel="external">网络问题排查方法</a>：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">$ ovs-appctl fdb/show br-ex <span class="comment"># 查看br-ex的mac表，可以查看mac地址学习状态</span></span><br><span class="line">$ ip netns <span class="built_in">exec</span> qrouter-<span class="number">4</span>bd0dc16-ae05-<span class="number">4304</span>-<span class="number">9483</span>-<span class="number">41</span>c0f9a67775 tcpdump -nn arp <span class="comment"># 在router上抓arp包</span></span><br><span class="line">$ ip netns <span class="built_in">exec</span> qrouter-<span class="number">4</span>bd0dc16-ae05-<span class="number">4304</span>-<span class="number">9483</span>-<span class="number">41</span>c0f9a67775 ip route <span class="comment"># 在router上执行路由命令</span></span><br><span class="line">$ ovs-vsctl list Interface gre-<span class="number">0</span>a006491 <span class="comment"># 查看 ovs交换机上某个端口的详细状态</span></span><br><span class="line">$ neutron agent-list <span class="comment"># 查看neutron上网络组件状态</span></span><br><span class="line">$ ovs-dpctl show <span class="comment">#查看ovs端口配置</span></span><br></pre></td></tr></table></figure>
<h3 id="虚拟机网络流量通道">虚拟机网络流量通道</h3><p><img src="http://docs.openstack.org/admin-guide-cloud/content/figures/14/a/a/common/figures/under-the-hood-scenario-1-ovs-compute.png" alt="ovs networking"><br>在创建虚拟机之后（可以部署完dashboard之后做更方便），可以<code>ip addr</code>看到oscomputeX上新增了4个设备，分别以<code>tap, qbr, qvb, qvo</code>开头，osnetwork上也也新增了<code>qdhcp</code>开头的namespace。oscomputeX上新增的tap设备用于连接虚拟机端口，qbr是一个普通linux bridge，qvb和qvo是veth pair，连接了qbr和br-int。不把tap直接挂到br-int的理由就是目前Openstack security group的实现方式是基于iptables对tap设备的控制，而openvswitch不支持iptables控制挂在它上面的tap设备。<br>官方引文如下，详细可以参考<a href="http://docs.openstack.org/admin-guide-cloud/content/under_the_hood_openvswitch.html" title="Networking Openvswitch on Openstack" target="_blank" rel="external">Networking Openvswitch on Openstack</a>:</p>
<blockquote>
<p>Security groups: iptables and Linux bridges<br>Ideally, the TAP device vnet0 would be connected directly to the integration bridge, br-int. Unfortunately, this isn’t possible because of how OpenStack security groups are currently implemented. OpenStack uses iptables rules on the TAP devices such as vnet0 to implement security groups, and Open vSwitch is not compatible with iptables rules that are applied directly on TAP devices that are connected to an Open vSwitch port.<br>Networking uses an extra Linux bridge and a veth pair as a workaround for this issue. Instead of connecting vnet0 to an Open vSwitch bridge, it is connected to a Linux bridge, qbrXXX. This bridge is connected to the integration bridge, br-int, through the (qvbXXX, qvoXXX) veth pair.</p>
</blockquote>
<h3 id="删除或变更GRE通道">删除或变更GRE通道</h3><p>当使用GRE tunnel时，所有network节点和compute节点之间是建立了full-mesh结构的点对点tunnel，当需要变动ip地址的时候（比如之前配置的时候把network节点上的local_ipx写错），需要手动把neutron数据库ip_addressb表中的错误ip删除，修改配置文件后重启network和计算节点傻姑上的<code>neutron-openvswitch-agent</code>服务。</p>
<h2 id="Dashboard">Dashboard</h2><p>Horizon是一个基于python2.6的Django app。需要安装Keystone和Nova服务之后才能使用dashboard。<br>在oskeystone上部署dashboard：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">$ yum install -y memcached python-memcached mod_wsgi openstack-dashboard</span><br></pre></td></tr></table></figure>
<p>根据<code>/etc/sysconfig/memcached</code>内容修改<code>/etc/openstack-dashboard/local_settings</code>：</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">CACHES = { 	'default': { 	'BACKEND' : 'django.core.cache.backends.memcached.MemcachedCache', 	'LOCATION' : '127.0.0.1:11211' 	} } ALLOWED_HOSTS = ['localhost', '192.168.182.154'] OPENSTACK_HOST = "10.0.100.149" # hostname of Identity Service</span><br></pre></td></tr></table></figure>
<p>启动服务：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">$ setsebool -P httpd_can_network_connect on</span><br><span class="line">$ getsebool httpd_can_network_connect <span class="comment"># check selinux status</span></span><br><span class="line">$ service httpd start</span><br><span class="line">$ service httpd start</span><br><span class="line">$ chkconfig httpd on</span><br><span class="line">$ chkconfig memcached on</span><br></pre></td></tr></table></figure>
<p>访问<code>http://192.168.182.149/dashboard</code>，Ops…看apache日志<code>tailf /var/log/httpd/error_log</code>：</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">... File "/usr/lib/python2.6/site-packages/keystoneclient/__init__.py", line 43, in <module> __version__ = pbr.version.VersionInfo('python-keystoneclient').version_string() ... File "/usr/lib/python2.6/site-packages/pbr/packaging.py", line 864, in get_version raise Exception("Versioning for this project requires either an sdist" Exception: Versioning for this project requires either an sdist tarball, or access to an upstream git repository. Are you sure that git is installed? ...</span><br></pre></td></tr></table></figure>
<p>看起来跟<code>python-keystoneclient</code>的加载有关，于是关闭apache，用本地命令执行方式启动horizon的Django应用，如果要切换回Apache等启动方式，记得删除<code>/tmp</code>下自动生成的<code>SECRET_KEY</code>（在<code>/etc/openstack-dashboard/local_settings</code>中配置）：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">$ <span class="built_in">cd</span> /usr/share/openstack-dashboard</span><br><span class="line">$ python manage.py runserver <span class="number">0.0</span>.<span class="number">0.0</span>:<span class="number">80</span></span><br></pre></td></tr></table></figure>
<p>发现错误信息一致，甚至不加参数执行<code>python manage.py</code>也会出现同样错误。Google到<a href="https://github.com/rackspace/pyrax/issues/450" target="_blank" rel="external">一个同样的问题</a>，更新<code>distribute</code>可解决，具体原因还没找到，初步估计跟打包方式及pbr有关：</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">$ pip install --upgrade distribute</span><br></pre></td></tr></table></figure>
<p>另，如果出现<code>ERROR: [Errno 113] No route to host</code>，可以检查Nova服务端的Host上iptables是否设置正确。</p>
<h2 id="Block_Service:_Cinder">Block Service: Cinder</h2><ul>
<li><code>cinder-api</code>: API</li>
<li><code>cinder-volume</code>: 通过driver与块设备交互</li>
<li><code>cinder-scheduler</code>: 选择最优块来创建卷</li>
</ul>
<h3 id="在oscontroller上安装Cinder_Service">在oscontroller上安装Cinder Service</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">$ yum install -y openstack-cinder</span><br><span class="line">$ openstack-config --set /etc/cinder/cinder.conf \</span><br><span class="line"> database connection mysql://cinder:<span class="number">791</span>ce55fa6888c065bf3@<span class="number">10.0</span>.<span class="number">100.149</span>/cinder</span><br></pre></td></tr></table></figure>
<p>在oskeystone上创建cinder的数据库：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">$ mysql -u root -pa904019ba8cc0b14bef2</span><br><span class="line">mysql> CREATE DATABASE cinder;</span><br><span class="line">mysql> GRANT ALL PRIVILEGES ON cinder.* TO <span class="string">'cinder'</span>@<span class="string">'%'</span> \</span><br><span class="line"> IDENTIFIED BY <span class="string">'791ce55fa6888c065bf3'</span>;</span><br></pre></td></tr></table></figure>
<p>同步数据库结构：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">$ su <span class="operator">-s</span> /bin/sh -c <span class="string">"cinder-manage db sync"</span> cinder</span><br></pre></td></tr></table></figure>
<p>创建cinder的keystone账户，配置<code>/etc/cinder/cinder.conf</code>：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><span class="line">$ keystone user-create --name=cinder --pass=<span class="number">89</span>edec5c2869dce44c61 --email=cinder@tecstack.org</span><br><span class="line">$ keystone user-role-add --user=cinder --tenant=service --role=admin</span><br><span class="line">$ openstack-config --set /etc/cinder/cinder.conf DEFAULT \</span><br><span class="line"> auth_strategy keystone</span><br><span class="line">$ openstack-config --set /etc/cinder/cinder.conf keystone_authtoken \</span><br><span class="line"> auth_uri http://<span class="number">10.0</span>.<span class="number">100.149</span>:<span class="number">5000</span></span><br><span class="line">$ openstack-config --set /etc/cinder/cinder.conf keystone_authtoken \</span><br><span class="line"> auth_host <span class="number">10.0</span>.<span class="number">100.149</span></span><br><span class="line">$ openstack-config --set /etc/cinder/cinder.conf keystone_authtoken \</span><br><span class="line"> auth_protocol http</span><br><span class="line">$ openstack-config --set /etc/cinder/cinder.conf keystone_authtoken \</span><br><span class="line"> auth_port <span class="number">35357</span></span><br><span class="line">$ openstack-config --set /etc/cinder/cinder.conf keystone_authtoken \</span><br><span class="line"> admin_user cinder</span><br><span class="line">$ openstack-config --set /etc/cinder/cinder.conf keystone_authtoken \</span><br><span class="line"> admin_tenant_name service</span><br><span class="line">$ openstack-config --set /etc/cinder/cinder.conf keystone_authtoken \</span><br><span class="line"> admin_password <span class="number">89</span>edec5c2869dce44c61</span><br></pre></td></tr></table></figure>
<p>配置MQ，在Keystone上注册Cinder Service：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><span class="line">$ openstack-config --set /etc/cinder/cinder.conf \</span><br><span class="line"> DEFAULT rpc_backend qpid</span><br><span class="line">$ openstack-config --set /etc/cinder/cinder.conf \</span><br><span class="line"> DEFAULT qpid_hostname <span class="number">10.0</span>.<span class="number">100.149</span></span><br><span class="line"><span class="comment"># API Version 1</span></span><br><span class="line">$ keystone service-create --name=cinder --type=volume --description=<span class="string">"OpenStack Block Storage"</span></span><br><span class="line">$ keystone endpoint-create \</span><br><span class="line"> --service-id=$(keystone service-list | awk <span class="string">'/ volume / {print $2}'</span>) \</span><br><span class="line"> --publicurl=http://<span class="number">192.168</span>.<span class="number">182.150</span>:<span class="number">8776</span>/v1/%\(tenant_id\)s \</span><br><span class="line"> --internalurl=http://<span class="number">10.0</span>.<span class="number">100.145</span>:<span class="number">8776</span>/v1/%\(tenant_id\)s \</span><br><span class="line"> --adminurl=http://<span class="number">10.0</span>.<span class="number">100.145</span>:<span class="number">8776</span>/v1/%\(tenant_id\)s</span><br><span class="line"><span class="comment"># API Version 2</span></span><br><span class="line">$ keystone service-create --name=cinderv2 --type=volumev2 --description=<span class="string">"OpenStack Block Storage v2"</span></span><br><span class="line">$ keystone endpoint-create \</span><br><span class="line"> --service-id=$(keystone service-list | awk <span class="string">'/ volumev2 / {print $2}'</span>) \</span><br><span class="line"> --publicurl=http://<span class="number">192.168</span>.<span class="number">182.150</span>:<span class="number">8776</span>/v2/%\(tenant_id\)s \</span><br><span class="line"> --internalurl=http://<span class="number">10.0</span>.<span class="number">100.145</span>:<span class="number">8776</span>/v2/%\(tenant_id\)s \</span><br><span class="line"> --adminurl=http://<span class="number">10.0</span>.<span class="number">100.145</span>:<span class="number">8776</span>/v2/%\(tenant_id\)s</span><br></pre></td></tr></table></figure>
<p>启动服务：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">$ service openstack-cinder-api start</span><br><span class="line">$ service openstack-cinder-scheduler start</span><br><span class="line">$ chkconfig openstack-cinder-api on</span><br><span class="line">$ chkconfig openstack-cinder-scheduler on</span><br></pre></td></tr></table></figure>
<h3 id="在osceph0上部署一个临时存储节点">在osceph0上部署一个临时存储节点</h3><p>安装配置cinder：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br></pre></td><td class="code"><pre><span class="line">$ yum install -y openstack-cinder scsi-target-utils</span><br><span class="line">$ openstack-config --set /etc/cinder/cinder.conf DEFAULT \</span><br><span class="line"> auth_strategy keystone</span><br><span class="line">$ openstack-config --set /etc/cinder/cinder.conf keystone_authtoken \</span><br><span class="line"> auth_uri http://<span class="number">10.0</span>.<span class="number">100.149</span>:<span class="number">5000</span></span><br><span class="line">$ openstack-config --set /etc/cinder/cinder.conf keystone_authtoken \</span><br><span class="line"> auth_host <span class="number">10.0</span>.<span class="number">100.149</span></span><br><span class="line">$ openstack-config --set /etc/cinder/cinder.conf keystone_authtoken \</span><br><span class="line"> auth_protocol http</span><br><span class="line">$ openstack-config --set /etc/cinder/cinder.conf keystone_authtoken \</span><br><span class="line"> auth_port <span class="number">35357</span></span><br><span class="line">$ openstack-config --set /etc/cinder/cinder.conf keystone_authtoken \</span><br><span class="line"> admin_user cinder</span><br><span class="line">$ openstack-config --set /etc/cinder/cinder.conf keystone_authtoken \</span><br><span class="line"> admin_tenant_name service</span><br><span class="line">$ openstack-config --set /etc/cinder/cinder.conf keystone_authtoken \</span><br><span class="line"> admin_password <span class="number">89</span>edec5c2869dce44c61</span><br><span class="line">$ openstack-config --set /etc/cinder/cinder.conf \</span><br><span class="line"> DEFAULT rpc_backend qpid</span><br><span class="line">$ openstack-config --set /etc/cinder/cinder.conf \</span><br><span class="line"> DEFAULT qpid_hostname <span class="number">10.0</span>.<span class="number">100.149</span></span><br><span class="line">$ openstack-config --set /etc/cinder/cinder.conf \</span><br><span class="line"> database connection mysql://cinder:<span class="number">791</span>ce55fa6888c065bf3@<span class="number">10.0</span>.<span class="number">100.149</span>/cinder</span><br><span class="line">$ openstack-config --set /etc/cinder/cinder.conf \</span><br><span class="line"> DEFAULT my_ip <span class="number">10.0</span>.<span class="number">100.142</span></span><br><span class="line">$ openstack-config --set /etc/cinder/cinder.conf \</span><br><span class="line"> DEFAULT glance_host <span class="number">10.0</span>.<span class="number">100.145</span> <span class="comment"># cinder使用glance images创建可启动卷</span></span><br><span class="line">$ openstack-config --set /etc/cinder/cinder.conf \</span><br><span class="line"> DEFAULT iscsi_helper tgtadm <span class="comment"># 使用tgtadm iSCSI service</span></span><br></pre></td></tr></table></figure>
<p>配置<code>/etc/tgt/targets.conf</code>，让<code>iSCSI target service</code>发现卷：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">include /etc/cinder/volumes/*</span><br></pre></td></tr></table></figure>
<p>模拟块设备:</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">$ dd <span class="keyword">if</span>=/dev/zero of=/opt/volumes/cinderVolumes bs=<span class="number">1</span>M count=<span class="number">10240</span></span><br><span class="line">$ losetup <span class="operator">-f</span> cinderVolumes <span class="comment"># 自动找到一个未使用的loop设备</span></span><br><span class="line">$ losetup <span class="operator">-a</span> <span class="comment"># 查看映射状态</span></span><br><span class="line">$ pvcreate /dev/loop0 <span class="comment"># Use your own loop device</span></span><br><span class="line">$ vgcreate cinder-volumes /dev/loop0 <span class="comment"># Use your own loop device</span></span><br></pre></td></tr></table></figure>
<p>编辑<code>/etc/lvm/lvm.conf</code>，让LVM不要扫描VM所用的卷：</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">devices { ... filter = [ "a/loop0/", "r/.*/"] # pvdisplay只能看到loop0的设备 ... }</span><br></pre></td></tr></table></figure>
<p>启动服务：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">$ service openstack-cinder-volume start</span><br><span class="line">$ service tgtd start</span><br><span class="line">$ chkconfig openstack-cinder-volume on</span><br><span class="line">$ chkconfig tgtd on</span><br></pre></td></tr></table></figure>
<p>验证测试：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">$ <span class="built_in">source</span> demorc</span><br><span class="line">$ cinder create --display-name myVolume <span class="number">1</span></span><br><span class="line">$ cinder list <span class="comment"># 如果状态为available则正常</span></span><br></pre></td></tr></table></figure>
<p>之后就可以在dashboard上把卷attach到instance上当块设备使用。</p>
<h2 id="创建Instance云主机">创建Instance云主机</h2><p>基于dashboard的很简单，看GUI操作很明确，基于命令也可以。</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line">$ <span class="built_in">source</span> demorc</span><br><span class="line">$ ssh-keygen -t rsa -P <span class="string">''</span> <span class="operator">-f</span> my.key</span><br><span class="line">$ nova keypair-add --pub-key ./my.key.pub demo-key</span><br><span class="line">$ nova keypair-list</span><br><span class="line">$ nova flavor-list</span><br><span class="line">$ nova image-list</span><br><span class="line">$ nova net-list</span><br><span class="line">$ nova secgroup-list</span><br><span class="line">$ nova boot --flavor m1.tiny --image cirros-<span class="number">0.3</span>.<span class="number">2</span>-x86_64 --nic net-id=b7d34c60-<span class="number">5439</span>-<span class="number">4</span>ec8-<span class="number">9468</span>-<span class="number">8</span>e2406801b98 \</span><br><span class="line"> --security-group default --key-name demo-key demo-instance1</span><br><span class="line">$ nova list</span><br><span class="line">$ nova get-vnc-console demo-instance1 novnc</span><br></pre></td></tr></table></figure>
<h2 id="Object_Service:_Swift">Object Service: Swift</h2><ul>
<li>swift-proxy-server: API，操作metadata, container，提供file和container列表，可以配合memcached提高性能；</li>
<li>swift-account-server: swift账号管理</li>
<li>swift-container-server: 容器管理</li>
<li>swift-object-server: 存储节点，要求支持<code>XATTRS</code>，推荐<code>XFS</code></li>
<li>定时任务进程，用于清理、一致性检验等</li>
</ul>
<h3 id="创建swift账号，在keystone上注册服务：">创建swift账号，在keystone上注册服务：</h3><p>192.168.182.128/25做为public网络，10.0.100.0/24做为数据内部同步网络。</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line">$ keystone user-create --name=swift --pass=d8e42c6a28bf6eb08b76 \</span><br><span class="line"> --email=swift@tecstack.org</span><br><span class="line">$ keystone user-role-add --user=swift --tenant=service --role=admin</span><br><span class="line">$ keystone service-create --name=swift --type=object-store \</span><br><span class="line"> --description=<span class="string">"OpenStack Object Storage"</span></span><br><span class="line">$ keystone endpoint-create \</span><br><span class="line"> --service-id=$(keystone service-list | awk <span class="string">'/ object-store / {print $2}'</span>) \</span><br><span class="line"> --publicurl=<span class="string">'http://192.168.182.144:8080/v1/AUTH_%(tenant_id)s'</span> \</span><br><span class="line"> --internalurl=<span class="string">'http://10.0.100.139:8080/v1/AUTH_%(tenant_id)s'</span> \</span><br><span class="line"> --adminurl=http://<span class="number">10.0</span>.<span class="number">100.139</span>:<span class="number">8080</span></span><br></pre></td></tr></table></figure>
<p>在<strong>所有节点</strong>上配置swift hash，用于确定swift ring的mapping，所有节点要一致且不可更改:</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">$ mkdir -p /etc/swift</span><br><span class="line">$ openssl rand -base64 <span class="number">12</span> <span class="comment"># 执行2次，生成2个随机字符串，用于swift_hash_path_*fix</span></span><br><span class="line">$ cat > /etc/swift/swift.conf << EOF</span><br><span class="line">[swift-hash]</span><br><span class="line"><span class="comment"># random unique string that can never change (DO NOT LOSE)</span></span><br><span class="line">swift_<span class="built_in">hash</span>_path_prefix = Fjt8GhQTi13Vr6Hc</span><br><span class="line">swift_<span class="built_in">hash</span>_path_suffix = <span class="number">7</span>uOcSBwQrK6pAtfx</span><br><span class="line">EOF</span><br></pre></td></tr></table></figure>
<h3 id="部署存储节点">部署存储节点</h3><p>每个osswift节点创建3个模拟的块设备，每个设备分配5G容量，在所有osswift上操作：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">$ mkdir -p /opt/swift_disk</span><br><span class="line">$ dd <span class="keyword">if</span>=/dev/zero of=/opt/swift_disk/swdisk0 bs=<span class="number">1</span>M count=<span class="number">5120</span></span><br><span class="line">$ dd <span class="keyword">if</span>=/dev/zero of=/opt/swift_disk/swdisk1 bs=<span class="number">1</span>M count=<span class="number">5120</span></span><br><span class="line">$ dd <span class="keyword">if</span>=/dev/zero of=/opt/swift_disk/swdisk2 bs=<span class="number">1</span>M count=<span class="number">5120</span></span><br><span class="line">$ losetup <span class="operator">-f</span> /opt/swift_disk/swdisk0</span><br><span class="line">$ losetup <span class="operator">-f</span> /opt/swift_disk/swdisk1</span><br><span class="line">$ losetup <span class="operator">-f</span> /opt/swift_disk/swdisk2</span><br><span class="line">$ losetup <span class="operator">-a</span> <span class="comment"># 查看映射状态</span></span><br></pre></td></tr></table></figure>
<p>安装部署：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">$ yum install -y openstack-swift-account openstack-swift-container \</span><br><span class="line"> openstack-swift-object xfsprogs xinetd</span><br><span class="line">$ <span class="keyword">for</span> i <span class="keyword">in</span> loop{<span class="number">0</span>,<span class="number">1</span>,<span class="number">2</span>};<span class="keyword">do</span> mkfs.xfs /dev/<span class="variable">$i</span>; <span class="built_in">echo</span> <span class="string">"/dev/<span class="variable">$i</span> /srv/node/<span class="variable">$i</span> xfs noatime,nodiratime,nobarrier,logbufs=8 0 0"</span> >> /etc/fstab; mkdir -p /srv/node/<span class="variable">$i</span> ; mount /srv/node/<span class="variable">$i</span>;<span class="keyword">done</span>;</span><br><span class="line">$ chown -R swift:swift /srv/node <span class="comment"># 注意每次重建xfs之后需要确认权限</span></span><br></pre></td></tr></table></figure>
<p>创建<code>/etc/rsyncd.conf</code>:</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">uid = swift gid = swift log file = /var/log/rsyncd.log pid file = /var/run/rsyncd.pid address = 10.0.100.139/140/141 # 各个存储节点的数据同步网IP [account] max connections = 2 path = /srv/node/ read only = false lock file = /var/lock/account.lock [container] max connections = 2 path = /srv/node/ read only = false lock file = /var/lock/container.lock [object] max connections = 2 path = /srv/node/ read only = false lock file = /var/lock/object.lock</span><br></pre></td></tr></table></figure>
<p>配置<code>/etc/xinetd.d/rsync</code>:</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">disable = no</span><br></pre></td></tr></table></figure>
<p>启动xinetd:</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">$ service xinetd start</span><br><span class="line">$ mkdir -p /var/swift/recon</span><br><span class="line">$ chown -R swift:swift /var/swift/recon</span><br></pre></td></tr></table></figure>
<p>配置<code>/etc/swift/account-server.conf</code>,<code>/etc/swift/container-server.conf</code>,<code>/etc/swift/object-server.conf</code>：</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">... bind_ip = 192.168.182.144/145/146 # 绑定到各自节点的public网络地址 ...</span><br></pre></td></tr></table></figure>
<h3 id="在osswift0上部署proxy_server">在osswift0上部署proxy server</h3><p>使用oskeystone上的memcached：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">$ yum install -y openstack-swift-proxy python-swiftclient python-keystone-auth-token</span><br></pre></td></tr></table></figure>
<p>配置<code>/etc/swift/proxy-server.conf</code>:</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><span class="line">...</span><br><span class="line">[filter:authtoken]</span><br><span class="line">paste.filter_factory = keystoneclient.middleware.auth_token:filter_factory</span><br><span class="line"><span class="comment"># Delaying the auth decision is required to support token-less</span></span><br><span class="line"><span class="comment"># usage for anonymous referrers ('.r:*').</span></span><br><span class="line">delay_auth_decision = <span class="literal">true</span></span><br><span class="line">auth_protocol = http</span><br><span class="line">auth_host = <span class="number">10.0</span>.<span class="number">100.149</span></span><br><span class="line">auth_port = <span class="number">35357</span></span><br><span class="line">admin_tenant_name = service</span><br><span class="line">admin_user = swift</span><br><span class="line">admin_password = d8e42c6a28bf6eb08b76</span><br><span class="line"></span><br><span class="line">[filter:cache]</span><br><span class="line">use = egg:swift <span class="comment">#memcache</span></span><br><span class="line"><span class="number">10.0</span>.<span class="number">100.149</span>:<span class="number">11211</span></span><br><span class="line">...</span><br></pre></td></tr></table></figure>
<p>配置<code>/etc/swift/object-expirer.conf</code>:</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">... [filter:cache] use = egg:swift#memcache memcache_servers = 10.0.100.149:11211 ...</span><br></pre></td></tr></table></figure>
<p>配置swift的ring：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line">$ <span class="built_in">cd</span> /etc/swift</span><br><span class="line">$ swift-ring-builder account.builder create <span class="number">18</span> <span class="number">3</span> <span class="number">1</span></span><br><span class="line">$ swift-ring-builder container.builder create <span class="number">18</span> <span class="number">3</span> <span class="number">1</span></span><br><span class="line">$ swift-ring-builder object.builder create <span class="number">18</span> <span class="number">3</span> <span class="number">1</span></span><br><span class="line">$ <span class="keyword">for</span> i <span class="keyword">in</span> loop{<span class="number">0</span>,<span class="number">1</span>,<span class="number">2</span>}; <span class="keyword">do</span> swift-ring-builder account.builder add z1-<span class="number">192.168</span>.<span class="number">182.144</span>:<span class="number">6002</span>R10.<span class="number">0.100</span>.<span class="number">139</span>:<span class="number">6005</span>/<span class="variable">$i</span> <span class="number">100</span>; swift-ring-builder container.builder add z1-<span class="number">192.168</span>.<span class="number">182.144</span>:<span class="number">6001</span>R10.<span class="number">0.100</span>.<span class="number">139</span>:<span class="number">6004</span>/<span class="variable">$i</span> <span class="number">100</span>; swift-ring-builder object.builder add z1-<span class="number">192.168</span>.<span class="number">182.144</span>:<span class="number">6000</span>R10.<span class="number">0.100</span>.<span class="number">139</span>:<span class="number">6003</span>/<span class="variable">$i</span> <span class="number">100</span>; <span class="keyword">done</span>; <span class="comment"># 把osswift0的设备加入zone1</span></span><br><span class="line">$ <span class="keyword">for</span> i <span class="keyword">in</span> loop{<span class="number">0</span>,<span class="number">1</span>,<span class="number">2</span>}; <span class="keyword">do</span> swift-ring-builder account.builder add z2-<span class="number">192.168</span>.<span class="number">182.145</span>:<span class="number">6002</span>R10.<span class="number">0.100</span>.<span class="number">140</span>:<span class="number">6005</span>/<span class="variable">$i</span> <span class="number">100</span>; swift-ring-builder container.builder add z2-<span class="number">192.168</span>.<span class="number">182.145</span>:<span class="number">6001</span>R10.<span class="number">0.100</span>.<span class="number">140</span>:<span class="number">6004</span>/<span class="variable">$i</span> <span class="number">100</span>; swift-ring-builder object.builder add z2-<span class="number">192.168</span>.<span class="number">182.145</span>:<span class="number">6000</span>R10.<span class="number">0.100</span>.<span class="number">140</span>:<span class="number">6003</span>/<span class="variable">$i</span> <span class="number">100</span>; <span class="keyword">done</span>; <span class="comment"># 把osswift1的设备加入zone2</span></span><br><span class="line">$ <span class="keyword">for</span> i <span class="keyword">in</span> loop{<span class="number">0</span>,<span class="number">1</span>,<span class="number">2</span>}; <span class="keyword">do</span> swift-ring-builder account.builder add z3-<span class="number">192.168</span>.<span class="number">182.146</span>:<span class="number">6002</span>R10.<span class="number">0.100</span>.<span class="number">141</span>:<span class="number">6005</span>/<span class="variable">$i</span> <span class="number">100</span>; swift-ring-builder container.builder add z3-<span class="number">192.168</span>.<span class="number">182.146</span>:<span class="number">6001</span>R10.<span class="number">0.100</span>.<span class="number">141</span>:<span class="number">6004</span>/<span class="variable">$i</span> <span class="number">100</span>; swift-ring-builder object.builder add z3-<span class="number">192.168</span>.<span class="number">182.146</span>:<span class="number">6000</span>R10.<span class="number">0.100</span>.<span class="number">141</span>:<span class="number">6003</span>/<span class="variable">$i</span> <span class="number">100</span>; <span class="keyword">done</span>; <span class="comment"># 把osswift2的设备加入zone3</span></span><br><span class="line">$ swift-ring-builder account.builder <span class="comment"># 注意检查上面的IP, 端口信息</span></span><br><span class="line">$ swift-ring-builder container.builder</span><br><span class="line">$ swift-ring-builder object.builder <span class="comment"># 检查</span></span><br><span class="line">$ swift-ring-builder account.builder rebalance</span><br><span class="line">$ swift-ring-builder container.builder rebalance</span><br><span class="line">$ swift-ring-builder object.builder rebalance <span class="comment"># Relance the ring.</span></span><br><span class="line">$ chown -R swift:swift /etc/swift</span><br></pre></td></tr></table></figure>
<p>把<code>account.ring.gz</code>,<code>container.ring.gz</code>,<code>object.ring.gz</code>复制到proxy-server和Storage Server的<code>/etc/swift</code>下。</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">$ <span class="built_in">cd</span> /etc/swift <span class="comment"># 由于proxy server部署在osswift0，在osswift0上生成ring配置文件。</span></span><br><span class="line">$ scp account.ring.gz container.ring.gz object.ring.gz root@osswift1:/etc/swift/</span><br><span class="line">$ scp account.ring.gz container.ring.gz object.ring.gz root@osswift2:/etc/swift/</span><br></pre></td></tr></table></figure>
<h3 id="启动服务：">启动服务：</h3><p>启动proxy节点上的服务：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">$ service openstack-swift-proxy start</span><br><span class="line">$ chkconfig openstack-swift-proxy on</span><br></pre></td></tr></table></figure>
<p>启动存储节点上的服务：</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">$ for service in \ openstack-swift-object openstack-swift-object-replicator openstack-swift-object-updater openstack-swift-object-auditor \ openstack-swift-container openstack-swift-container-replicator openstack-swift-container-updater openstack-swift-container-auditor \ openstack-swift-account openstack-swift-account-replicator openstack-swift-account-reaper openstack-swift-account-auditor; do \ service $service start; chkconfig $service on; done # 也可以用swift-init all start启动所有服务进程</span><br></pre></td></tr></table></figure>
<p>检验效果：（确保osswift各个节点的iptables配置正确）</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">$ <span class="built_in">source</span> adminrc</span><br><span class="line">$ swift <span class="built_in">stat</span></span><br><span class="line">$ <span class="built_in">cd</span>; <span class="built_in">echo</span> <span class="string">"Hello test"</span> > test.txt; <span class="built_in">echo</span> <span class="string">"Hello test2"</span> > <span class="built_in">test</span>2.txt</span><br><span class="line">$ swift upload myfiles test.txt</span><br><span class="line">$ swift upload myfiles <span class="built_in">test</span>2.txt</span><br><span class="line">$ swift download myfiles</span><br></pre></td></tr></table></figure>
<p>swift默认的log比较奇怪，是在/var/log/messages里。</p>
<h2 id="Heat">Heat</h2><p>流程编排服务，类似于vagrant，适合自动化系统规划部署：</p>
<ul>
<li>heat: client</li>
<li>heat-api: REST API</li>
<li>heat-api-cfn: AWS Query API</li>
<li>heat-engine: 根据模板执行</li>
</ul>
<p>在oskeystone上准备heat的数据库：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">$ mysql -u root -pa904019ba8cc0b14bef2</span><br><span class="line">mysql> CREATE DATABASE heat;</span><br><span class="line">mysql> GRANT ALL PRIVILEGES ON heat.* TO <span class="string">'heat'</span>@<span class="string">'%'</span> \</span><br><span class="line">IDENTIFIED BY <span class="string">'51eb27f53983633f3337'</span>;</span><br></pre></td></tr></table></figure>
<p>在oscontroller上安装heat：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br></pre></td><td class="code"><pre><span class="line">$ yum install -y openstack-heat-api openstack-heat-engine \</span><br><span class="line"> openstack-heat-api-cfn</span><br><span class="line">$ openstack-config --set /etc/heat/heat.conf \</span><br><span class="line"> database connection mysql://heat:<span class="number">51</span>eb27f53983633f3337@<span class="number">10.0</span>.<span class="number">100.149</span>/heat</span><br><span class="line">$ openstack-config --set /etc/heat/heat.conf DEFAULT qpid_hostname <span class="number">10.0</span>.<span class="number">100.149</span></span><br><span class="line">$ openstack-config --set /etc/heat/heat.conf keystone_authtoken \</span><br><span class="line"> auth_uri http://<span class="number">10.0</span>.<span class="number">100.149</span>:<span class="number">5000</span>/v2.<span class="number">0</span></span><br><span class="line">$ openstack-config --set /etc/heat/heat.conf keystone_authtoken \</span><br><span class="line"> auth_host <span class="number">10.0</span>.<span class="number">100.149</span> <span class="comment"># 官方</span></span><br><span class="line">$ openstack-config --set /etc/heat/heat.conf keystone_authtoken \</span><br><span class="line"> auth_port <span class="number">35357</span></span><br><span class="line">$ openstack-config --set /etc/heat/heat.conf keystone_authtoken \</span><br><span class="line"> auth_protocol http</span><br><span class="line">$ openstack-config --set /etc/heat/heat.conf keystone_authtoken \</span><br><span class="line"> admin_tenant_name service</span><br><span class="line">$ openstack-config --set /etc/heat/heat.conf keystone_authtoken \</span><br><span class="line"> admin_user heat</span><br><span class="line">$ openstack-config --set /etc/heat/heat.conf keystone_authtoken \</span><br><span class="line"> admin_password <span class="number">3817</span>bfface1b24918d4b</span><br><span class="line">$ openstack-config --set /etc/heat/heat.conf ec2authtoken \</span><br><span class="line"> auth_uri http://<span class="number">10.0</span>.<span class="number">100.149</span>:<span class="number">5000</span>/v2.<span class="number">0</span> <span class="comment"># 可以暂时不配置。</span></span><br><span class="line">$ su <span class="operator">-s</span> /bin/sh -c <span class="string">"heat-manage db_sync"</span> heat</span><br></pre></td></tr></table></figure>
<p>创建Heat Service的keystone账号，注册服务：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br></pre></td><td class="code"><pre><span class="line">$ keystone user-create --name=heat --pass=<span class="number">3817</span>bfface1b24918d4b \</span><br><span class="line"> --email=heat@tecstack.org</span><br><span class="line">$ keystone user-role-add --user=heat --tenant=service --role=admin</span><br><span class="line">$ keystone service-create --name=heat --type=orchestration \</span><br><span class="line"> --description=<span class="string">"Orchestration"</span></span><br><span class="line">$ keystone endpoint-create \</span><br><span class="line"> --service-id=$(keystone service-list | awk <span class="string">'/ orchestration / {print $2}'</span>) \</span><br><span class="line"> --publicurl=http://<span class="number">192.168</span>.<span class="number">182.150</span>:<span class="number">8004</span>/v1/%\(tenant_id\)s \</span><br><span class="line"> --internalurl=http://<span class="number">10.0</span>.<span class="number">100.145</span>:<span class="number">8004</span>/v1/%\(tenant_id\)s \</span><br><span class="line"> --adminurl=http://<span class="number">10.0</span>.<span class="number">100.145</span>:<span class="number">8004</span>/v1/%\(tenant_id\)s</span><br><span class="line">$ keystone service-create --name=heat-cfn --type=cloudformation \</span><br><span class="line"> --description=<span class="string">"Orchestration CloudFormation"</span></span><br><span class="line">$ keystone endpoint-create \</span><br><span class="line"> --service-id=$(keystone service-list | awk <span class="string">'/ cloudformation / {print $2}'</span>) \</span><br><span class="line"> --publicurl=http://<span class="number">192.168</span>.<span class="number">182.150</span>:<span class="number">8000</span>/v1 \</span><br><span class="line"> --internalurl=http://<span class="number">10.0</span>.<span class="number">100.145</span>:<span class="number">8000</span>/v1 \</span><br><span class="line"> --adminurl=http://<span class="number">10.0</span>.<span class="number">100.145</span>:<span class="number">8000</span>/v1</span><br><span class="line">$ keystone role-create --name heat_stack_user <span class="comment"># Orchestration创建用户的默认角色</span></span><br><span class="line">$ openstack-config --set /etc/heat/heat.conf \</span><br><span class="line"> DEFAULT heat_metadata_server_url http://<span class="number">10.0</span>.<span class="number">100.145</span>:<span class="number">8000</span></span><br><span class="line">$ openstack-config --set /etc/heat/heat.conf DEFAULT \ heat_waitcondition_server_url http://<span class="number">10.0</span>.<span class="number">100.145</span>:<span class="number">8000</span>/v1/waitcondition</span><br></pre></td></tr></table></figure>
<p>启动服务：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">$ service openstack-heat-api start</span><br><span class="line">$ service openstack-heat-api-cfn start</span><br><span class="line">$ service openstack-heat-engine start</span><br><span class="line">$ chkconfig openstack-heat-api on</span><br><span class="line">$ chkconfig openstack-heat-api-cfn on</span><br><span class="line">$ chkconfig openstack-heat-engine on</span><br></pre></td></tr></table></figure>
<p>验证效果：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">$ <span class="built_in">source</span> demorc</span><br><span class="line">$ vim <span class="built_in">test</span>-stack.yml</span><br></pre></td></tr></table></figure>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">heat_template_version: 2013-05-23 description: Test Template parameters: ImageID: type: string description: Image use to boot a server NetID: type: string description: Network ID for the server resources: server1: type: OS::Nova::Server properties: name: "Test server" image: { get_param: ImageID } flavor: "m1.tiny" networks: - network: { get_param: NetID } outputs: server1_private_ip: description: IP address of the server in the private network value: { get_attr: [ server1, first_address ] }</span><br></pre></td></tr></table></figure>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">$ NET_ID=$(nova net-list | awk <span class="string">'/ demo-net / { print $2 }'</span>)</span><br><span class="line">$ heat stack-create <span class="operator">-f</span> <span class="built_in">test</span>-stack.yml \</span><br><span class="line"> -P <span class="string">"ImageID=cirros-0.3.2-x86_64;NetID=<span class="variable">$NET_ID</span>"</span> <span class="built_in">test</span>Stack</span><br><span class="line">$ heat stack-list</span><br></pre></td></tr></table></figure>
<h2 id="Ceilometer">Ceilometer</h2><p>监控CPU、网络等指标，通过REST API访问。</p>
<ul>
<li>ceilometer-agent-compute: 部署在每个计算节点，目前主要针对计算节点采集信息；</li>
<li>ceilometer-agent-central: 非计算节点信息采集；</li>
<li>ceilometer-collector: 采集信息汇聚。</li>
<li>ceilometer-alarm-notifier: 告警设置。</li>
<li>ceilometer-api: 接受查询请求。</li>
<li>后端存储，比如mongodb。</li>
</ul>
<h3 id="在osmeter部署Ceilmeter_Service：">在osmeter部署Ceilmeter Service：</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">$ yum install -y openstack-ceilometer-api openstack-ceilometer-collector \</span><br><span class="line"> openstack-ceilometer-notification openstack-ceilometer-central openstack-ceilometer-alarm python-ceilometerclient</span><br></pre></td></tr></table></figure>
<p>配置MongoDB：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">$ yum install -y mongodb-server mongodb</span><br><span class="line">$ vim /etc/mongodb.conf <span class="comment"># edit the bind_ip to osmeter internal ip:10.0.100.150</span></span><br><span class="line">$ service mongod start</span><br><span class="line">$ chkconfig mongod on</span><br><span class="line">$ mongo --host <span class="number">10.0</span>.<span class="number">100.150</span> --eval <span class="string">'</span><br><span class="line">db = db.getSiblingDB("ceilometer");</span><br><span class="line">db.addUser({user: "ceilometer",</span><br><span class="line"> pwd: "072aac393486f9b29235",</span><br><span class="line"> roles: [ "readWrite", "dbAdmin" ]})'</span></span><br></pre></td></tr></table></figure>
<p>配置Ceilmeter服务：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br></pre></td><td class="code"><pre><span class="line">$ openstack-config --set /etc/ceilometer/ceilometer.conf \</span><br><span class="line"> database connection mongodb://ceilometer:<span class="number">072</span>aac393486f9b29235@<span class="number">10.0</span>.<span class="number">100.150</span>:<span class="number">27017</span>/ceilometer</span><br><span class="line">$ CEILOMETER_TOKEN=$(openssl rand -hex <span class="number">10</span>)</span><br><span class="line">$ <span class="built_in">echo</span> <span class="variable">$CEILOMETER_TOKEN</span></span><br><span class="line">$ openstack-config --set /etc/ceilometer/ceilometer.conf publisher metering_secret <span class="variable">$CEILOMETER_TOKEN</span></span><br><span class="line">$ openstack-config --set /etc/ceilometer/ceilometer.conf \</span><br><span class="line"> DEFAULT rpc_backend ceilometer.openstack.common.rpc.impl_qpid</span><br><span class="line">$ openstack-config --set /etc/ceilometer/ceilometer.conf DEFAULT qpid_hostname <span class="number">10.0</span>.<span class="number">100.149</span></span><br><span class="line">$ openstack-config --set /etc/ceilometer/ceilometer.conf \</span><br><span class="line"> DEFAULT auth_strategy keystone</span><br><span class="line">$ openstack-config --set /etc/ceilometer/ceilometer.conf \</span><br><span class="line"> keystone_authtoken auth_host <span class="number">10.0</span>.<span class="number">100.149</span></span><br><span class="line">$ openstack-config --set /etc/ceilometer/ceilometer.conf \</span><br><span class="line"> keystone_authtoken admin_user ceilometer</span><br><span class="line">$ openstack-config --set /etc/ceilometer/ceilometer.conf \</span><br><span class="line"> keystone_authtoken admin_tenant_name service</span><br><span class="line">$ openstack-config --set /etc/ceilometer/ceilometer.conf \</span><br><span class="line"> keystone_authtoken auth_protocol http</span><br><span class="line">$ openstack-config --set /etc/ceilometer/ceilometer.conf \</span><br><span class="line"> keystone_authtoken auth_uri http://<span class="number">10.0</span>.<span class="number">100.149</span>:<span class="number">5000</span></span><br><span class="line">$ openstack-config --set /etc/ceilometer/ceilometer.conf \</span><br><span class="line"> keystone_authtoken admin_password eb8ed86ef2178168c458</span><br><span class="line">$ openstack-config --set /etc/ceilometer/ceilometer.conf \</span><br><span class="line"> service_credentials os_auth_url http://<span class="number">10.0</span>.<span class="number">100.149</span>:<span class="number">5000</span>/v2.<span class="number">0</span></span><br><span class="line">$ openstack-config --set /etc/ceilometer/ceilometer.conf \</span><br><span class="line"> service_credentials os_username ceilometer</span><br><span class="line">$ openstack-config --set /etc/ceilometer/ceilometer.conf \</span><br><span class="line"> service_credentials os_tenant_name service</span><br><span class="line">$ openstack-config --set /etc/ceilometer/ceilometer.conf \</span><br><span class="line"> service_credentials os_password eb8ed86ef2178168c458</span><br></pre></td></tr></table></figure>
<p>注册Keystone服务：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">$ keystone service-create --name=ceilometer --type=metering \</span><br><span class="line"> --description=<span class="string">"Telemetry"</span></span><br><span class="line">$ keystone endpoint-create \</span><br><span class="line"> --service-id=$(keystone service-list | awk <span class="string">'/ metering / {print $2}'</span>) \</span><br><span class="line"> --publicurl=http://<span class="number">192.168</span>.<span class="number">182.155</span>:<span class="number">8777</span> \</span><br><span class="line"> --internalurl=http://<span class="number">10.0</span>.<span class="number">100.150</span>:<span class="number">8777</span> \</span><br><span class="line"> --adminurl=http://<span class="number">10.0</span>.<span class="number">100.150</span>:<span class="number">8777</span></span><br><span class="line">$ keystone user-create --name=ceilometer --pass=eb8ed86ef2178168c458 --email=ceilometer@tecstack.org</span><br><span class="line">$ keystone user-role-add --user=ceilometer --tenant=service --role=admin</span><br></pre></td></tr></table></figure>
<p>启动服务：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">$ <span class="keyword">for</span> svc <span class="keyword">in</span> openstack-ceilometer-{api,notification,central,collector,alarm-evaluator,alarm-notifier}; <span class="keyword">do</span> service <span class="variable">$svc</span> start; chkconfig <span class="variable">$svc</span> on; <span class="keyword">done</span>;</span><br></pre></td></tr></table></figure>
<h3 id="给计算节点部署Agent：">给计算节点部署Agent：</h3><p>在oscompute1和oscompute2上安装：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">$ yum install -y openstack-ceilometer-compute python-ceilometerclient python-pecan</span><br><span class="line">$</span><br></pre></td></tr></table></figure>
<p>编辑<code>/etc/nova/nova.conf</code>:</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">$ openstack-config --set /etc/nova/nova.conf DEFAULT \</span><br><span class="line"> instance_usage_audit True</span><br><span class="line">$ openstack-config --set /etc/nova/nova.conf DEFAULT \</span><br><span class="line"> instance_usage_audit_period hour</span><br><span class="line">$ openstack-config --set /etc/nova/nova.conf DEFAULT \</span><br><span class="line"> notify_on_state_change vm_and_task_state</span><br></pre></td></tr></table></figure>
<p>多值的参数，直接修改文件如下：</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[DEFAULT] ... notification_driver = nova.openstack.common.notifier.rpc_notifier notification_driver = ceilometer.compute.nova_notifier ... # qpid 和qpid_hostname之前已经配置过，这里采用一样</span><br></pre></td></tr></table></figure>
<p>编辑<code>/etc/ceilometer/ceilometer.conf</code>:</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br></pre></td><td class="code"><pre><span class="line">$ openstack-config --set /etc/ceilometer/ceilometer.conf publisher \</span><br><span class="line"> metering_secret <span class="number">711</span>e1a83f278a83de1f8 <span class="comment"># 在ceilometer上用openssl生成的</span></span><br><span class="line">$ openstack-config --set /etc/ceilometer/ceilometer.conf DEFAULT rpc_backend ceilometer.openstack.common.rpc.impl_qpid</span><br><span class="line"><span class="comment"># openstack-config --set /etc/ceilometer/ceilometer.conf DEFAULT qpid_hostname 10.0.100.149</span></span><br><span class="line">$ openstack-config --set /etc/ceilometer/ceilometer.conf \</span><br><span class="line"> keystone_authtoken auth_host <span class="number">10.0</span>.<span class="number">100.149</span></span><br><span class="line">$ openstack-config --set /etc/ceilometer/ceilometer.conf \</span><br><span class="line"> keystone_authtoken admin_user ceilometer</span><br><span class="line">$ openstack-config --set /etc/ceilometer/ceilometer.conf \</span><br><span class="line"> keystone_authtoken admin_tenant_name service</span><br><span class="line">$ openstack-config --set /etc/ceilometer/ceilometer.conf \</span><br><span class="line"> keystone_authtoken auth_protocol http</span><br><span class="line">$ openstack-config --set /etc/ceilometer/ceilometer.conf \</span><br><span class="line"> keystone_authtoken admin_password eb8ed86ef2178168c458</span><br><span class="line">$ openstack-config --set /etc/ceilometer/ceilometer.conf \</span><br><span class="line"> service_credentials os_username ceilometer</span><br><span class="line">$ openstack-config --set /etc/ceilometer/ceilometer.conf \</span><br><span class="line"> service_credentials os_tenant_name service</span><br><span class="line">$ openstack-config --set /etc/ceilometer/ceilometer.conf \</span><br><span class="line"> service_credentials os_password eb8ed86ef2178168c458</span><br><span class="line">$ openstack-config --set /etc/ceilometer/ceilometer.conf \</span><br><span class="line"> service_credentials os_auth_url http://<span class="number">10.0</span>.<span class="number">100.149</span>:<span class="number">5000</span>/v2.<span class="number">0</span></span><br></pre></td></tr></table></figure>
<p>重启服务：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">$ service openstack-nova-compute restart</span><br><span class="line">$ service openstack-ceilometer-compute start</span><br><span class="line">$ chkconfig openstack-ceilometer-compute on</span><br></pre></td></tr></table></figure>
<h3 id="给Glance、Cinder、Swift部署Agent：">给Glance、Cinder、Swift部署Agent：</h3><p>在oscontroller上编辑<code>/etc/glance/glance-api.conf</code>，并重启服务:</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">$ openstack-config --set /etc/glance/glance-api.conf DEFAULT notification_driver messaging</span><br><span class="line">$ openstack-config --set /etc/glance/glance-api.conf DEFAULT rpc_backend qpid</span><br><span class="line">$ openstack-config --set /etc/glance/glance-api.conf DEFAULT qpid_hostname <span class="number">10.0</span>.<span class="number">100.149</span></span><br><span class="line">$ service openstack-glance-api restart</span><br><span class="line">$ service openstack-glance-registry restart</span><br></pre></td></tr></table></figure>
<p>在oscontroller和osceph0上编辑<code>/etc/cinder/cinder.conf</code>，并重启服务：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">$ openstack-config --set /etc/cinder/cinder.conf DEFAULT control_exchange cinder</span><br><span class="line">$ openstack-config --set /etc/cinder/cinder.conf DEFAULT notification_driver cinder.openstack.common.notifier.rpc_notifier</span><br><span class="line">$ service openstack-cinder-api restart <span class="comment"># 在oscontoller上</span></span><br><span class="line">$ service openstack-cinder-scheduler restart <span class="comment"># 在oscontroller上</span></span><br><span class="line">$ service openstack-cinder-volume restart <span class="comment"># 在osceph0上</span></span><br></pre></td></tr></table></figure>
<p>在osswift0上安装：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">$ yum install -y python-ceilometer</span><br></pre></td></tr></table></figure>
<p>另，ceilometer需要<code>ResellerAdmin</code>角色来获取数据：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">$ keystone role-create --name=ResellerAdmin</span><br><span class="line">$ keystone user-role-add --tenant service --user ceilometer \</span><br><span class="line"> --role $(keystone role-list | awk <span class="string">'/ ResellerAdmin / {print $2}'</span>)</span><br></pre></td></tr></table></figure>
<p>在osswift0上修改<code>/etc/swift/proxy-server.conf</code>:</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">... [filter:ceilometer] use = egg:ceilometer#swift ... [pipeline:main] pipeline = healthcheck cache authtoken keystoneauth ceilometer proxy-server ...</span><br></pre></td></tr></table></figure>
<p>重启<code>openstack-swift-proxy</code>服务:</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">$ service openstack-swift-proxy restart</span><br></pre></td></tr></table></figure>
<h3 id="检验ceilometer效果">检验ceilometer效果</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">$ ceilometer meter-list</span><br><span class="line">$ glance image-download <span class="string">"cirros-0.3.2-x86_64"</span> > cirros.img <span class="comment"># 下载一个镜像</span></span><br><span class="line">$ ceilometer meter-list</span><br><span class="line">$ ceilometer statistics -m image.download -p <span class="number">60</span></span><br></pre></td></tr></table></figure>
<p>如果遇到采集指标不齐问题，可以检查MQ上的消息队列，可以用<code>qpid-tools</code>工具（其他MQ对应也有其他工具）：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">$ yum install -y qpid-tools</span><br><span class="line">$ qpid-tool <span class="number">10.0</span>.<span class="number">100.149</span></span><br><span class="line">qpid > list</span><br><span class="line">qpid > list queue/exchange</span><br><span class="line">qpid > show ID_xxx</span><br></pre></td></tr></table></figure>
<h2 id="总结">总结</h2><p>Openstack主要的组件部署不复杂，文档目前也已经比较成熟，只不过涉及到linux和python dev相关内容较多，细节容易出错，多动动手基本也可以很快熟悉起来。但如果要深入内部，还得从log、运行原理等切入，直至跟进组件社区开发状态和进展，非一日之功。</p>
<p>参考：</p>
<ol>
<li><a href="http://docs.openstack.org/icehouse/" title="Openstack docs" target="_blank" rel="external">Openstack docs</a></li>
<li><a href="http://promisejohn.github.io/2015/04/15/PythonDevEnvSetting/" title="Python开发环境搭建">Python开发环境搭建</a></li>
<li><a href="http://techglimpse.com/openstack-installation-errors-solutions/" title="openstack安装错误方案" target="_blank" rel="external">openstack安装错误方案</a></li>
<li><a href="http://www.gossamer-threads.com/lists/openstack/operators/34627" title="Problem configuring openvswitch br-ex" target="_blank" rel="external">Problem configuring openvswitch br-ex</a></li>
<li><a href="https://www.rdoproject.org/Neutron_with_existing_external_network" title="Neutron_with_existing_external_network" target="_blank" rel="external">Neutron_with_existing_external_network</a></li>
<li><a href="http://docs.openstack.org/openstack-ops/content/network_troubleshooting.html" title="Network troubleshooting" target="_blank" rel="external">网络问题排查方法</a></li>
<li><a href="http://docs.openstack.org/admin-guide-cloud/content/under_the_hood_openvswitch.html" title="Networking Openvswitch on Openstack" target="_blank" rel="external">Networking Openvswitch on Openstack</a></li>
</ol>
]]></content>
<summary type="html">
<![CDATA[<h2 id="参考架构及部署规划">参考架构及部署规划</h2><p><img src="http://docs.openstack.org/icehouse/install-guide/install/yum/content/figures/1/a/common/figures/openstack_havana_conceptual_arch.png" alt="参考架构"></p>
<ul>
<li>操作系统：CentOS6.5</li>
<li>第三方yum源：epel, rdo</li>
</ul>
<p>节点部署角色：</p>
<table>
<thead>
<tr>
<th>节点名</th>
<th>internal ip</th>
<th>public ip</th>
<th>Role</th>
</tr>
</thead>
<tbody>
<tr>
<td>oscontroller</td>
<td>10.0.100.145</td>
<td>192.168.182.150</td>
<td>nova, glance, cinder, image, neutron, dashboard, heat</td>
</tr>
<tr>
<td>osnetwork</td>
<td>10.0.100.146</td>
<td>192.168.182.151</td>
<td>ML2, OVS, L2 Agent, L3 Agent, DHCP Agent</td>
</tr>
<tr>
<td>oscompute1</td>
<td>10.0.100.147</td>
<td>192.168.182.152</td>
<td>nova-compute</td>
</tr>
<tr>
<td>oscompute2</td>
<td>10.0.100.148</td>
<td>192.168.182.153</td>
<td>nova-compute</td>
</tr>
<tr>
<td>oskeystone</td>
<td>10.0.100.149</td>
<td>192.168.182.154</td>
<td>qpid/rabbitmq, keystone, mysql， memcached</td>
</tr>
<tr>
<td>osmeter</td>
<td>10.0.100.150</td>
<td>192.168.182.155</td>
<td>ceilometer, mongodb</td>
</tr>
<tr>
<td>osswift0</td>
<td>10.0.100.139</td>
<td>192.168.182.144</td>
<td>swift0, swift-proxy-server</td>
</tr>
<tr>
<td>osswift1</td>
<td>10.0.100.140</td>
<td>192.168.182.145</td>
<td>swift1</td>
</tr>
<tr>
<td>osswift2</td>
<td>10.0.100.141</td>
<td>192.168.182.146</td>
<td>swift2</td>
</tr>
<tr>
<td>osceph0</td>
<td>10.0.100.142</td>
<td>192.168.182.147</td>
<td>ceph0</td>
</tr>
<tr>
<td>osceph1</td>
<td>10.0.100.143</td>
<td>192.168.182.148</td>
<td>ceph1 # 暂时不用</td>
</tr>
<tr>
<td>osceph2</td>
<td>10.0.100.144</td>
<td>192.168.182.149</td>
<td>ceph2 # 暂时不用</td>
</tr>
</tbody>
</table>
<p><img src="http://docs.openstack.org/icehouse/install-guide/install/yum/content/figures/1/figures/installguide_arch-neutron.png" alt="参考部署架构"></p>]]>
</summary>
<category term="ceilometer" scheme="http://promisejohn.github.io/tags/ceilometer/"/>
<category term="cinder" scheme="http://promisejohn.github.io/tags/cinder/"/>
<category term="glance" scheme="http://promisejohn.github.io/tags/glance/"/>
<category term="neutron" scheme="http://promisejohn.github.io/tags/neutron/"/>
<category term="nova" scheme="http://promisejohn.github.io/tags/nova/"/>
<category term="openstack" scheme="http://promisejohn.github.io/tags/openstack/"/>
<category term="swift" scheme="http://promisejohn.github.io/tags/swift/"/>
<category term="Tech" scheme="http://promisejohn.github.io/categories/Tech/"/>
</entry>
<entry>
<title><![CDATA[Ruby开发环境搭建]]></title>
<link href="http://promisejohn.github.io/2015/04/17/RubyDevEnvSetting/"/>
<id>http://promisejohn.github.io/2015/04/17/RubyDevEnvSetting/</id>
<published>2015-04-17T02:48:49.000Z</published>
<updated>2015-06-13T07:06:01.243Z</updated>
<content type="html"><![CDATA[<h3 id="Ruby开发环境搭建">Ruby开发环境搭建</h3><p>多版本管理<code>RVM</code>：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">$ gpg --keyserver hkp://keys.gnupg.net --recv-keys <span class="number">409</span>B6B1796C275462A1703113804BB82D39DC0E3</span><br><span class="line">$ curl <span class="operator">-s</span>SL https://get.rvm.io | bash <span class="operator">-s</span> stable</span><br><span class="line">$ <span class="built_in">source</span> /etc/profile.d/rvm.sh</span><br><span class="line">$ sed -i <span class="string">'s!cache.ruby-lang.org/pub/ruby!ruby.taobao.org/mirrors/ruby!'</span> <span class="variable">$rvm_path</span>/config/db <span class="comment"># 使用taobao源下载ruby</span></span><br><span class="line">$ rvm list known</span><br><span class="line">$ rvm install <span class="number">2.1</span>.<span class="number">4</span></span><br><span class="line">$ rvm docs generate-ri <span class="comment"># 生成ruby文档</span></span><br><span class="line">$ rvm use <span class="number">2.1</span>.<span class="number">4</span> --default <span class="comment"># 设定默认ruby版本</span></span><br><span class="line">$ rvm list <span class="comment"># 查询已安装版本</span></span><br></pre></td></tr></table></figure>
<p>用<code>gemset</code>建立独立环境：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">$ rvm use <span class="number">2.1</span>.<span class="number">4</span></span><br><span class="line">$ rvm gemset create rails42</span><br><span class="line">$ rvm use <span class="number">2.1</span>.<span class="number">4</span>@rails42</span><br><span class="line">$ rvm gemset list</span><br></pre></td></tr></table></figure>
<p><code>Gem</code>管理ruby包：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">$ gem sources --remove https://rubygems.org/</span><br><span class="line">$ gem sources <span class="operator">-a</span> https://ruby.taobao.org/</span><br><span class="line">$ gem sources <span class="operator">-l</span></span><br><span class="line">$ gem search rails <span class="comment"># search，install, etc</span></span><br></pre></td></tr></table></figure>
<p>参考：</p>
<ol>
<li><a href="https://rvm.io/" title="RVM official site" target="_blank" rel="external">RVM official site</a></li>