-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathvault-continuous-secret-deployment-openrnday.html
476 lines (392 loc) · 24.7 KB
/
vault-continuous-secret-deployment-openrnday.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Deploying your application secrets: Hashicorp Vault and continuous delivery</title>
<meta name="description" content="Managing application secrets, like database credentials, passphrases, salts and private keys, is hard. The availability of those elements are critical to the application, yet they need to be properly secured to reduce the attack surface on your system. Most secret management systems, like Hashicorp Vault, are used as a centralized database, but it creates a single point of failure and it requires extra care in hardening the security of that system. How about deploying your secrets, in Hashicorp Vault, alongside your application? By leveraging your build infrastructure, you can deploy a copy of your secrets in a Vault that is secured using a one-time token, accessible only by your application. In this presentation, we'll show a continuous delivery pipeline that enables that approach, talk about the implications of handling secrets in your build infrastructure, and use threat modeling to verify the security of the deployed Vault.">
<meta name="apple-mobile-web-app-capable" content="yes" />
<meta name="apple-mobile-web-app-status-bar-style" content="black-translucent" />
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no">
<link rel="stylesheet" href="bower_components/reveal.js/css/reveal.css">
<link rel="stylesheet" href="bower_components/reveal.js/lib/css/zenburn.css">
<link rel="stylesheet" href="bower_components/code-prettify/styles/desert.css">
<link rel="stylesheet" href="css/theme-open-rnday.css" id="theme">
<script>
if (window.location.search.match(/print-pdf/gi)) {
var link = document.createElement('link');
link.rel = 'stylesheet';
link.type = 'text/css';
link.href = 'css/print/pdf.css';
document.getElementsByTagName('head')[0].appendChild(link);
}
</script>
</head>
<body class="noFullScreen">
<div id="footer" class="footer show">
<img class="logo-lesfurets" src="img/logo_lesfurets_blanc.png">
<img class="logo-conference" src="img/logo_open_rnday.png">
</div>
<div id="reveal" class="reveal">
<div class="slides">
<section>
<img class="logo herve-francois" width="33%" src="img/lf_com_herve_francois.png">
<h1>Deploying your application secrets: Hashicorp Vault and continuous delivery</h1>
<p>Alexandre DuBreuil<br/>Gilles Di Guglielmo</p>
</section>
<section data-transition="fade-in slide-out">
<h2>Previously at Open R&Day</h2>
<img class="logo herve-francois" width="90%" src="img/jeremy_courtial_open_rrnday.png">
</section>
<section>
<section>
<h2>Context</h2>
</section>
<section data-transition="slide-in fade-out">
<img width="66%" src="img/lf_car_journey.png">
</section>
<section data-transition="fade-in slide-out">
<img width="66%" src="img/lf_com_car_price_sheet.png">
</section>
<section>
<h3>Web application secrets</h3>
<p>We define a <strong>secret</strong> as information that can be used to access sensitive data. Pretty much any information that we cannot put on a public repository. That includes:</p>
<ul>
<li>Insurer web service credentials (username, password)</li>
<li>Encryption keys and key passphrases</li>
<li>Database credentials (username, password)</li>
<li><strong>Out of scope:</strong> customer credentials, PII</li>
</ul>
</section>
<section>
<h3>Secret in Java file</h3>
<div class="code-wrapper">
<pre class="prettyprint">
<code class="code lang-java">
public class ClientPasswordCallback implements CallbackHandler {
private static final String USERNAME = <mark>"lesfurets"</mark>;
private static final String PASSWORD = <mark>"hunter2"</mark>;
@Override
public void handle(Callback[] callbacks) {
final WSPasswordCallback pc = (WSPasswordCallback) callbacks[0];
if (USERNAME.equals(pc.getIdentifier())) {
pc.setPassword(PASSWORD);
}
}
}
</code>
</pre>
</div>
</section>
<section>
<h3>Secret in Tomcat server.xml</h3>
<div class="code-wrapper">
<pre class="prettyprint">
<code class="code lang-xml">
<?xml version='1.0' encoding='utf-8'?>
<Server port="1234" shutdown="SHUTDOWN">
<!-- ... -->
<GlobalNamingResources>
<Resource name="jdbc/b2b2cDatabase"
username=<mark>"dev"</mark>
password=<mark>"hunter2"</mark>
url="localhost:2345"
type="javax.sql.DataSource"
driverClassName="org.mariadb.jdbc.Driver"
jdbcInterceptors="..."/>
</GlobalNamingResources>
<!-- ... -->
</Server>
</code>
</pre>
</div>
</section>
<section>
<h3>Today's objective</h3>
<p>Remove secrets from code and production machines</p>
</section>
<section>
<h3>Overview of password life-cycle</h3>
<p>Our objective is to have a life-cycle that works like this:</p>
<ul>
<li><strong class="color-zen-blue">Developer</strong> use password key in code (ex: <code>insurer_password</code>)</li>
<li><strong class="color-zen-blue">Developer</strong> puts development value in code (ex: <code>testpass</code>)</li>
<li><strong class="color-zen-orange">Security admin</strong> adds production secret value in secret system</li>
<li><strong class="color-zen-green">Release manager</strong> deploys app without seeing the production value</li>
<li><strong class="color-zen-red">Production application</strong> uses the secret</li>
</ul>
<p>From code to production, different person with different access rights are handling secrets.</p>
</section>
<section>
<h3>Prerequisite: Infrastructure as code</h3>
<p>If you do <strong>infratructure as code</strong>, you probably have secrets in your source code. We want to keep infra as code, but remove the secrets.</p>
<!-- https://image.slidesharecdn.com/deepdive-infrastructureascode-150702131406-lva1-app6892/95/deep-dive-infrastructure-as-code-17-638.jpg?cb=1435843061 -->
<img width="33%" src="img/infra_as_code.jpg">
</section>
<section>
<h3>Prerequisite: Infrastructure automation</h3>
<p>Our machine provisioning and deployment is done with Ansible. It makes <strong>staging possible</strong> by facilitating the creation of new environment and enables <strong>disposable infrastructure</strong>.</p>
<img width="10%" src="img/logo_ansible.jpg">
<img width="10%" src="img/logo_chef.jpg">
<img width="10%" src="img/logo_puppet.jpg">
</section>
<section>
<h3>Prerequisite: Continuous delivery</h3>
<p>At LesFurets we deliver code to production at least daily. Continuous delivery means that it is <strong>easy to push a feature to production</strong>, and also easy to push an <strong>old version in case of emergency</strong>.</p>
<img width="10%" src="img/logo_jenkins.jpg">
<img width="10%" src="img/logo_teamcity.jpg">
</section>
</section>
<section>
<section>
<h2>Security</h2>
</section>
<section>
<h3>Choosing a tool</h3>
<p>Many tools are available for secrets management, yet not all will fit your purpose. Making your own <strong>custom solution</strong> might not be a good idea given how hard it is.</p>
<ul>
<li><strong class="color-zen-red">Ansible, Chef, etc.:</strong><br>do not remove secrets on production machine</li>
<li><strong class="color-zen-orange">Square Keywhiz:</strong><br>very similar to Vault and could have been a good choice</li>
<li><strong class="color-zen-yellow">Amazon KMS</strong>, <strong class="color-zen-yellow">Azure Key Vault</strong>, <strong class="color-zen-yellow">Google KMS:</strong><br>somewhat similar to Vault but tied to specific ecosystem</li>
</ul>
</section>
<section>
<h3>Buildtime secrets vs Runtime secrets</h3>
<p>You can fetch the secrets at:</p>
<p><strong class="color-zen-blue">Buildtime</strong> which means the production machine will have a cleartext copy of the secret</p>
<p><strong class="color-zen-orange">Runtime</strong> which means the production machine will dynamicaly get the secret, use it, then discard it, resulting in increased security</p>
</section>
<section>
<h3>Hashicorp Vault</h3>
<p>Lightweight, performant, open-source and battle hardened.</p>
<ul>
<li><strong>Seal</strong> and <strong>unseal</strong> makes your Vault safe</li>
<li><strong>Wrap</strong> secrets to distribute them safely</li>
<li><strong>Authenticate</strong> with different methods</li>
<li><strong>One-time token</strong> by combining token auth and wrap</li>
<li><strong>Audit log</strong> out of the box and easy to use</li>
</ul>
<p><img width="20%" src="img/logo_vault.jpg"></p>
</section>
<section>
<h3>Our Vault usage context</h3>
<p>Deploying <strong class="color-zen-orange">multiple copies</strong> of Vault instead<br> of using it as a <strong class="color-zen-white">central database</strong>.</p>
<img width="30%" src="img/vault_centralized.svg">
<img width="30%" src="img/vault_decentralized.svg">
</section>
<section>
<h3>Why use Vault decentralized?</h3>
<p>We are looking for very specific advantages:</p>
<ul>
<li><strong class="color-zen-orange">Disposable infrastructure</strong>, <strong class="color-zen-orange">continuous delivery</strong> and <strong class="color-zen-orange">version migration</strong> <br> it's easier to replace than modify.</li>
<li><strong class="color-zen-orange">Attack surface</strong> and <strong class="color-zen-orange">staging</strong> <br> deploy specific secrets for a specific env</li>
<li><strong class="color-zen-orange">SLA</strong> and <strong class="color-zen-orange">performance</strong> <br> network issues are hard and one local Vault per JVM is super-fast</li>
</ul>
</section>
<section>
<h3>Team Password Manager</h3>
<p>TPM is a password manager (like Vault) containing our secrets, <strong class="color-zen-orange">but it is never used directly by the production servers</strong>.</p>
<ul>
<li><strong>Additional failsafe layer:</strong> if it fails, it doesn't impact the system</li>
<li><strong>Easier to migrate:</strong> since the production doesn't depend on it</li>
<li><strong>Can be any database system:</strong> ever another Vault</li>
</ul>
</section>
<section>
<h3>Storage of key -> values</h3>
<img width="75%" src="img/tpm_content_01.png">
</section>
<section>
<h3>Uses permissions and audit logs</h3>
<img width="75%" src="img/tpm_content_02.png">
</section>
</section>
<section>
<section>
<h2>Threat model</h2>
</section>
<section>
<h3>What is a threat model?</h3>
<p>A process by which <strong class="color-zen-red">potential threats</strong> can be identified, enumerated, and prioritized.</p>
<p></p>
<h3>Why a threat model?</h3>
<p>To design a system with <strong class="color-zen-green">security in mind</strong>.</p>
</section>
<section>
<h3>How to do a threat model?</h3>
<p>There are many ways to do a threat model,<br>today we'll use the popular <strong class="color-zen-orange">STRIDE method</strong>.</p>
<p>(see <a href="https://en.wikipedia.org/wiki/Stride">wikipedia.org/wiki/Stride</a>)</p>
</section>
<section>
<h3 class="threat-model">Threat model<br>Vault Startup</h3>
</section>
<section data-transition="slide-in fade-out" data-background="img/vault_threat_model_key_01.svg" data-background-size="contain" data-background-color="#f3f3f3">
<h3 class="threat-model key">Threat model<br>Vault startup</h3>
</section>
<section data-transition="fade-out" data-background="img/vault_threat_model_key_02.svg" data-background-size="contain" data-background-color="#f3f3f3">
<h3 class="threat-model key">Threat model<br>Vault startup</h3>
<h4 class="threat-model key text">Denial of service<br>TPM</h4>
</section>
<section data-transition="fade-out" data-background="img/vault_threat_model_key_03.svg" data-background-size="contain" data-background-color="#f3f3f3">
<h3 class="threat-model key">Threat model<br>Vault startup</h3>
<h4 class="threat-model key text">Information Disclosure<br>Unseal key</h4>
</section>
<section data-transition="fade-out" data-background="img/vault_threat_model_key_04.svg" data-background-size="contain" data-background-color="#f3f3f3">
<h3 class="threat-model key">Threat model<br>Vault startup</h3>
<h4 class="threat-model key text">Information Disclosure<br>Unseal key</h4>
</section>
<section data-transition="fade-in slide-out" data-background="img/vault_threat_model_key_05.svg" data-background-size="contain" data-background-color="#f3f3f3">
<h3 class="threat-model key">Threat model<br>Vault startup</h3>
<h4 class="threat-model key text">Information Disclosure<br>Key in transit</h4>
</section>
<section>
<h3 class="threat-model">Threat model<br>Tomcat Startup</h3>
</section>
<section data-transition="slide-in fade-out" data-background="img/vault_threat_model_ott_01.svg" data-background-size="contain" data-background-color="#f3f3f3">
<h3 class="threat-model ott">Threat model<br>Tomcat startup</h3>
</section>
<section data-transition="fade-out" data-background="img/vault_threat_model_ott_02.svg" data-background-size="contain" data-background-color="#f3f3f3">
<h3 class="threat-model ott">Threat model<br>Tomcat startup</h3>
<h4 class="threat-model ott text">Elevation of privilege<br>One time token</h4>
</section>
<section data-transition="fade-out" data-background="img/vault_threat_model_ott_03.svg" data-background-size="contain" data-background-color="#f3f3f3">
<h3 class="threat-model ott">Threat model<br>Tomcat startup</h3>
<h4 class="threat-model ott text">Elevation of privilege<br>One time token</h4>
</section>
<section data-transition="fade-out" data-background="img/vault_threat_model_ott_04.svg" data-background-size="contain" data-background-color="#f3f3f3">
<h3 class="threat-model ott">Threat model<br>Tomcat startup</h3>
<h4 class="threat-model ott text">Denial of service<br>One time token</h4>
</section>
<section data-transition="fade-in slide-out" data-background="img/vault_threat_model_ott_05.svg" data-background-size="contain" data-background-color="#f3f3f3">
<h3 class="threat-model ott">Threat model<br>Tomcat startup</h3>
<h4 class="threat-model ott text">Spoofing<br>Session token</h4>
</section>
<section>
<h3>Security design implications: decryption key</h3>
<p>There is only <strong class="color-zen-orange">one decryption key</strong> that can unseal the Vault. It should never be written to disk. If the Vault is sealed (manually or not), it cannot be unsealed again.</p>
<p>If that happens, the application needs to be redeployed.</p>
</section>
<section>
<h3>Security design implications: authentication</h3>
<p>There is only <strong class="color-zen-orange">one, single use, wrapped token</strong> than can provide the <strong>session token</strong>. Once the wrapped token is used, there is no other way of connecting to the Vault.</p>
<p>If the connexion to the Vault is lost for too long, the lease for the session token expires and the app cannot authenticate anymore.</p>
<p>If that happens, the application needs to be redeployed.</p>
</section>
</section>
<section>
<section>
<h2>In practice</h2>
</section>
<section>
<h3>Overview of delivery pipeline</h3>
<img width="50%" src="img/jenkins_pipeline_vault_packaging.jpg">
<img width="75%" src="img/jenkins_pipeline_vault_deploy.jpg">
</section>
<section data-background="img/vault_infra_01.svg" data-background-size="contain" data-background-color="#f3f3f3"></section>
<section data-background="img/vault_infra_02.svg" data-background-size="contain" data-background-color="#f3f3f3"></section>
<section data-background="img/vault_infra_03.svg" data-background-size="contain" data-background-color="#f3f3f3"></section>
<section data-background="img/vault_infra_04.svg" data-background-size="contain" data-background-color="#f3f3f3"></section>
<section data-background="img/vault_infra_05.svg" data-background-size="contain" data-background-color="#f3f3f3"></section>
<section data-background="img/vault_infra_06.svg" data-background-size="contain" data-background-color="#f3f3f3"></section>
<section data-background="img/vault_infra_07.svg" data-background-size="contain" data-background-color="#f3f3f3"></section>
<section data-background="img/vault_infra_08.svg" data-background-size="contain" data-background-color="#f3f3f3"></section>
<section data-background="img/vault_infra_09.svg" data-background-size="contain" data-background-color="#f3f3f3"></section>
<section data-background="img/vault_infra_10.svg" data-background-size="contain" data-background-color="#f3f3f3"></section>
<section>
<h3>No more secrets in code!</h3>
<div class="code-wrapper">
<pre class="prettyprint">
<code class="code lang-java">
public class ClientPasswordCallback implements CallbackHandler {
private static final VaultService VAULT = CoreServiceFactory.getInstance().getVaultClient();
@Override
public void handle(Callback[] callbacks) {
final WSPasswordCallback pc = (WSPasswordCallback) callbacks[0];
if (<mark>VAULT.getSecret("insurer_username")</mark>.equals(pc.getIdentifier())) {
pc.setPassword(<mark>VAULT.getSecret("insurer_password")</mark>);
}
}
}
</code>
</pre>
</div>
</section>
<section>
<h3>Java design implications</h3>
<p>Read <strong><a href="https://www.owasp.org/index.php/OWASP_Secure_Coding_Practices_-_Quick_Reference_Guide">OWASP Secure Coding Practices</a></strong> and make sure it is known in the development team. A secure system needs a secure codebase.</p>
<p>Java isn't a secure language but for our use case using <strong class="color-zen-orange">short lived secrets</strong> (stack memory, not heap memory) is a good start.</p>
<p>Using a security static code analysis tool like <strong>Checkmarx</strong> is also recommended.</p>
</section>
</section>
<section>
<section>
<h2>Operations</h2>
</section>
<section>
<h3>Performance / Scalability</h3>
<p>Using Vault decentralized makes it easier to manage and <strong class="color-zen-orange">performance is not an issue</strong> if each JVM has it own Vault</p>
<p>We rely <strong class="color-zen-red">heavily</strong> on Vault, since each PII encryption needs an encryption keys in Vault.</p>
<p>It's also <strong class="color-zen-orange">easier to scale</strong> by adding new Vaults and more <strong class="color-zen-orange">resilient to network failures</strong>.</p>
</section>
</section>
<section>
<section>
<h2>Conclusion</h2>
</section>
<section>
<h3>In retrospective: good specific solution?</h3>
<p>Remember our goals, mainly: disposable infrastructure, continuous delivery, version migration, reduced operation, performance (speed and network).</p>
</section>
<section>
<h3>In retrospective: <strong class="color-zen-red">disadvantages</strong></h3>
<p><strong class="color-zen-red">Complex solution</strong>: compared to a single Vault, this is more complicated to implement, but easier to automate and maintain.</p>
<p><strong class="color-zen-red">Requires strong automation</strong>: we had to port old Bash deployment to Ansible, but it is a healthy approach that benefits the whole system.</p>
<p><strong class="color-zen-red">Impossible application restart</strong>: this is disposable infrastructure, it is not a problem if redeployment is fast.</p>
</section>
<section>
<h3>In retrospective: <strong class="color-zen-green">advantages</strong></h3>
<p><strong class="color-zen-green">Continuous delivery</strong> and <strong class="color-zen-green">disposable infrastructure</strong>: easier to replace than migrate.</p>
<p><strong class="color-zen-green">DevOps</strong>: no additional infrastructure, less work for the operations team, and more freedom for the devs.</p>
<p>We have no <strong class="color-zen-green">network failures</strong>, no <strong class="color-zen-green">migration</strong>, <strong class="color-zen-green">excellent performance</strong> and <strong class="color-zen-green">easy staging</strong> for new environment.</p>
</section>
<section>
<h3>Thank you!</h3>
</section>
</section>
</div>
</div>
<script src="bower_components/reveal.js/lib/js/head.min.js"></script>
<script src="bower_components/reveal.js/js/reveal.js"></script>
<script>
// Full list of configuration options available here:
// https://github.com/hakimel/reveal.js#configuration
Reveal.initialize({
controls: false,
progress: true,
history: true,
center: true,
embedded: true,
mouseWheel: true,
slideNumber: false,
slideNumber: 'c/t',
viewDistance: 5,
width: 1280,
height: 900,
margin: 0,
transition: Reveal.getQueryHash().transition || 'linear', // default/cube/page/concave/zoom/linear/fade/none
// Optional libraries used to extend on reveal.js
dependencies: [
{ src: 'bower_components/reveal.js/lib/js/classList.js', condition: function () { return !document.body.classList; } },
{ src: 'bower_components/reveal.js/plugin/markdown/marked.js', condition: function () { return !!document.querySelector('[data-markdown]'); } },
{ src: 'bower_components/reveal.js/plugin/markdown/markdown.js', condition: function () { return !!document.querySelector('[data-markdown]'); } },
//{ src: 'bower_components/reveal.js/plugin/highlight/highlight.js', async: true, callback: function() { hljs.initHighlightingOnLoad(); } },
{ src: 'bower_components/reveal.js/plugin/zoom-js/zoom.js', async: true, condition: function () { return !!document.body.classList; } },
{ src: 'bower_components/reveal.js/plugin/notes/notes.js', async: true, condition: function () { return !!document.body.classList; } }
]
});
</script>
<script src="bower_components/jquery/dist/jquery.slim.min.js"></script>
<script src="js/script.js"></script>
<script src="bower_components/code-prettify/loader/run_prettify.js?skin=desert&callback=prettyCallback"></script>
</body>
</html>
<!-- vim: set tabstop=2 softtabstop=2 shiftwidth=2 expandtab smarttab: -->