-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathindex.html
212 lines (175 loc) · 7.82 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
<!DOCTYPE html>
<html>
<head>
<title>Title</title>
<meta charset="utf-8">
<style>
@import url(https://fonts.googleapis.com/css?family=Droid+Serif:400,700,400italic);
@import url(https://fonts.googleapis.com/css?family=Ubuntu+Mono:400,700,400italic);
@import url(https://fonts.googleapis.com/css?family=Source+Sans+Pro|Bungee+Inline|Scope+One);
body { font-family: 'Source Sans Pro'; }
h1, h2, h3, h4 {
font-family: 'Bungee Inline', 'Source Sans Pro';
font-weight: normal;
color: purple;
}
.remark-code, .remark-inline-code {
font-family: 'Ubuntu Mono';
}
a:link {
text-decoration: none;
}
</style>
</head>
<body>
<textarea id="source">
class: center, middle
# Generative Adversarial Networks
by _Roozbeh Farhoodi_
**download tutorial:**
[Github repository](https://github.com/KordingLab/lab_teaching_2016)
[IPython notebook](https://github.com/KordingLab/lab_teaching_2016/blob/master/session_4/Generative%20Adversarial%20Networks.ipynb)
---
### Generative models
“Generative modeling is a broad area of machine learning which deals with
models of distributions `\(\mathbb{P}(X)\)`, defined over datapoints X in some potentially
high-dimensional space `\(\Theta\)`."
### Motivations
- New samples from a dataset: generating natural images, ...
- completing the incomplete data
.center[<img src='images/completion.gif' width="250px"/>]
---
### variational autoencoder (VAE)
.center[<img src='images/vae.png' width="700px"/>]
[tutorial](https://arxiv.org/pdf/1606.05908v2.pdf)
---
### GAN
[Goodfellow et al, 2014](https://arxiv.org/pdf/1406.2661v1.pdf)
`\(D\)` and `\(G\)` play the following two-player minimax game with value function `\(V(G, D)\)`:
`\(\min_G \max_D V(D, G) = \mathbb{E}_{x \sim p_{\text{data}}({x})}[\log D({x})] + \mathbb{E}_{{z} \sim p_{{z}}({z})}[\log (1 - D(G({z})))]\)`
.center[<img src='images/gan.jpg' width="260px"/>]
---
### Example
.left[<img src='images/fig1.pdf' width="170px"/> <img src='images/fig2.pdf' width="170px"/><img src='images/fig3.pdf' width="170px"/><img src='images/fig4.pdf' width="170px"/>]
The lower horizontal line is the domain from which z is sampled.
The horizontal line above is part of the domain of x.
The upward arrows show how the mapping x = G(z) imposes the non-uniform distribution `\(p_g\)` on
transformed samples. G contracts in regions of high density and expands in regions of low density of pg. (a)
Consider an adversarial pair near convergence: pg is similar to pdata and D is a partially accurate classifier.
(b) In the inner loop of the algorithm D is trained to discriminate samples from data, converging to D(x) =
pdata(x)
pdata(x)+pg(x)
. (c) After an update to G, gradient of D has guided G(z) to flow to regions that are more likely
to be classified as data. (d) After several steps of training, if G and D have enough capacity, they will reach a
point at which both cannot improve because pg = pdata. The discriminator is unable to differentiate between
the two distributions, i.e. D(x) = 1/2
.
---
.center[**Generative and Discriminative**]
.center[<img src='images/twogames.png' width="550px"/>]
---
## Variations
.left[<img src='images/variation.jpg' width="800px"/>]
- dcGAN
- InfoGAN
- AC-GAN
- EB-GAN
- LAPGAN...
---
## DC-GAN
[**Alec Radford and Luke Metz et al, 2016**](https://arxiv.org/pdf/1511.06434.pdf)
.center[<img src='images/dcgan.jpg' width="650px"/>]
"We propose and evaluate a set of constraints on the architectural topology of Convolutional
GANs that make them stable to train in most settings. We name this class of architectures
Deep Convolutional GANs (DCGAN)"
---
## DC-GAN
Architecture guidelines for stable Deep Convolutional GANs
- Replace any pooling layers with strided convolutions (discriminator) and fractional-strided
convolutions (generator).
- Use batchnorm in both the generator and the discriminator.
- Remove fully connected hidden layers for deeper architectures.
- Use ReLU activation in generator for all layers except for the output, which uses Tanh.
- Use LeakyReLU activation in the discriminator for all layers.
---
## Info-GAN
[**Xi Chen et al, 2016**](https://arxiv.org/pdf/1606.03657v1.pdf)
"The GAN formulation uses a simple factored continuous input noise vector z, while imposing no
restrictions on the manner in which the generator may use this noise. As a result, it is possible that
the noise will be used by the generator in a highly entangled way, causing the individual dimensions
of z to not correspond to semantic features of the data."
---
## Info-GAN
Solution: Introducing the latent variable `c` and modifying:
.center[<img src='images/infogan-equation1.jpg' width="550px"/>]
the second term can be approximated by:
.center[<img src='images/infogan-equation2.jpg' width="550px"/>]
Which can be approximated again by Monte Carlo simulations.
.center[<img src='images/infoGAN.jpg' width="550px"/>]
---
## AC-GAN
[**Augustus Odena et al, 2016**](https://arxiv.org/pdf/1610.09585v1.pdf)
"GANs struggle to generate globally coherent, high resolution samples - particularly from datasets with high variability."
---
## EBGAN
[**Junbo Zhao et al, 2016**](https://arxiv.org/pdf/1609.03126v2.pdf)
---
## LAPGAN
[**Emily Denton et al, 2015**](https://arxiv.org/pdf/1506.05751.pdf)
---
### Remedies
- **Freezing learning:** Frequently either the generator or discriminator will outpace the other
and the GAN will get stuck in a degenerate solution. One simple remedy is to freeze
learning in one model when it begins to get too strong. [See here](https://arxiv.org/pdf/1610.01945.pdf?utm_content=buffer13ebb&utm_medium=social&utm_source=facebook.com&utm_campaign=buffer)
- **Label smoothing:** A simple trick to prevent gradients from vanishing when the discriminator’s
predictions are very confident, label smoothing replaces `\(0 : 1\)` labels with `\(\epsilon : 1 - \epsilon\)`,
which guarantees the generator will always have informative gradients.
- **Minibatch discrimination:** To prevent the generator from collapsing onto a single
sample, minibatch discrimination extends the role of the discriminator from classifying
single images to classifying entire minibatches.[See here](https://arxiv.org/abs/1606.03498)
---
### Remedies
- **Batch normalization:** Batch normalization has been critical for scaling GANs to
deep convolutional networks, and virtual batch normalization has extended this by using a
constant reference batch to prevent correlations in predictions due to the other elements of
the minibatch.[See here](https://arxiv.org/pdf/1511.06434v2.pdf) and [here](https://arxiv.org/pdf/1502.03167v3.pdf)
.center[<img src='images/dist.png' width="400px"/>]
---
### Evaluation of Generative models
https://arxiv.org/pdf/1511.01844v3.pdf
---
### tutorial's talk
.center[<iframe width="560" height="315" src="https://www.youtube.com/embed/KeJINHjyzOU" frameborder="0" allowfullscreen></iframe>]
---
### let's start painting!
.center[<img src='images/painting1.png' width="800px"/>]
---
class: center, middle
# Questions and Discussion
</textarea>
<script src="https://gnab.github.io/remark/downloads/remark-latest.min.js"></script>
<script src="http://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS_HTML&delayStartupUntil=configured" type="text/javascript"></script>
<script>
var slideshow = remark.create({
ratio: '4:3',
navigation: {
scroll: false,
touch: true,
click: false
}
});
// Setup MathJax
MathJax.Hub.Config({
tex2jax: {
skipTags: ['script', 'noscript', 'style', 'textarea', 'pre']
}
});
MathJax.Hub.Queue(function() {
$(MathJax.Hub.getAllJax()).map(function(index, elem) {
return(elem.SourceElement());
}).parent().addClass('has-jax');
});
MathJax.Hub.Configured();
</script>
</body>
</html>