2024/07/14: Sleep-wake

[shemlem]

After a small detour these past two weeks, it was decided to get back again to reading the seminal papers in Generative AI.

The paper chosen for next Wednesday 14th of August is the original GANs paper by Ian Goodfellow et al. :

https://arxiv.org/abs/1406.2661

gans.pdf

The wake-sleep algorithm

[agosztolai]

Who’s presenting this? If you have time, could you please also briefly go through the (old paper) Sleep-wake algorithm by Hinton et al. ?

[moritzschaefer]

If no-one else signs up, I could present it (and potentially the sleep-wake one in the same go)

[shemlem]

I volunteered to present given that there were no volunteers, but I'm happy for Moritz or any other one to present

[moritzschaefer]

Ok fine, I can take over this week if you haven't started preparing yet.

[shemlem]

Sure!

[moritzschaefer]

Dear all,

please take the time to watch the lecture (13 mins) by Hinton on his algorithm until tomorrow.

We'll try to split the session tomorrow into the wake-sleep algorithm and GANs. I might probably even focus a little bit more on the former

[moritzschaefer]

Interesting questions that came up:

• How does wake sleep relate to autoencoders; How to VAEs?
• Can wake-sleep be expanded to non-Bernoulli?

Regarding the seven, I guess all it would take would be to define the cost C for continuous-value distributions, which is well defined (shannon works not only on discrete values)

[agosztolai]

Here's a paragraph from the VAE paper:

"The wake-sleep algorithm [HDFN95] is, to the best of our knowledge, the only other on-line learning method in the literature that is applicable to the same general class of continuous latent variable models. Like our method, the wake-sleep algorithm employs a recognition model that approximates the true posterior. A drawback of the wake-sleep algorithm is that it requires a concurrent optimization of two objective functions, which together do not correspond to optimization of (a bound of) the marginal likelihood. An advantage of wake-sleep is that it also applies to models with discrete latent variables. Wake-Sleep has the same computational complexity as VAE per datapoint.'

So, it seems (as we already noted) that the advantage of VAE is that, through the reparametrisation trick, it allows backpropagating through the recognition and generative models. Doing so yields a single optimisation function, which lower bounds the marginal likelihood.