Normalization & Gamma

[jubueche]

I don't understand two points in your code:

if l == L: #Convert z and apply softmax for the last layer. (TODO: Only for prediction or if we pass through encoder?) h = tf.nn.softmax(weights['gamma'][l-1] * (z+weights['beta'][l-1])) else: h = tf.nn.relu(z + weights['beta'][l-1]) #TODO: No gamma?

Why do you leave out the gamma in the second calculation? I neither Valpola2015 nor Bengio2016 I can see a reason for this. Maybe you can explain.

Second:
def eval_batch_norm(): # Evaluation batch normalization # obtain average mean and variance and use it to normalize the batch mean = ewma.average(running_mean[l-1]) var = ewma.average(running_var[l-1]) z = batch_normalization(z_pre, mean, var) return z

I don't understand, why we are using the std and mean from the labeled data that we have accumulated (is this correct?). I thought for evaluation we are using the batchmean and batchstd.

Also another thing: I am sure I was just too lazy to understand, but are we also saving the labeled data in the dictionary under 'unlabeled'?
The reason I think this is, that in the decoding phase we are only using the data 'unlabeled', but we also must learn on the labeled data (take the recons. cost into account). Especially if we have only labeled data.
I hope there is still somebody active here!