question about GPU memory

[bravelyw]

First thanks for your contribution. In your introduction, you said you use a GPU with 12 GB, but I run the program in a GPU with 24 GB and use nearly 18 GB. I dont know why, can you help me?
And the GPU Fan mabye is broken, but it is not the reason.

[LIKP0]

Hello! I'm also following this project now but I can't run the code. Are you willing to share your email with me and have some debates?

[xamyzhao]

@bravelyw I'm sorry, I need more context to figure out your memory usage. Which models are you loading? Does this happen after you load the model, or only once training starts?

[bravelyw]

Hello! I'm also following this project now but I can't run the code. Are you willing to share your email with me and have some debates?

ok, share your email and I will send the code to you.

[bravelyw]

@bravelyw I'm sorry, I need more context to figure out your memory usage. Which models are you loading? Does this happen after you load the model, or only once training starts?

The flow-fwd model, and throughout the training phase.

[LIKP0]

12232132@mail.sustech.edu.cn. Thank you!
Actually I met this problem after I use the OASIS dataset, and I can't figure it out.

[bravelyw]

12232132@mail.sustech.edu.cn. Thank you! Actually I met this problem after I use the OASIS dataset, and I can't figure it out.

I have sent to you. And I guess mabye the dataset is not correct.

[bravelyw]

@bravelyw I'm sorry, I need more context to figure out your memory usage. Which models are you loading? Does this happen after you load the model, or only once training starts?

And I want to manually import the last traing result when OOM, but it seems no init_model_weights method in exp, how should I do?

[xamyzhao]

@bravelyw what batch size are you using? How large are your volumes?

You're right, looks like init_model_weights wasn't included in this repo. You should be able to implement your own init_model_weights function that uses standard model loading code from your desired path (e.g. https://www.tensorflow.org/guide/keras/save_and_serialize)