Discrete adjoint divergence issues

[WallyMaier]

Hi All,

I am running into some issues regarding my extension of the discrete adjoint for NEMO cases. There are two main issues I am running into:
1.) I am not aware of a method to debug recording/working of the code. NEMO is quite non-linear in its use of variables, and its hard to understand which variables that are being recorded. This required to reinitialize some variable structures after setting the recorded variables.

2.) Even after reaching 10+ orders of magnitude of convergence on the primal solution, the discrete adjoint diverges slowly. I'm not sure how to approach this problem, or which "knobs to turn" to get convergence.

[thomasdick]

Hi,

regarding the two potential issues.

1. Reinitializing after recording, i.e. overwriting registered input variables, can lead to AD issues.
   I assume that you are using SU2_TYPE::GetDerivative. This uses the index stored in the su2double type to look up the derivative on the AD tape. You could check if these are still the correct ones. There is a function 'SetIndex' in ad_structure.hpp. Maybe store the index of an input when it is registered and check if it is still the same when the derivative is extracted.

2. From my experience with fluid adjoints, it can be hard to tell when the primal is converged 'enough'. But 10+ orders of magnitude should do it.
   Can you maybe upload a plot showing the convergence of the primal residual? Does it converge to machine precision, or are there some oscillations in the end?

[WallyMaier]

Hi,

Thanks for your reply!

Regarding the indexing information, It seems like "SetIndex" function is called in CVariable
(ad_index) AD::SetIndex((*ad_index)(iPoint,iVar),variable(iPoint,iVar));`
Ive added a few print statements and everything seems to line up fine.

Ive also run another case that seems to work fine and matches with the base solver fairly well. The adjoint solution looks to follow the same trend. Which makes this problem even more confusing.

Sorry for the poor format....my laptop was struggling

.

[thomasdick]

Hi,

if the indices line up well and your other test case matches your expectations, then some AD error where the recording does not work seems unlikely to me.

With regards to the convergence plot, there does not seem to be any oscillatory behavior in the end. In my experience, if the residuals are oscillating, then adjoint often times does not work. Since it linearizes in the last step and this is not a fixed point. But from your plot I do not think this is an issue here either.

Have you tried Pedro's suggestions and checked if the restart files store the right information?
Also, does the Newton-Krylov method help?

[pcarruscag]

Check if you have exact restarts, if you don't, it is possible that you have more state variables than you think (some intermediate quantity that is derived from the solution variables but lags by one iteration for example).

Turn off preaccumulation (PREACC=NO) for testing.

Have a look at how the fluid problems are setup in the discadj_fsi folder (NEWTON_KRYLOV=YES) you need a multizone problem with one zone but it significantly helps convergence.

[WallyMaier]

EDIT:
Hi all,

I have some updates regarding my issues.
I have tried to follow discadj_fsi flow problems using NEWTON_KRYLOV and PREACC=NO, and the cases are running. I am seeing some trends that are expected. However, the energy seems to blowup.

I have run several cases with varying flow physics to see if I can isolate the issue.
1.) M=8 NACA0012.
2.) M=5 Wedge
3.) M=15 Blunt body
4.) M=8 Viscous Cylinder.

All the cases were run with a single-species (ARGON), 2-species (N2) and 5 species (AIR-5) gas. I was able to get all cases to work with the discrete adjoint using ARGON. I was able to get convergence to the wedge with all 3 cases. A more "comprehensive" document is attached.
AdjointComps.pdf

[pcarruscag]

Hmm you don't need to run the primal simulation with those options to use them in the adjoint, they do not change the primal discretization.

[WallyMaier]

OHHHHHH. Thank you!