Please help -- stopping criterion for very basic and simple nonlinear ODE inverse problem

[sophiaxxiao]

Hello everyone.
I'm experimenting with a basic FNN for solving an ODE inverse problem. Specifically, given noisy state observations (with Gaussian additive noise) from a nonlinear system of ODE's where observations are from an equispaced time-grid, I want to use the NN to infer both true trajectories and parameters.

The system of ODEs I consider is non-linear with fixed true parameters. It has trajectories which are highly smooth, very small Lipschitz constant over time-domain of interest. So I am using a very basic, small, and shallow FNN.

My problem is: I don't know what to use as a stopping criterion. Stopping when loss plateaus results in overfitting to noise in observations..

Background:
I have followed the example from the docs with the Lorentz system (https://deepxde.readthedocs.io/en/latest/demos/pinn_inverse/lorenz.inverse.html).
My use case is also highly similar to the experiments in the paper "Systems biology informed deep learning for inferring parameters and hidden dynamics" (https://doi.org/10.1371/journal.pcbi.1007575).

** Architecture used: **

• 1 input neuron, 3 hidden layers w/64 neurons
• Xavier initialization
• tanh activation function

** Training hyperparameters **

• Adam optimizer
• Learning rates 0.1, 0.01, 0.001

** Details about problem **
I was advised to stop training when overall loss (smoothed) plateaus. This results in overfitting to noise in the observations. As with the Lorentz example, I didn't use a train/test split.

Potential solutions I thought of:

1. Try manual weighting of the ODE and data loss to be of the same magnitude.
2. try increasing regularization from ODE loss. Increase the number of collocation points for the ODE loss
3. Remove hidden layers
4. Early stopping.

Any help would be much appreciated.