NaNs in Fourcastnet

[Isaaciwd]

There are NaNs present in both the input and output of Fourcastnet. For the single-layer variables (tcwv, u10, v10, t2m, u100, v100, msl, tp, sp), there are no NaNs at the 0th timestep, which represents the initial conditions. However, at latitude = -90, every value is NaN for all longitudes at all subsequent timesteps.

For variables with pressure as a coordinate (z, t, u, v, r), the pattern is less consistent. These variables contain NaNs at the 0th timestep, typically at the highest level (50 hPa). Additionally, for all these variables except z, all values at 50 hPa are NaNs at every timestep. There are other instances of NaNs, but I have not yet identified a clear pattern for them.

I am using the latest versions of ai-models (0.6.1) and ai-models-fourcastnet (0.0.7). I also saw similar behavior with ai-models 0.4.0 and ai-models-fourcastnet 0.0.2. I tested this on two machines: one running openSUSE Leap 15.4 and the other on Red Hat 8.8, and I ran on both CPU and GPU.

[1301970063]

The reason is that you didn't provide the correct initial conditions.

[Isaaciwd]

Yes, that could be it. But the ai-models package downloads, processes and provides the initial conditions so this is still an issue with the package. I should have added that I'm running using CDS input, I'd be interested to know if this problem still arises with MARS input.

[1301970063]

Mars requires an authorized account to access, and I haven't used it. I also tried downloading the ERA5 data myself and found that it can run successfully.

[Isaaciwd]

Okay, so just to confirm, using ERA5 data you downloaded yourself you were able to get fourcastnet to run without producing NaNs?