4. Visualisations and Analysis

There are several functions within fspnet.utils.plots that can help with analysing the results from the network as well as a function to get the reduced Poisson data, Gaussian background statistics (PGStat).

Plotting Loss per Epoch using plot_performance

plot_performance will plot the loss per epoch for a network.
To generate the plots, it requires at least one epoch of training.

Example code:

from fspnet.utils import plots

# Initialise the decoder and data loader
d_loaders, decoder = init()[1:3]

# Train the decoder
decoder.training(num_epochs, d_loaders)

# Plot the performance of the decoder
plots.plot_performance(
    'Loss',
    decoder.losses[1],
    plots_dir='path/to/plots/directory/',
    train=decoder.losses[0],
)

Comparing Parameter Predictions Against Fitted Parameters

If you have previously fitted parameters that you can treat as the ground truth, then you can compare these with the predictions from the encoder.
The functions for parameter comparison are:

• plot_param_comparison: Compares predicted parameters against the ground truth
• plot_param_distribution: Compares the distribution of each parameter for the predicted parameters and the ground truth
• plot_param_pairs: Compares each parameter against every other in a pair plot for the predicted parameters and the ground truth

For the last two functions, they by default only support the plotting of one set of data; however, the function plot_multi_plot can be used to call these functions multiple times with different sets of data to compare different distributions.

Example code:

from fspnet.utils import plots

# Initialise the decoder and data loader
e_loaders, *_, net = init()

# Train the encoder
net.net.net[1].requires_grad_(False)
net.training(num_epochs, e_loaders)

# Generate predictions where data['targets'] are the ground truth and data['latent'] are the parameter predictions
data = net.predict(e_loaders[-1])

# Plot parameter comparisons
plots.plot_param_comparison(
    log_params,
    param_names,
    data['targets'],
    data['latent'],
    plots_dir='path/to/plots/directory/',
)

# Plot parameter distributions using plot_multi_plot
plots.plot_multi_plot(
    ['Target', 'Prediction'],
    [data['targets'], data['latent']],
    plots.plot_param_distribution,
    plots_dir='path/to/plots/directory/and/prefix',
    y_axis=False,
    log_params=log_params,
    param_names=param_names,
)

# Plot parameter pair plot using plot_multi_plot
plots.plot_multi_plot(
    ['Targets', 'Predictions'],
    [data['targets'], data['latent']],
    plots.plot_param_pairs,
    plots_dir='path/to/plots/directory/and/prefix',
    log_params=log_params,
    param_names=param_names,
)

Calculating & Plotting the Saliency

Saliency maps can be used to infer the influence of the input on the output.
This can be useful to see what the network is trying to improve the most, but it can be quite ambiguous.

There are two functions to calculate the saliency that come from fspnet.utils.analysis, decoder_saliency and autoencoder_saliency.
The autoencoder saliency can be plotted using plot_saliency from fspnet.utils.plots.

Example code:

from fspnet.spectrum_fit import init
from fspnet.utils.plots import plot_saliency
from fspnet.utils.analysis import autoencoder_saliency, decoder_saliency

# Initialise networks and data loaders
e_loaders, d_loaders, decoder, net = init()

# Calculate saliencies
decoder_saliency(d_loaders[1], decoder.net)
saliency_output = autoencoder_saliency(e_loaders[1], net.net)
plot_saliency(*saliency_output, plots_dir='path/to/plots/directory/')

Plotting Linear Weights using plot_linear_weights

To try to help with interpretability, the multiplication of weights in the linear layers that correspond to each input parameter in the decoder can provide an idea of how each parameter contributes to the construction of the spectrum.

Example code:

from fspnet.utils import plots

# Initialise decoder
decoder = init()[-2]

# Plot linear weights
plots.plot_linear_weights(param_names, decoder.net, plots_dir='path/to/plots/directory/')

Reduced PGStat of the Encoder's Predictions using pyxspec_tests

pyxspec_tests can be used to obtain the reduced PGStat for different scenarios.
The ideal reduced PGStat is 1, similar to $\chi^2_\nu$.
pyxspec_tests will use PyXspec to test 5 different scenarios and get the median reduced PGStat for:

• Encoder predictions: Predicted parameters from the encoder
• Ground truth: Target reduced PGStat due to imperfections in the data and model and a comparison to traditional methods
• Defaults: Upper limit of reduced PGStat as if no optimisation was performed
• Encoder predictions with PyXspec fitting: Encoder predictions provide initial first guess, then PyXspec refines the predictions
• PyXspec fitting: Default initial starting point, then PyXspec refines the predictions

First, several parameters in config.yaml need to be set in spectrum-fit:

• training:
  • cpus: Number of threads to perform PyXspec reduced PGStat calculations and fitting, higher is generally better, but the maximum number depends on personal hardware
  • python-path: If using a virtual environment, path to the python file
• data:
  • spectra-directory: Path to the directory containing the spectra .fits files
• model:
  • iterations: Number of iterations of PyXspec fitting
  • model-name: Name of the model to fit
  • custom-model-name: Name of model if a custom model is used and requires loading
  • model-directory: Path to the directory containing custom models
  • fixed-parameters: Which parameters in the model are fixed
  • default-parameters: Default parameters, should be what Xspec uses
• output:
  • worker-directory: Temporary directory to save data for communication between worker threads

Then simply generate the predictions from the network and call pyxspec_tests.
The results will be saved to worker-directory with a name corresponding to the test type as a .csv file containing the spectra names, predicted parameters, and reduced PGStat.

Example code:

from fspnet.spectrum_fit import init, pyxspec_tests

# Initialise autoencoder and data loader
e_loaders, *_, net = init()

# Generate predictions
data = net.predict(e_loaders[-1])

# Start PyXspec tests
pyxspec_tests(data)