R2 values of -inf when no neutralization is observed (i.e. a flat line at 1)

[anloes]

When a virus is not neutralized, this results in fraction infectivity values which can be fit by flat line at 1.0. In the current iteration of neutcurve, it appears that this type of data can result in a fit with an r2 value of negative infinity. This means that filtering on a minimum r2 value would remove these values from analysis. This is not ideal, as non-neutralization of a given strain is a reasonable and expected result in many cases.

[jbloom]

This really reflects a conceptual as much as technical issue.

The r2 value is the coefficient of determination, which reflects how much of the total variation in the data (the total sum of squares of the data, $\sum_i \left(y_i - \langle y \rangle\right)$) is explained by the fit sigmoid.

The total variation in the data is just the summed squared difference of all of the data points (fraction infectivity at each concentration) from a straight horizontal line drawn at the mean fraction infectivity.

The fraction of the variation not explained by the fit is the sum squared residuals.

So when all the data points are on a perfectly straight line, both the variation in the data and the residuals are zero. So arguably, yes, this should give a value of 1 for the r2 rather than negative infinity, and I will fix this.

But more generally, when the data fall along a straight line with just a tiny bit of noise (jitter), then the r2 will be zero (or close to zero) since a sigmoid cannot fit the data better than a straight line.

But really, if we are looking at data with no neutralization, we don't want to classify this as a bad fit.

So I think really we need QC to include two quantities:

• the coefficient of determination (r2)
• the root mean squared deviation (rmsd), which quantifies the absolute amount that the points vary from the data

For non-neutralized data, we could have a good fit with a very poor r2, but then the rmsd will be very small. So I will add computation of that from the curve fits, and then curves can be QC-ed by looking at both the r2 and rmsd.
A fit should be considered good if either r2 is close to one rmsd is close to zero.