Calibrator.Fit output problems

[mattodeg]

Good morning. I am using RASCAL as a tool for line identification and fitting for a project at my University.

I found no problem using the whole MWE provided in the doc, apart form the fit function used to fit lines from a spectrum provided by an atlas.

The doc -- and the MWE provided there -- suggests to use fit_coeff, rms, residual, peak_utilisation = c.fit(max_tries=1000, fit_deg=4) but when I try to run it it gives me an error of too many values to unpack. After few tries I found that it returns only one array, so there is no need to give the 4 variables as before.

But when looking better into the array, I discovered it has a length of 7.

So my question is: what are these 7 values?

Thanks.

[cylammarco]

Hi, the document is a bit outdated as we are preparing for the next major upgrade of rascal.

The 7 outputs are:

        fit_coeff: list
            List of best fit polynomial fit_coefficient.
        matched_peaks: list
            Peaks used for final fit
        matched_atlas: list
            Atlas lines used for final fit
        rms: float
            The root-mean-squared of the residuals
        residual: float
            Residual from the best fit
        peak_utilisation: float
            Fraction of detected peaks (pixel) used for calibration [0-1].
        atlas_utilisation: float
            Fraction of supplied arc lines (wavelength) used for
            calibration [0-1].

Hope that helps.

[mattodeg]

Thank you for the update. When will the update be released? Deguidi Matteo

…

On Mon, Jun 12, 2023, 23:20 Marco ***@***.***> wrote: Hi, the document is a bit outdated as we are preparing for the next major upgrade of rascal. The 7 outputs are: fit_coeff: list List of best fit polynomial fit_coefficient. matched_peaks: list Peaks used for final fit matched_atlas: list Atlas lines used for final fit rms: float The root-mean-squared of the residuals residual: float Residual from the best fit peak_utilisation: float Fraction of detected peaks (pixel) used for calibration [0-1]. atlas_utilisation: float Fraction of supplied arc lines (wavelength) used for calibration [0-1]. Hope that helps. — Reply to this email directly, view it on GitHub <#90 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AR3DJULIPVB2X5ICJO6FO4LXK6B23ANCNFSM6AAAAAAZAZR3GM> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[cylammarco]

@jveitchmichaelis

[jveitchmichaelis]

Hi @mattodeg

Currently we don't have a fixed timeline - at least I can't promise a release on a specific date. However we're revamping a lot of the way that Rascal is configured and Atlases are handled, which should things simpler.

Are you getting a decent fit currently?

Cheers

[mattodeg]

Hi @jveitchmichaelis! Yes it seems reasonable, although it recongizes only 20 lines over 330 provided - with IRAF I was able to detect almost every line.

But it is just for a small Uni-related project so it doesn't matter too much.

I was asking about the release because I may have used it, but don't worry at all. WIsh you the best!

[jveitchmichaelis]

Hmm maybe @cylammarco has some suggestions there. In theory once you have your fit, you can transform all your lines to detector coordinates and check to see if they line up (even if rascal doesn't "match" them for you).

[cylammarco]

Hmm, the peak finding depends on the usage of the find_peaks(). I would try setting the prominence and/or heights arguments to get more peaks.

[mattodeg]

How can I tansform lines into detector coordinates?

[jveitchmichaelis]

Probably easier to go the other way - take your full list of peaks and pass them to c.polyval(peaks, fit_coeff) where fit_coeff is returned by the fit function. That should give you a list of wavelengths which you can compare to your atlas. There are some other convenience functions for matching in the calibrator, but I'm not sure how mature they are in the latest release vs what we're working on in the development branch. Another question for Marco I think.

[cylammarco]

@jveitchmichaelis, the question was from lines to detector coordinates. Would you recommend using this https://github.com/alvarosg/pynverse on the polynomial function c.polyfit?

[jveitchmichaelis]

Yeah I was going to suggest pynverse, but for a comparison with an atlas I'm not sure it matters which way you go (either you convert peak to wavelength and see, or you try and see where distinct wavelengths would fall).

[mattodeg]

Thx for the suggestions - I will try what you said.

My main problem is that I'm trying to do a porject previously done with IRAF in Python and the I am struggling with the identify, reidentify and fitcoord commands. If you have any suggestion, feel free to share them.

[cylammarco]

If you can share with us the arc or an image of it, we may be able to give better suggestions.

[mattodeg]

I uploaded the images in this folder. Tell me if you have the rights to see the images. The images must be summed together by pairs (like 100097 an 100099). They are calibration images taken using an Iron-Argon lamp.

Tell me also if you need other info.

https://drive.google.com/drive/folders/1afmIg_wcO11f5hH8hF1TNOL7sy_muDQF?usp=sharing

[cylammarco]

Hi, I have not managed to get a good fit yet, but here's the update:

By setting the height and prominence to some smaller values, many lines can be recovered

temperature = fits_file.header["TEMP-INT"]
pressure = fits_file.header["ATMPRESS"] * 100.0
relative_humidity = fits_file.header["HUM-INT"]

# Collapse into 1D spectrum between row 110 and 120
spectrum = np.median(spectrum2D[100:200], axis=0)

# Identify the peaks
peaks, _ = find_peaks(spectrum, height=300, prominence=5, distance=5)
peaks = util.refine_peaks(spectrum, peaks, window_width=5)

v0.3 doesn't contain the Fe lines, that's why you are getting a very small number of line matches. To get the iron lines, they have to be manually provided. Here is how you can do it:

First, get the lines at
https://github.com/jveitchmichaelis/rascal/blob/new_ransac/src/rascal/arc_lines/nist_clean_Fe_I.csv
https://github.com/jveitchmichaelis/rascal/blob/new_ransac/src/rascal/arc_lines/nist_clean_Fe_II.csv

Then add them and fit:

# Initialise the calibrator
c = Calibrator(peaks, spectrum=spectrum)
c.plot_arc()
c.set_hough_properties(
    num_slopes=5000,
    range_tolerance=200.0,
    xbins=200,
    ybins=200,
    min_wavelength=5800.0,
    max_wavelength=7010.0,
)
c.set_ransac_properties(sample_size=5, top_n_candidate=5, filter_close=True)

fe1 = np.array(list(csv.reader(open("nist_clean_Fe_I.csv"), delimiter=',', quotechar='"', skipinitialspace=True))[1:])
fe2 = np.array(list(csv.reader(open("nist_clean_Fe_II.csv"), delimiter=',', quotechar='"', skipinitialspace=True))[1:])

# filtering the weak lines and use the appropriate wavelength ranges
fe1 = fe1[fe1[:,2].astype('float')>20.0]
fe1 = fe1[fe1[:,1].astype('float')>5900]
fe1 = fe1[fe1[:,1].astype('float')<7100]

fe2 = fe2[fe2[:,2].astype('float')>20.0]
fe2 = fe2[fe2[:,1].astype('float')>5900]
fe2 = fe2[fe2[:,1].astype('float')<7100]


atlas = Atlas(
    pressure=pressure,
    temperature=temperature,
    relative_humidity=relative_humidity,
)
atlas.add_user_atlas(
    fe1[:,0],
    fe1[:,1].astype('float'),
    pressure=pressure,
    temperature=temperature,
    relative_humidity=relative_humidity,
)
atlas.add_user_atlas(
    fe2[:,0],
    fe2[:,1].astype('float'),
    pressure=pressure,
    temperature=temperature,
    relative_humidity=relative_humidity,
)
c.set_atlas(atlas, candidate_tolerance=5.0)

c.do_hough_transform()

# Run the wavelength calibration
(
    best_p,
    matched_peaks,
    matched_atlas,
    rms,
    residual,
    peak_utilisation,
    atlas_utilisation,
) = c.fit(max_tries=10000)

One caveat here, the wavelength limits of the calibrator do not apply to the manual lines (but the limits for the set_hough_properties are relevant), so the wavelength and intensity filterings were done manually above.

@jveitchmichaelis ?

[jveitchmichaelis]

Thanks @mattodeg we're going to try with our newer branch (very beta) which has more extensive built in line lists and some improved atlas handling.

Given the line density of your emission spectrum, it's likely that providing a good manual list will yield better results as there are probably too many degenerate (low error) bad fits. Though it may work, one advantage of spectra like this is that maximising the number of recovered lines is normally the best metric to evaluate whether the fit is good or not.

We might also be able to fix a couple of lines at each end (since you have good coverage), which will constrain the fit.

[mattodeg]

I totally missed the email updates and I want to apologize with both of you. Thanks for the solution and the update, I will try to implement it in my code with your suggestions.

Thanks also for developing RASCAL, it is very interesting and does a very promising job"