Add VERITAS data

[cdeil]

VERITAS has published a lot of results (images, lightcurves, spectra) in FITS format:

http://veritas.sao.arizona.edu/veritas-science/veritas-results-mainmenu-72

We should add it here.

Ideally this should be done via Python scripts that download and re-format the data to our formats, so that it's reproducible. Or course contributions that manually add the data in our format would be welcome as well.

[cdeil]

Apparently there are plans to collect VERITAS data at HEASARC.

See #11 (comment) by @GernotMaier .

@GernotMaier -- Do you know if there's an ETA for this? Is this coming soon or is this something that might come in a year or two or never?

[GernotMaier]

You know how collaborations work: we form a working group now, we discuss with the HEASARC the format, and then we need to collect all the data from the publications. So I wouldn't expect anything before Spring.

[cdeil]

OK. So just if someone else comes across this issue:

• Feel free to add any VERITAS data to gamma-cat, let's not wait.
• Personally I probably won't work on this much, but I probably will add available info on Galactic sources to have complete TeV data for a study we're working on.

[cdeil]

@GernotMaier - FYI: Since yesterday, @pdeiml is doing data entry for gamma-cat.

He's doing the CTA 1 VERITAS publication now:
http://adsabs.harvard.edu/abs/2013ApJ...764...38A

Can you provide spectral points for this paper?
I don't see something to download here?
http://veritas.sao.arizona.edu/veritas-science/veritas-results-mainmenu-72/350-cta1

Actually I haven't been able to find any VERITAS spectral point data in electronic format yet.
Could you please point out a few examples where to find them, or even a list of all available spectral point data from VERITAS (or main interest is Galactic sources at the moment, and once that is complete, move on to extragalactic)

[GernotMaier]

The spectral points are in Table 1 in the paper.

I completely agree that VERITAS is very bad in providing their data in electronic format. We have in principle that rule, but somehow it does not seem to work.

[cdeil]

OK, I'll copy & paste the spectral points from the PDF now.

I've looked around a bit on the http://veritas.sao.arizona.edu/veritas-science page and haven't been able to find any data in electronic format. So if you could point out data that is available in electronic format to us, we'll reformat it and add to gamma-cat. Otherwise, we'll but the VERITAS corresponding authors or you from time to time.

[GernotMaier]

Some results have FITS or text files (e.g. http://veritas.sao.arizona.edu/veritas-science/veritas-results-mainmenu-72/423-template). But we don't have a well defined file format, most of the data is anyway a simple table.

[cdeil]

Some results have FITS or text files (e.g. http://veritas.sao.arizona.edu/veritas-science/veritas-results-mainmenu-72/423-template).

Thanks for this link!

But we don't have a well defined file format, most of the data is anyway a simple table.

That doesn't matter much. We have to reformat and add some meta info on data entry anyways.
Having data in any electronic format is often still much more convenient than having to copy & paste numbers from a PDF.

[cdeil]

@GernotMaier added a lot of VERITAS data in the past months and plans to add more.

Gernot - I'm assigning this general reminder issue to "add VERITAS data" to you.
Feel free to close it any time if you don't find it helpful, or keep it open as a reminder if you have some goal in mind where this task could be considered mostly complete.

[GernotMaier]

Good - this will still take a while (about 50% done).

Below the current statistics ('done' means includes in gamma-cat; not counting proceedings):

year	total number of papers	number of non-UL papers	status 'done'
2008	3	3	1
2009	13	11	4
2010	7	6	2
2011	12	9	3
2012	9	4	2
2013	8	7	5
2014	10	9	7
2015	7	5	2
2016	10	5	4
2017	8	6	2
total	87	65	32

[cdeil]

@GernotMaier - Could you please add folders and very small info.yaml files for all VERITAS papers you think should be added to gamma-cat.

Examples: 7586417
Schema: https://github.com/gammapy/gamma-cat/blob/master/input/schemas/dataset_info.schema.yaml
Discussion for MAGIC publications that lead to @pdeiml and me inventing the info.yaml files:
#157 (comment)

If we have those info.yaml files, we will be able to make lists of how complete we are (I'll write the scripts and expose this on a webpage soon), which data we still want, and possibly distribute the work of doing data entry for gamma-cat some more.

I'll try to find the time to add the missing publications / info.yaml files for HESS soon.

Let me know if you have any questions or comments about the info.yaml files.

[GernotMaier]

Will do - but will take a while.

[GernotMaier]

I see that info.yaml files have been added to the existing VERITAS papers: is it correct that you set by default for all of them

status: incomplete
reviewed: no
?

I just want to know what my starting point is.

[cdeil]

Yes, we put reviewed: no for all of them. It's not used for now, but at some point there might be a review of data entry completeness or accuracy. Or maybe never, there it's not clear if this will be useful.

The status is supposed to be set in a useful way, with the following logic:

• missing = this is a dummy entry, a paper we want in gamma-cat, but no data entry yet
• complete = data entry complete, no further work needed for now (well, mostly, a bit of a judgement call)
• incomplete = data entry incomplete. I've put this for most cases, namely the ones where a YAML file with fitted model parameters is missing or an ECSV file with spectral points is missing. In some cases I might have gotten it wrong, and really incomplete can be changed to complete.

There is a single "notes:" entry that can be added for a given paper, where you can leave notes if you think it'll be useful to your future self or others, e.g. "spectral points missing" or "this paper has no spectral points" or "have to check value XYZ with Jimbo".

This is just a starting point, of course we're open to feedback and improvements to info.yaml. The main motivation for now was to get an overview of what data we're still missing, i.e. to add the "status: missing" new info.yaml files and make a list of missing papers and try to get new contributors. (I'll make a poster for the gamma-cat at the CTA PHYS meeting in 10 days.)

[GernotMaier]

An update on the status of the VERITAS papers:

• all non-UL/non technical VERITAS papers have now an entry
• 36 of 64 papers of these kind are complete
• for 25 out of the remaining papers, the 'only' items missing are the light curve data
• will add the UL papers as soon as we agreed on the format

year	total number of papers	non-UL/non-technical papers	status 'done'	status 'lc points missing'
2008	3	3	1	2
2009	13	11	4	7
2010	7	6	3	3
2011	12	9	5	4
2012	9	4	2	1
2013	8	7	5	2
2014	10	9	7	2
2015	7	5	2	2
2016	11	6	5	1
2017	8	5	4	1
total	88	64	36	25

[GernotMaier]

Progress table

(please ignore the last column)

year	total number of papers	non-UL/non-technical papers	status 'done'	ignore this column...'
2008	3	3	3	2
2009	13	11	7	1
2010	7	6	4	0
2011	12	9	6	0
2012	9	4	3	0
2013	8	7	6	0
2014	10	9	7	0
2015	7	5	3	0
2016	11	6	6	0
2017	8	5	5	0
total	88	66	50	3