add "similarity search" capability based on just a GPS time

[reedessick]

An extremely useful REST API feature would be to specify a GPS time and an IFO and obtain glitch class predictions for each family as well as performing a similarity search for similar times. This functionality will have a "guaranteed customer" in the automated detection checklist and will have an immediate impact for vetting of online/low-latency candidates. The run-time latency for this tool does not need to be particularly fast (ie: tens of minutes will suffice).

[scottcoughlin2014]

@duncanmmacleod I am of two minds on this as I think this a really useful query.
1.) I set up the gravityspytools server with enough software in order to allow it to have LIGO kerberos credential so that it can remotely get the data and run a small targeted gravityspy instance to accomplish the stated goal above. Concerns could be that as a non super dope webserver box like LDVW perhaps between the search queries and making qScans on the fly the server gets sad.

2.) I am thinking it could be an additional class on the Gravity Spy Table, if you do not think that would be too burdensome. I think I can code it in such a way that it will not be too obnoxious (i.e. the class will not require an unruly amount of gravityspy specific imports just to accomplish the targeted goal) and that way if 20 people do it at the same time they are using their own (or CIT or whatever) processing power and just hitting NDS2 hard which it is designed for.

When you get the the office let me know what you think.

[scottcoughlin2014]

@reedessick and @duncanmmacleod Alright it is very clear that it makes sense given the extra dependencies to go with route 1.) above. Before committing an initial go at doing this via the user interface (which shockingly works pretty well), I am now realizing that credentials and proprietary stuff might be a concern. Because as of now since the kinit is on the server side any old person can run this using a GPS time and get back results. Is that a big deal? I am guess we will want to make this UI and restful API behind kerberos or something?

[reedessick]

@scottcoughlin2014 , I've been thinking about this too and I'm not sure we need to add dependencies into the core Gravity-Spy code.

Perhaps it makes sense to provide a separate tool that generates a PNG/figure/whatever given a gps time, and then Gravity-Spy itself just has to provide an API that ingests that figure and runs the similarity search rather than ingesting a GPS time and building a figure internally. You then require users to generate their own figures, which they can conceivable only do if they can authenticate to get to the data somehow. An additional benefit is that I could upload pictures of actual koi fish (i.e.: the animal) to Gravity-Spy and see if they're classified as koi glitches, for example.

I think as long as you can isolate the authentication issues to a single executable/function call that lives in a module that can easily be dropped later on (i.e.: a simple gw_data_find | plotting script), you should be good to go. That being said, the software security folks will probably appreciate hearing about this; I think Warren Anderson is the main point of contact there.

[scottcoughlin2014]

@reedessick I like this idea a lot.

1.) I think Warren Anderson should be contacted and I will do so

2.) I think your idea will take away what the gravityspytools server is authenticated to do or not do completely which is a really smart idea in general because although it may be slightly more "user friendly" it does mean that gravityspytools has to be monitored closer/bogged down in authentication more than is probably desirable.

3.) it should be easy to take the first go I have written here locally and rearrange it so it is an easy little executable available via detcharsoft or the like.

4.) also since there is already ldvw ability to generate 1 second spectrograms I will try to make it compatible with that as well.

[scottcoughlin2014]

@reedessick

I just remembered that I actually have a webserver that is already behind LIGO authentication

https://gravityspy.ciera.northwestern.edu/

It is hosting an app called Kibana which I had hoped to be a powerful interactive gravityspy tool but I just have never gotten to fully deploying it and I think having LIGO specific tools on this server and then the public version on the other server may work!

I wont get to emailing Warren or thinking super critically about this until next week.

Scotty

[reedessick]

The final implementation choice is obviously up to you, but I do think it could be fun to use an API which allowed me to upload an arbitrary image and look for similar images. Something similar in spirit to this Google thing I found.

There are probably benefits to outreach for such a tool, but they may not justify the time required to implement it.

I'm happy as long as there's something in place by O3 that can be plugged into the Data Quality Report for GraceDb.