Loop path queries: imjs silently swallows them

[julie-sullivan]

See intermine/intermine#2053

Try running this query in flymine:

<query name="" model="genomic" view="Gene.symbol Gene.homologues.homologue.symbol" longDescription="" sortOrder="Gene.symbol asc">
  <constraint path="Gene" op="=" loopPath="Gene.homologues.homologue"/>
</query>

You get an error. However you don't get an error in MouseMine (older version) or in FlyMine's iodocs (doesn't use imjs). The error looks like it's dropped the loopPath value, and so we get an invalid XML error.

It doesn't look like imjs has a test for loopPaths. Can you add one?

This is higher priority because loopPaths aren't working in the UI right now.

[yochannah]

Just to clarify - do we think the bug is in InterMine core / java and we just need to test it, or do we also think there is a bug in imjs that needs to be fixed?

[julie-sullivan]

no idea. I am only desperately trying to shift blame.

Having a passing imjs loop query test would help though, even if it's not an imjs bug.

Java has passing loop query tests. iodocs works (no imjs), UI does not (imjs). So imjs is a suspect. However, loopqueries were working last summer (at least) but I don't see any imjs changes that could have caused this issue.

[yochannah]

Okay, that's good to know. We might also need to broaden the scope to im-tables depending on what investigations reveal

[yochannah]

@LakshSingla Would you be able to take a look at this? It might be that this is (or isn't) imjs or im-tables. Let me know if anything isn't clear or if it seems like it's a problem on the intermine side :)

[LakshSingla]

According to me, loop constraints are still functional in imjs and InterMine. I tried running the following code and it works on REPL. Can you point me to the webapp's code so I can take a look at it?

const intermine = require('imjs');

const flymine = new intermine.Service({ root: 'www.flymine.org/query' });
const query = {
  from: 'Gene',
  select: [
    'exons.symbol',
    'chromosome.primaryIdentifier',
    'exons.chromosomeLocation.start',
    'exons.chromosomeLocation.end'
  ],
  where: [
    ["Gene.homologues.homologue", "=", "Gene"]
  ]
};

flymine.rows(query).then(function (rows) {
  console.log("No. of exons: " + rows.length);
  //Print first 100 rows, if present
  for (let i = 0; i < 100; ++i)
    console.log("[" + rows[i][0] + "] " + rows[i][1] + ":" + rows[i][2] + ".." + rows[i][3])
});

Following is the output of the given query:

No. of exons: 7
[eIF-2gamma:1] 3R:15257844..15258442
[eIF-2gamma:2] 3R:15257984..15258442
[eIF-2gamma:3] 3R:15258510..15259383
[eIF-2gamma:4] 3R:15259450..15259686
[eIF-2gamma:5] 3R:15262547..15262721
[eIF-2gamma:6] 3R:15262797..15262937
[eIF-2gamma:7] 3R:15263282..15263458

PS: Am I able to correctly understand the problem ?

[julie-sullivan]

@LakshSingla That's great! What I am worried about is this, I searched GitHub and don't see 'loopPath' anywhere in imjs:

We couldn’t find any code matching 'loopPath' in intermine/imjs

So how do you convert the pathqueries you get from the webapp to be loop queries? If you don't ever handle the loopPath param?

Here's an example query:

<query name="" model="genomic" view="Gene.symbol Gene.homologues.homologue.symbol" longDescription="" sortOrder="Gene.symbol asc">
  <constraint path="Gene" op="=" loopPath="Gene.homologues.homologue"/>
</query>

Shouldn't loopPath be in this list: https://github.com/intermine/imjs/blob/dev/src/query.coffee#L123

(sorry if a stupid question, i have no idea how imjs works!!)

[julie-sullivan]

(Also loop queries have worked for a long time so this is a recent bug.)

[julie-sullivan]

So the problem isn't running a Loop Query in imjs, it's translating from the XML that seems to be the problem.

Run the above query in flymine, and you will get an error saying the query looks like this:

<query name="" model="genomic" view="Gene.symbol Gene.homologues.homologue.symbol" longDescription="" sortOrder="Gene.symbol asc">
  <constraint path="Gene" op="="/>
</query>

The loopPath param is being lost somewhere.

[julie-sullivan]

Error: "XML is not well formatted. Got <query model="genomic" view="Gene.symbol Gene.homologues.homologue.symbol Gene.id" sortOrder="Gene.symbol ASC" ><constraint path="Gene" op="=" code="A" /></query>."
    blocking http://cdn.intermine.org/js/intermine/imjs/3.15.0/im.js:139
    emit http://cdn.intermine.org/js/intermine/imjs/3.15.0/im.js:8281
    handle http://cdn.intermine.org/js/intermine/imjs/3.15.0/im.js:13263
    onreadystatechange http://cdn.intermine.org/js/intermine/imjs/3.15.0/im.js:13026
im.js:3128:12
    DEFAULT_ERROR_HANDLER http://cdn.intermine.org/js/intermine/imjs/3.15.0/im.js:3128
    lib$es6$promise$$internal$$tryCatch http://cdn.intermine.org/js/intermine/imjs/3.15.0/im.js:6264
    lib$es6$promise$$internal$$invokeCallback http://cdn.intermine.org/js/intermine/imjs/3.15.0/im.js:6276
    lib$es6$promise$$internal$$publish http://cdn.intermine.org/js/intermine/imjs/3.15.0/im.js:6247
    lib$es6$promise$$internal$$publishRejection http://cdn.intermine.org/js/intermine/imjs/3.15.0/im.js:6197
    lib$es6$promise$asap$$flush http://cdn.intermine.org/js/intermine/imjs/3.15.0/im.js:6058

[julie-sullivan]

That first error line is parsing a JSON query. Is that what imjs gets from the webapp? Or do you get XML then translate to JSON?

[LakshSingla]

Sorry for the late response!! I have been investigatingimjs and it seems that converting the query to XML back is causing the loopPath to drop. The query works only if I use the simple object (as above) in the Service class. Creating Query from the simple object (as shown below) causes the shown error. So I think it is a problem in the imjs library only. Adding a function which checks for such constraints would be helpful. Taking a look at the code for the web app might be helpful, to see how it deals with the XML query. Can you tell me where it is located?

const q = new intermine.Query(query)
console.log(q.toXML())

Output of following piece of code (illegal XML)

<query view="Gene.exons.symbol Gene.chromosome.primaryIdentifier Gene.exons.chromosomeLocation.start Gene.exons.chromosomeLocation.end" >
  <constraint path="Gene.homologues.homologue" op="=" value="Gene" />
</query>

[julie-sullivan]

READING XML

PathQueryHandler.java parses the XML and creates a PathConstraintLoop object if it finds a loopPath.

WRITING XML

Then when the XML is written out, if there is a PathConstraintLoop, a loopPath attribute is set. See PathQueryBinding.java

That's how we end up with the valid query with a loopPath attribute:

<query name="" model="genomic" view="Gene.symbol Gene.homologues.homologue.symbol" longDescription="" sortOrder="Gene.symbol asc">
  <constraint path="Gene" op="=" loopPath="Gene.homologues.homologue"/>
</query>

Does that help?

Thanks so much!! Great detective work.

[LakshSingla]

Ohh!! Lemme see what can be done from the imjs side to rectify the issue. BTW can you also show me the front end code of the webapp (flymine), where the XML query fails? I want to see how the imjs library fits between the XML and the backend service.

[julie-sullivan]

Import that query into beta FlyMine and click on Submit.

You are now in the "query builder" with a legal query. Scroll down, and on the right is a big green "Show Results" button. Under that is a link that says "export xml". If you click on that, you will see the XML is still correct.

Now click on the big green button. ERROR!

XML is not well formatted. Got <query model="genomic" view="Gene.symbol Gene.homologues.homologue.symbol" sortOrder="Gene.symbol ASC" ><constraint path="Gene" op="=" code="A" /></query>.

XML is now wrong.