SPARQL UPDATE error, body limit exceeded.

[aindlq]

Is there a way to increase request body limit for SPARQL UPDATE requests?

Qlever fails to process INSERT DATA request with ~4000 non prefixed statements.

To reproduce download https://gist.githubusercontent.com/aindlq/8b2588b613eddf8e8fe76900e98d863f/raw/ebfd4a20276bccb8c08bbc4f34389b9296688a69/input.sparql and then execute:

curl -X POST -H "Content-Type: application/x-www-form-urlencoded" -H "Authorization: Bearer yourToken" --data-urlencode "update@input.sparql" http://localhost:7020

error:

ERROR: body limit exceeded [beast.http:9 at /usr/include/boost/beast/http/impl/basic_parser.ipp:467:13 in function 'void boost::beast::http::basic_parser::finish_header(boost::beast::error_code&, std::true_type)']: body limit exceeded

[hannahbast]

@aindlq We had exactly that problem two weeks ago and discussed solutions.

1. Increasing the body limit is easy: HOW TO increase the request size limit #1762
2. The next problem is the limit on the size of the command line; this can be solved using curl's @ feature, like in your example
3. The next problem is the SPARQL parser: the grammar for parsing a list of triples or quads is recursive https://github.com/ad-freiburg/qlever/blob/master/src/parser/sparqlParser/generated/SparqlAutomatic.g4#L194-L198, hence each new triple is a recursive call in the code and the call stack overflows when there are very many triples
4. Even what is solved (by changing the grammar or the parsing), the SPARQL parser is very slow for many triples

The solution we envisioned is to parse large sets of update triples using on of the parser we use for parsing the input. In particular, those are fast and can also parse in parallel, with a speed of 4 M triples / second and more.

@Qup42 Did I forget anything?

[aindlq]

@hannahbast thanks! In our use case increase of request size should do the job for now.

[hannahbast]

@aindlq OK, let me turn this into a proper PR with an option to make the request size configurable

[Qup42]

An addition to the curl options: The option --data-binary is the way to go for large payloads. Assuming the SPARQL update or Graph Store RDF payload is in a file foo, use curl --data-binary @foo ....

• --data-urlencode only works for relatively small payloads (<200k lines, 30M)
• -d/--data works for larger payloads (<1.8M lines, 323M)
• --data-binary works for even larger payloads (though the file is loaded into memory, which limits the size). The difference between --data-binary and --data is that the latter strips out carriage returns, newlines and null bytes, while the former changes nothing.