Indexing a Turtle file fails (because of regex for end of statement not being matched in parsed chunk?)

[LorenzBuehmann]

Hi Qlever folks,

before raising an issue I'm asking here because I might be doing something wrong ...

Setup:

• latest Qlever Docker pulled via Qlever CLI
• file to index is a 5GB Turtle file, nothing special but it does make use of Turtle syntax of course, which also means syntactic sugar to omit redundant subjects resp. subject-predicate mentions.

Qlever config file does nothing more than a cat of the uncompressed file and using a batch size of 500k

Output of qlever index is

Executing "index":

docker run -it --rm -u 10026:10000 -v /etc/localtime:/etc/localtime:ro -v /home/LorenzBuehmann/tools/qlever:/index -w /index --entrypoint bash --name qlever.tiva.index-build adfreiburg/qlever -c "ulimit -Sn 1048576; cat tiva/exgr_bsci_2018_updated_1000_2.ttl | IndexBuilderMain -F ttl -f - -i tiva -s tiva.settings.json --stxxl-memory-gb 10 | tee tiva.index-log.txt"

2023-07-06 17:09:28.677	- INFO:  QLever IndexBuilder, compiled on Sat Jun 17 20:22:30 UTC 2023 using git hash 04cccc
2023-07-06 17:09:28.678	- INFO:  You specified the input format: TTL
2023-07-06 17:09:28.678	- INFO:  Locale was not specified in settings file, default is en_US
2023-07-06 17:09:28.678	- INFO:  You specified "locale = en_US" and "ignore-punctuation = 0"
2023-07-06 17:09:28.678	- INFO:  You specified "num-triples-per-batch = 500,000", choose a lower value if the index builder runs out of memory
2023-07-06 17:09:28.678	- INFO:  Integers that cannot be represented by QLever will throw an exception (this is the default behavior)
2023-07-06 17:09:28.678	- INFO:  Processing input triples from /dev/stdin ...
2023-07-06 17:09:28.916	- ERROR: The regex which marks the end of a statement was not found at all within a single batch that was not the last one. Please increase the FILE_BUFFER_SIZE or choose a different parser

I get the impression that the parser does chunk (100MB) read the file, and I got unlucky with a split end which hasn't closed the last Turtle statement? If so, that would be rather inconvenient for users, right? If that's not the reason for my problem, then I'm stuck atm.

Any suggestions or hints how to proceed? Anything else I could provide here?
Thanks in advance.

Cheers,
Lorenz

[hannahbast]

Hi Lorenz, can you try "ascii-prefixes-only": false for SETTINGS_JSON in the Qleverfile or directly in the settings.json file.

And have you checked that turtle --check from the Apache Jena libraries runs through without error?

[LorenzBuehmann]

Hi Hannah,

tried with your suggestion, same error.

For completeness, here are the current indexer settings:

# Indexer settings
DB                = "tiva"
RDF_FILES         = "${DB}.ttl"
CAT_FILES         = "cat tiva/exgr_bsci_2018_updated_1000_2.ttl"
WITH_TEXT_INDEX   = false
SETTINGS_JSON     = '{ "num-triples-per-batch": 500000, "ascii-prefixes-only": false }'
#GET_DATA_CMD      = "Write a one-liner here for how obtain your data (or comment this line out)"
INDEX_DESCRIPTION = "RDF data for ${DB}"
STXXL_MEMORY_GB   = 10

Checked in advance already with Jena riot, but to make check I ran again

turtle --sink --check tiva/exgr_bsci_2018_updated_1000_2.ttl

which terminates with no parse errors.

I guess it's rather hard to reproduce my issue, I mean, would think it could happen that your Turtle parser fails because of some unfortunate splits? From source code I can see that the error is in a parallel parser, and I assume it tries to process chunks in parallel, maybe in my file the boundaries of a statement occur in different chunks?

Here is also a sample of the data:

<https://data.coypu.org/trade/tiva/origin_of_value_added_in_gross_exports/HRV_D36T39_FIN_D20_2018_9.0> a tiva:ExgrBsci;
                coy:hasValue "9.0"^^xsd:decimal;
                tiva:hasExport <https://data.coypu.org/trade/tiva/exports/HRV_D36T39_FIN_D20_2018_9.0>;
                tiva:hasValueAddedOrigin <https://data.coypu.org/trade/tiva/value_added_origin/HRV_D36T39_FIN_D20_2018_9.0>;
                coy:hasYear "2018"^^xsd:gYear.

I hope your parser doesn't fail because of the dot char in the URI, e.g. https://data.coypu.org/trade/tiva/value_added_origin/HRV_D36T39_FIN_D20_2018_9.0 and a dot being the Turtle statement terminator.

[joka921]

Hi Lorenz,
Indeed we by default (maybe it shouldn't be...) we parse in parallel which puts some assumptions on the input, Most notably that you frequently find the regex \. *\\n (A dot followed by optional whitespace and a newline) and that this regex also marks the end of a triple.
This means that it only works for inputs that

• Have an ending of a triple group every now and then (every dozens of MBs at least) that ends with a .
• There is a newline after this dot.
• There are no multiline string literals with unescaped newline characters which could lead to false positives.

(The sample of your data looks unsuspicous in that way, but maybe there is somewhere else). Does your file use Linux style line breaks (\n and not \r\n?)

Note that this is a strict subset of the Turtle standard, most notably you can write your whole file into a single line and still be valid, but this wouldn't work anymore.
So the approach would be to use a different parser, that parses the file serially. Unfortunately I just saw, that there is currently no command line option to choose this parser.
If you compile QLever from sources, you can replace TurtleParallelParser by TurtleStreamParser inside src/index/IndexImpl.cpp (lines 80 - 100, 2 occurences).

If you just use a downloaded Docker container, I will set up a fix soon to expose this option, thanks for pointing this out.

[LorenzBuehmann]

Hi Johannes,

thanks for the insights.

I checked the file

• Have an ending of a triple group every now and then (every dozens of MBs at least) that ends with a .
• There is a newline after this dot.
• There are no multiline string literals with unescaped newline characters which could lead to false positives.

I didn't generate this file from CSV, but I know who and it looks like it was a Windows user ...

file -k tiva/exgr_bsci_2018_updated_1000_2.ttl

and it turns out that is indeed DOS/Window CRLF

tiva/exgr_bsci_2018_updated_1000_2.ttl: ASCII text, with CRLF line terminators

So I did a

dos2unix -n tiva/exgr_bsci_2018_updated_1000_2.ttl tiva/exgr_bsci_2018_updated_1000_2_new.ttl

and now the file can be parsed and indexed.
Looks like I have to do this for all files I have to load -> good time to get some coffee while waiting.

---

Maybe we could document this somewhere, though I know it's rather hind to find good places. Maybe as a hint at a place for supported formats or something, such that the rare group of scientific Windows users and also all others are aware of this?

Tnak you very much for the fast support! Good job @hannahbast and @joka921 !

[joka921]

Thanks for trying this out and finding this subtlety. At least I can think of the following TODOs:

• Fix this regex, s.t. it supports all kinds of whitespaces and newline flavors, and not just "space" and \n.
• When someone gets this exact error, then they get exactly the information that I gave you about the exact requirements.
• This information (maybe in a shorter form) is also presented at the beginning, when we know that this parallel parser is chosen.
• We need a way to choose the non-parallel parser which doesn't have those restrictions, because sometimes it might not be feasible to fix the file (e.g. if there are in fact multiline literals, then this is much harder than fixing line endings).