Bug: Buffer manager exception for simple queries on large data

[NeuralFlux]

Kùzu version

v0.8.0

What operating system are you using?

MacOS Sequoia 15.2

What happened?

I'm using the Python API. I have sample biomedical data with about 150k nodes and 1 million edges. Most queries I've tried fail with buffer manager exception. Is this expected? Am I hitting the limits of KuzuDB (for my machine)?

Sample queries that work -

MATCH (a) RETURN COUNT(a);  // count total nodes

MATCH (g:Gene) WHERE g.name CONTAINS 'kinase' RETURN g;  // property filtering for a node

Sample queries that fail -

MATCH ()-[r]->() RETURN COUNT(r);  // count total edges

MATCH (n)-[r]-(m) WHERE n.id = 'C0429886' AND m.id = 'C0015943' RETURN r;  // get all edges between 2 nodes

Some more info - I have around 30 node tables and 1800 rel tables, MacBook Pro 3 with 18GB memory

Are there known steps to reproduce?

Here's the Kuzu export in CSV format to help reproduce the errors. semmed_export.zip

[ray6080]

hi @NeuralFlux thanks for reporting this. will take a look asap.

[ray6080]

hi @NeuralFlux first, let me clarify that you're not hitting the limit of Kuzu 😄 instead this is a performance bug, and thanks for spotting this for us. The bm exception is due to a non-optimized memory usage when scanning from multiple labelled rel tables. We will optimize this away soon. 😄
As a workaround for now, can you try reduce num of threads, e.g., CALL threads=1;, or explicitly specify labels if applicable?

[NeuralFlux]

Thanks for the quick response, @ray6080 ! Limiting the number of threads did the job. I was mistaken about Kuzu's performance :)

[NeuralFlux]

If it's not proprietary, may I know how you debugged this? Just curious, it might help sort issues by myself in the future, haha.

[ray6080]

If it's not proprietary, may I know how you debugged this? Just curious, it might help sort issues by myself in the future, haha.

Ah, sorry, this slipped my mind. No, it's not proprietary at all. First, we were aware of the potential bottleneck when scanning from multiple labeled relationship tables. So, we did some profiling within our MemoryManager to track where memory is being allocated and used.

[ray6080]

@NeuralFlux this should be fixed now on master. Could you try it on our next dev build (0.8.3.dev4)?

[NeuralFlux]

@ray6080 thanks for the feedback! Sure, I will test it once it's out.

[aracardan]

Hi @NeuralFlux
I'm Ardan - the CPO of Kuzu. I'm very interested in learning more about how Kuzu is used in Life Sciences in general and bio medicine in particular. My email is ardan@kuzudb.com. If you can email me I'd love to ask you a few questions about your project and of course hear your feedback about Kuzu.

[ray6080]

hi @NeuralFlux not sure if you got the chance to test it on your side, but just want to let you know that the dev build is out, and you can pull our latest one.