Sqlite vs DuckDb

[MyraBaba]

Hi,
@wagga40
would you consider / possible to use duckDb instead sqlite ?

for 500MB json :

DuckDB ingestion time: 1.3920 seconds
DuckDB memory usage: 12.11 MB
SQLite ingestion time: 14.3287 seconds
SQLite memory usage: 1105.72 MB

[wagga40]

Why not.

The most important thing to understand about Zircolite is that it is based on a pysigma backend instead of reimplementing an specific engine. Which mean that if we use something other than SQLite, we may need to create a new backend.

To be honest I did not have a thorough look at DuckDB.

The code is for now quite deeply coupled with SQLite, but feel free to submit some PR with DuckDB support.

[MyraBaba]

Will look into pysigma to better understand .

…

On 22 Oct 2024, at 23:00, Wagga ***@***.***> wrote: Why not. The most important thing to understand about Zircolite is that it is based on a pysigma backend <https://github.com/SigmaHQ/pySigma-backend-sqlite> instead of reimplementing an specific engine. Which mean that if we use something other than SQLite, we may need to create a new backend. To be honest I did not have a thorough look at DuckDB. The code is for now quite deeply coupled with SQLite, but feel free to submit some PR with DuckDB support. — Reply to this email directly, view it on GitHub <#91 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEFRZH5MVJCKP4GXP4RN2L3Z42VFDAVCNFSM6AAAAABQMTBM5CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMZQGE2DCMBRGI>. You are receiving this because you authored the thread.

[wagga40]

Please use discussions instead of issues of you have no bug or problem with Zircolite.