Benchmarks & speeds and improvements

[MyraBaba]

Hi,

I cam a cross your project today and wll have a try.

We wonder the benchmarks of the processing 2000 sigma rules over 5gig log ie ? Do you have such benchmarks ?

How we can improve the processing speed ? We can re-write critical part in c++ if it helps ?

Best

[wagga40]

Hi,

I just ran Zircolite on 15 GB of EVTX files (Sysmon, Security etc.) :

• With multiple files and the use of GNU Parallel, It took 1 min 41 s with 2723 rules
• By using Zircolite on all the files at once, it took 9 min 54s with 2723 rules (from the sigma official repository) on Python 3.10 (newer version of Python will be faster).

For the same dataset, Hayabusa took 8 min. Both tools use alot of memory.

The main part of Zircolite (the detection process) is only performed on one CPU Core. I am currently working on a way to speed things up by using all the cores.

[MyraBaba]

Almost same as hayabusa .. Below 15 min is not bad at all. The memory usage is high 15GB EVTX ~30GB - 45GB memory I will look closely the code. I assume only one python file. Best

…

On 21 Oct 2024, at 18:46, Wagga ***@***.***> wrote: Hi, I just ran Zircolite on 15 GB of EVTX files (Sysmon, Security etc.) : With multiple files and the use of GNU Parallel, It took 1 min 41 s with 2723 rules By using Zircolite on all the files at once, it took 9 min 54s with 2723 rules (from the sigma official repository) on Python 3.10 (newer version of Python will be faster). For the same dataset, Hayabusa took 8 min. Both tools use alot of memory. The main part of Zircolite (the detection process) is only performed on one CPU Core. I am currently working on a way to speed things up by using all the cores. — Reply to this email directly, view it on GitHub <#90 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEFRZHZQURQZLGUSLMAUPSLZ4UOUDAVCNFSM6AAAAABQJ53CJSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMRXGA2TSMJZGY>. You are receiving this because you authored the thread.

[wagga40]

Thanks.

The memory usage is high 15GB EVTX ~30GB - 45GB memory

For the aforementioned test, without GNU Parallel, memory usage peaked at 25 GB.

[wagga40]

Just to let you know that the future version of zircolite takes 1 min 51 s to execute with the same 15 GB dataset on the very same computer.

[MyraBaba]

@wagga40 is it 1 core or multiple core result 1 min 51 s ?

sqlite insert looks last longer..

[wagga40]

multiple cores.

[wagga40]

sqlite insert looks last longer..

You should not focus to much on SQLite Insertion, on the actual version (single core) for the same dataset takes only 25 secs...

Most of the execution time is on executing rules.

[MyraBaba]

When are you going to release new version :) ?

…

On 25 Oct 2024, at 17:55, Wagga ***@***.***> wrote: sqlite insert looks last longer.. You should not focus to much on SQLite Insertion, on the actual version (single core) for the same dataset takes only 25 secs... — Reply to this email directly, view it on GitHub <#90 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEFRZH6NM3TEGTXCGZ75PCLZ5JLU7AVCNFSM6AAAAABQJ53CJSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMZYGA2DEMZSHE>. You are receiving this because you authored the thread.

[wagga40]

I still have some things I need to adjust but I think this week end.