exportForTimesketch.tmpl Not producing the correct CSV Fields

[mtreanor-r7]

Hi,

I'm trying to produce a use case of processing key assets for an IR in Zircolite and uploading to TS for timeline analysis, I've tried using the template with outputting to CSV and unable to have the correct header fields for datetime, message and timestamp_desc.

I understand the fields have the header required fields appended but shouldn't need to massage the csv to get it to process correctly.

Thoughts?

[mtreanor-r7]

Tested out the .json output with the template and looks like TS is indexing, just had issues with outputting to csv, I'll keep you posted on this issue.

[wagga40]

Ok thanks 👍🏻

[mtreanor-r7]

Indexing is still in progress, using another sigma tool, the csv with the required fields datetime, timestamp_desc and message was ingested quite fast, unsure if there is just a lot more data to sift through with the Zircolite output in json but I'll check tomorrow and can revert but thought I'd post an update.

It's the TS integration that we're aiming for to do analysis at scale.

[wagga40]

Whenever I can, I avoid using CSV especially with EVTX logs (there are too much fields with a lot of annoying characters that can make a parser fail). This is why I created the JSON template for Timesketch.

[mtreanor-r7]

I've just refreshed the Time Sketch after uploading the json file yesterday using the template, still not indexed, unsure which side might be impacting it. Happy to close it out if it's too complicated to test but would be interested if others here have exported their processed data to Time Sketch successfully?

[wagga40]

Thanks for your feedback.
Don't close, I will do some tests this weekend.

[mtreanor-r7]

Thank you so much, didn't want to name drop a competing sigma tool but https://github.com/Yamato-Security/hayabusa/blob/main/doc/TimesketchImport/TimesketchImport-English.md worked out of the box and indexed very fast. Hope that helps to pivot/compare potentially.

What I'm trying to do is present a use case for either/both tools for our analysis workflow for key compromised assets.

[wagga40]

No problem, they cite Zircolite on their github, I should have done the same (there is also Chainsaw).

There is no « competition » just different goals (using Python, using a sigma backend, handling auditd logs etc.). Moreover, they are a team and I am working alone (but I accept contributions 😅) on Zircolite.

[wagga40]

Hi, I did run some tests :

1. I created a new VM and installed Timesketch following these instructions : https://timesketch.org/guides/admin/install/
2. I cloned the EVTX-ATTACK-SAMPLES repository and ran Zircolite against it :

• git clone git clone https://github.com/sbousseaden/EVTX-ATTACK-SAMPLES.git
• python3 zircolite.py -e EVTX-ATTACK-SAMPLES/ -r rules/rules_windows_sysmon_full.json --template templates/exportForTimesketch.tmpl --templateOutput EVTX-ATTACK-SAMPLES-Timesketch.json

3. I loaded the EVTX-ATTACK-SAMPLES-Timesketch.json file into Timesketch
4. It took only a few seconds to index and it seems everything was fine (but I am not using Timesketch a lot)

A screen capture (mp4 - 1,5 MB) of the step 3 and 4 is available here

Note : I have just tried with a big dataset of EVTX (8,2 GB), Indexing in timesketch only took 20 seconds.

[mtreanor-r7]

Hi,

Very much appreciate this update, I replicated your instructions above and it's hanging on 'Indexing in progress..' which is possibly leading to a Timseketch version issue on our end (we might be 3-4 months behind), the manager who looks after this is on leave and will see if we can test on a more recent version once he returns.

Happy for this to be closed off and I'll reference this issue/instructions later on once we get a more recent version spun up, out of interest, I'm assuming you installed Timesketch based on the latest release?

[wagga40]

Yes, I installed the latest release.

[mtreanor-r7]

Great thanks, I may not be able to come back to you with an update until early April due to his Paid Time Off.

[wagga40]

Ok, I will leave the issue open until then.

[mtreanor-r7]

Hi,

I have an update, the solution was to rename the .json to .jsonl and it worked on our end, it most probably is a version control issue but TS makes it clunky to upgrade our production server, this will be a short term fix to just rename files.

Thank you for the support with testing this.

[wagga40]

Oh 😳… That’s quite a surprise, I will do update the docs. Thank you very much !