Best practices of converting multiple .json files

[goonhoon]

Hi!

I currently work with a large contract dataset and usually:

1. annotate each file (a whole contract) in paragraph-sized units
2. export to .json
3. every individual .json runs through a .json to .spacy convertor
4. the model is then trained on a directory of .spacy files

Does anyone recommend a clever way to speed up step 3.? Is it not recommended to convert multiple .json documents into one .spacy file?

Best,
AKH

[adrianeboyd]

Typically 1) and 4) are the time-consuming steps and although converting JSON files might take a few minutes, it's not expected to be a huge factor overall in a project?

Is there a particular convert command that you're using? If you're doing something during your conversion like adding sentence boundaries with a model, there might be ways to speed this up or switch to a faster alternative.

With a custom converter, you can convert JSON to DocBin/.spacy however you'd like, combining multiple JSON files if that's faster or preferred for some reason.

For spacy train it shouldn't make a very big difference whether you have one .spacy file or multiple .spacy files in a directory for the same data.