Game plan, thoughts and discussions

[mguidoti]

Ok, so, this is a summary of what I discussed with @slint on skype today - and this discussion is a catch up of what we previously discussed in the last Arcadia Sprint meeting at CERN (Feb/2020). This is a Plazi-Zenodo join effort that aims a complete re-do of Lycophron in order to deliver a tool that can handle any use case, not only specific ones, with more performance and reliability.

• Lycophron should have a separate module to handle the Zenodo communication;
• It should load/export data using Pandas Dataframes;
• It must be a CLI tool (Click), accepting commands as: upload, update, publish, delete, and uninstall, with some parameters to turn in/off sandbox mode, define the export file, edit sensible information (e.g. API token), and so on;
• It should use .env (python-dotenv) to keep sensible information and other eventual parameters of the tool;
• Schema-based data validation (possible libs: Pydantic, Marshmallow);
• It should be to auto-match provided columns with Zenodo fields, asking, if not an absolute match, if the user agrees before doing any API call;
• We should use Celery to 'multithreading';

First step is building the Zenodo communication module, the next step would be implementing the first commands for the CLI.

Tomorrow I'll work on setting labels, milestones and creating templates for issues, and the README (at least the skeleton).

What do you think, @slint ?

Cheers!

[slint]

• Lycophron should have a separate module to handle the Zenodo communication;

Something more of a Zenodo "client", where one can do e.g.:

client = ZenodoClient(token='<your-api-token>')

record = client.records.get('12345')
record['title']
# 'My record title'

deposit = client.deposits.create()
deposit.metadata.update({'title': 'some title', ... })  # updates the metadat
deposit.files.add(path='/path/to/figure.png')  # uploads a file
deposit.publish()  # publishes the deposit

• It should load/export data using Pandas Dataframes;

A lighter alternative to Pandas (since we're not doing any strictly computational work) is also openpyxl... Not sure if there are major speed/ease-of-use differences.

• It should use .env (python-dotenv) to keep sensible information and other eventual parameters of the tool;

Another helpful option for globally storing CLI application config is appdirs (which is also cross-platform).

• It should be to auto-match provided columns with Zenodo fields, asking, if not an absolute match, if the user agrees before doing any API call;

There's also the Zenodo Deposit JSON Schema we can cross-check against using the jsonschema library before submitting things.

[mguidoti]

I think you had one additional suggestion regarding schema validation, other than jsonschema, right?

[slint]

I think you had one additional suggestion regarding schema validation, other than jsonschema, right?

Yup, it was Frictionless Data. They have some nice concepts around validating complex data and it's both a tool + library, which means we can extend it with custom rules (e.g. validate license IDs, custom keywords, etc.).

It produces a nice "visual" output as well for errors, so it's a bit more user-friendly.