zen4R deposit record, multiple confusions with settings and strange behaviours

[kikislater]

Hi,

I played with geoflow and zen4R and I have some questions.

First of all is it will be interesting to add some workflows

• The first one from here : https://github.com/r-geoflow/geoflow/blob/master/inst/extdata/workflows/config_metadata_gsheets_zenodo.json doesn't work.
  Uploading a deposit needs at least one communties. By default NA is applied whereas NA doesn't exists in regular zenodo. I created one in sandbox but it will be removed due to sandbox reset ...
  https://sandbox.zenodo.org/communities/na

• So below is my working configuration to upload and send a deposit on zenodo and keep it as draft (not published):
  https://github.com/IRDG2OI/geoflow-g2oi/blob/sylvain/Zenodo.json

Then I tried others configurations like :

• updating my draft (attached csv are in same repository) :
  https://github.com/IRDG2OI/geoflow-g2oi/blob/sylvain/20230811-183228_zenodo_update-config.json

• publish deposit then updating record which differ from draft configuration:
  https://github.com/IRDG2OI/geoflow-g2oi/blob/sylvain/20230811-183228_zenodo_update-published-config.json
  Related isssue explaining in depth : [zen4R_deposit_record] When depositWithFiles is false and update_metadata is true, metadata are not updated #345

• adding a new version with complete change of title and description, uploading new files:
  https://github.com/IRDG2OI/geoflow-g2oi/blob/sylvain/20230811-183228_zenodo_processed-config.json
  So this configuration doesn't work at first launch. It returns one error from console:

Error in upload_file(path) : file.exists(path) n'est pas TRUE
Erreur dans withVisible(...elt(i)) : 
  Workflow failed during execution, check logs at: job-logs.txt

with bucket url in log and nothing more interesting. So again update_files and depositWithFiles needs to be set to true otherwise nothing happen. file.exists in my understanding check in repository and tried to upload updated file on the remote server. Finally deleteOldFiles:true does nothing and fail the process. Even with deleteOldFiles:false, files are removed from newversion.
If we change nothing and rerun the same workflow with same csv and same json, it works.

Regarding these scenarios, could you help me understand what is wrong or what works. Honestly, I'm totally lost, some of the logic behind the options doesn't seem intuitive (update_files, deleteOldFiles ...).

@juldebar

[eblondel]

As first stage, I need to perform a full review of the zen4R API interface and underlying action in geoflow. Some work was funded and done in the past, but through years there were changes starting with the Zenodo API, and the work needs some adaptive maintenance. Some was done already in zen4R, but no maintenance was done on geoflow because this stayed orphaned let's say, without any request for support.

Regarding the depositWithFiles, you need to be careful, because it's not really an option, rather an internal param used in actions that are tagged as generic uploaders. It's the case of zenodo action, but also dataverse and dataone. The fact it is set as option is confusing, and part of the work I have to do here is to distinguish well what can be set by users as option.

For generic uploaders, there is a specific and more complex behavior in the sense they are generators of PIDs (of different type depending on the action, a doi in case of Zenodo and Dataverse, a packageId in case of Dataone). You can see which actions are PID generators, ie generic uploaders, with geoflow::list_actions()
The behavior for that is that if a generic uploader is set as action, we are going to trigger it twice:

• a first time at the beginning (ideally as 1st action) where we are going to generate the PID (eg. DOI) that we also want to propagate in underlying actions (eg. recycle DOI in ISO 19115). For now there is control to make sure this action is position as first (Relates to Investigate the question of workflow actions order #156 which approaches the question of workflow actions ordering). At this stage we expect depositWithFiles be FALSE, the role of this first action trigger is essentially to create the PID.
• a second time where we are going to deposit the files (hence geoflow set depositWithFiles to TRUE), but also where we are going to update the metadata, since the entity might have been enriched during the workflow actions (eg. new relations), and we want to inherit this in the final metadata that is deposited on Zenodo.

All options we have then were implemented to cover requirements, and have rationale (before trying to be intuitive) and their role is documented through action utils, see geoflow::list_action_options("zen4R-deposit-record"). For those you mention:

• update_files it allows eventually not to redo the the files upload when you just want to run a metadata update over a deposit
• deleteOldFiles, which allows deactivating the deletion of files, to cover the need where we want to append files and keep the existing ones.

[kikislater]

Ok this explain some mysterious behaviours.
So if I wanted to create new version then upload some files before publishing I ended with these actions:

    "actions": [
        {
            "id": "zen4R-deposit-record",
            "options": {
                "update_files": true,
                "communities": "uav",
                "depositWithFiles": false,
                "publish": false,
                "update_metadata": true,
                "strategy": "newversion",
                "zipEachDataFile": false,
                "deleteOldFiles": false
            },
            "run": true
        },
        {
            "id": "zen4R-deposit-record",
            "options": {
                "update_files": true,
                "communities": "uav",
                "depositWithFiles": true,
                "publish": false,
                "update_metadata": true,
                "strategy": "newversion",
                "zipEachDataFile": false,
                "deleteOldFiles": false
            },
            "run": true
        }        
    ]