Add retry mechanism to many database requests

[carmichaelong]

Should help issues such as #189 and #192 that use the requests library. Requests are attempted one time and can fail if there are intermittent connectivity issues between the worker and database. This leads to errors that cannot be traced on the database, since it lost connection and cannot post to the database.

This adds a wrapper function to attempt multiple requests to these calls with an exponential backoff (e.g., attempt after 1 second, 2 seconds, 4 seconds, 16 seconds, 32 seconds).

@antoinefalisse @AlbertoCasasOrtiz I'm tagging you both (and would like a review from both) since it touches many important calls to the database. Some notes below:

Testing that's been done:

• Added tests for the new wrapper function
• Reprocessed a completed trial on dev. This, however, does not test all of the functions that have been updated.

Notes for review:

• What other testing should be done (both now, before it gets merged into dev, and also before we merge into main)? At least a full session with new data collection on dev should be done before main.
• This adds a tests folder, which required removing it from the .gitignore file. Will this mess up someone's workflow?
• Would you like to see a smaller set of changes first? The change has been applied pretty broadly at this point, but we can use a smaller set of changes first and monitor if that's helpful.

[antoinefalisse]

@carmichaelong I've wanted something like this for so long!!! Thanks. I will review by the end of the week.

[antoinefalisse]

@carmichaelong LGTM

What other testing should be done (both now, before it gets merged into dev, and also before we merge into main)? At least a full session with new data collection on dev should be done before main.

I feel like a full session with new data on dev should be sufficient

This adds a tests folder, which required removing it from the .gitignore file. Will this mess up someone's workflow?

I don't think so, old stuff.

Would you like to see a smaller set of changes first? The change has been applied pretty broadly at this point, but we can use a smaller set of changes first and monitor if that's helpful.

I think it is good as is. I double checked that you changed all the requests.

[AlbertoCasasOrtiz]

I just reviewed the code. LGTM!