Merge pull request #162 from RyanQuey/feature/crowd-in-auto-ingest

Add script for downloading crowdin entries and moving them to proper directory

scripts/crowdin/.env.sample

@@ -0,0 +1,2 @@
+CROWDIN_API_TOKEN=
+STEP_CROWDIN_PROJECT_ID=10716

scripts/crowdin/.gitignore

@@ -0,0 +1,184 @@
+# This project specific stuff: 
+.env
+.env*
+!.env.sample
+step_project.json
+all_projects.json
+tmp/*/*.properties
+tmp/unzipped/**
+
+
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+#   For a library or package, you might want to ignore these files since the code is
+#   intended to run in multiple environments; otherwise, check them in:
+# .python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# UV
+#   Similar to Pipfile.lock, it is generally recommended to include uv.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#uv.lock
+
+# poetry
+#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
+#poetry.lock
+
+# pdm
+#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
+#pdm.lock
+#   pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
+#   in version control.
+#   https://pdm.fming.dev/latest/usage/project/#working-with-version-control
+.pdm.toml
+.pdm-python
+.pdm-build/
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+
+# PyCharm
+#  JetBrains specific template is maintained in a separate JetBrains.gitignore that can
+#  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
+#  and can be added to the global gitignore or merged into this file.  For a more nuclear
+#  option (not recommended) you can uncomment the following to ignore the entire idea folder.
+#.idea/
+
+# Ruff stuff:
+.ruff_cache/
+
+# PyPI configuration file
+.pypirc

scripts/crowdin/README.md

@@ -0,0 +1,110 @@
+# Description
+## Purpose
+To be ran in 
+
+## What it does
+Executes process to:
+1) Download zip file from Crowdin with translation files for all languages
+2) Unzip crowdin file 
+3) Rename files to match expected filenames for our STEP Bible process
+
+
+# Instructions
+## 1) Setup env
+### 1a) Acquire API Key
+See here to learn more: https://support.crowdin.com/enterprise/account-settings/#access-tokens
+
+Basically you just need to sign in and create a token. I named mine "STEP-Bible"
+- https://crowdin.com/settings#api-key
+
+#### Required Permissions for your API Token: 
+I created a token with read-only access to the following (probably more than I needed):
+- `Projects (List, Get, Create, Edit)`
+- `Translation Status` (probably not needed)
+
+AND read AND write access to the following: 
+- `Translations` (needed, so can execute a `build` on the translations)
+
+This is the token you'll use in step `1b` below. 
+
+### 1b) Set Env Vars
+
+
+#### OPTION #1: Set directly in your terminal
+
+```
+export CROWDIN_API_TOKEN=<your API token from previous step>
+```
+
+NOTE: Make sure you know what you're doing with this one, since if you don't set it in the right session, the env var might not carry over when you actually run the script. But in general it should work just fine as long as you're only working in a single terminal session and don't restart your computer etc. Otherwise you'll need to do this : 
+
+#### OPTION #2: Use .env file
+- First, copy the sample .env file
+    ```
+    cp .env.sample .env 
+    ```
+- Next, set the appropriate values
+    - `CROWDIN_API_TOKEN` for your crowdin API token from the previous step. 
+
+(TODO if we want to, we can use `python-dotenv` lib instead in the future)
+
+### 1c) Install Python Pre-reqs
+**If don't have it already**, install `pip install virtualenv`
+```
+pip3 install virtualenv
+```
+
+Create and activate a virtual env for this subproject
+```
+python3 -m venv venv
+source venv/bin/activate
+```
+
+Then install the requirements
+```
+pip3 install -r requirements.txt
+```
+
+## 2) Run the script
+```
+# probably just `python download.py` should work, but it wasn't on my machine so putting down full path to the venv python. 
+./venv/bin/python download_bundle.py
+```
+
+### Options:
+#### Option: Skip Download
+Will not download zip file, but just use existing zip file instead. If this is set, will not run a build either
+```
+./venv/bin/python download_bundle.py --skip-download
+```
+
+#### Option: Force Rebuild
+Will not check for existing builds, will just build a new one
+```
+./venv/bin/python download_bundle.py --force-rebuild
+```
+
+#### Option: Path
+Specify a path to put `*.properties` files. Defaults to `step/scripts/crowdin/tmp/bundle_out` 
+
+```
+./venv/bin/python download_bundle.py --path ../../step-core/src/main/resources/
+```
+
+(This is what we would do during our build process)
+
+#### Option: Skip Existing File Check
+Doesn't error out if there's a file in target dir already
+```
+./venv/bin/python download_bundle.py --skip-existing-file-check
+```
+
+
+### Use Case: Use updated `*.properties` files for project build
+Basically make sure to skip existing file check and specify the `resources` dir path, and then the script will leave files that don't have replacements, but replace all `*.properties` files that do have replacements. 
+```
+./venv/bin/python download_bundle.py --path ../../step-core/src/main/resources/ --skip-existing-file-check
+```
+
+# TODOs
+- [ ] See download_bundle.py and all the TODOs there