feat(batch-jobs): Add step to pass API server env vars to spark jobs

[deadlycoconuts]

Description

This simple PR simply adds a tiny feature to allow the API server to pass certain pre-configured environment variable values from its environment to the Spark drivers and executors (for batch prediction) that it spins up.

Modifications

• api/batch/resource.go - Addition of a step to add pre-configured API server environment variables to the Spark driver/executor manifest
• api/config/batch.go - Addition of a new config field to specify the environment values mentioned above

Tests

Checklist

• Added PR label
• Added unit test, integration, and/or e2e tests
• Tested locally
• Updated documentation
• Update Swagger spec if the PR introduce API changes
• Regenerated Golang and Python client if the PR introduces API changes

Release Notes

NONE

[vinoth-gojek]

hi @deadlycoconuts the changes lgtm!

could you also share some background on why we need to pass this env vars to spark jobs? this will help me visualise the changes in the e2e flow

[deadlycoconuts]

Thanks for the quick review @vinoth-gojek !

could you also share some background on why we need to pass this env vars to spark jobs? this will help me visualise the changes in the e2e flow

Ahh, the main motivation was to allow the API server to pass credentials (needed to access external stuff) to the Spark executors/drivers which may already exist as environment variables in the API server's environment, instead of having to specify these credentials explicitly again as configuration for the spark jobs.

[deadlycoconuts]

I'll be merging this PR now! I've just added a couple of unit tests since then but nothing else changed. Thanks a lot for the review once again!