question on packaging missing dependencies

[malcolmm83]

This package looks really nice and fills a critical need. I see from the demo that it works well on DataBricks. I'm hoping you can offer some suggestions on how I get it going in my local environment. Might be helpful for some other folks as well. I've got my own Kubernetes cluster, and I've been deploying with this helm chart https://github.com/helm/charts/tree/master/stable/spark . It includes Spark and Zeppelin. The Zeppelin container uses Conda, but the Spark containers don't. The ability to deploy python libraries at runtime on the cluster is pretty limited, and any manual configurations on running containers are lost when you redeploy. I've tried to build and zip all the dependencies using sc.addPyFile and that works until I get to pandas which has some C dependencies.

ImportError: C extension: No module named tslib not built.

I can continue down that path, try to find a spark image that includes the right version of Conda, or build my own image. Any suggestions before I chew up a lot of time chasing this?