Helm and hardening

[srudin]

We are trying to get osm-seed to run on kubernetes using the existing helm chart. Unfortunately we have a number of security guidelines that caused hardening policies on the server such as requiring a security context other than root or making the containers readonly. Especially the latter is causing a lot of troubles as many osm containers try to modify files or even build code during startup.

I accept that these security guidelines are not relevant to everyone. I also accept that our company demands them to be followed. We are willing to spend the time to adjust the helm templates and create pull requests for changes whenever they can be integrated in a meaningful way hoping that they find their way into the main repo.

The questions I have are:

• Did someone already do any work such as trying to mount any of the images readonly that we could build on?
• Could I ask for support when we run into issues that require deeper knowledge of the internals?
• Is there any interest to have these adjustments merged into the main repo?

[batpad]

Hi @srudin - thanks much for the issue and really happy to know you might find it useful !

I think in general we'd be super happy to support and merge in work that helps with security hardening / allowing OSM-Seed to be installable in environments with stricter policies.

Whilst saying this, the requirement that images be read-only is a bit new to me and I'm not sure I'm able to grok or meaningfully think about how one might do this in the osm-seed context. If you do have clear ideas on how one might do this, super happy to discuss.

Did someone already do any work such as trying to mount any of the images readonly that we could build on?

I don't believe so. I think we definitely wanted to spend more time looking at security considerations better - providing more options in the Helm chart to set affinity rules, RBAC rules, etc., and I think would be very keen to merge code that adds that flexibility to the Helm charts. Read-only images definitely seems really tricky to me, and I'm not sure we'd be keen to merge in something that could disrupt existing workflows to support this. If you do have any links or reading material around why this security consideration exists, I'd be super happy to read so I can understand this better.

Could I ask for support when we run into issues that require deeper knowledge of the internals?

Especially as you seem able to contribute meaningfully with pull requests, absolutely :-) . None of us are exactly helm / kubernetes experts, so have also been learning as we go along -- super happy for ideas on doing things better, and very happy to clarify anything that seems unclear with the current setup.

Is there any interest to have these adjustments merged into the main repo?

As I said above, doing better security hardening of our Helm templates, and especially offering more options for cluster administrators with regards to security policies and container placements is definitely something we've wanted to do, so there, I think I'd be very keen to work together if this is something you folks feel like you can contribute to. For the read-only containers use-case, I'd have to understand the rationale behind it better, and ideally see if it can be done in a way that does not negatively impact existing workflows / can be done with a flag or so so as not to affect setups that do not need this. And currently it's very fuzzy to me how one would even accomplish this: the OSM software can be a bit weird to work with and most of the things we do at startup time is because for one reason or another, were not possible / we could not figure out how to do at docker build time. We do have this open issue to try and avoid doing the assets precompile at run-time: #25 - we would definitely be grateful for a good solution, we have just not figured one out :-) . There are several other things like modifying configuration files based on envirionment variables at run-time, etc. that I just don't know how one would achieve with a read-only file-system other than making drastic modifications to the osm-website code-base, which is something we try to avoid since it makes keeping up-to-date with upstream harder.

Super happy to continue talking, and even if we are unable to merge all the changes, I'd be super happy to talk to try and figure out a good way where you don't need to completely fork the code-base, and we could still benefit from other improvements you and your team might make that could be useful to the project.

(Just saw the other bit about requiring a security context other than root -- I think this is probably less controversial, and if you were able to make modifications that meant we did not run things as root on the container, I think we'd be very happy to work together and get those merged in as a best practice.)

Thanks again for the ticket and hope osm-seed works out for your use-case.

[srudin]

Thanks for the extended response ;-) I think we will find a good solution that benefits all of us then.

About why making containers readonly makes sense you may want to have a quick look at the post here: https://benlobaugh.medium.com/how-to-secure-docker-containers-with-a-read-only-filesystem-b27230f6efb. This is only one short example and searching the net you should easily find a number of other posts explaining why this is a suggested security guideline.

However please note that it's not me who came up with this and the other guidelines. I work for a pretty big company and some security cracks decided these are our guidelines so I have to comply no matter what I personally think about them. The good thing is that we have a support channel where I can request support from people who have specialized about these k8 topics.

Having said that I can confirm that I already have (generic) solutions for some of the problems mentioned - I might start to create PRs soon then we can discuss the changes in there. However it's going forward in very, very small steps as I need to find out and solve every single issue individually and your support might be very useful there as I am not familiar with the internals of the solution that cause these problems.

Instead of bloating up this post with a lot of different and specific information about this I suggest I will create new issues or PRs instead but add a link to all of them here. This way we have a separation per thread as well as an overview over all the threads related to these security improvements. Ok for you?

[batpad]

About why making containers readonly makes sense you may want to have a quick look at the post here: https://benlobaugh.medium.com/how-to-secure-docker-containers-with-a-read-only-filesystem-b27230f6efb.

This makes sense, but also (to me) seems a bit like one of these enterprisey security requirements that are not so practical for all use-cases :-( - like I can totally see how this is a nice idea if running many wordpress sites or so, but am extremely skeptical of the amount of work-arounds that might be needed to work with the OSM software ecosystem, most of which does expect the ability to be able to write to the file-system. I do worry that this might not really work for us / add complexity that we don't really need. But if this can happen with some options to turn on / off in the Helm chart or so, super happy to see if we can work things out in a way that wouldn't require you to fork. Just want to be upfront here a bit that I think I'd probably be a bit wary to add lots of complexity into osm-seed to support this - I am extremely sympathetic to security requirements in large orgs, and super happy to merge something in that doesn't negatively impact our existing clusters, but just to be upfront: I don't see this as a real use-case for us at the moment. (for eg. if we had the time / money to invest in greater security for osm-seed, I'd probably look at other things right now that I think would be more urgent / important than read-only containers). Of course, I totally understand your situation :-) .

Having said that I can confirm that I already have (generic) solutions for some of the problems mentioned - I might start to create PRs soon then we can discuss the changes in there. However it's going forward in very, very small steps as I need to find out and solve every single issue individually and your support might be very useful there as I am not familiar with the internals of the solution that cause these problems.

This sounds great and small steps sounds wonderful.

Instead of bloating up this post with a lot of different and specific information about this I suggest I will create new issues or PRs instead but add a link to all of them here. This way we have a separation per thread as well as an overview over all the threads related to these security improvements. Ok for you?

That sounds perfect, thank you.

Looking forward to collaborating on this. If it helps at some point to get on a quick call, happy to do so at some point.

[srudin]

extremely skeptical of the amount of work-arounds that might be needed

Know exactly what you mean but hey, I'll have to do it and you just need to approve / deny... ;-)

I think there are decent options for the readonly thing that won't mess up anything if not required, let me push some suggestions and we can discuss them there.

[srudin]

Links to PRs related to this discussion:

• Add security context with fallback #282
• Handle readonly containers #284

[srudin]

After some deeper investigations I had to come up with a different solution:

• osm-seed is too complex for what we need - a simple tile server fits our requirements
• We have a large number of security guidelines to follow. I had to realize that following all of them would be nearly impossible based on existing Docker containers and Helm charts.

So I created my own OSM tiles server according to the switch2osm documentation, then created my own Helm chart based on the company guidelines. I would have preferred to make more intensive use of open source solutions such as osm-seed but I wasn't able to fit them into our restrictive guidelines.

I apologize for all the fuss in the first place...