✨ Add OpenAPI v3 schema support to the Virtual Workspace framework

[xmudrii]

Summary

This PR continues work done in #3059. Huge thanks to @palnabarun for all the work on this feature so far!

This PR adds OpenAPI v3 schema support to the Virtual Workspace dynamic framework.

A new openAPIHandler has been implemented that utilizes the kube-openapi library to generate and serve the OpenAPI v3 specs. The specs are generated for the following routes:

• /api and /apis - information about API groups
• /apis/<group> - information about API versions
• /api/v1 and /apis/<group>/<version> - specs for the concrete GroupVersion

Example calls:

# The root spec containing all other OpenAPI v3 URLs
k get --raw '/services/apiexport/<Cluster ID>/<APIExport Name>/clusters/<Cluster ID>/openapi/v3

# Information about API groups
k get --raw '/services/apiexport/<Cluster ID>/<APIExport Name>/clusters/<Cluster ID>/openapi/v3/apis

# Specs for APIs in stable.example.com/v1
k get --raw '/services/apiexport/<Cluster ID>/<APIExport Name>/clusters/<Cluster ID>/openapi/v3/stable.example.com/v1

Specs are generated for all APIDefintions available in that virtual workspace. In other words, everything that's served by GetAPIDefinitionSet function is available in the OpenAPI v3 spec in that virtual workspace.

The implementation is a bit different to what the regular API servers are doing due to the way how are virtual workspaces implemented. The regular API servers are building the OpenAPI v3 specs from web services registered in that API server. However, this is not an option for the virtual workspaces API server because it only has three web services registered: /api, /apis, and /version. None of these three web services have any subroute registered, so there's no information about e.g. /apis/<group> or /apis/<group>/<version>. That makes generating the OpenAPI v3 specs using the "usual approach" impossible.

I mitigated that by:

• Generating specs for /api, /apis/, and /version from the respective web services
• Utilizing API discovery to generate web services for each available /apis/<group>, and then generate the OpenAPI v3 specs for these web services
• Converting the available APIDefinitions to CustomResourceDefintions, and then using the appropriate function from the kube-openapi library to generate the specs

At the end, all these specs are merged into one and served. This is heavily inspired by #3246

This is also a bit different than idea in #3059, as #3059 tried to manually implement each OpenAPI v3 route, while this heavily relies on the kube-openapi library.

To make this more performant, a simple caching solution has been implemented. I'm generating a hash out of the APIDefinition's spec, and then use it to generate a key. That key is used for the LRU cache where I store the OpenAPI v3 service. This is almost copy-paste of implementation from #3246.

As requested by the community, I implemented a simple way to disable the OpenAPI v3 generation by setting the SkipOpenAPIInstallation extra configuration option to true.

Finally, unlike the original implementation in #3059, this implementation doesn't tap into the CreateServingInfoFor function. Thinking about that, I think it would have been possible and it could have enabled more effective caching, but because of the complexity, I decided to go with a "simpler" initial implementation, and we can consider doing this in the future.

The only missing part are E2E tests. This is something that I plan on adding later, potentially as a follow up PR if that's okay.

Related issue(s)

Fixes #3242

Release Notes

Add OpenAPI v3 schema support to the Virtual Workspace framework

[kcp-ci-bot]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[kcp-ci-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign clubanderson for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[sttts]

/retest

e2e-sharded flake real?

[xmudrii]

@sttts I believe it's a flake, it's now green and I didn't touch anything related to that (I think). I'm now working on adding an E2E test for this feature too and it appears to be pretty stable (I still didn't push it, will likely do it tomorrow).

[sttts]

this is a heavy operation, isn't it? Shouldn't we lock the saved key somehow and let other parties with the same interest wait? This even applies to one client collecting specs for example.

A common pattern for that: store an item struct in the lru cache with a wait channel:

type item struct {
  done <-chan struct{}
  err error
  service http.Handler
}

Every cache hit will wait for done before accessing err or service. The go routine owning this will add it to the lru cache before starting the heavy work. It will set err or service and then close done to wake all other waiters up.

[sttts]

This needs an e2e test.

[sttts]

We could also have a ttl cache 🤔

[embik]

Just for my understanding here to match it with the PR description: Your intention was that developers of virtual workspaces that use the virtual framework can disable OpenAPI, not that kcp admins can disable OpenAPI on their default kcp VWs, right?

[xmudrii]

To put it the other way around, what do we actually want and/or need more? My understanding of this implementation is that it's more towards those who run virtual workspaces, i.e. kcp-admins. Do we want both options?