Add metrics related to virtual threads

[tjquinno]

Description

Resolves #9533

New meters related to virtual thread usage

• current number of active virtual threads (disabled by default)
• total number of virtual thread starts (disabled by default)
• total number of pinned threads
• timer (with histogram) of pinned threads' durations
• number of failed attempts to submit virtual threads to platform threads

The virtual thread count meters are disabled by default for performance reasons. Enable them by setting metrics.virtual-threads.count.enabled=true in configuration, but be aware doing so can degrade the server's performance.

New meter related to platform threads

Helidon also now exposes the new system meter thread.starts which displays the total number of platform thread starts performed in the JVM since server start-up.

---

The PR adds a new MetersProvider implementation to the helidon-metrics-system-meters component. The new implementation registers for selected Java Flight Recorder events to track data related to virtual threads.

The three virtual thread meters that are enabled by default should be rare so monitoring the JFR events for them adds very little overhead. In contrast, to maintain the current number of active virtual threads and the total number of virtual thread starts the added code must register for and respond to virtual thread start and end events which can be costly. That's why those two meters--and the registration of listeners for those events from JFR--are disabled by default.

Documentation

Small additions to the SE and MP metrics guide and doc pages.

[spericas]

Some general questions @tjquinno ...

• current number of active virtual threads (disabled by default)
• total number of virtual thread starts (disabled by default)
• total number of pinned threads

Is there also a counter for platform threads?

• distribution summary (history) of pinned threads
• number of failed attempts to submit virtual threads to platform threads

Under what conditions would this counter be incremented?

[spericas]

Just a copyright update?

[tjquinno]

This is a remnant from an earlier iteration of the change I was contemplating. Not needed.

[tjquinno]

Fixed. File no longer in the changed list.

[tjquinno]

Some general questions @tjquinno ...

• current number of active virtual threads (disabled by default)
• total number of virtual thread starts (disabled by default)
• total number of pinned threads

Is there also a counter for platform threads?

The existing Helidon built-in meters created in `SystemMetersProvider.java include some platform-thread related ones:

• thread count
• daemon thread count
• maximum thread count

This JMX MBean makes a number of other thread-related values available:
https://docs.oracle.com/en/java/javase/21/docs/api/java.management/java/lang/management/ThreadMXBean.html

Probably the most useful value that's available we don't currently expose is the total started thread count. We could add that.

• distribution summary (history) of pinned threads
  (I edited the description to change "history" to "histogram" which I originally intended!)

Do you have an example?

I don't have an interesting example of the value.

Here is an example of the output, taken from the updated doc that's part of this PR (obviously with no interesting output):

  "vthreads.recentPinned": {
      "count": 0,
      "max": 0.0,
      "mean": 0.0,
      "total": 0.0,
      "p0.5": 0.0,
      "p0.75": 0.0,
      "p0.95": 0.0,
      "p0.98": 0.0,
      "p0.99": 0.0,
      "p0.999": 0.0
    },

• number of failed attempts to submit virtual threads to platform threads

Under what conditions would this counter be incremented?

Here's what the Java Flight Recorded doc says about the value it reports about pinned virtual threads.

• distribution summary (history) of pinned threads
• number of failed attempts to submit virtual threads to platform threads

Under what conditions would this counter be incremented?

See the previous link.

[tjquinno]

The latest push accomplished:

• Removed an unchanged file (except for copyright date) from the changed file list.
• Added the new non-virtual thread system meter thread.starts to the built-in system meters
• Changed the meter type for vthreads.recentPinned from distribution summary to timer (the two are basically equivalent except that the timer type displays an elapsed time instead of the more general data value).