quark.7

.Dd $Mdocdate$
.Dt QUARK 7
.Os
.Sh NAME
.Nm quark
.Nd unified system process telemetry library
.Sh DESCRIPTION
.Nm
is a library that provides a way to retrieve and listen to process events in
linux systems.
Its main purpose is to abstract different backends and to provide a common
API for listening to system-wide events like
.Xr fork 2 ,
.Xr exec 3 ,
.Xr exit 3
and others.
.Pp
.Nm quark
not only provides an API for listening to events, but also handles ordering,
buffering and aggregation of said events.
In its most basic form, a short lived process consisting of
.Xr fork 2
+
.Xr exec 3
+
.Xr exit 3
will be aggregated into one
.Vt quark_event .
An internal process cache is also kept that can be looked up via
.Xr quark_process_lookup 3 .
.Sh QUICKSTART
Clone the repository, compile and run quark's test utility
.Xr quark-mon 8 :
.Bd -literal
$ git clone --recursive https://github.com/elastic/quark
$ cd quark
$ make
$ sudo ./quark-mon
.Pp
On another shell, create any process like:
$ ls -1 /tmp | wc -l
.Ed
.Pp
See
.Sx BUILDING
for a list of dependencies if you're having trouble building.
Also see
.Sx INCLUDED BINARIES
and
.Xr quark-mon 8 .
.Sh FEATURES
.Bl -ohang
.It Em ORDERING
.Nm
tries to guarantee event ordering as much as possible.
Ordering must be done in userland for some backends, notably anything that
uses perf-rings.
.Nm
uses two
.Em Rank Balanced Trees
for ordering and aggregation.
.Pp
The first tree is basically a priority queue, ordered by the time of the
event.
The second tree is ordered by time of the event + pid and it's used for event
aggregation.
.It Em AGGREGATION
.Nm
buffers and aggregates related events that happened close enough.
The common case is generating a single event for the triple:
.Xr fork 2 ,
.Xr exec 3 ,
.Xr exit 3 .
There are rules on what can be aggregated, and only events of the same pid are
aggregated.
For example:
.Nm
won't aggregate two
.Xr exec 3
events, otherwise we would lose the effects of the first one.
These rules will be exposed and configurable in the future.
.It Em BUFFERING
For aggregation and ordering to work,
.Nm
needs to be able to buffer events, this means holding them before presenting
them to the user.
.Nm
employs an ageing timeout that is a stepped function of the number of currently
buffered events, the more events you have, the shorter the timeout will be, so
memory can be bound.
A
.Vt quark_event
is only given to the user when it has a certain age.
From quark.c:
.Bd -literal -offset indent
/*
 * Target age is the duration in ns of how long should we hold the event in the
 * tree before processing it. It's a function of the number of items in the tree
 * and its maximum capacity:
 * from [0; 10%]    -> 1000ms
 * from [90%; 100%] -> 0ms
 * from (10%; 90%)  -> linear from 1000ms -> 100ms
 */
.Ed
.It Em ENRICHMENT
The library tries to give as much context for an event as possible.
Depending on the backend, the events we read from the kernel can be limited in
context.
.Nm
maintains an internal process table with what has been learned about the process
so far, this context is then included in each event given to the user.
The process table can also be queried, see below.
.It Em PROCESS CACHE
An internal cache of processes is kept that can be looked up via
.Xr quark_process_lookup 3 .
This cache keeps soon-to-be-purged elements for a little while so that you can
still lookup a process that just exited.
The table is initialized by scraping
.Pa /proc .
.It Em TRANSPARENCY
.Nm
tries to be as transparent as possible about what it knows, there are counters
for lost events, and each piece of information of a
.Vt quark_event
is guarded by a flag, meaning the user might get incomplete events in the case
of lost events, it's the user responsability to decide what to do with it.
.Pp
Depending on load, the user might see an event as the aggregation of multiple
events, or as independent events.
The content remains the same.
.It Em LANGUAGE BINDINGS
.Nm
is written in C, but Go bindings are also provided.
Ideally we will be able to provide bindings for other languages in the future.
.It Em MULTIPLE BACKENDS
Currently, EBPF and a kprobe-based backend are provided, but we would like
to add AUDIT support as well.
The backend in use is transparent to the user and unless specified,
.Nm
will try to use the EBPF, falling back to KPROBE if it failed.
.El
.Sh BUILDING
.Nm
can be built natively or via a container, native is preferred and depends on:
.Bl -bullet
.It
bpftool
.It
clang
.It
gnumake
.It
gcc
.It
mandoc (for docs).
.It
m4
.El
.Pp
Make sure to clone the repository recursively:
.Em "git clone --recursive" .
.Pp
.Em make
builds the repository, including
.Pa quark-mon ,
.Pa libquark_big.a
and a
.Pa libquark.a .
.Pp
.Pa libquark_big.a
includes all needed dependencies in one big archive.
This includes a
.Pa libbpf.a ,
.Pa libelf_pic.a
(from the elftoolchain project, BSD license),
and a
.Pa libz.a
(see zlib/LICENSE).
See
.Sx LINKING
to learn how to link either.
.Pp
While
.Nm
doesn't build
.Em elastic/ebpf ,
it does use the EBPF programs from that repository, only the files needed are
included in
.Nm ,
as
.Em elastic/ebpf
is quite big.
.Pp
Other useful build targets include:
.Bl -tag -width "docker-cross-arm64"
.It Em clean
Clean object files from
.Nm .
.It Em docker
Builds
.Nm quark
inside a docker container, so you don't have to worry about
having build dependencies.
.It Em docker-cross-arm64
Builds
.Nm quark
for arm64 inside a docker container.
.It Em centos7
Builds
.Nm quark
inside a centos7 docker container, useful for linking against
ancient glibc-2.17.
.It Em test
Builds and runs
.Xr quark-test 8 .
.It Em test-kernel
Runs
.Xr quark-test 8
over all kernels in
.Pa kernel_images/ .
.It Em test-all
Shortcut for test + test-kernels.
.It Em btfhub
Regenerates
.Pa btfhub.c .
Usage:
.Bd -literal
$ make btfhub BTFHUB_ARCHIVE_PATH=/my/path/to/btfhub-archive
.Ed
.It Em clean-all
Clean all object files, including the ones from
.Em libbpf ,
.Em libz
and
.Em libelf .
.It Em docs
Lints and generates all the documentation from manpages in
.Pa docs/ .
.It Em svg
Builds an SVG out of the DOT files produced by
.Xr quark-mon 8 .
.It Em README.md
Generates
.Pa README.md
out of
.Pa quark.7 .
.It Em eebpf-sync
Copies the files from EEBPF_PATH used by
.Nm .
Usage:
.Bd -literal
$ make eebpf-sync EEBPF_PATH=/my/path/to/elastic/ebpf
.Ed
.It Em initramfs.gz
Builds an initramfs file containing all quark binaries so that it can be run as
the init process on boot, useful for testing any kernel under qemu.
See
.Sx TESTING .
.El
.Pp
All the targets above can generate debug output by specifying
.Em V=1 ,
as in:
.Bd -literal
$ make V=1
.Ed
.Sh LINKING
.Bd -literal
$ cc -o myprogram myprogram.c libquark_big.a
OR
$ cc -o myprogram myprogram.c libquark.a libbpf/src/libbpf.a elftoolchain/libelf/libelf_pic.a zlib/libz.a
.Ed
.Sh TESTING
.Xr quark-test 8
is the main test utility ran by the CI, can be invoked via
.Em make test .
All tests are self-contained in this binary.
.Pp
Some included kernels can be tested in qemu via
.Em make test-kernel .
Any
.Nm quark
utility can be run on a custom kernel via the
.Pa krun.sh
script, as in:
.Bd -literal
$ make initramfs.gz
$ ./krun.sh initramfs.gz kernel-images/amd64/linux-4.18.0-553.el8_10.x86_64 quark-test -vvv
.Ed
.Pp
Note that you can pass arguments to the utility and you have to make
.Pa initramfs.gz
first.
.Sh INCLUDED BINARIES
.Xr quark-mon 8
is a program that dumps
.Vt quark_events
to stdout and can be used for demo and debugging.
It has a neat feature: can be run without priviledges, while useless in this
small program, it aims to demonstrate how a user could implement the same.
.Pp
.Xr quark-btf 8
is a program for dumping BTF information used by
.Nm .
.Pp
.Xr quark-test 8
is a program for running tests during development.
.Sh CONVENTIONS
.Bl -bullet
.It
Library calls fail with -1 unless otherwise stated, and
.Va errno
is set.
.It
Quark returns pointers to internal state, which must not be modified and/or
stored.
In the case of multithreading, these pointers should not be accessed if another
thread is driving
.Nm quark
through
.Xr quark_queue_get_event 3 .
.It
No threads are created, the library is driven solely through
.Xr quark_queue_get_event 3 .
.It
Access to a
.Vt quark_queue
must be synchronized by the user in the case of multithreading.
.El
.Sh BASIC USAGE
The ball starts with
.Xr quark_queue_open 3 .
.Pp
.Xr quark_queue_open 3
initializes a
.Vt quark_queue
which holds the majority of runtime state used by library, this includes
perf-rings, file descriptors, EBPF programs buffering data-structures and the
like.
It must be paired with a
.Xr quark_queue_close 3
on exit.
.Pp
.Xr quark_queue_get_event 3
is the main driver of the library, it does the buffering, per-ring scanning,
aggregation and event cache garbage collection.
In case there are no events it
returns NULL and the user is expected to call
.Xr quark_queue_block 3
or equivalent.
.Sh EXAMPLES
.Bd -literal -offset indent
#include <err.h>
#include <quark.h>
#include <stdio.h>

int
main(void)
{
	struct quark_queue	 	 qq;
	const struct quark_event	*qev;

	if (quark_queue_open(&qq, NULL) == -1)
		err(1, "quark_queue_open");

	for (; ;) {
		qev = quark_queue_get_event(&qq);

		/* No events, just block */
		if (qev == NULL) {
			quark_queue_block(qq);
			continue;
		}

		quark_event_dump(qev, stdout);
	}

	quark_queue_close(&qq);

	return (1);
}
.Ed
.Sh API
.Bl -tag -width "quark_queue_default_attr(3)"
.It Xr quark_queue_open 3
open a queue to receive events, initial library call.
.It Xr quark_queue_default_attr 3
get default attributes of
.Xr quark_queue_open 3 .
.It Xr quark_queue_get_event 3
get event, main library call.
.It Xr quark_process_lookup 3
lookup a process in quark's internal cache
.It Xr quark_event_dump 3
dump event, mainly a debugging utility.
.It Xr quark_queue_get_epollfd 3
get a descriptor suitable for blocking.
.It Xr quark_queue_block 3
block for an unspecified amount of time.
.It Xr quark_queue_get_stats 3
basic queue statistics.
.It Xr quark_queue_close 3
close a queue.
.El
.Sh FURTHER READING
.Xr quark_queue_get_event 3
is the meat of the library and contains further useful documentation.
.Pp
.Xr quark-mon 8
is the easiest way to get started with
.Nm .
.Pp
.Xr quark_queue_open 3
describes initialization options that can be useful.
.Sh SEE ALSO
.Xr quark_event_dump 3 ,
.Xr quark_process_lookup 3 ,
.Xr quark_queue_block 3 ,
.Xr quark_queue_close 3 ,
.Xr quark_queue_get_epollfd 3 ,
.Xr quark_queue_get_event 3 ,
.Xr quark_queue_get_stats 3 ,
.Xr quark_queue_open 3 ,
.Xr quark-btf 8 ,
.Xr quark-mon 8 ,
.Xr quark-test 8
.Sh LICENSE
.Nm
is released under the Apache-2.0 license and contains code under BSD-2, BSD-3,
ISC, and zlib Licenses.
.Sh HISTORY
.Nm quark
started in April 2024.