gitopper.8

.\" Generated by Mmark Markdown Processer - mmark.miek.nl
.TH "GITOPPER" 8 "March 2023" "System Administration" "Git Operations"

.SH "GITOPPER"
.SH "NAME"
.PP
gitopper - watch a git repository, pull changes and reload the server process

.SH "SYNOPSIS"
.PP
\fB\fCgitopper [OPTION]...\fR \fB\fC-c\fR \fBCONFIG\fP

.SH "DESCRIPTION"
.PP
Gitopper is GitOps for non-Kubernetes folks it watches a remote git repo, pulls changes and HUP the
server (service) process.

.PP
A sparse (but with full history) git checkout will be done, so each service will only see the files
it will actually need. Several bind mounts are then setup to give the service access to the file(s)
in Git. If the target directories don't exist, they will be created, with the current user - if
specified.

.PP
This tool does little more than just pull the repo, but the little it brings to the table allows for
a GitOps workflow without resorting to Kubernetes like environments.

.PP
The Git repository that you are using to provision the services must have at least one
(sub)directory for each service.

.PP
Gitopper will install packages if told to do so. It will not upgrade or downgrade them, assuming
there is a better way of doing those.

.PP
The remote interface of gitopper uses SSH keys for authentication, this hopefully helps to fit in,
in a sysadmin organisation.

.PP
The following features are implemented:

.IP \(bu 4
\fIMetrics\fP: are included see below, they export a Git hash, so a rollout can be tracked.
.IP \(bu 4
\fIDiff detection\fP: possible using the metrics or gitopperctl.
.IP \(bu 4
\fIOut of band rollbacks\fP: use gitopperctl to bypass the normal Git workflow.
.IP \(bu 4
\fINo client side processing\fP: files are used as they are in the Git repo.
.IP \(bu 4
\fICanarying\fP: give a service a different branch to check out.


.PP
The options are:

.TP
\fB-h, --hosts strings\fP
hosts (comma separated) to impersonate, local hostname is always added
.TP
\fB-c, --config string\fP
config file to read
.TP
\fB-s, --ssh string\fP
ssh address to listen on (default ":2222")
.TP
\fB-m, --metric string\fP
http metrics address to listen on (default ":9222")
.TP
\fB-d, --debug\fP
enable debug logging
.TP
\fB-r, --restart\fP
send SIGHUP to ourselves when config changes
.TP
\fB-o, --root\fP
require root permission, setting to false can aid in debugging (default true)
.TP
\fB-t, --duration duration\fP
default duration between pulls (default 5m0s)


.PP
For bootstrapping gitopper itself the following options are available:

.TP
\fB-U, --upstream string\fP
use this git repo to clone and to bootstrap from
.TP
\fB-D, --directory string\fP
directory to sparse checkout (default "gitopper")
.TP
\fB-B, --branch string\fP
check out in this branch (default "main")
.TP
\fB-M, --mount string\fP
check out into this directory, -c is relative to this directory
.TP
\fB-P, --pull\fP
pull (update) the git repo to the newest version before starting


.SH "QUICK START"
.IP \(bu 4
Generate a toy SSH key: \fB\fCssh-keygen -t ed25519\fR and make it write to an \fB\fCid_ed25519_gitopper\fR file.
.IP \(bu 4
Put the path to the \fIPUBLIC\fP key (ending in .pub) in the \fB\fCkeys\fR fields of \fB\fC[global]\fR of the
following config.toml file.
.IP \(bu 4
Start as root: \fB\fCsudo ./gitopper -c config.toml -h localhost\fR


.PP
.RS

.nf
[global]
upstream = "https://github.com/miekg/gitopper\-config"
branch = "main"
mount = "/tmp/gitopper"
keys = [
    { path = "keys/id\_ed25519\_gitopper.pub", ro = true },
]

[[services]]
machine = "localhost"
service = "prometheus"
user = "prometheus"
package = "prometheus"
action = "reload"
dirs = [
    { local = "/etc/prometheus", link = "prometheus/etc" },
]

.fi
.RE

.PP
And things should work then. I.e. in /etc/prometheus you should see the content of the
\fImiekg/gitopper-config\fP repository. Note that the prometheus package is installed, because \fB\fCpackage\fR
is mentioned in the config file.

.PP
The checked out git repo in /tmp/prometheus should \fIonly\fP contain the prometheus directory thanks to
the sparse checkout. Changes made to any other subdirectory in that repo do not trigger a prometheus
reload.

.PP
Then with gitopperctl(8) you can query the server:

.PP
.RS

.nf
\&./gitopperctl \-i <path\-to\-your\-key> list service @localhost
#  SERVICE     HASH       STATE  INFO  CHANGED
0  prometheus  606eb576   OK           Fri, 18 Nov 2022 09:14:52 UTC

.fi
.RE

.SH "SERVICES"
.PP
A service can be in 5 states: OK, FREEZE, ROLLBACK (which is a FREEZE to a previous commit) and
BROKEN/DIFF.

.PP
These states are not carried over when gitopper crashes/stops (maybe we want this to be persistent,
would be nice to have this state in the git repo somehow?).

.IP \(bu 4
\fB\fCOK\fR: everything is running and we're tracking upstream.
.IP \(bu 4
\fB\fCFREEZE\fR: everything is running, but we're not tracking upstream.
.IP \(bu 4
\fB\fCROLLBACK\fR: everything is running, but we're not tracking upstream \fIand\fP we're pinned to an older
commit. This state is quickly followed by FREEZE if we were successful rolling back, otherwise
BROKEN (systemd error)  of DIFF (git error)
.IP \(bu 4
\fB\fCBROKEN\fR: something with the service is broken, we're still tracking upstream. I.e. systemd error.
.IP \(bu 4
\fB\fCDIFF\fR: the git repository can't be reconciled with upstream. I.e. git error.


.PP
ROLLBACK is a transient state and quickly moves to FREEZE, unless something goes wrong then it
becomes BROKEN, or DIFF depending on what goes wrong (systemd, or git respectively).

.PP
.RS

.nf
 +\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-+
 |                         |
 v                         |
*OK \-\-\-\-\-\-\-> ROLLBACK \-\-\-> FREEZE
 |          /    \\         |
 |         /      \\        v
 |        |        |       |
 |        v        v       |
 |      BROKEN     DIFF    |
 |        |         |      |
 |        |         |      |
 +\-\-\-\-\-\-\-\-+\-\-\-\-\-\-\-\-\-+\-\-\-\-\-\-+

.fi
.RE

.IP \(bu 4
\fB\fC*OK\fR is the start state
.IP \(bu 4
from \fB\fCOK\fR and \fB\fCFREEZE\fR we can still end up in \fB\fCBROKEN\fR and \fB\fCFREEZE\fR and vice versa.


.SH "CONFIG FILE"
.PP
.RS

.nf
# global options are applied if a service doesn't list them
[global]
upstream = "https://github.com/miekg/gitopper\-config"  # repository where to download from
mount = "/tmp"                                     # directory where to download to, mount+service is used as path
# ssh keys that are allowed in via authorized keys
keys =[
    { path = "keys/miek\_id\_ed25519\_gitopper.pub" },
    { path = "keys/another\_key.pub", ro = true },
]

# each managed service has an entry like this
[[services]]
machine = "prometheus"        # hostname of the machine, so a host knows when to pick this up.
service = "prometheus"        # service identifier, if it's used by systemd it must be the systemd service name
action = "reload"             # call systemctl <action> <service> when the git repo changes, may be empty
branch = "main"               # what branch to check out
package = "prometheus"        # as used by package mgmt, may be empty (not implemented yet)
user = "prometheus"           # do the check out with this user
# what directories or files from the repo to mount under the local directories
dirs = [
    { local = "/etc/prometheus", link = "prometheus/etc" },   # prometheus/etc *in the repo* should be mounted under /etc/prometheus
    { local = "/etc/caddy/Caddyfile", link = "caddy/etc/Caddyfile", file = true },   # caddy/etc/Caddyfile *in the repo* should be mounted under /etc/caddy/Caddyfile
]

.fi
.RE

.PP
Note that \fB\fCmachine\fR above should match either the machine name ($HOSTNAME) or any of the values you
give on the \fB\fC-h\fR flag. This allows you to create services that run everywhere, by defining a service
that have name (say) "localhost" and then deploying gitopper with \fB\fC-h localhost\fR on every machine.

.PP
Options for each service:

.IP \(bu 4
\fB\fCmachine\fR: the machine where this service should be active. By default \fB\fCgitopper\fR will know the
current hostname, but multiple aliases may be given to it via the \fB\fC-h\fR flag.
.IP \(bu 4
\fB\fCservice\fR: what systemd unit file is used to call \fB\fCaction\fR on. If service contains an \fB\fC@\fR a
service template unit is assumed and gitopper will then run \fB\fCsystemctl enable <service>\fR to enable
the service template.
.IP \(bu 4
\fB\fCaction\fR: action to use when calling \fB\fCsystemctl <service>\fR. If empty no systemd command will be
issued when the repo changes.
.IP \(bu 4
\fB\fCbranch\fR: what branch to use in the checked out repo. Note different branches that use the \fIsame\fP
repository on disk, will error on startup.
.IP \(bu 4
\fB\fCpackage\fR: what package to install for this service. If empty, no package will be installed.
.IP \(bu 4
\fB\fCuser\fR: what user should the git repository belong to.
.IP \(bu 4
\fB\fCdirs\fR: describe the mapping between directories and files in the repository and on the local
disk. \fB\fClocal\fR is the \fIon disk\fP name, and \fB\fClink\fR is the \fIrelative\fP path of the directory or file in
the git repo. If a single file is used, \fB\fCfile\fR should be set to true.


.SS "HOW TO BREAK IT"
.PP
Moving to a new user, will break git pull, with an error like 'dubious ownership of repository'. If
you want a different owner for a service, it's best to change the mount as well so you get a new
repo. Gitopper is currently not smart enough to detect this and fix things on the fly.

.SH "INTERFACE"
.PP
Gitopper opens two ports: 9222 for metrics and 2222 for the rest-protocol-over-SSH. For any
interaction with gitopper over this port your key must be configured for it.

.PP
The following services are implemented:

.IP \(bu 4
List all defined machines.
.IP \(bu 4
List services run on the machine.
.IP \(bu 4
List a specific service.
.IP \(bu 4
Freeze a service to the current git commit.
.IP \(bu 4
Unfreeze a service, i.e. to let it pull again.
.IP \(bu 4
Rollback a service to a specific commit.


.PP
For each of these gitopperctl(8) will execute a "command" and will parse the returned JSON into a nice
table.

.SH "METRICS"
.PP
The following metrics are exported:

.IP \(bu 4
gitopper_service_state{"service"} <state>
.IP \(bu 4
gitopper_service_change_time_seconds{"service"} <epoch>
.IP \(bu 4
gitopper_machine_git_errors_total - total number of errors when running git.
.IP \(bu 4
gitopper_machine_git_ops_total - total number of git runs.


.PP
Metrics are available under the /metrics endpoint on port 9222.

.SH "EXIT CODE"
.PP
Gitopper has following exit codes:

.PP
0 - normal exit
2 - SIGHUP seen (signal to systemd to restart us)

.SH "BOOTSTRAPPING"
.PP
There are a couple of options that allow gitopper to bootstrap itself \fIand\fP make gitopper to be
managed by gitopper. Basically those options allow you to specify a service on the command line.
Gitopper will check out the repo and then proceed to read the config \fIin that repo\fP and setup
everything from there.

.PP
I.e.:

.PP
.RS

.nf
\&... \-c config.toml \-U https://github.com/miekg/gitopper\-config \-D gitopper \-M /tmp/

.fi
.RE

.PP
Will sparse check out (only the \fB\fCgitopper\fR (-D flag) directory) of the repo \fIgitopper-config\fP (-U
flag) in /tmp/gitopper (-M flag, internally '/gitopper' is added) and will then proceed to parse the
config file /tmp/gitopper/gitopper/config.toml and proceed with a normal startup.

.PP
Note this setup implies that you \fImust\fP place config.toml \fIinside\fP a \fB\fCgitopper\fR directory, just as
the other services must have their own subdirectories, gitopper needs one too.

.PP
The gitopper service self is \fIalso\fP added to the managed services which you can inspect with
gitopperctl(8).

.PP
Any keys that have \fIrelative\fP paths, will also be changed to key inside this Git managed directory
and pick up keys \fIfrom that repo\fP.

.PP
The \fB\fC-P\fR flag can be given to pull the repository even if it already exists, sometimes you need to
the newest version to properly bootstrap. For normal services the "git pull" routine will
automatically rectify it and restart the service.

.SH "SEE ALSO"
.PP
See this design doc
\[la]https://miek.nl/2022/november/15/provisioning-services/\[ra], and gitopperctl(8).