This tool helps to upgrade large SSR deployments.
In general every SSR upgrade consists of two phases:
- Download the target release from an online software repository to the selected router
- Upgrade the software of the selected router to the target release
In order to perform a software upgrade of an SSR deployment, the t128-bulk-upgrade
tool connects to the conductor server. The conductor has its own asset repository of all routers of a deployment and provides functionality to trigger software downloads and upgrades.
The t128-bulk-upgrade
command line tool is designed to run in chunks of configurable size. All routers in a chunk will download the target software release first, and, if everything was successful, all routers will be upgraded.
When a chunk was successfully upgraded, the tool continues with the next chunk until all (selected) routers are done or the maximum amount of routers (see --max
commandline parameter) was reached.
The connection between t128-bulk-upgrade
and the conductor is always HTTPS.
However, there are two ways of authenticating an API user:
- using username and password (in the same fassion like using a web browser)
- using a locally stored authentication file on the conductor.
Running t128-bulk-upgrade
from an external computer requires IP connectivity to the conductor (works if the conductor can be reached from a web browser) and a Python (version 3) installation.
The following command line parameters are relevant to connect to the conductor:
--host HOST Conductor/router hostname
--user USER Conductor/router username (if no key auth)
--password PASSWORD Conductor/router password (if no key auth)
To test connectivity the following command can be used to connect to a conductor running at IP address 10.0.0.128:
python3 t128-bulk-upgrade.pyz --host 10.0.0.128 --user admin --password 128tRoutes --list-releases
The tool should list all available releases depending on the installed license.
Similar to external connections, the script can be ran directly from the conductor's Linux shell (typically with t128
user). In this case sudo
privileges are required to access the authentication file:
sudo python3 t128-bulk-upgrade.pyz --list-releases
By default the tool upgrades all routers in Running
state, unless a filter is provided using the --filter
commandline parameter (or in short -f
).
The filter expect the following format: key=PATTERN
.
The the following keys are supported:
- name.startswith
- name.contains
- name.equals
- version.startswith
- version.equals
where name
refers to the router name and version
to currently running version of a router:
Examples are:
sudo python3 t128-bulk-upgrade.pyz --filter name.contains=headend --release 5.4.11 --max 1
sudo python3 t128-bulk-upgrade.pyz --filter version.startswith=5.4 --release 5.5.8
Another way to select routers is a text file with router names, one router per line. This file can be specified with the --router-file
parameter. For example:
$ cat routers.txt
headend1
branch1
branch2
$ sudo python3 t128-bulk-upgrade.pyz --router-file routers.txt --release 5.5.8
As all routers in a chunk are triggered in parallel for a download and upgrade, the number of routers per chunk is configurable by the parameter --parallel
(or in short -p
).
The default chunk size is 1
(one router).
During the download and upgrade process, the t128-bulk-upgrade
tool declares a chunk not to be successful when a timeout has exceeded and t128-bulk-upgrade
stops further processing.
The default timeout for downloads and upgrades is one hour (3600
seconds). It can be overridden by the --timeout
parameter (or in short -t
) followed by an integer value. The timeout value is in seconds.
The parameter --download-timeout
allows to adjust the timeout for downloads only.
For example to allow more time for an upgrade (e.g. 2h = 7200s):
sudo python3 t128-bulk-upgrade.pyz --release 5.4.11 --timeout 7200
The tool prints its actions into the terminal to standard output. To get a better view over the upgrade progress, a status file can be written with the --status-file
parameter. The file is and so the router states are updated on a regular basis when the tool is running and can help to identify a failed router in case of errors.
These are the possible states of a router:
NOOP
(no operation) = No action was needed for the router (typically when the router was already upgraded before)UNKNOWN
= The running version of a router could not be determined (typically when a router is offline or has connection issues with the conductor)DOWNLOAD_IN_PROGRESS
= download has not yet finishedDOWNLOAD_COMPLETED
= download has finished, but upgrade has not yet startedUPGRADE_IN_PROGRESS
= upgrade has not yet finishedUPGRADE_COMPLETED
= upgrade has finished
Especially during tests, some parameters are useful to adjust the upgrade process:
--dry-run
does not perform any download or upgrade action, but shows what would be performed.--download-only
(or in short-d
) does perform only download actions, but no upgrades. This allows to pre-download the software on routers to have it available at a later point (e.g. maintenance) in order to reduce the time for an actual upgrade window.--wait-running
- wait until all routers in a chunk come back intoRunning
state after an upgrade and before continuing with the next chunk. This should avoid upgrading all routers having provisioner issues. In such a severe situation the tool would stop after the first chunk.--ignore-download-errors
- in some cases it may be desired to allow upgrades of all routers, even if some of them cannot download the target release. This parameter skips the failed routers and continues with the uprade for all other routers in that chunk.