Skip to content

Latest commit

 

History

History
5676 lines (3397 loc) · 114 KB

CMSSDT.md

File metadata and controls

5676 lines (3397 loc) · 114 KB
Raw

CMSSDT

View description: None

View type: List View

Projects:

check-cms-container-certificates

Description: This job checks for HOST certificate in cmssw/cms:rhelX images, and fail if host certificate is going to expire in less than CERT_EXPIRY_DAYS days.
In such case, we need to check if there is already an updated osg-ca-certs package available and rebuild the container.

One can rerun the job with LOCAL_CERTIFICATE=/cvmfs/grid.cern.ch/etc/grid-security/certificates and CHECK_LOCAL_CERTIFICATE=true to see it new certificates are already available.
If the job still fails, this means that new root certificates are still not available.

Note that the job fails some days before the actual expiration (CERT_EXPIRY_DAYS).

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

H H * * *

check-cvmfs-releases-map

Description: None

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

H H/4 * * *

check-future-commit-prs

Description: Periodicaly runs https://github.com/cms-sw/cms-bot/blob/master/fix-backport-labels.py to check if the PR on master, which backport has been requested, is merged. If the original PR has been merged, it changes all opened backport PRs of it from backport(blue) to backport-ok(green)

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

H/15 * * * *

check-pending-job

Description: 

The project checks for a running job and shows the status of process running on a remote machine.
This can be useful to see if any process is handing and kill it. To run this job

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

check-unused-cmsdist-packages

Description: This job check for unused cmsdist files so that one can do the cleanup.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

check-zombie

Description: Connects to the slave and look for runaway process. This job fails when it finds such processes.
In case of failure, we need to check the runaway process list (e.g. see the job log) and make sure that these processes are really runaway (e.g. cmsRun jobs running with parent PID 1). If all looks good, then one can restart the job with KILL_HANGING_PROCESSES=true.

Note: A runaway process is a process that enters an infinite loop and spawns new processes.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

cleanup-cmsrep

Description: This cleans up old cms.weekN.PR_* repositories from cmsrep.cern.ch server.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

cleanup-cvmfs-ci

Description: This job cleans up old cms.weekN.PR_* repositories from cmsrep.cern.ch server.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

cleanup-release-build

Description: Connect to selected slave and cleans the auto-builds directories

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

cms-build-package

Description: This job builds a cmssw external package ( e.g. crab , SCRAMV1 ) using the master cmsdist/pkgtools branches. It then trigger the cvmfs installation job to install the newly build packages.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

cms-spack-build-singularity

Description: None

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

cms-spack-ib

Description: This jobs starts an Integration Build (IB). Base on state of CMSDIST/CMSSW git repositories, it builds either a full release or patch release.
Build Full IB if:

  • There are changes in CMSDIST
  • There is no full IB available based on current CMSDIST
  • Previous full IB had build errors

Otherwise build a patch release.



Q/A

  • Q: The job is scheduled with a clock simbol. Jenkins also complains that there are not agents with labels THIS and THIS.
  • A: Jenkins should automaticly launch agents with specific labels for the job. However, for some reason it does not work for this job. For now, launch agent manually.
  • Q: How to manually build an IB?
  • A: Go to upstream job (ib-tag-and-schdule) and start a job.
  • Q: What to do if it fails?
  • A: If it fails due to network/github issues then just re-try it, but if it fails to build then we need to understand the build issues before retry.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

cms-spack-install-cvmfs

Description: None

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

cmsrep-eos-backup

Description: Copy unused Os/archs RPMs to EOS

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

H 10,22  * *  *

cmssw-afs-eos-comparison

Description: None

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

crab-cleanup

Description: This job runs once a day to clean up the /eos/cms/store/user/cmsbot/CRAB_PrivateMC directory e.g. delete every thing older than X days.
In case of failure , just re-run it.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

H 0 * * *

cvmfs-check-cuda-compatible-runtime

Description: This job checks and updates cuda-compatible-runtime. This is used by CMS pilot jobs to find available gpu on the worker node.

Requestsed by marco.mascheroni@cern.ch and andrea.bocci@cern.ch

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

cvmfs-deploy-artifacts

Description: Copy baseline results from cmssdt for an IB

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

cvmfs-gc

Description: This runs CVMFS GC (once a week)

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

#Run once on Thursday at 23h05
H 19  * *  2

cvmfs-install-pr

Description: To install PR externals

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

cvmfs-list-catelogs

Description: None

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

cvmfs-publish-artifacts

Description: Copy baseline results from cmssdt for an IB and deploy them on cvmfs

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

cvmfs-run-gc

Description: None

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

cvmfs-run-gw-gc

Description: This runs CVMFS GC on the Gateway, make sure that top level path "/" has a lease.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

cvmfs-update-proot

Description: None

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

cvmfs-update-qemu

Description: None

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

cvmfs_publish_remote_dir

Description: This jobs install an IB on /cvmfs/cms-ib.cern.ch.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

docker-issue-comment

Description: avalenzu: reporting back on existing issues

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

docker-packages-info

Description: avalenzu: testing docker image comparison

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

docker-watch-pkgs

Description: avalenzu: testing watching concrete pkgs

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

H 15 * * 0

dxr-cmssw-cleanup

Description: None

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

H 7 * * *

es-cmssw-afs-eos

Description: None

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

H H * * *

es-insert-failure-data

Description: Insert record into ES linking Issue/PR and error in IB

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

find-cmssw-build-packages

Description: None

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

git-notify-build-results

Description: Triggered when new IB results are available and send a Mattermost notification if there are new failures

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

ib-check-baseline-requests

Description: This jobs look for on demand baseline generation requests (made by ib-run-pr-tests job) and start the baseline generation job ( if not already started).

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

H/3 * * * *

ib-check-siteconf

Description: This check for /cvmfs/REPO/SITECONF contents and triggers ib-install-siteconf (for each repo) if https://github.com/cms-sw/siteconf has updates. Thsi jobs runs every hour

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

H * * * *

ib-inject-profiling-data

Description: Inject the FastTimerService files to CMSMonit

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

ib-install-cvmfs

Description: This jobs install an IB on /cvmfs/cms-ib.cern.ch. As this job runs on the CVMFS Stratum 0, so only one job can run at a time. Sometimes ( specially IBs for non-86-64 archs) are stuck and do nothing. In that case better to kill the job and re-try it. This gives chance to other IBs to get installed and at the end the re-tried job will re-run.

There is no automatic re-try setup for this job. It rarely fails but in case it fails then just re-try the failed job and either delete the failed job instance or update the "Build information" and mentioned that it has been re-tired. This allows others to not re-try it.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

ib-install-siteconf

Description: Update the contents of/cvmfs/REPO/SITECONF from https://github.com/cms-sw/siteconf

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

ib-monitor-crab

Description: This job is triggered by ib-run-crab or ib-run-pr-crab once a test analysis is submitted to CRAB. It monitors the status of the CRAB job by using curl calls until the job finishes.
Finally, it generates a status file that is visible in the IB page (https://cmssdt.cern.ch/SDT/jenkins-artifacts/ib-run-crab).

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

ib-run-baseline

Description: This job generates the IB baseline. It is two phase job

  • First it runs on any suitable node and look for workflows for which the baseline should be generated. It then re-run itself with the list of workflows and build an upload the actual baseline.
  • In case of failure, just retry this job (Unless there are network/github issues)

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

ib-run-class-versions

Description: Runs edm class version checks for IB

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

ib-run-crab

Description: This job integrates CRAB into the CI system.

Note: CRAB is the CMS Computing tool to submit CMSSW users' analysis jobs to distributed computing resources (https://twiki.cern.ch/twiki/bin/view/CMSPublic/SWGuideCrab).

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

ib-run-hlt-p2-timing

Description: None

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

ib-run-pr-addon

Description: Build mutiple pull requests. Run cmssw addON (HLT tests)

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

ib-run-pr-dasgoclient

Description: Job to run das client and cache the results in github to be used by IBs. Ignore any failed job if a newer job has succeeded.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

ib-run-pr-external_checks

Description: Build mutiple pull requests. test cmssw externals packages static checks e.g. hardcoded build paths etc.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

ib-run-pr-hlt_p2_integration

Description: None

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

ib-run-pr-hlt_p2_timing

Description: None

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

ib-run-pr-profiling

Description: Build mutiple a pull requests. Same puprose as ib-any-integration, just different script is called.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

ib-run-pr-relvals

Description: Build mutiple a pull requests. Same puprose as ib-any-integration, just different script is called.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

ib-run-pr-unittests

Description: Build mutiple pull requests. Run cmssw GPU unit tests

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

ib-run-pr-wait-deployment

Description: Same puprose as ib-any-integration, just different script is called.
This job waits until the build has been deployed into cvmfs (cms-ci.cern.ch repository).
If the job fails, it could be because of a failure in the previous job (cvmfs-install-pr) that produces and uploads the build. Therefore, the current job will time out.
The way to proceed in case of failure is to look for the PR number present in the build name and trace back what has happened. It could be that the PR test were stuck and the build was never produced or uploaded.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

ib-run-profiling-mem

Description: Runs Jemalloc memory profiling on the RECO and PAT steps for high pileup workflow.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

ib-run-profiling-mem-GC

Description: Runs FastTimerService and Igprof on the RECO and PAT steps for high pileup workflow.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

ib-test-lto

Description: None

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

ib-test-lto-openstack

Description: None

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

ib-test-lto-openstack-hlt

Description: avalenzu: To be removed soon. HLT test can run as part of https://cmssdt.cern.ch/jenkins/job/ib-test-lto-openstack/

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

install-qemu

Description: None

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

jenkins-test-retry

Description: This job retries a build given the job name and the build number to retry.
It copies the build parameters from the given build and adds a retry counter as an environmental variable to keep track of the number of retries (a maximum of 3 retries per build has been set).
This job is automatically triggered by the job "jenkins-test-parser" when a build fails due to a known issue and a retry action is needed.
This job is expected to fail if the same build has been retried more than 3 times. If it fails due to another issue, we need to take the appropiate action.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

lxr-repair-table

Description: Run this if there is any MySQL table to repair e.g. if lxr-remove-idex fails with error
ERROR 144 (HY000) at line 3: Table './lxr/lxr_usages' is marked as crashed and last (automatic?) repair failed
then run it with LXR_TABLE=lxr_usages

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

pr-publish-cmssw

Description: None

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

process-relval-logs-cleanup

Description: This job process partial logs of Relvals and place files accordingly.
There is no need to re-try this is a newer run is successful. If it keeps on failing then one need to check the logs and find out the reason of failure. In that case some manual work is needed to cleanup cmssdt.cern.ch logs.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

query-new-data-repo-issues

Description: Processes a github issue to check if it is requesting the creation of a new data repository. If the issue is not requesting any release, it ignores it.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

reco-profiling-eos-cleanup

Description: None

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

release-analyze-reco-profiling

Description: This job processes data from release-run-reco-profiling and prepares https://cms-reco-profiling.web.cern.ch/cms-reco-profiling/

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

release-rsync-reco-profiling

Description: None

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

release-run-reco-profiling

Description: Profiling jobs that are submitted manually for each CMSSW release. The performance is summarized on https://cms-reco-profiling.web.cern.ch/. Managed by cms-offline-conveners-reco@cern.ch.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

run-container-stageout

Description: This job tests submission to multiple sites. Some of them are expected to fail.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

H 0,12 * * *

run-container-tests

Description: None

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

run-vtune-profiling

Description: Profiling jobs that are submitted manually for each CMSSW release. The performance is summarized on https://cms-reco-profiling.web.cern.ch/. Managed by cms-offline-conveners-reco@cern.ch.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

s3-speet-test

Description: None

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

set-shifter

Description: Run this job at the start of your shift to receive MM pings from failed builds/tests

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

sync-profile-data

Description: None

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: ['ib-run-igprof', 'ib-run-profiling', 'ib-run-profiling-mem', 'ib-run-profiling-mem-GC']

Periodic builds:

Not periodically build

test-containter-singularity

Description: None

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

test-new-cvmfs-publisher

Description: None

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

test-os

Description: This job generates property files for triggering os test jobs (e.g., test-os-cs8, test-os-alma8, etc) for any os_arch_compiler combination.
The operating systems tested are rhel8, rocky8, ubi8, el8, cs8, alma8 and lxplus8 os.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

H 20 * * *

test-os-alma8

Description: None

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

test-os-cs8

Description: None

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

test-os-el8

Description: None

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

test-os-lxplus8

Description: None

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

test-os-rhel8

Description: None

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

test-os-rocky8

Description: None

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

test-os-ubi8

Description: None

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

tmp-node-get-config

Description: None

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

update-circle-dataset

Description: None

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

update-circles-eos

Description: None

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

update-cmssw-l2-histroy

Description: None

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

update-pr-status

Description: None

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

vtune-check

Description: None

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

H/20 * * * *

vtune-cmssw-cleanup

Description: None

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

H 7 * * *

abort-jenkins-job

Description: Kill a running job (by default it is 'ib-run-pr-tests' and jobs started by it).

The idea is that if pull request was updated, all the test should be restarted. This job will kill all the running jobs for that PR (with matching parameters). It will ignore the given job ID - the ID of upstream job that started this job.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

abort-pr-tests

Description: This job basically triggers the abort-jenkins-jobs job for all the jobs related to PR tests.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

abort-release

Description: Aborts and kills a release building process.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

auto-forward-port

Description: This is triggered by github webhook for each cmssw/cmsdist branch merge event. This is just a place holder job to trigger one sub-job per destionation branch for which the forward porting should be done. If this fails then this means that one of its sub-job failed. There is no need to re-try this job. Retry the failed sub-jobs only.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

auto-forward-port-branch

Description: This job forward ports git changes from one branch to another. If it fails due to network of github related issues then just re-try it. Most of the times this job fails due to merge conflicts e.g. cmsdist master branch has different version of root as compare of rootmaster rootnext branch. If merge conflicts are just the version differences, then re-build the job with "STRATEGY=ours" as parameter, but if there are complex conflicts then better to resolve the issue by hand and directly push changes to github

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

backport-pull-request

Description: Manually executed job for backporting pull requests given the repo name, the branch for which the PR should be created and the PR to be backported.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

build-any-ib

Description: This jobs starts an Integration Build (IB). Base on state of CMSDIST/CMSSW git repositories, it builds either a full release or patch release.
Build Full IB if:

  • There are changes in CMSDIST
  • There is no full IB available based on current CMSDIST
  • Previous full IB had build errors

Otherwise build a patch release.



Q/A

  • Q: The job is scheduled with a clock simbol. Jenkins also complains that there are not agents with labels THIS and THIS.
  • A: Jenkins should automaticly launch agents with specific labels for the job. However, for some reason it does not work for this job. For now, launch agent manually.
  • Q: How to manually build an IB?
  • A: Go to upstream job (ib-tag-and-schdule) and start a job.
  • Q: What to do if it fails?
  • A: If it fails due to network/github issues then just re-try it, but if it fails to build then we need to understand the build issues before retry.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

build-docker-container

Description: This job builds container images and uploads them to dockerhub.

Containers are built when either their base image is updated or if changes were made to dockerfile.
Failures may arise from the changes in the base image. In this case, the best approach is to log into the machine, reproduce the issue and try to adapt the dokerfile accordingly (e.g., changing dependencies, ordering, etc).

If the job fails due to a glitch in the infrastructure, retry should work. If the container image is changed, we need to retrigger this job from check-docker-container job (so that the correct checksum is passed to build-docker-container job).


Since the built container images are left on the machine, this job can also fail due to disk full.
There is a separate jenkins job (clean-build-docker-machine) cleaning the images left on the device that runs once a week. In any case, one can always log into the machine and run the docker cleanup as follows:

$ docker image prune
$ docker system prune --volumes

If the ./singularity folder is big enough ($ du -hs /build/cmsbld/jenkins/workspace/.singularity), it can be also removed.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

build-fwlite

Description: This job builds a CMSSW_X_Y_Z_FWLITE release based on an existing CMSSW_X_Y_Z release. CMSSW_X_Y_Z should be already build/uploaded before building its FWLITE version.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

build-fwlite-ib

Description: This job is responsible for building FWLITE release for each Integration Build (IB). Results of this build can be seen via CMSSDT IB page (old page).

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

build-release

Description: This job actually builds a release. It is triggered by cms-bot after the Release build Issue is approved by the release manager.

  • In case the job fails:
    Depending on the type of failure, most of the time re-try works. In some cases (specially for aarch64 builds), it fails due to disk quota issues as one of the aarch64 machine does not have enought disk. In that case, one needs to login to the corresponding aarch64 machine and cleanup old/unused stuff and then re-try this job.
  • In case there is a build error in a release:
    It can happen that the job succeed, but there have been build errors that the bot reports in the corresponding issue. In this case, the Rebuild button can be used to re-trigger the build. The bot will automatically remove the build-error labels and inform that a new build has been started.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

check-cmsdev-disk

Description: This job cleans up disk space on cmsdev machines.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

check-docker

Description: This job connects to the slave and checks if docker service is useable.
Job can fail for three reasons:

  1. user is not in docker group
  2. docker service is not running on the machine
  3. docker daemon reports that there isn't space in the machine ("no space left on device").

If machine is in our control, we should take the appropriate action (e.g. login into the machine and manually run the ssh command), but if machine is in CERN IT control (e.g OpenLab machines), we should open a SNOW ticket.
In case of no space on the device, one can manually run the job clean-check-docker indicating the name of the machine as an input parameter.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

check-docker-container

Description: This job checks for changes in the parent image. If there are changes, it triggers the build-docker-container job so that our based image is updated and uploaded to the registry.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

clean-build-docker-container

Description: This job prevents running out of disk in the machines used to build container images (e.g., the ones used for running the build-docker-container job).

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

H 9 * * 1

clean-check-docker

Description: This job cleans the indicated machine when docker daemon reports errors of the type "no space left on device" in job check-docker.
It is necessay to indicate the machine name as an input parameter.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

cleanup-auto-build

Description: This job deletes the release build areas after three days.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

cleanup-cms-sw-io-history

Description: This job cleanup the cms-sw/cms-sw.github.io repository history. Specially the data and _data directories history.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

H 1 * * 0

cleanup-cmssdt

Description: This job cleans up disk space on cmssdt.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

H 0 * * *

cleanup-docker-tags

Description: This job removes outdated tags from docker repository.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

H 0 * * *

cleanup-tags

Description: Cleanup IB tags from the official CMSSW repository.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

H 0 * * *

cms-auto-build-container

Description: None

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

H 10 * * *

cms-bot

Description: This is cms bot job which is triggered by github webhooks ( https://cmssdt.cern.ch/SDT/cgi-bin/github_webhook ) for every valid comment added to github PRs.

Will kill a scheduled/running job acording to comments.



Q/A

  • Q: Job 'cms-sw/cmssw #****' failed. What to do?
  • A: Job is configured to do few re-tries, so in case of github or network glitches the job will recover. If it keeps on failing then one need to check the logs. Mostly it fails due to github/ssh connection, so it will automatically recover.
  • Q: Job failed with ssh timeout?
  • A: Nothing to do, job will re-try 3 time to recover. If it keeps on failing for few hours then better to disable it an re-enable once issue is solved e.g. github is back or CERN network is working again.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

cms-containers-checks-tags

Description: This project search for tags.yaml files in cms-sw/cms-docker repostory and create new image tags if needed

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

H 2 * * *

cms-containers-run-cmssw-test

Description: None

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

cms-containers-schedule-cmssw-test

Description: None

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

cms-prs-cache

Description: This job collects metadata for pull requests since reading github directly when necessary might become time consuming or rejected by github on reached limit hits/hour grounds.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

H 20 * * *

cms-prs-files

Description: On every CMSSW PR, this job dumps list of files that are going to be modified by open PRs to files_changed_by_prs.json .

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

H H/2 * * *

cms-tag-container

Description: None

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

cmsbuild-access

Description: None

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

H 14 * * *

cmspkg-clone

Description: None

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

cmspkg-test

Description: None

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

cmsrep-webhook

Description: This is cms bot job which is triggered by github webhooks ( https://cmssdt.cern.ch/SDT/cgi-bin/github_webhook ) for every valid comment added to github PRs. Also it runsevery 30mins to make sure any webhooks were not missed.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

cmssw-container

Description: None

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

cmssw-doxygen

Description: Generate doxygen documentation for CMSSW project

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

cmssw-to-stitched

Description: None

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

comp-bot

Description: This is cms bot job which is triggered by github webhooks ( https://cmssdt.cern.ch/SDT/cgi-bin/github_webhook ) for every valid comment added to github PRs. This is to process bot commands for cmsdist comp branches

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

compare-material-budget

Description: Comopare results of material budget of two releases using Validation/Geometry/test/TrackerMaterialBudgetComparison.C macro

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

compare-root-files-short-matrix

Description: Download the files of the baseline IB and compare them with the results of the ones of a pull request. Rate of failure of this job is very low and when failed then just re-try it and either delete the failed job or update the build information that it has been re-tried.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

das-query

Description: Job to run das client and cache the results in github to be used by IBs. Ignore any failed job if a newer job has succeeded. Retry if last run is failed.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

H 9,21 * * *

dataset-to-ibeos

Description: Copies root files for a given dataset localy on eos to be used from local cache (eos) if they diasappear from the site where they resided.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

deploy-cms-repo

Description: Updates cms-bot clone on Jenkins master

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

deploy-cmsdoxygen

Description: This job deploys doxygen scripts to web server. It is later used to serve CMSSW documentation
Could be automated by github hooks.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

docker-dmwm-CMSKubernetes

Description: None

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

docker-images-to-github

Description: This job migrates exisiting images in dockerhub to the github registry.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

H H * * *

dockerhub_synchronization

Description: Synchronizes Docker Hub Organization's setup (repositories/teams/members/permissions) with docker configuration yaml file.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

H 0 * * *

dxr-cmssw-index

Description: Index CMSSW documentation using DXR.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

es-open-indexes

Description: This job keeps last 4 weeks of data in Elasticsearch open, and it closes older indexes (archive it). We do not care about older data. By doing it we make Elasticsearch faster.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

es-push-pkginfo

Description: None

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

H 3 * * *
H 15 * * *

find-cmssw-install-packages

Description: avalenzu(testing): dummy job for builid cmssw images.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

fix-backport-prs

Description: Periodicaly runs https://github.com/cms-sw/cms-bot/blob/master/fix-backport-labels.py to check if the PR on master, which backport has been requested, is merged. If the original PR has been merged, it changes all opened backport PRs of it from backport(blue) to backport-ok(green)

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

H * * * *

gh-teams

Description: printing some python file

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

H 22 * * *

git-mirror-repository

Description: Mirror one git repository.
Q/A:

  • Q: Failed due to `remote: GitLab: Failed to authorize your Git request: internal API unreachable`
  • A: Problem with GitLab. Just file a ticket.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

git-reference-cvmfs

Description: Create GIT Reference for cms-sw/cmssw repository in /cvmfs/cms-ci.cern.ch/git/cms-sw. This is automatically triggered by "git push" to cmssw repo.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

git-repo-size

Description: None

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

5 11,23 * * *

github-hooks

Description: Adds and configures web hooks in github repositories. The web hooks are then used to send some events to a specified url and then a jenkins job is triggered based on the information passed from the web hook , as jenkins parameters.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

H 0 * * *

github-labels

Description: Adds and configures web hooks in github repositories. The web hooks are then used to send some events to a specified url and then a jenkins job is triggered based on the information passed from the web hook , as jenkins parameters.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

H 1 * * *

github-monitor-api

Description: PyGitHub API at: https://pygithub.readthedocs.io/en/latest/github.html

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

* * * * *

github-push-hook

Description: Updates cms-bot clone on cmssdt. This job is also triggered via github web hook. Please do not add/remove any parameters of this job otherwise github web hooks will not be able to triiger the job

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

H */6 * * *

GitHub-refactor-cmssw

Description: Refactors the CMSSW repo.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

GitHub-refactor-cmssw-module

Description: Refactors one module of CMSSW and makes an PR.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

github-webhook

Description: This is cms bot job which is triggered by github webhooks ( https://cmssdt.cern.ch/SDT/cgi-bin/github_webhook ) for every valid comment added to github PRs. Also it runsevery 30mins to make sure any webhooks were not missed.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

ib-build-logs

Description: It is build periodically (H/30 * * * *). Runs on cmssdt. Projects to build: update-github-pages

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

H/30 * * * *

ib-cache-to-eos

Description: Job to run das client and cache the results in github to be used by IBs.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

H 10,22 * * *

ib-reproduce-unittests

Description: None

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

ib-run-additional-tests

Description: None

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

ib-run-addons

Description: Runs addons test on IB. Results are shown as "Other test" in the IB page for the IB.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

ib-run-cfipython

Description: This job gets the cfipython files for a cmssw release and push the changes to cms-sw/cmssw-cfipython repo.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

ib-run-check-headers

Description: Check for missing headers and parse the log for all errors (clang modules)

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

ib-run-flawfinder

Description: Runs Flawfinder test on IB.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

ib-run-HLT

Description: Appends job's time out information into jenkins.log file.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

ib-run-igprof

Description: Runs IgProf on IB. Results are available on IB page.


run-ib-igprof is executed with `pp` flag for `performance profiling`.



Q/a:

  • Q: Error: near line 63: unrecognized token: "�"
  • A: Igprof has a bug that from time to time it returns unrecognizable character which fails the job. There is no easy fix, so we glance over it. However, job will execute all steps and then fail.
  • Q: `IOError: [Errno 2] No such file or directory: 'runall-report-step123-.log'`
  • A: No idea... Yet.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

ib-run-invalid-includes

Description: Runs iwyu logs parsing for each IB

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

ib-run-iwyu

Description: Runs iwyu logs parsing for each IB

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

ib-run-lizard

Description: Runs Lizard, a Cyclomatic Complexity Analyzer, on IB. Results are available on IB page.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

ib-run-material-budget

Description: Runs Validation/Geometry/test/runP_Tracker_cfg.py and MaterialBudget.C for an IB

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

ib-run-pr-crab

Description: PR Unit Test for CRAB updates.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

ib-run-pr-tests

Description: This is CMSSw CI Job which runs the build and unit tests part of CI. It first kills any other job whcih is running for same PullRequests/Architecture. and then starts the build process. Once the build process if done then it trigger other jobs to start the deployment of the build artifects on CVMFS and start the tests. at the end it run the unit tests. Unit tests are run here as it needs the CMSSW/tmp directory which is not deployed on CVMFS.

Q/A: In case this job fails then in most cases just re-try it. In case it is failed due to network/github issues then it is better to wait for the service to get healthy.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

ib-run-profiling

Description: Runs FastTimerService and Igprof on the RECO and PAT steps for high pileup workflow.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

ib-run-python3

Description: Runs iwyu logs parsing for each IB

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

ib-run-qa

Description: Runs Quality Assurance (QA) test on IB. Results are available at IB page's Q/A section.
If this job fails, logs can be found under https://cmssdt.cern.ch/buildlogs//os_arch_compiler/www/week_day/...

Sometimes it can hang during scp copy / ssh. This failure can be found by looking for a gap of several hours in the log file during scp / ssh.
If the job runs for more than several hours, the procedure is to login to build node, check which job is hanging and take the appropiate action, if necessary.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

ib-run-relvals

Description: The job runs release validations, as validations are separated on pieces (1of6 2of6 etc).
It can fail due to connection issues (e.g. Remote call on machine-XYZ failed).

Sometimes it can also hang during scp copy / ssh (which is usually the last job). This failure can be found by looking for a gap of several hours in the log file where the Jenkins job is trying to finish the last test job.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

ib-run-static-checks

Description: Runs a few tests only for the IB, for comparison with those ran by the pull request.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

ib-run-valgrind

Description: This job runs valgrind tool for selected IBs when build IB job is complete.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

ib-schedule

Description: This is a warpper job which runs daily at 11h and 23h (except no IB at 23h Sat & 11h Sun but a special IB at Sun 00h) to triger 'ib-tag-and-schdule' sub-project.
This was created to avoid the issue with Jenkins Dynamic Parameters.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

#Special IB at 0h on Sunday
5  0   * *  0
#We skip Sunday 11h IB to leave resources for special Sunday's 0h IB
5 11  * * 1,2,3,4,5,6
#We do not run SAT 23H IB instead we start a special SUN 00h00 IB.
5 23  * *  0,1,2,3,4,5

ib-schedule-additional-tests

Description: Wrapper project to schedule aditonal test such as Flawfinder, Lizard, IgProf, etc.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

ib-schedule-pr-tests

Description: Build a pull request

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

ib-tag-and-schdule

Description: This job tags and schedules all the IBs.
It reads the cms-bot/config.map to find all available CMSSW release cycles and tag them in github. Once tags is done then it triggers 'build-any-ib' sub-job to actually build IBs for all possible architectures (mentioned in cms-bot/config.map). For Developement IB, it also triggers the generation of LXR indexes

For Sunday's 00h IBs, it also resets the CMSREP weekly repositories by building a dummy package and uploading it to cms.weekN. It also then triggers 'ib-install-cvmfs' sub-job to get the new cms.weekN deployed on the /cvmfs/cms-ib.cern.ch

This job re-tries itself on failure. So there is no need to re-try it manually (unless all re-tries failed).

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

ib-validation

Description: This job waits for the recently installed IB on CVMFS to be visible on CERN Stratum 1. Once the IB is available on Stratum 1 then it triggers various validation jobs. If this job fails or timeout then just re-run it. Failure rate is very low but on Wednesday CERN IT runs CVMFS Stratum 1 garbage collection which can add extra few hours delay and may cause this job to timeout.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

kill-build-release

Description: Aborts and kills a release building process.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

kill-hanging-jobs

Description: This project checks all selected host for process running longer then set treshold (default 2 days) and kill them.

Q/A:

  • Q: The job is taken more time then is supposed to (status bar becomes red).
  • A: Most likely one of the machine is in a weird state (last time it was issues with CMSFS) and job hangs when connecting to it. IF it is a cmsbuildXX machine, just delete it and restart the job.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

H H/8 * * *

lfn-to-ibeos

Description: Job to run das client and cache the results in github to be used by IBs.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

lumi-image-update

Description: Update sif image for LUMI HPC

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

lxr-check

Description: This job checks if https://cmssdt.cern.ch/lxr/ is acessable and if not, starts lxr-run-container job to restart the service.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

H/20 * * * *

lxr-generate-index

Description: This job sets modification timestamp of CMSSW source code according to commit history before indexing it using LXR.

LXR index files based on modification timestamp. <code>git clone</code> ,however, sets files timestamps to command's execution time. Without it, LXR would index every file, increasing jobs execution duration and database size.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

lxr-remove-index

Description: Deletes index of of IB's older then treshold on LXR ( default 14 days). If specified, it also deletes Release index.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

H 5  * * *

lxr-run-container

Description: This job stops and deletes old docker contaner of LXR service and starts new one.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

material-budget-ref

Description: Runs Validation/Geometry/test/runP_Tracker_cfg.py and MaterialBudget.C for an IB

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

monitor-openstack-vms

Description: This job run periodically to monitor the running state of vms in openstack , if stopped , creates an email alert.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

@hourly

nodes-sanity-check

Description:

This job runs a check that makes sure /afs, /cvmfs, /cvmfs/cms.cern.ch, /cvmfs/cms-ci.cern.ch, /cvmfs/cms-ib.cern.ch, /cvmfs/grid.cern.ch and /cvmfs/unpacked.cern.ch are mounted and that singularity can start a container in the node. It runs on lxplus7, lxplus8 and lxplus9 considering the available hosts at the time of running and both aarch64 and ppc64le machines.

If any of the checks fails, it sends an email notification and writes the node name into a blacklist. For lxplus nodes, this blacklist is checked before connecting any node to make sure Jenkins does not connect to a blacklisted node. For aarch64 and ppc64le machines, it takes the corresponding nodes offline.

The blacklist is cleaned up for lxplus nodes if the tests run again successfully in any of the nodes or if the host is not in the list of available hosts anymore. The aarch64 and ppc64le nodes must be taken online manually after receiving the email notification and fixing the issue.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

H/30 * * * *

nodes-status-summary

Description: This job checks the status of the Jenkins nodes every day to verify that none of them has been manually disconnected/or disconnected by CLI and not bring back online.
It also notifies about the blacklisted nodes.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

H 16 * * *

process-deprecate-releases

Description: This job marks selected realeases depricated in realease map. Grid jobs will not use depricated release and posibly uninstall it. This is useful if a specific release contains and needs to be removed.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

process-external-elastic-stats

Description: This job process resource metrics jsons for externals

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

H 22-23 * * *

process-relval-logs

Description: This job process partial logs of Relvals and place files accordingly.
There is no need to re-try newer run is successful or running. If it keeps on failing then one need to check the logs and find out the reason of failure. In that case some manual work is needed to cleanup cmssdt.cern.ch logs.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

H/20 * * * *

query-build-release-issues

Description: Processes a github issue to check if it is requesting the build of a new release. If the issue is not requesting any release, it ignores it.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

H/15 * * * *

reindex-elasticsearch-indexes

Description: Job to reindex existing indexes whenever the template schema is changed or for whatever other reason

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

release-produce-changelog

Description: Posts a message in the github issue that triggered the build. Structure of the message depends on the option used.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

run-cvmfs-commands

Description: None

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

run-pr-code-checks

Description: This run "scram build code-checks" for a cmssw PR to find out if it comply with cmssw code checks. In a CMSSW dev area, it runs git cms-merge-topic -u PR scram build code-checks

This job re-tries twice to recover. If it still fails after two retires then try to understand the failure and re-try again.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

slaves-checks

Description: A wrapper project that starts a workspace-cleanup-slave and test-docker jobs on selected slaves.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

H 07 * * *

summary-of-merged-prs

Description: Generates statistics for each IB ( merged pull request since last IB, test result summary, .etc) as well as structure of release que and stores it in .json files. It then push it to cms-sw.github.io repo as well as deploys on the web server. It is used to generate IB page.



Q/A

  • Q: The job failed.
  • A: Most likely Github rejected push request because other job pushed after `git pull --rebase` . Do not worry - job builds quite often and next build shouls be succesful. But if it keeps on failing (over 3 times) then try to re-run the job with "RESET" enabled.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: ['update-github-pages']

Periodic builds:

Not periodically build

test-build-periodic

Description: None

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

H/15 * * * *

test-build-release

Description: avalenzu: Testing build error in release cms-sw/cmssw#41976 for el8_amd64_gcc12

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

trigger-docker-packages-check

Description: This job checks for changes in the parent image. If there are changes, it triggers the build-docker-container job so that our based image is updated and uploaded to the registry.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

update-categories-page

Description: Generates categories.json file and uploads it to this github repo.
The date is later used to generate this page.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

H 0 * * *

update-config-guess

Description: None

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

H 10,22 * * *

update-das-queries

Description: Job to run das client and cache the results in github to be used by IBs.
Job could fail due to github issues or disk related issue. Normally it is safe to just retry it once github/disk issues are resolved.
There is no need to retry if it has been rerun successfully after the last failure.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

update-data-tag-on-pr-merge

Description: This job creates new release and tag it on cms-data PR merge. It also creates cmsdist PR with the new release tag

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

update-github-pages

Description: This job update contents of the "data" directory in cms-sw.github.io. It downloads our github pages, parses the log files, updates the documentation accordingly, and uploads the pages again. In this case, the job is automatically triggered after some minutes. If it fails, one just needs to check that the next build was successful so that pages are updated correctly.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: ['cleanup-cms-sw-io-history', 'ib-build-logs']

Periodic builds:

Not periodically build

update-ibeos-cache

Description: Job to run das client and cache the results in github to be used by IBs.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

H */6 * * *

update-release-map

Description: This project adds the release information in cms-bot/releases.map file. CVMFS installation is started once a release is available in this file.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

update-release-notes-collection

Description: Generates CMSSW release notes using Jekyll.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

upload-release

Description: This job uploads a release on cmsrep server once approved by release manager.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

upload-release-setup

Description: This job uploads a release on cmsrep server once approved by release manager.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

web-update-cmssdt-ib

Description: Job used to transpile cmssdt ib page from ECMAscript6 to regular javascript and push changes to github.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

web-update-logReader

Description: Transpiles Logreader to pure js/html/css and deploys to cmssdt-web.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build

workspace-cleanup-slave

Description: Connect to selected slave and cleans (deletes) workspace.

Project is enabled.

Upstream projects:

Downstream projects:

Sub-projects:

Triggers from: []

Periodic builds:

Not periodically build