kerbrose ticket
kinit username@FNAL.GOV
username@cmslpc-el9.fnal.gov
Eos area
eos root://cmseos.fnal.gov
EOS area is not provided by default. A detailed instructions can be found here.
How to get an EOS area enabled or linked to CERNusername
Note: this linking is also used for submitting cmslpc condor jobs
- You can get your EOS area linked to your CMS grid certificate by filling a simple form called "CMS Storage Space Request" in the LPC Service Portal form: "CMS Storage Space Request" (Request for creation of EOS area or for increases to EOS space)
- Use your Fermilab Services credentials to login
- Select "Enable" and fill out the form
- Your DN is the result of voms-proxy-info --identity after authenticating your grid certificate with the cms voms on a linux system
- Put in your CERNusername (the username that you use to login to lxplus.cern.ch)
Note: Your ServiceNow request will be completed automatically within a minute. However, the information is propagated to all nodes with a system that only runs during FNAL business hours. This may take 1-3 hours during FNAL business day.
Note: If you change your CERN fullname, when you renew your CERN grid certificate it will have a different identity, for instance if you went from Firstname MiddleName Lastname to Firstname Lastname in the CERN database. Then your CERN grid certificate will look different, and thus the new certificate will need to be linked to your EOS area. To resolve this, open a new "CMS Storage Space Request" (Enable), using your services account to login, as described above.
-
For lxplus:
source lxplus.cern.ch cd ~/Analysis/Analysis_HH-bbgg conda activate higgs-dna #Go to HiggsDNA pip install -e . python scripts/pull_all.py --all
-
lpc:
login on fermilab cd ~/Analysis/Analysis_HH-bbgg conda activate higgs-dna #Go to HiggsDNA pip install -e . python scripts/pull_all.py --all
source /cvmfs/cms.cern.ch/cmsset_default.sh
export CMSSW_GIT_REFERENCE=/cvmfs/cms.cern.ch/cmssw.git.daily
cd /eos/user/<first-letter-of-username>/<username>
mkdir YOURWORKINGAREA
cd YOURWORKINGAREA
### If you are using Bash shell
export SCRAM_ARCH=el9_amd64_gcc11
### Alternatively, If you are using the default tcsh shell (or csh shell)
setenv SCRAM_ARCH el9_amd64_gcc11
### Then, in both cases:
cmsrel CMSSW_13_0_17
cd CMSSW_13_0_17/src
cmsenv
other methods can be
source /cvmfs/cms.cern.ch/cmsset_default.sh
cmsrel CMSSW_X_Y_Z # Replace X_Y_Z with the desired release version
cd CMSSW_X_Y_Z/src
cmsenv
further, compile and test using
scram b -j 8 # Adjust the number 8 depending on the number of cores available
To be able to check out specific CMSSW packages from GitHub, you will need to configure your local account. You only have to do this command once for any given cluster you are working on, such as lxplus:
git config --global user.name "[Name]"
git config --global user.email [Email]
git config --global user.github [Account]
To see your current git configuration you can use the following command:
git config --global -l
Now you can initialize the CMSSW area as a local git repository:
git cms-init
This last command will take some time to execute and will produce some long output, be patient.
When you get the prompt again, run the following command:
echo $CMSSW_BASE
-
HTCondor If your cluster uses HTCondor:
- Check the status of all your jobs:
condor_q
- Check the status of specific jobs by ID:
condor_q <job_id>
- Check the status of all your jobs:
-
Slurm If your cluster uses Slurm:
- Check the status of all your jobs:
squeue
- Check the status of a specific job by ID:
squeue -j <job_id>
- Check the status of all your jobs:
-
Grid Engine If your cluster uses Grid Engine:
- Check the status of all your jobs:
qstat
- Check the status of a specific job by ID:
qstat -j <job_id>
- Check the status of all your jobs:
- Ensure you have the OpenSSL tool installed.
- Obtain a valid
.p12
certificate file from your certificate authority.
If you have a .p12
file, extract the user certificate and key using OpenSSL:
openssl pkcs12 -in myCertificate_lpc.p12 -out usercert.pem -clcerts -nokeys
openssl pkcs12 -in myCertificate_lpc.p12 -out userkey.pem -nocerts -nodes
Set premission for the key file:
chmod 400 userkey.pem
Set Environment variables:
export X509_USER_CERT=$HOME/.globus/usercert.pem
export X509_USER_KEY=$HOME/.globus/userkey.pem
Initialize a VOMS proxy certificate with the following command:
voms-proxy-init --rfc --voms cms -valid 192:00
To view information about your VOMS proxy certificate:
voms-proxy-info --all
If you need to remove an old or expired VOMS proxy:
voms-proxy-destroy
To regenerate or update a VOMS proxy certificate:
voms-proxy-regen --voms cms
voms-proxy-update --voms cms
ssh -L localhost:8899:localhost:8899 sraj@cmslpc-el9.fnal.gov
source ~/.bashrc
# Env setup
conda activate hhbbgg-awk
##Navigate to the specified directory
jupyter notebook --no-browser --port=8899 --ip 127.0.0.1
# Notify the user that the script has finished
echo "Initialized mamba, activated hhbbgg-awk environment, and changed directory to hhbbgg_AwkwardAnalyzer."
Basic File Operations:
ls -l
cd /path/to/directory
cp source_file destination_file
mv old_name new_name
rm -rf file_or_directory
File Permissions and Ownership:
chmod 755 file
chown user:group file
Viewing File Contents:
cat file
less file
head file
tail file
tail -f file
```bash date
uptime
whoami
df -h
free -m ```
Process Management:
ps aux
top
kill PID
pkill process_name
Network Commands:
ifconfig
netstat -tuln
ping hostname_or_ip
wget url
scp user@host:/path/to/remote/file /path/to/local/destination
Package Management (Debian/Ubuntu):
sudo apt update
sudo apt upgrade
sudo apt install package_name
sudo apt remove package_name
apt search package_name
Vim Commands:
vim file
i
:wq
:q!
Symbolic Links and Search:
ln -s target link_name
find /path/to/search -name filename
man command
conda activate higgs-dna
- On LPC
source ~/.bashrc
# Env setup
conda activate hhbbgg-awk
##Navigate to the specified directory
cd /uscms/home/sraj/nobackup/Hh-bbgg/Analysis_HH-bbgg/hhbbgg_AwkwardAnalyzer
#
- lxplus
- navigate manually
# Env setup
conda activate hhbbgg-awk
command to merge all produced .parquet
files:
python3 prepare_output_file.py --input /afs/cern.ch/user/s/sraj/Analysis/output_parquet/ --merge --root --ws --syst --cats --args "--do_syst"
To convert merged .parquet
to .root
using
in general( HiggsDNA folder)
python scripts/postprocessing/convert_parquet_to_root.py --input_parquet_files --output_parquet_file_output/file_name.root mc
eg.
python scripts/postprocessing/convert_parquet_to_root.py ../../../output_parquet/merged/NMSSM_X300_Y100/nominal/NOTAG_NOTAG_merged.parquet ../../../output_root/NMSSM_X300_Y100/NMSSM_X300_Y100.root mc
Container instructions: https://cms-ml.github.io/documentation/software_envs/containers.html LCG environemnt: https://cms-ml.github.io/documentation/software_envs/lcg_environments.html
- Update and upgrade your system
sudo apt update
sudo apt upgrade
- Install dependencies
sudo apt install dpkg-dev cmake g++ gcc binutils libx11-dev libxpm-dev libxft-dev libxext-dev python3 libssl-dev
- Install optional dependencies
sudo apt install libpng-dev libjpeg-dev libgif-dev libtiff-dev libxml2-dev libssl-dev libgsl-dev
- Download ROOT. ROOT can be downloaded from ROOT website with latst realease, https://root.cern.ch/doc/master/group__Tutorials.html, but it is preferred to use the git instructions.
git clone https://github.com/root-project/root.git
- Build and install ROOT
cd root
mkdir build
cd build
configure the build with Cmake:
cmake ..
To build ROOT (this may take some time):
cmake --build . -- -j$(nproc)
Once the build process is complete, you can install ROOT by running:
sudo cmake --build . --target install
- Set Up Environment Variables Add ROOT to your environment variables by adding the following lines to your .bashrc or .bash_profile:
source /path/to/your/root/build/bin/thisroot.sh
Make sure to replace /path/to/your/root/build/ with the actual path to your ROOT installation.
Then apply changes,
source ~/.bashrc
- Verify the installation
root
brew update
brew install cmake
Install root
brew install root
Check the root version using
root -v
To clone/pull all branch at once from the github, as for example we are taking our hhbbgg_analyzer file, git@github.com:raj2022/hhbbgg_AwkwardAnalyzer.git
git checkout --track "branch name"
Examples,
git checkout --track origin/NMSSM
and
git checkout --track origin/v2_higgsdna
git config --global user.name "Your Name"
: Sets your name in Git configuration.git config --global user.email "you@example.com"
: Sets your email in Git configuration.git config --global color.ui auto
: Enables colored output in Git.
git init
: Initializes a new Git repository.git clone <repo-url>
: Clones an existing repository from a remote source.
git status
: Displays the state of the working directory and staging area.git add <file>
: Stages a file.git add .
: Stages all changes in the current directory.git commit -m "Commit message"
: Commits staged changes with a message.git commit -am "Commit message"
: Adds and commits changes in one step.
git branch
: Lists all branches in your repository.git branch <branch-name>
: Creates a new branch.git checkout <branch-name>
: Switches to the specified branch.git checkout -b <branch-name>
: Creates and switches to a new branch.git merge <branch-name>
: Merges the specified branch into the current branch.git branch -d <branch-name>
: Deletes the specified branch.git branch -D <branch-name>
: Force-deletes the specified branch.
git remote add <name> <url>
: Adds a remote repository.git remote -v
: Lists the URLs of all remote repositories.git fetch <remote>
: Fetches changes from the remote repository.git pull <remote> <branch>
: Fetches and merges changes from the remote branch into the current branch.git push <remote> <branch>
: Pushes committed changes to the remote branch.git push origin --delete <branch-name>
: Deletes a remote branch.
git checkout -- <file>
: Discards changes in the working directory.git reset HEAD <file>
: Unstages a file without discarding changes.git reset --hard
: Discards all local changes.git revert <commit>
: Creates a new commit that undoes changes made in the specified commit.git reset --soft <commit>
: Resets the index to the specified commit without changing the working directory.
git log
: Shows commit history.git log --oneline
: Displays the commit history in a condensed format.git log --graph --oneline --all
: Visualizes the commit history as a graph.git diff
: Shows changes between commits, commit and working tree, etc.git show <commit>
: Displays changes introduced by a specific commit.
git stash
: Stashes the current changes in the working directory.git stash list
: Lists all stashes.git stash apply
: Applies the most recent stash.git stash apply stash@{n}
: Applies a specific stash from the stash list.git stash drop
: Deletes the most recent stash.git stash drop stash@{n}
: Deletes a specific stash.
git submodule add <repo-url>
: Adds a submodule to your repository.git submodule update --init
: Initializes, fetches, and checks out the content of the submodule.git submodule update --remote
: Updates the submodule to the latest commit.
git tag
: Lists all tags.git tag <tag-name>
: Creates a new tag.git tag -d <tag-name>
: Deletes a local tag.git push origin <tag-name>
: Pushes a tag to the remote repository.git push origin --tags
: Pushes all tags to the remote repository.
git cherry-pick <commit>
: Applies the changes from a specific commit.git rebase <branch>
: Applies commits from another branch onto the current branch.git reflog
: Shows a log of all the references in the local repository.git bisect start
: Starts a bisecting session to find a specific commit that introduced a bug.
git merge --abort
: Aborts the merge process and attempts to reconstruct the pre-merge state.git mergetool
: Opens a merge tool to resolve conflicts.git add <file>
: After resolving conflicts, stage the resolved file(s).git commit
: Completes the merge after resolving conflicts.
git clean -f
: Removes untracked files.git clean -fd
: Removes untracked files and directories.
git blame <file>
: Shows who modified each line of a file and when.git shortlog
: Summarizesgit log
output by author.
h
: Move cursor left.j
: Move cursor down.k
: Move cursor up.l
: Move cursor right.0
: Move to the beginning of the line.$
: Move to the end of the line.w
: Move to the beginning of the next word.b
: Move to the beginning of the previous word.G
: Go to the end of the file.gg
: Go to the beginning of the file.:n
: Go to linen
(e.g.,:10
to go to line 10).Ctrl + f
: Move forward one screen.Ctrl + b
: Move backward one screen.
i
: Insert mode (start inserting text before the cursor).I
: Insert mode at the beginning of the line.a
: Insert mode (start inserting text after the cursor).A
: Insert mode at the end of the line.o
: Insert a new line below the current line and enter insert mode.O
: Insert a new line above the current line and enter insert mode.Esc
: Return to normal mode.v
: Enter visual mode (select text).V
: Enter visual line mode (select whole lines).Ctrl + v
: Enter visual block mode (select a block of text).
x
: Delete the character under the cursor.dd
: Delete the current line.dw
: Delete from the cursor to the end of the word.d$
: Delete from the cursor to the end of the line.d0
: Delete from the cursor to the beginning of the line.u
: Undo the last action.Ctrl + r
: Redo the last undone action.yy
: Copy (yank) the current line.yw
: Copy (yank) from the cursor to the end of the word.y$
: Copy (yank) from the cursor to the end of the line.p
: Paste after the cursor.P
: Paste before the cursor.r
: Replace the character under the cursor.R
: Enter replace mode (overwrite characters).
/pattern
: Search forpattern
in the file (e.g.,/foo
to search for "foo").n
: Move to the next occurrence of the search pattern.N
: Move to the previous occurrence of the search pattern.:%s/old/new/g
: Replace all occurrences ofold
withnew
in the file.:%s/old/new/gc
: Replace all occurrences with confirmation.
:w
: Save the current file.:wq
: Save and exit Vim.:q
: Quit Vim.:q!
: Quit without saving changes.:wq!
: Force save and exit (useful when the file is read-only).ZZ
: Save and quit (equivalent to:wq
).ZQ
: Quit without saving (equivalent to:q!
).
v
: Start visual mode.V
: Start visual line mode.Ctrl + v
: Start visual block mode.y
: Yank (copy) the selected text.d
: Delete the selected text.p
: Paste the yanked text after the selection.>
: Indent the selected text.<
: Unindent the selected text.
>>
: Indent the current line.<<
: Unindent the current line.=
: Auto-indent the selected text or current line.
:e filename
: Openfilename
for editing.:bnext
or:bn
: Go to the next buffer.:bprev
or:bp
: Go to the previous buffer.:bd
: Close the current buffer.:ls
: List all open buffers.
:split filename
: Openfilename
in a horizontal split.:vsplit filename
: Openfilename
in a vertical split.Ctrl + w, s
: Split the current window horizontally.Ctrl + w, v
: Split the current window vertically.Ctrl + w, w
: Switch between split windows.Ctrl + w, q
: Close the current split window.:tabnew filename
: Openfilename
in a new tab.gt
: Go to the next tab.gT
: Go to the previous tab.:tabclose
: Close the current tab.
.
: Repeat the last command.:!command
: Run an external shell command (e.g.,:!ls
).:set number
: Show line numbers.:set nonumber
: Hide line numbers.:set ignorecase
: Ignore case in searches.:set noignorecase
: Make searches case-sensitive.:set hlsearch
: Highlight search results.:set nohlsearch
: Disable search result highlighting.
"a
: Access registera
."ap
: Paste the contents of registera
."ayy
: Yank into registera
.:reg
: View the contents of all registers.
qa
: Start recording a macro into registera
.q
: Stop recording the macro.@a
: Play the macro stored in registera
.@@
: Replay the last played macro.
jk
: Mapjk
to exit insert mode (can be set in your.vimrc
).
-
ls
: List directory contents.ls -l
: List with detailed information.ls -a
: List all files, including hidden ones.ls -lh
: List with human-readable file sizes.
-
cd <directory>
: Change the current directory.cd ..
: Move up one directory level.cd ~
: Change to the home directory.cd -
: Go back to the previous directory.
-
pwd
: Print the current working directory. -
mkdir <directory>
: Create a new directory.mkdir -p <path>
: Create directories along the specified path.
-
rmdir <directory>
: Remove an empty directory. -
rm <file>
: Remove a file.rm -r <directory>
: Remove a directory and its contents recursively.rm -f <file>
: Force removal of a file without prompting.rm -rf <directory>
: Force remove a directory and its contents.
-
cp <source> <destination>
: Copy files or directories.cp -r <source> <destination>
: Copy directories recursively.cp -p <source> <destination>
: Preserve file attributes during copy.
-
mv <source> <destination>
: Move or rename files or directories. -
touch <file>
: Create an empty file or update the timestamp of an existing file. -
cat <file>
: Concatenate and display the content of a file.cat file1 file2 > combined_file
: Concatenate multiple files into one.
-
more <file>
: View the content of a file one screen at a time. -
less <file>
: View the content of a file with backward navigation. -
head <file>
: Display the first 10 lines of a file.head -n <number> <file>
: Display the first<number>
lines.
-
tail <file>
: Display the last 10 lines of a file.tail -n <number> <file>
: Display the last<number>
lines.tail -f <file>
: Follow the file as it grows, useful for log files.
-
grep <pattern> <file>
: Search for a pattern in a file.grep -r <pattern> <directory>
: Recursively search for a pattern in files under a directory.grep -i <pattern> <file>
: Case-insensitive search.grep -v <pattern> <file>
: Invert match; display lines that do not match the pattern.grep -n <pattern> <file>
: Display line numbers with matching lines.
-
sed 's/old/new/g' <file>
: Replace all occurrences ofold
withnew
in a file.sed -i 's/old/new/g' <file>
: In-place replacement in a file.
-
awk '{print $1, $2}' <file>
: Print specific columns from a file. -
cut -d '<delimiter>' -f <fields> <file>
: Cut out sections from a file using a delimiter. -
sort <file>
: Sort lines in a file.sort -r <file>
: Sort lines in reverse order.sort -n <file>
: Sort lines numerically.
-
uniq <file>
: Report or filter out repeated lines.uniq -c <file>
: Count and display occurrences of repeated lines.
-
wc <file>
: Count the number of lines, words, and characters in a file.wc -l <file>
: Count lines.wc -w <file>
: Count words.wc -c <file>
: Count characters.
-
chmod <permissions> <file>
: Change the file permissions.chmod 755 <file>
: Give the owner full access and others read and execute access.chmod +x <file>
: Make a file executable.
-
chown <owner>:<group> <file>
: Change the owner and group of a file.chown -R <owner>:<group> <directory>
: Change ownership recursively.
-
chgrp <group> <file>
: Change the group ownership of a file.
-
ping <host>
: Check connectivity to a host. -
ifconfig
: Display or configure network interfaces. -
curl <url>
: Transfer data from or to a server.curl -O <url>
: Download a file from a URL.curl -I <url>
: Fetch the headers only.
-
wget <url>
: Download files from the web.wget -c <url>
: Continue a previous download.
-
scp <source> <user@host:destination>
: Securely copy files between hosts.scp -r <source> <user@host:destination>
: Recursively copy directories.
-
ssh <user@host>
: Securely connect to a remote server.
-
ps
: Display currently running processes.ps aux
: Display all processes with detailed information.
-
top
: Display real-time system resource usage and running processes. -
htop
: Enhanced version oftop
with more features (if installed). -
kill <pid>
: Kill a process by its PID.kill -9 <pid>
: Force kill a process.
-
killall <process_name>
: Kill all processes with a specific name. -
bg
: Move a job to the background. -
fg
: Bring a job to the foreground. -
jobs
: List all jobs.
#!/bin/bash
: Shebang line to specify the script should be run in Bash.echo "text"
: Print text to the terminal.read <variable>
: Read user input into a variable.$(command)
: Command substitution; captures the output of a command.var=value
: Assign a value to a variable.if [ condition ]; then ... fi
: If statement.for var in list; do ... done
: For loop.while [ condition ]; do ... done
: While loop.function_name() { ... }
: Define a function.chmod +x script.sh
: Make a script executable../script.sh
: Run a script.
command > file
: Redirect output to a file (overwrite).command >> file
: Append output to a file.command 2> file
: Redirect error output to a file.command1 | command2
: Pipe the output ofcommand1
tocommand2
.command < file
: Use a file as input to a command.
tar -cvf archive.tar <files>
: Create a tar archive.tar -xvf archive.tar
: Extract a tar archive.tar -czvf archive.tar.gz <files>
: Create a compressed tar archive with gzip.tar -xzvf archive.tar.gz
: Extract a compressed tar archive with gzip.tar -cjvf archive.tar.bz2 <files>
: Create a compressed tar archive with bzip2.tar -xjvf archive.tar.bz2
: Extract a compressed tar archive with bzip2.zip archive.zip <files>
: Create a zip archive.unzip archive.zip
: Extract a zip archive.gzip <file>
: Compress a file using gzip.gunzip <file.gz>
: Decompress a gzip file.
df -h
: Display disk space usage in human-readable format.du -h <directory>
: Display disk usage of a directory and its contents.du -sh <directory>
: Display total disk usage of a directory.
sudo apt update
: Update the package list.sudo apt upgrade
: Upgrade all installed packages.sudo apt install <package>
: Install a package.sudo apt remove <package>
: Remove a package.sudo apt autoremove
: Remove unnecessary packages.sudo apt search <package>
: Search for a package in the repository.sudo apt list --installed
: List all installed packages.
-
man <command>
: Display the manual page for a command. -
history
: Show command history.!n
: Re-execute command numbern
.!!
: Re-execute the last command.
-
alias name='command'
: Create an alias for a command. -
unalias name
: Remove an alias. -
clear
: Clear the terminal screen. -
date
: Display or set the system date and time. -
cal
: Display a calendar. -
uptime
: Show how long the system has been running. -
whoami
: Display the current user. -
sudo <command>
: Run a command as the superuser. -
exit
: Exit the current shell session. -
echo $SHELL
: Display the current shell. -
uname -a
: Display system information.
Vim commands can be found at, https://github.com/raj2022/_Practice_/tree/main/vim
This document lists commonly used HTCondor commands for submitting and managing batch jobs on CERN's LXPLUS.
condor_submit <submit_file>
Submit a job using a Condor submit description file.
condor_submit_dag <dag_file>
Submit a Directed Acyclic Graph (DAG) job.
condor_q
Show the status of jobs in the queue for the current user.
- View all jobs:
condor_q -all
- Check jobs for a specific user:
condor_q <username>
- View held jobs:
condor_q -hold
- Detailed information for a specific job:
condor_q -long <job_id>
condor_status
Check the status of Condor pool resources.
- View scheduler information:
condor_status -schedd
- Detailed resource information:
condor_status -long
condor_rm <job_id>
Remove a job from the queue.
- Remove jobs from a specific schedd:
condor_rm -name <name>
- Remove jobs matching a constraint:
condor_rm -constraint <expression>
condor_hold <job_id>
Place a job on hold.
condor_release <job_id>
Release a held job.
condor_suspend <job_id>
Suspend a running job.
condor_continue <job_id>
Resume a suspended job.
condor_history
View information about completed jobs.
- Details for a specific job:
condor_history <job_id>
- Show the last
n
completed jobs:condor_history -limit <n>
condor_userlog
Parse and display events from the user log file.
condor_fetchlog <hostname> <daemon>
Fetch log files from Condor daemons on remote machines.
condor_analyze
Analyze why a job isn't running.
condor_ping
Ping Condor daemons for communication checks.
condor_config_val <parameter>
Query configuration values for Condor.
condor_top
Monitor Condor in real time.
condor_compile <executable>
Compile a program for checkpointing.
condor_checkpoint
Manually checkpoint a job.
condor_reschedule
Force the scheduler to reevaluate the job queue.
condor_kbdd
Monitor keyboard and mouse activity for interactive jobs.
- HTCondor is configured on CERN’s LXPLUS for the LHC computing grid.
- Submit jobs using
condor_submit
with specific requirements for computing resources. - Monitor jobs using
condor_q
and check logs usingcondor_history
.
- Submit jobs using
For more details, consult the HTCondor Manual.