-
Notifications
You must be signed in to change notification settings - Fork 0
Home
Maximilian Swiatlowski edited this page Nov 1, 2021
·
51 revisions
Welcome to the Organization wiki! Here we can organize information on the atlas-calo-ml project.
Aaron has just introduced a new format for our datasets, described in these slides.
Full samples are available for the new nearby particles using this format! On the grid, these are:
user.angerami.mc16_13TeV.900147.PG_singleDelta_logE5to2000.e8312_e7400_s3170_r12383.v01-45-gaa27bcb_OutputStream
user.angerami.mc16_13TeV.900148.PG_singlerho_logE5to2000.e8312_e7400_s3170_r12383.v01-45-gaa27bcb_OutputStream/
Single particles are also ready!
user.angerami.mc16_13TeV.900246.PG_singlepi0_logE0p2to2000.e8312_e7400_s3170_r12383.v01-45-gaa27bcb_OutputStream/
user.angerami.mc16_13TeV.900247.PG_singlepion_logE0p2to2000.e8312_e7400_s3170_r12383.v01-45-gaa27bcb_OutputStream/
All samples are on EOS at /eos/user/a/angerami/ML4Pions/MLTreeData.
All-cell version of single rho dataset available on EOS at:
/eos/user/a/akong/ml4pions/singlerho-percell/
Jet samples are also ready!
user.mswiatlo:user.mswiatlo.361021.Pythia8EvtGen_A14NNPDF23LO_jetjet_JZ1W.recon.ESD.e3569_s3170_r10788_v01-45_OutputStream
user.mswiatlo:user.mswiatlo.361024.Pythia8EvtGen_A14NNPDF23LO_jetjet_JZ4W.recon.ESD.e3668_e5984_s3170_r10788_v01-45_OutputStream
user.mswiatlo:user.mswiatlo.361025.Pythia8EvtGen_A14NNPDF23LO_jetjet_JZ5W.recon.ESD.e3668_e5984_s3170_r10788_v01-45_OutputStream
user.mswiatlo:user.mswiatlo.361026.Pythia8EvtGen_A14NNPDF23LO_jetjet_JZ6W.recon.ESD.e3569_e5984_s3170_r10788_v01-45_OutputStream
user.mswiatlo:user.mswiatlo.361027.Pythia8EvtGen_A14NNPDF23LO_jetjet_JZ7W.recon.ESD.e3668_e5984_s3170_r10788_v01-45_OutputStream
user.mswiatlo:user.mswiatlo.361028.Pythia8EvtGen_A14NNPDF23LO_jetjet_JZ8W.recon.ESD.e3569_e5984_s3170_r10788_v01-45_OutputStream
user.mswiatlo:user.mswiatlo.361029.Pythia8EvtGen_A14NNPDF23LO_jetjet_JZ9W.recon.ESD.e3569_e5984_s3170_r10788_v01-45_OutputStream
These are located on EOS at: /eos/user/m/mswiatlo/images/v01-45/jets
Sample calo geometry tree extended to include neighbour information.
/eos/user/a/angerami/ML4Pions/MLTreeData/CellGeo.neighbours.root
new branches cell_geo_next* and cell_geo_previous* for Phi, Eta, Samp, SubDet and SuperCalo. Values are the index in the cell tree corresponding to the neighbour. e.g. cell_geo_eta[cell_geo_nextInEta[10]] is the eta value of cell adjacent to the 10th cell. Following the convention in athena, next and previous correspond to adjacent cells that are farther/nearer from the origin. Thus for eta values that are positive, the eta of the nextInEta neighbour will be larger, while for eta values that are negative the nextInEta neighbour will be smaller (although larger in absolute value).
These use the 'old format' archived here: https://github.com/atlas-calo-ml/MLTree/tree/v01.1
user.mswiatlo:user.mswiatlo.900147.PG_singleDelta_logE5to2000_flatM1p2to5.recon.ESD.e8312_e7400_s3170_r12383.images_v01.1_OutputStream
user.mswiatlo:user.mswiatlo.900148.PG_singlerho_logE5to2000_flatM0p5to5.recon.ESD.e8312_e7400_s3170_r12383.images_v01.1_OutputStream
user.mswiatlo:user.mswiatlo.900246.PG_singlepi0_logE0p2to2000.recon.ESD.e8312_e7400_s3170_r12383.images_v01.1_OutputStream
user.mswiatlo:user.mswiatlo.900247.PG_singlepion_logE0p2to2000.recon.ESD.e8312_e7400_s3170_r12383.images_v01.1_OutputStream
They're on EOS at /eos/user/m/mswiatlo/images/v01-1/
Jet samples are available at:
user.mswiatlo:user.mswiatlo.361021.Pythia8EvtGen_A14NNPDF23LO_jetjet_JZ1W.recon.ESD.e3569_s3170_r10788_v01.1_OutputStream
user.mswiatlo:user.mswiatlo.361024.Pythia8EvtGen_A14NNPDF23LO_jetjet_JZ4W.recon.ESD.e3668_e5984_s3170_r10788_v01.1_OutputStream
user.mswiatlo:user.mswiatlo.361025.Pythia8EvtGen_A14NNPDF23LO_jetjet_JZ5W.recon.ESD.e3668_e5984_s3170_r10788_v01.1_OutputStream
user.mswiatlo:user.mswiatlo.361026.Pythia8EvtGen_A14NNPDF23LO_jetjet_JZ6W.recon.ESD.e3569_e5984_s3170_r10788_v01.1_OutputStream
user.mswiatlo:user.mswiatlo.361027.Pythia8EvtGen_A14NNPDF23LO_jetjet_JZ7W.recon.ESD.e3668_e5984_s3170_r10788_v01.1_OutputStream
user.mswiatlo:user.mswiatlo.361028.Pythia8EvtGen_A14NNPDF23LO_jetjet_JZ8W.recon.ESD.e3569_e5984_s3170_r10788_v01.1_OutputStream
user.mswiatlo:user.mswiatlo.361029.Pythia8EvtGen_A14NNPDF23LO_jetjet_JZ9W.recon.ESD.e3569_e5984_s3170_r10788_v01.1_OutputStream
We don't have these on EOS as they're quite big. But if you download them, let us know so we can post things here!
Old version from Max and Nicholas, including pre-sampler and phi bug fix and tracks and jets:
user.mswiatlo.428000.ParticleGun_single_pi0_logE0p2to2000.images_v10_wTrack_wJets_v01-36-g99e8a7a_OutputStream/
user.mswiatlo.428001.ParticleGun_single_piplus_logE0p2to2000.images_v10_wTrack_wJets_v01-36-g99e8a7a_OutputStream/
user.mswiatlo.428002.ParticleGun_single_piminus_logE0p2to2000.images_v10_wTrack_wJets_v01-36-g99e8a7a_OutputStream/
And on EOS:
/eos/user/m/mswiatlo/images/v10_wJets_wTrack (let me know if you need access to this directory-- everyone on the email list should be able to see it)
The older version used for the publication was the v7 images located in the same directory.
November 1, 2021
- Good technical discussion with contributions from LBNL and LLNL on graph architectures and data formats
- Useful to see the code people are using!
- Also updates from Piyush on various approaches and results with his combined network
- The "geometric" graph performs marginally better, hard to tell if things are consistent
- Jan: might it be informative to look at differences of performance in eta regions, etc.?
- Kaela had looked at this initially with DeepSets
- Agreed to look at this
- Currently these results are not eta-restricted
- Aaron: performance of classification as a function of energy could be useful as well
- Max: is the bumpy response distribution expected? Worse than before? What's a better metric?
- Piyush: could just be a feature of the data
- Jan: studied this in CNN, looked pretty flat-- but that was with a central eta only!
- Could be eta dependent?
- But we should figure out if it's:
- A feature of the dataset (eta?)
- A feature of our loss function
- A feature of our metric..
- Something else?
October 25, 2021
- Announcements: need contributors for central framework development
- PFlow outputs, more truth hit per particle info?
- Round table next
- Jan: merged in new update with tf.dataset loading examples
- Lots of trial and error to get this working
- may be useful example for others
- Piyush: next week can show the "custom dataset generator pickling" which makes things work faster
- First time it reads and processes as pickle files, then further times are much faster
- First epoch is slow, then everything after is fast
- No tf.dataset usage, but works well
- Next week: slides from various efforts on dataset loading to compare techniques?
- Russell has his memory mapped tf.dataset technique as well
- Adam met with Russell to discuss example coe: great!
- Russell: feedback on graph structure: any consensus on what is best?
- Max: kNN and geometric tree? anything best?
- Piyush: tried many options, kNN 9 for example working well
- Russell: tried energy in edges?
- Not yet, only euclidean distance
- Russell: working on pointnet and segmentation for EM vs NonEM
- Do still want datasets that have truth info per particle, instead of just EM vs non EM
October 18, 2021
- Albert-- playing with DM, but hard to train with PFN network
- Will iterate with Russell
- Adam-- having trouble with training, posted on Slack
- Will take a look
- Dewen-- still implementing GNN, report soon
- Dhanush-- haven't had a lot of time on the "new cluster collection"
- Was helping Mark with ONNX in r22
- Max reminds that others are interested in the cluster collection-- thanks for this!
- Mariel and Ming
- Mariel getting started, welcome!
- Ming exploring features
- https://twiki.cern.ch/twiki/bin/viewauth/AtlasComputing/ClusterMoments
- Piyush: working on simultaneous classification and regression with kNN graphs and structured graphs
- Various steps ongoing, optimizing
- Russell: working on segmentation, but have the weird issues of overlapping classes of energy in each cells
- Trying PointNet and GACs
- Would be interesting to see what graphs are working best down the line to help with GACs
- Trying PointNet and GACs
- Elham work is ongoing, replicating the energy flow example work and then starting up the MDN additions
August 9, 2021
- Nothing specific
- Looking at images for olverapping particles, as as well as cluster isolation
- Isolation is complementary to the overlap R variable
- Interesting information available
- Low response jets have been fixed! Hooray
- Was just a bug in the code
- Would like to make a MR soon with improved functions
- PFN works well at identifying the hadrons, even for non-pion hadrons
- Overlapping EM components is more difficult...
- Takes quite some selection to find the pure sample of the pi0
- Sanmay: look at profiles/shower properties?
- Did look at this in the start, but not with the latest selections-- can do that now
- Would be interesting to help undrestand the consistency of the training on different datasets, etc.
July 26, 2021
-
https://arxiv.org/pdf/2107.10207.pdf
- Doesn't reference us, but nice result
- Max: should add pflow to outputs
- Should update to-do list more immediately!
- Looking at images, for events with sharing
- Sharing defined by the "r ratio": how much energy is in each cluster, from different kinds of truth particles
- Want "medium" values of r
- Splitting in this particular event did separate pi+ and pi0, but energy is still dominantly in one
- Ratio can still be a good variable for understanding overlaps
- Ben: color of the markers?
- Darkness indicates higher energy
- But could also be other clusters overlapping... need to double check
- Ben: symbols indicate location of truth particles?
- Yup
- Max: which cluster is front/back?
- C0 is the "earlier" one
- Low r indicates nearly fully hadronic energy-- so tile one is this one
- Discussion: how to handle these overlapping cases?
- Jan: Would it make any sense to train a network to learn this ratio? You could then apply charged/neutral energy regressions accordingly. Or just train a new regression on these overlapping clusters.
- Max: looking at nearby clusters? In the back to indicate things?
- Aaron: did the charged pion deposit a lot in the front? Or photons shower late?
- Max: let's just make the larger per cell dataset, will launch this
- A new task: classifying these overlapping cases?
- Using split_CNN_EMB with 0.978 as classifer, then Jan's regressions binned in energy
- Jet response measurement!
- There's one peaked at 1-- great!
- There's one peaked at 0.2-- oooof
- Definitely there is something going wrong with these
- What's going wrong?
- The 20 GeV cluster bin has a huge discontinuity
- Caused by 'off diagonal' clusters: low Etrue, high Ereco
- Not seen in the jet sample-- which has the opposite: high Etrue, low Ereco
- Could this be caused by some kind of bug in the calib hits?
- will investigate these clusters/events
- Or can filter the training set of data for the regression
- Ben: don't think the bias could cause this-- the binning could be an issue instead
- 2D plots are fairly smooth...
- Some kind of dataset selection issue? something else?
July 19, 2021
- Graphs with KNN5 being used for classification
- Sanmay: what's the message passing step?
- Ben doesn't know offhand-- will follow up
- Great set of regression results for pi0 and pi+ using graphs!
- Very conclusive set of studies
- Sanmay: event to event comparison of truth vs prediction?
- Response plots
- Max: interesting eta dependence. Doesnt' quite compare to the classificaiton results from Kaela?
- Aaron: any ideas on what the weird feature is in the 2D plots?
- This could be driving the response down in weird ways
- Max: isolation? pick out the highest energy cluster per event?
- Ben: pick out n_celsl? could be anomouslly low?
- Mark: incorporate tracking?
- Max: next talk will have results!
- Lots of different ways to set this up
- Lots more data, and the regression starts to work really well
- Still erratic loss curve, but seems to be converging
- Extremely good energy performance!!! both at low and high energy
- Some discussion with Mark on setup of data
- Ben: why does it start at 1 GeV?
- Would expect 500 MeV pions to be making it
- Should double check what cut is causing this
- Dhanush: why is it better compared to last weeks?
- Better ML: more epochs, higher learning rate, more data
- Have taught the network to listen to the calorimetry at high energy!
- Ben: will be very interesting to also study the less than ideal situations
- Kaela looking at Deltas-- very fun
July 12, 2021
- Idea is to use this as a sample of 'overlapping' or close by particles
- How well does it do at that?
- Lots of close by particles!
- But nearly all the time the clusters are 'pure' and actually separated
- At extremely high energy, there is some mixing
- Cluster splitting is doing a good job!
- But may want to investigate 'close by' clusters-- no mixing, but borders are affected by neighbors
- But nearly all the time the clusters are 'pure' and actually separated
- Sanmay: did you check how many times the two photons are coming from the neutral pion?
- Maximum overlap at cell level
- look at events where charged pion is right between photons
- Max: could run the "per cell truth" information
- Aaron: could have had two separate photon clusters and this wouldn't show up?
- Yes, these visualizations don't display those differences
- But we do know these are separate clusters
- Aaron: given the truth particles, how well is the splitting working?
- Very often there were 'baby' clusters at the sides
- Aaron:
- Are they different clusters, or splitting?
- Seems like splitting, but can't be sure
- Russell: question on the looking up-- done by cluster_hitsTruthIndex
- Aaron: the cluster_hitsTruthE is a sum per cell. The total energy from the calibHits doesn't sum the cluster level calib hits, that should have been the same. Some slight mismatch here?
- Some very impressive initial results-- great resolution at low pT and great response!
- But incredibly sensitive to details-- can get much worse reults very easily (and still some issues with those plots-- issues at high energy)
- Also seems clear that more epochs can help, and more data: need more!
- Continuing literature search on optimization practices here
- Trainiable aggregation functions could be a neat future extension
- Piyush: this is similar to the self-attention networks, quite interesting and connected to Sanmay/Piyush's work
- Sanmay may have some code for multi-headed self attention-- a next step after baseline setup from Russell
- Piyush: this is similar to the self-attention networks, quite interesting and connected to Sanmay/Piyush's work
- Trained on jet samples: trained much longer
- Needs bigger network too
- SOme discussion on truth energy-- learning clusterCalibE
- Incoming presentation to discuss targets in jet samples
- May need to be careful with pileup
July 5, 2021
- Thanks to Sanmay and Jan for presentations at ML workshop and ATLAS week
- Main questions from Sanmay were on execution time
- For Jan, Mario asked baout data validation
- Can use minbias, W->tau nu, and normal 'jet level' validation
- First combined regression with track and calorimeter information!
- Looking nice, but definitely need more stats-- next on the agenda
- Ben: what's the difference in training samples on one of the early slides?
- Tight track+calo matching on the right
- Max: in terms of resolution comparison:
- This is hard-- could do cluster-only from paper, but need to scale x-axis?
- Could do PFlow-- need PFlow objects output to ntuple, this is a good idea
- Sanmay: batch normalization may help the jumpiness on s8
- Ben suggests increasing batch size, slower learning rate
- Ben: what about the target being the ratio of something?
June 14, 2021
- Ben: call from ML forum for next meeting, June 24
- Attention and deep sets
- One talk or two? Can see how it goes
- Piyush: GraphNet library for energy calibration studies
- One-hot encoding relationships to other cells
- next/previous is one big list of one hot variables
- Full GN block, 3 of them, concatanate, then indepdenet block
- Sanmay: is it fully connected or what?
- No, uses the adjacent cells defined in the GeoTree
- Predicted/true is looking very good!
- Some differences at low energy, can sstill be improved
- These models are 10k parameters: very efficient, almost on par with millions of parameters
- Previous week just had issue with loss function definition
- Sanmay: tried on jet sample? Not yet
- Max: introducing Kaela to group!
- Jan: using regression with CNN
- Instead of calib_tot, learn the calibration factor
- REsolution seems to be much easier to improve using this ratio as the target
- Suggestions from Max:
- Log plot in x
- Study Pi+
- Question: why divide by median?
- If you have an addititive offset, could make arbitrarily low resolution
- Alternative metrics:
- Efficiency of a threshold
- Resolutions of invariant masses
- Sanmay: Standard preprocessing pipeline. Jan has a pull request and will present slides next week for discussion.
- Instead of calib_tot, learn the calibration factor
- Albert: numpy gets converted to akward? -> will followup on slack.
June 7, 2021
- Start with event displays with tracks
- Multiple topoclusters in different colors: cool! And track overlays
- Had to have some workarounds for geometry tree, which can get large or has duplicate entries when hadd'ed
- Currently working with 'platinum dataset'-- good match, no additional clusters around tracks
- First regression-- not amazing results yet, but works!
- Includes track as special point in the cloud, marked as a flag
- Also working on calo-only regression with DeepSets
- Will try various settings, loss functions
- Piyush: what does the 10% energy veto mean?
- This is in the context of a cleaning cut in the 'platinum dataset'
- Basically suppress additional pion showers, just to start
- Will need to include these eventually!
- Piyush: has a file with single geotree for full detector available
- Mark: why not match to multiple clusters?
- Max: just to start somewhere
- Mark: sum of DR of 0.2 around the track works well
- Jan made the PR with features-- thanks! Will merge
- Sanmay-- did some attention with graphs progress. Next week will show
- Would like to discuss also whether to flag topoclusters in jets next time
- Piyush-- also working on the GraphNets. Had issues with training, was due to some issues in loss function; getting close to image performance, but haven't performed too much analysis yet
May 31, 2021
- Update on image based classifiers
- The "new data old format" and "old data" and "new data new format" all seem to be compatible now
- "new noisy data" has a little bit better format
- Is the noise correlated with energy deposited?
- Max: shouldn't be-- it's just electronics noise
- Thoughts on other ways to pick out network stability, etc.
- Sanmay: did you look at the average images?
- Did check individual, they seemed good; will check average
- Ben: topoclustering noise suppression makes this funny
- Won't capture everything but still nice to check
- Feedforward of noise from original image might be a problem?
- Could pre-train with original samples without noise?
- Ben: probably don't have enough data for something this big?
- Ben: some additional questions on differences between samples-- does seem weird but consistent that noise helps
- Introduction to Deep Sets for Particle Flow Networks
- Use a basic setup for now for latent space network and general function
- PFN outperforms EFN; already have 0.98 AUC without much work!
- Dropout and L2 regularization can be tried
- Some studies of latent space dimension: 150 seems best?
- Phi module also stuided, but need bigger steps
- Ben: Definitely don't worry about EFN, only nice thing is the kernel visualiation
- Add global eta and phi as global features to the F network
- Could add energy as well?
- Sanmay: any plan for attention based networks?
- Could do, but haven't tried, no initial plan
- Could we use other Deep Sets packages?
- In principle this doesn't have any particular advantages, just 'nice' interface
- Piyush: do you know the maximum number of cells vs the padding length?
- We don't know :| Will check
- The newest version on GitHub doesn't need zero padding
- Using jet samples, see non-zero center for the training regression: need to watch out a bit. Low energies could be a problem here somehow
May 24, 2021
- Use a MDN instead of the DNN-- instead of one value, output several values
- Previously, was working pretty well but bit worse at low energy
- Add 3 Gaussians for final model-- performs nearly identically to DNN now
- Now also take the predicted resolution from the Gaussian, and can compare to 'real' resolution
- They compare extremely well! That's awesome
- Can create a pull distribution : (predicted - true) / predicted
- Indeed looks pretty Gaussian at center, but has long tails, especially on left
- Max: this is great. Will be useful for PFlow training?
- Ben: should we make this default? Seems pretty good...
- Alex: in practice, converges a bit slower
- Alex: NB that learninging the StdDev right now, as functionality is a bit limited
- Max: let's make a PR!
- Elham: any other functions/loss function?
- Maybe show the resolution distribution -- can you predict the non-Gaussian part of the resolution?
- (so not just the medians of the distributions, but actually overlay resolution in an energy bin)
- Limited to 3 Gaussians because of memory limits on google colab?
- Ben: some truth studies with Ming
- Max: Deep sets first steps working
- Making a tf.dataset setup for using the full dataset
- Jan: the new datasets are performing extremely, extremely well
- The "new data old format" performance is marginally better than before
- "new data new format" still doing much better-- maybe that's a good thing!
- Christina: working on training, having some problems with the setup
- Some advice on the setup
- Try COlab, reduce to 500k events
- Worst case, can get access at TRIUMF?
- Piyush: testing models with GraphNets
- Having issues with training-- learns for a little bit and then loss stops changing
- Outputs are predicting a very small number
- Ben: Ming is able to train using GraphNets, does seem to work, but training is very complicated-- gets better for a while, then stabilizes, then plummets
- Seems to be a feature and not a bug-- may be common with graphs?
- Sanmay: Optimizers can change this a lot... ADAM is standard, playing with learning rate may help
- Max saw this as well with CNNs
- Ben: doing standardization, etc.?
- Piyush will have an intern working on this too
May 17, 2021
- Max, made the "new samples" with "old datasets"
- Will announce shortly
- Should be useful for Christina and Nicholas, and validation
- Dhanush: how best to format the data for athena?
- Some discussion on this
- Max to be in touch with ideas on technical implementation
- Make an algorithm that converts the clusters into arrays
- Then pass that onto ONNX
- AOD or ESD? Either is fine as long as AODs have cells
- Sanmay: made an event display of a dijet event: cool! Cells are beautiful
- Should we add information to clusters about the jets?
- Ben: this could be useful, but makes the learning less general
- Knowing density of nearby clusters could be helpful?
- Ben: this could be useful, but makes the learning less general
- Max: training on this, vs training on single pions, would be really interesting
- Ben: could do pileup mitigation if we train on this!
- Should we add information to clusters about the jets?
- Russell: some first studies of tracks in these datasets
- Event displays with tracks and cells: great overlaps! (center of cluster and track is agreeing very well)
- Problem he's running into: cluster center doesn't align with the /cell locations/
- Next steps: apply track/cluster matching, and get into a deep set
- Albert: cluster eta/phi followup?
- Will double check what's going on, and see if he can reproduce
- Sanmay: in general, we can have more than one track pointing to cluster? may need to handle down the line
- Jan: been using image based classification on the new dataset (converted format)
- A few details to be ironed out-- energy distributions needed to be aligned a little bit; classifier was learning just energy differences?
- 20 million clusters-- why so many more?
- eta cut would reduce this...
- noise cut...
- Is that enough?
-
https://its.cern.ch/jira/browse/ATLMCPROD-9116 is the request
- oh dang we did request 10 million lol, so that's a good part of it
May 10, 2021
- Sanmay, question on jet dataset and strategy
- Max: best to remake jets from scratch, then match afterwards
- (This is more of a fair comparison)
- Should double check the code from Jan on this
- Max: Russell has been looking at tracking and accessing that data
- Dilia: looking at EFN
- Sanmay may provide TF dataset functionality
- Christina and Matt
- Making progress on taus
- Christina would like to get next stage inputs-- needs the NN!
- Good reminder: we should remake the NEW datasets in the OLD format-- Christina could then train the new NNs
- Piyush
- Making progress on the GNN tests
- Questiosn about cluster moments-- max points out CENTER_MAG, FIRST_ENG_DENS, CENTER_LAMBDA
- Jan will post the data on EOS, Dhanush will copy for Piyush
- Max: make OLD format on NEW datasets
- Jan: post images to EOS
- Jan: post jet code
- Everyone: PHYSICS
May 3, 2021
- Max: jet datasets are finished, will post coordinates on grid (some problems downloading to EOS as they're a little large)
- Also looked at tracks- information should be usable (need to do matching, etc.)
- Graph datasets look 'more concentrated' compared to the older dataset
- EMB1 in the 'new' image set has a slightly different shape
- This is a bit concerning: could be a centering mismatch?
- Max: We would have expected these things to be wider if anything, with the calo noise being turned on
- Should we run the new samples through the old code?
- Max: Yes, should compare. Max can launch samples, pass to Al for comparison
- Ben: absolute scale is a bit different: 8% vs 2% in the old one?
- Al: wouldn't put too much stock into it, as energy binning isn't enforced carefully in new plot
- Max: eta cut?
- Al: not applied
- Max: need to apply 0.7 to keep it apples to apples: "concentration" could be coming from the image binning being enforced, but calo granularity getting much coarser
- Two models of graphs being studied; interesting results!
- Both show good performance at high energy
- INteresting comparison of performance: we should do this in other contexts more often
- Ratio of how 'in agreement' the models are
- Big disagreements at low energy-- expected to be difficult!
- Max: Look at log plots?
April 26, 2021
- Max: dijet samples nearly produced! Will anounce when ready
- Modified old 'image based' ML4Tree to get tau information and matching for samples, etc.
- Lots of trickiness with cluster formats and so on, but making good progress to retrieve tau information
- Ben and Max: nice job! Hard to get through all these technical details
- Ben: NB, calibration only applies |eta| to < 0.7
- Will double check
- Questions from Christina:
- We feed in 'six images per cluster'
- Maybe makes the most sense to start with 1p0n-- should be identical to what we're doing
- Vertex correction-- does this affect energy, or only position?
- Corrects the pT
- Shower distribution will look 'tilted'
- But maybe the origin correction is what un-tilts? not sure
- Suggest to choose one to start (maybe corrected), and check differences
- How big should output be?
- Suggest to run on grid
- We feed in 'six images per cluster'
- Max and Dilia: playing around with DeepSets/PFN
- Sanmay: should be similar to 'fully connected' network from his setup
- Ben: working with graphs and tensorflow, setup similar to Sanmay but in TF. Good to figure out how things go
- Using DeepMind graphs package
April 19, 2021
- Staged data to disk-- can launch MLTree this week?
- C++ and python versions of graph -> image code. Both running! Will compare results.
- C++ faster, python more flexible/slower
April 12, 2021
- Max: Albert and Jan working on converting graph format to images. This is really slow.
- Matt: Yeah, that's what I've seen as well :(
- Probably will need to consider pre-processing steps in the future
- Max: Nicholas will be giving a talk at ML workshop tomorrow-- will send out slides later
- Using TensorFlow implementation of MDN-- seems to work pretty well! Nice!
- Max: Slightly reduced performance at very low energy-- may need hyperparameter tuning? Optimization of starting parameters? Learning rate, etc.? Initialization of sigma especially?
- Ben: compare resolution measured to real resolution?
- They already have plots of the IQR for example
- Should make residual/pull plots, etc.
- Alternative graph model: static graph, use NNs to train Q/K/V tensors per node
- then pass messages using these tensors
- Initial performance possible-- looks decent without any optimization, so something is happening
- Next steps: interpretability, phase space investigation, optimization, etc.
- Could also do things like combine the MDN + Graph to learn resolutions, even in the graph context
- Everything updated on GitHub-- thanks!
- Max: was this in the paper?
- Similar things, this multi-head attention is more sophisticated
- Sanmay: status of jet sampels?
- This is on Max-- need to update
March 29, 2021
- Albert merged his "create images from graph inputs" code-- thanks Albert!
- Jan to merge this into his code: may need to update later
- Action for Mattew S and Albert: compare approaches and 'centering'
- Need to get jet samples processed: may need to fetch them from tape first
- Action item for Aaron/Max
- Reminder: x/y/z for tracks?
- Could actually use the track extrapolator to interact with cells directly-- could be useful! Expected energy lost could be included as well
- Aaron has some test samples with this, Sanmay can take a quick look
- Sanmay can take a look at the event displays for example
- Sanmay: around 30% of the events have 0 clusters-- do we understand why?
- Aaron: In the past, we had an eta cut... don't know why now though.
- Max: make truth particle plots for events that fail
- Could actually use the track extrapolator to interact with cells directly-- could be useful! Expected energy lost could be included as well
- Starting with kNN graph topology-- simple, has fewer reduncies than 'fixed'
- Also include many 'features' from each cluster, copy to each node
- Use this as an input to a 'dynamic graph model': learn a representation of the space
- Then learn the truth energy from that, with just MLP
- Basic performance does seem to work!
- Some tail on the right side; too high predicted energy
- Implemented three structurally independent models, testing them now
- Max: high tail on s7 is not crazy; probably a result of under-training and low energy bias in sample. bin in energy?
- Piyush: How are you pooling the nodes together on s5?
- Mean-pool: make a single nude with wthese tagged features
- Why max/mean of the s4 Graph construction?
- Many tested in previous paper, this was effective
- Sanmay: current implementation in PyTorch
- Piyush: some additional questions on implementation-- can provide more details
March 22, 2021
- New organization wiki available: https://github.com/atlas-calo-ml/Organization/wiki/Ongoing-and-Past-Work
- Please list your work here!
- Aaron and Sanmay: new repos for graphs, or use old repos and add?
- Max: if you have a repo already, can import it, no problem
- Small preference to consolidate in one place, for the sake of utilities
- (May want to separate utilities into a separate git package so the 'common' development is easily accessible regardless of repo)
- Max: if you have a repo already, can import it, no problem
- Graph inputs are just cell lists: if we want to run our 'old' CNNs we have to convert back to images
- Write a 'binning' image algorithm that transforms a cell list into an image (array) just like it used to be
- Should be 'plug and play' with old networks!
- Quick look at average images: they look pretty-- general trends look very similar!
- Max: which bin is set to center? EMB1 not peaked in 0,0?
- Could just depend on the algorithm performance chosen?
- Double check with Joakim implementation
- Aaron: old code had some 'aliasing' issues as well-- so this could be normal? Something to consider
- Aaron: is DeltaEta calculated with respect to cluster Eta, or cell's Eta?
- It is cluster Eta/phi
- Aaron: we took the cell that was the center, then calculated DEta and DPhi wrt that
- Albert: can check
- Max: which bin is set to center? EMB1 not peaked in 0,0?
- Matthew: also working on something similar; great!
- Let's merge these!
- Matthew's images: https://hep.ph.liv.ac.uk/~msullivan/TauCP/images/
- Next plans:
- Verify bit more
- Make available in ml_utils
- Check the old CNN performance
- Sanmay: plan for jet sample?
- Jan already looking into this; bad results so far, we think due to noise settings
- Will followup offline
March 15, 2021
- Aaron's single particle request finished: thanks!!!
- Aaron will launch ntuples
- Aaron is placing some documentation on 'neighbors' and representation in the roundtable: check there for more info
- So far, we've considered images
- Can also consider data as other things: point clouds
- Points can have no relationship to others: point clouds like deep sets, etc.
- Graphs could encode 'neighbor' relationships: neighbors in a grid are listed
- Even more complicated, could have 'distance' between cells (geometric or otherwise?) encoded
- Q: what about edges between layers?
- Could also consider different type of blocks, and what you want to learn: could try to learn geometry first, then other things later
- s4 in ben's slides are a useful graphical explanation
- Aaron adds some points in a similar way: lot of different configurations are possible!
- Sanmay: some practical attempts at graph construction
- kNN and fixed Radius Algorithms investigated
- Made some event displays-- code from Jonathan? Nice!
- kNN looks neat: we see connections only in layers, not between layers-- need more neighbors (k=10) to start to get some connections
- Ben: what distance? Sanmay: physical, using the x/y/z location
- But the very dense cells of the ECal mena you don't see connections between layers there
- fixed radius also seems a bit problematic in some places
- Nice comparison of fixedR and kNN
- fixedR has huge redundancy in edges; kNN is sparser (but this depends on settings?)
- All code on github, thanks!
- Plans: also can do dynamic edge determination
- Max: scales are very different! need to be careful with fixed R
- Learnable radius parameter?
- Ben: we could re-define 'closeness' as well; is x/y/z the right unit? or x/y/lambda ? x/y/layer
- Sanmay: good idea on the node features Ben included
- could we get 3D info for tracks (not just eta/phi/layer). How best to do this?
- David: philosophical question; what constraints do we impose, what do we let it learn?
- Sanmay: Impose constraints via loss function (lagrange multipliers, etc. on loss function). Or if not imposed, try to understand what is learned
- Russell: once you get to large fixed radius, missing connections at inner layer-- can you elaborate?
- First layer trying to connect to 3, because of the R=1 surface it is using; really it should have been connected to something shorter. Dynamic would be better.
- Max: let's keep a list of who is playing with what, and remember who did what
- Trained with the old data, but nice new results
- lots of comparisons of different options
- RESNet doing well, CNN_all doing well as well
- EMB1 DNN also pretty great
- Also can plot learnable parameters vs performance (AUC)
- simpleCombine with resnet has even more parameters, does even better
- CNN does well for few parameters
- simpleCombine with resnet has even more parameters, does even better
- Max: need to bin these in energy
- Tile will have much more impact at higher energies: try a cut at 10/20 GeV to see
- Jan: CNNs may not be the right thing for the Tile, though
- Current method Jan is using for interpolation (bilinear interpolation) may not be very physical; need to adjust?
March 8, 2021
-
We will have a discussion next week about the space of things to study for graph nets, focusing in particular on the graph construction. Aaron now has the functionality implemented to save the nearest neighbors.
-
Piyush showed some slides (to be posted on indico) with a first look at ParicleNet (EdgeConv) applied to the data. To be discussed more next week.
-
Albert talked about the timing for converting the graph structure back into images. It currently takes a long time (many hours for 17 events!). We talked about using the akward arrays directly instead of first converting to Pandas/Numpy (which may be slower because there are arrays of uproot types whereas it is naturally serialized in akward). There was some discussion about if we should add in information from the geometry tree to the hit ID, but Aaron pointed out that if the dictionary is done efficiently, then this should be no faster than storing the information from the geometry tree.
-
Dhanush mentioned that he is pausing to see how the graph structures work because it could be much easier on the ATHENA side (e.g. if the graph construction is part of the model, then we don't need to do a lot of data manipulation before running inference).
March 1, 2021
- Samples from aaron being tested by Ming and Albert
- Close by samples done!
- Should run on this with the new code
- Should we merge the latest changes and go for it?
- Aaron needs a couple more days to finalize the last batch, then we merge and launch
- Max will poke Haifeng to get fixed single particles
- Sanmay: request to join github, max does so
- And gives others ownership
- Jan: working on cyclic learning rate to help resnet, which seems to get stuck in local minima
- Max: try lowering learning rate as well? Did help, but still working through
- Dewen: working with dead material. Presampler should help for some types of dead material; some other dead material corrections have parameterizations related to last EM layer and first tile layer. OK to use this?
- Sure: might be 'double using' but shouldn't be double counting energy, as long as training targets are different
- Piyush: trying out graphs, not too great a result yet, but exciting
- Could you present distance metrics next week? Yes, will show
- ParticleNet model was used; Sanmay used a variation of that
- Also dynamic graphics were used, and Piyush is trying as well
- Next week: aim for a discussion on spaces and distance metrics, how to represetn data in the graphs, etc.
- Albert, Ming, Piyush, Sanmay for discussion?
February 22, 2021
- Dataset production ongoing-- thanks Aaron!
- Last time: classifier was losing information with the noise threshold, but regression didn't look like it was losing much-- what's up with that?
- Sanity check: blank images completely
- Classifier: 0 input
- regression-- decent at 1-2 GeV, quite bad below
- 'swoop' shape, seen by others
- Dedicated check: seems like things make sense
- blank images are still information, when other images are not blank
- 5 MeV still seems like a decent baseline
- Directionality bug in previous images
- The conclusions we drew are still correct, but some differences do exist
- Will discuss with subconveners and other convener on whether to update PUB
- Matching trees with cluster/tracks is very slow it turns out
- Test samples available
- Better to merge trees in creation step already?
- Aaron was going to do this: definitely proceed
- Jan suggests "TTreeIndex" for fast lookups in Trees for sorting/matching
- Update on treemaker: combine the cluster and event trees
- Also can keep track of which particles contributed how much energy to each cell
- Thaaat's cool
- Removed some things like the 'duplicates' branches-- were these useful?
- Joakim confirms can be removed
- Is the track calo extension used?
- Yes-- for PFlow super critical!
- Need to generalize to other layers however beyond barrel
- How many particle links to keep? 3 like PFlow?
- David: more if wanted for HL-LHC
- Keep it configurable for now?
- Thing to check: how is truth handled in 21.9 samples? Eric brings up truth pileup is possible but crashes
- Aaron: DigiTruth is also available for pileup samples
- Sometimes
cluster_hitsTruthEvalues are too low-- why? - G4 Particles in truth list?
- Check with Mark
- Logarithms have weird properties for small inputs
- Can limit precision for low energy particles, which are very important for us
- Piecewise function seems to work well
- Good comparison of various options
- Tested using Densenet, 20 epochs
- Max: may want to double check with 'DNN all' model? Or is that what you were using?
- Indeed: just using DNN. May want to double check # of epochs-- converged?
- Some methods lead to overflows: have to be careful when exponentiating results
- Albert: why is linear better than log?
- Some discussion on this: spacing is better, less compression of information
- Wanted to implement ResNet: one of these big classic models
- Classification looking good, but perhaps overtraining
- Implemented fast rebinning: use integrals to re-normalize images after using TF's fast rebinning functions
- Diagram of current setup: currently using energy and eta concatanated at the end
- Performance is not super great
- Noticeable offset in Epred/ETrue
- Could be not trained long enough?
- Wojtek: could also require larger dataset: 100k can get overtrained
- Monitor validation loss within the epoch (not just once per epoch)
- Is the scaling done online?
- yes: slower, but better with memory
- Is tile included?
- Yes
- May want to change the structure of layers: mix energy into the image
- Piyush: wasn't actually using full DenseNet, used 'densenet-like' to be simpler
- Much smaller number of parameters
- Could be simpler to do it that way
- Piyush: which ResNet model?
- ResNet 50: relatively large implementation
- Wojtek: could do augmentation online?
- Flip horizontally/vertically to get 'free' additional data
February 15, 2021
- Max: new meeting time: 6 PM. Unfortunately need to still alternate (need to work with Australian time zone). Alternating time will be 9 or 10 PM, will confirm ASAP and send agenda. Thank you for being flexible to allow for better collaboration across the globe! <3
- New 'graph data structure' seems to be working!
- Truth information is much more granular: available at cell level now
- cellID is a hash for a cell location, can be looked up in a separate 'meta' tree
- Noise in cells is also available!! Cool
- Flag for old clusterImage format as well
- All this code ready in branch-- will hold off on merging while final decisions on some bugfixes comes in
- Sample output file is already available, and ROOT macro for understanding this should be available
- Some problems observed when debugging this: overlapping cells sometimes not handled perfectly
- Maybe easiest to abandon the 'old' format, and just make a macro to make images from the cells output in the graph format?
- Other questions: in the longer term, truth is our target: need to make some decisions on how to associate this truth information
- "which truth particle created these hits" is a question Athena can help us answer
- We don't have PV's here-- if there's a truth vertex spread, probably could want to have truth vertex information to correct for this?
- Also validated new overlapping particle samples: things look good in general! DigiNoise looks correct, etc.
- Some cool effects with hadronic showers 'early in detector' being reconstructed as multiple clusters: very neat
- Ben: question on the truth matching. Aaron describes PFlow's tools for this-- convenient functions for finding what truth particles are interacting
- Ben: what exactly should our regression target be? Given these fun hadronic showers on s8?
- This is going to be important to consider
- Early showers we should be able to identify
- Aaron: also some situations with a large number of tracks: even earlier showers? But not correlated in number of clusters... OPEN QUESTION. Max: could be a weird low energy pion shower? Doesn't get to calo?
- Discussion:
- Don't worry about the bug in the old format. Use the new format. Can also make 'symmetric' images better?
- keep all the varities of truth or no?
- Ask Peter and Mark for opinions: Ben thinks to sum it and just go with that.
- Max suggests to see if the size is manageable... might still be ok.
- Aaron: no cuts in cells. Currently comparable to noise. Apply 5 MeV cut from Albert's studies? Can add the optino to have the cut.
- Jan: what about labeling in the overlapping samples?
- Truth links?
- Based on the samples?
- Label them as "multi particle samples"
- Could be labeled by fractions, classification without labels, etc.
- Labeling individual clusters is really hard... hmm....
- Label them as "multi particle samples"
- Come up with a proposed labeling for now?
- Less ambiguous in sinlge particle...
- Can cross-check single particle 'sample' labeling with the proposed barcode labeling?
- Particle links might be sorted by energy... taking the first would be a first method
- Could label those as 'bad' or 'ambiguous' somehow if there are doubts
- Ultimatley we don't care too much about the label, so maybe this is ok... just a better way of training...
- Some people may want more PID information for physics if available, and if this was high quality
February 8, 2021
- Discussion on formats, especially heading to the forward region
- Aaron: great summary of benefits of Athena from CellNavigation
- Proposal: make 2 trees, one with energy/CellID, 2nd tree has CellID info with eta/phi/size/etc.
- Ben/Jonathan: Should add relative shift information to images as well
- Discusion on checking in taus:
- R22-- this is where cells exist in xAOD, need to check this out
- could use existing Z->tau ESD?
- Liverpool-- interest in Athena collaboration?
- Trying out models, etc., plugging things into Athena, etc.
- Talk from Dhanush will be in core software ~this week (not last week like I thought)
- Christina to check in taus? Nicholas also interested
- Next steps for Pflow discussion
- Sanmay: can use Pflow's cluster/track association
- Aaron: for truth energy per cell: could be helpful to engage Peter/Sven to get the truth energy per cell, it's a bit complicated to do this
- Mark did this! Can help!
January 25, 2021
- https://arxiv.org/pdf/2101.08578.pdf interesting paper
- TauCP is interested: quick discussion with Bertrand, tauCP convener. Could be done in r22 using cells in xAODs that exist. Nice!
- They're interested in both classifying and energy calibrations. Great! Should collaborate!
- Training models for regression including reco dead material correction (the correction from LC)
- Probably a better idea to not include the LC result: directly regress to the dead material energy
- This could be an interesting comparision however
- Max will send variables to target
- Goal: compare true dead material, LC dead material, and regressed dead material
January 18, 2021
- Working on conversion of TF models to onnx and importing into Athena
- Some initial issues in converting to onnx because of TensorFlow version issues
- However, this was sorted out
- Implementation of data-handling code: some differences in the C++ for Athena compared to usual TF python code
- Implementation in Athena seems solid now with the DenseNet model: nice! Both Median and IQR
- Next steps:
- Documentation on experience, guidelines for future users
- Figure out optimal input of calorimeter images: right now is one pion at a time
- Importing from ESDs
- Optimal batching, etc.
- Ben: Is it possible to compare shower by shower? See if you get the same answer
- Yes, should be
- Max: IQR is a bit off at the high end?
- Double check those are working
- Questions from Dhanush:
- Where to present? Where to document?
- Document for now on GitHub wiki? Easy to port elsewhere
- Where to present? Get in touch with Mark to check
- Present in AML soon? Sure, why not
- Where to present? Where to document?
- Remove cells with energy cuts: how does the classifier and regression do?
- Classifier degrades, regression degrades
- Ben: high energy cut on regression-- how is it so ok?
- Check images and check resolution
- Russell: kicks up on the right hand for the 2 GeV-- any understanding?
- Max: thinks it is a training artificat
- Using our calibrations on jets
- Last time: was getting hugely wrong energies. way too low.
- A number of fixes: 4-vector math handled better, scalers used consistently, etc.
- Looks a little better, but still not Great
- NB: energy distributions between single pions and jet samples are very different
- Need to lower training energy cut from single pions, that's one important change
- Does isolation have an impact on performance?
- Hard to tell, but doesn't look like it
- But what about reweighting-- does correcting to the training dataset, or in the other way, help?
- Energies are a little better, but not 100% still
- Next steps: lower energy cut for sure
- Other datasets: train on jets?
- But that's hard to label...
- Could train classifiers without labels!
- Use unsupervised learning!
- Other datasets: train on jets?
- Ben: probably do want to make a dataset of clusters that have labels in the normal way
- May need some work to find the real origin/matching/etc.
- Jan did already try matching via pion truth record (via the nearest pions)
- Didn't work very well: bad at classification
- Ben: focus on the ones with clear, 'golden mode' matches
- Jan did already try matching via pion truth record (via the nearest pions)
- Self-supervised learning: super hot right now
- Ben will make some technical suggestions offline
- This can be particularly helpful for non-ideal clusters
- May need some work to find the real origin/matching/etc.
- Aaron: is there a fundamental problem in the images being different?
- Aaron brought this up as a dataset problem before
- That's probably the biggest difference
- Training sample has fewer cells with energy in them
- Really have to track this down now
- Could also use Aaron's 2-particle samples
December 14, 2020
- What happens when we change the noise threshold, which is a 5 MeV cut in the cell energy?
- Can try a few different energies, and also remove (or keep) the negative energy (we remove by default)
- Did some validation: indeed see differences from the various different settings in the images we produce
- But the result is pretty consistent: the noise threshold doesn't impact the performance of the networks
- Piyush: any reason to stop at 10 MeV: try higher? Looks to be increasing (very small though)
- Could increase the cut significantly, remove low energy clusters-- simplify for trigger?
- Could also study performance binned in energy
- COuld also study a cut that's a 'fraction of energy' instead of hard cut
- Aaron: maybe not unexpected-- right now, this is defined by the 420 clustering
- Should understand the rational of the '0'
- Difference between classification and regression
- How does the energy regression work ?
- Should understand the rational of the '0'
- Aaron made some new data, better treatment of the eta/phi bug at edges, and including the presampler
- Slightly higher IQR at low end, slightly reduced at high end (high end is more significant)
- Weird banding effect in 2D energy plots: need to understand what those are
- QuadDense vs TriDense: using presampler or no. Seems like small improvement to regerssion at low energy
- Max: presampler might have a bigger imapct w/ the presampler?
- Aaron: could be useful for dead material?
- Max: don't worry too much about 100 MeV, but the banding is still weird and should be understood
- Can look at average images in that area
- Piyush: important to see the improvement at the high end energy
- Comparing to LC would be great: we were only comparable there before
December 7, 2020
- Retraining the network, then going to apply things for jets
- Currently having problems with ML application, but also replicating original EM jets
- Ben: is the energy a spike at 0, or just a small value?
- Need to double check
- Jan + Aaron: may be a problem with truth jet pT, may want to update this
- Wojtek: could recompute the 4-vector using ML calibration?
- Could use a deltaR matching or something
- Max: probably some discrepancy coming from pT cut on EM jets
- Added lots of nice developments for tracking: v1 dataset with tracks launched!
- Huge thanks to Nicholas!
- Wojtek: TRT high threshold? Pixel dE/dx? Charge?
- V2 should contain these
- Ben to provide some pseudocode
- Max: jobs actually finished! can store this on EOS and rucio
- Max: Interest from tau CP: storing now cells in xAOD from reconstructed taus. Can hopefully use Dhanush's Athena framework to calibrate/classify.
- Ben: what flitering? will need to be careful.
- Aaron: what data structure? Max: think it's just a cellcontainer. Dhanush's code "should just work"
- Aaron: truth info? Max: think no :(
- Our formats will still be useful/necessary for this...
November 16, 2020
- Goal: make the tracking information available!
- Change the data pipeline to include some level of 'matched tracks'
- Example script to output numpy arrays of matched clusters and tracks
- Ready to run on grid and produce central datasets
- Currently: choose track closest to cluster in DR, in cases of overlap
- Ben:
- pT/eta/phi are at the origin, right? --> Yes
- Also including the global eta/phi, at each layer
- Track selection? Laura: 9 silicon hits, pT cut, will double check (more in line with tight primary)
- PT > 500 MeV, 9 silicon hits, no missing pixel hits
- Action: still double check with Mark
- pT/eta/phi are at the origin, right? --> Yes
- Sanmay:
- question on compiling: email the list is best (Max and Albert working on small tweaks to README recently)
- Wojtek:
- do we have local information on the centroid per layer?
- Could calculate this ourselves?
- Could we include some quality variables for offline analysis?
- Chi2, nHits, nSi, TRT extrapolation info...?
- d0 and z0 might be good?
- Covariance matrix at each layer?
- For later (along with track/cluster assocation, would be main longer-term improvements)
- Could also include eloss, or eloss per layer? (idea from aaron)
- do we have local information on the centroid per layer?
- Using decision trees for ooc and dead material
- Detailed feature set analysis
- Using new datasets, slightly new training
- Wrap around in phi fixed, but other problems could ahve been introduced...
- Weird features in dataset: strange?
- Interesting to see that DeadMaterial and OOC are actually... decent??
- Aaron: NCells seems to be helpful, at least in some cases (not uniformly in pi0 vs pi+)
- David: nCells as proxy for density? Maybe remove nCells, see if density ranks up?
- Aaron: out of cluster prediction for pi0-- this is the worst. See multiple maxima in the output distribution-- weird. Cluster splitting? True calib out also has a second maximum...
- Wojtek: Maybe conversions? Is there a truth handle for this? ACTION
- Isolate events and understand?
- Follow up with Peter/Sven?
- Wojtek: Maybe conversions? Is there a truth handle for this? ACTION
- Aaron: NCells seems to be helpful, at least in some cases (not uniformly in pi0 vs pi+)
November 9, 2020
- Larger convolutions: not so helpful: 2x2 (or even 1x1) seems to be maximizing performance (at least in terms of AUC)
- Also now tried out the 3D convolution technique: also seems pretty similar in performance
- Ben: very interesting-- could be saturated, but maybe we don't have enough data to take advantage of the structure?
- One structure: AUC vs nsamples. Should be a quick check.
- Wojtek: what upscale?
- Not at full EMB1 scale: Laura is working on this
- Generator is working, but other leaks?
- David: are the channels connected when doing the 2D convolutions?
- There should be information in the 3D convolution approach... How is the 2D approach done?
- Convolution moves through 2D space, across all channels (and then summed); vs through 3D space
- Punchline: Both can learn "shower depth", which is part of the convolution in the 3D case and is in the summing across layers in the 2D case. In both cases, the dense network at the end links the layers and the information is connected via back prop.
- 3D shower profile better matched by 2D case? Weight sharing is different
- NB: just linking adjacent layers in the 3D case (since filter size is 2 in the longitudinal direction)
- More ideas?
- Ben: can we reduce parameters in the network? 3D conv should be "smaller", more weight sharing. Could remove dense layers, could do "more with less"
- Wojtek: full granularity, coordinate with Laura
- Aaron/Max: 3x2x2 convolutions?
- Aaron: should we be padding? Especially in the first layer?
- Ben: not doing topo-clustering right?
- Eric: Global actually gets all cells: could do more
- SC: do we have a baseline LUT for calibration? Eric: people are pretty convinced NN is simpler/smaller, but yeah, everything is in flux
- Sanmay: two step process discussion. Then discussion about classes-- do we want more for electron and photon? Could be useful-- need to see if it's helpful
- Mark: interested, also interested in the EF level, and PFlow and so on
- Alison: topoclusters as inputs, so centroids aren't in the middle: could you scale/use mapping to link information between layers? (more related to Albert's talk)
- Aaron: How do we trust this in the trigger? how difficult to update?
- Liza: changing a LUT is easy between runs, but do you want to change between runs?
- Eric: The hope is that a stripped down NN like this can be incorporated to a single network, it could be /more/ efficient
- Doesn't have to be perfect! Already can gain a lot iwth first order
- Wojtek: what are the plans for CPU/GPU streaming?
- SC: still too early to say. Have to decide what's the coprocessor. We could decide to have a GPU per CPU, or dedicated GPU farms, etc... still need to iterate and decide.
- Definitely makes sense to have both levels of ML, need to build on L0's strengths.
- HLT might be more flexible: still have some time to adapt
- Mark: Fast portable inference workshop coming up at FastML workshop (https://indico.cern.ch/event/924283/)
- Dec 3 parallel session
- SC: CMS seems a little ahead on this: need to do more :)
November 2, 2020
- Apply the ML-based calibration/classification to calo clusters, PFlow objects, etc. in Athena
- Ultimately explore 'batching' to make GPU use more efficient
- Planning on using ONNX runtime
- This has been implemented already into Athena and AnalysisBase
- Converting keras models to ONNX, but there is some problem with keras models with custom activiation
- Going Keras -> TensorFlow -> ONNX does seem to work
- Aiming for validation comparisons of the implementation inside Athena, to the existing inference outside of ATLAS
- Wojtek: does the athena package support batching over event boundaries?
- Currently, the example package does not: but the goal is to extend it so that it can go over multiple events
- Aaron: There will be some engineering required for this, so goal is to learn how to do this
- Ben: batching across the event may not be necessary, as we have large number of clusters per event already
- Mark: Is it possible to get clusters from two events at once?
- Indeed, not possible right now
- But probably Ben's point is the relevant one...
- Wojtek: calculate throughput, etc. to get a motivation for larger batching? This may be sufficient
- How many clusters per second infered, as a function of batch size?
- Sanmay: what stage where this be implemented?
- Has to be at ESD -> later
- Starting with the cluster trees, indexed by cluster
- Jan writing a script to regroup things by event number
- Can be used to remake jets!
- Million topocsluters in 15 minutes
- Aaron will figure out where the jet samples are, so Jan can use those
- Add truth jets to the trees?
- Aaron: event tree already has the truth jets
October 28, 2020
Agenda: https://indico.cern.ch/event/969219/
- Albert reported in JetDef: https://indico.cern.ch/event/907045/
- Suggested to check the 'crossing' IQR images one by one to see if they're pathological?
- Ben: let's make sure to store all these results: should go in next publication
- Aaron made some images with presampler, and more moments: cool!
- Added presampler and fixed some bugs in the handling of the phi coordinate in the images. Any low-level comparisons between these and previous runs would be useful
- user.angerami.428002.single_piminus_logE0p2to2000.e7279_s3411_r11281.v01-24-g0f4cdca_OutputStream/
- user.angerami.428001.single_piplus_logE0p2to2000.e7279_s3411_r1128.v01-24-g0f4cdca_OutputStream/
- user.angerami.428000.single_pi0_logE0p2to2000.e7279_s3411_r11281.v01-24-g0f4cdca_OutputStream/
- 1x1 convolutions: can this gives us 'pure' depth information that is useful?
- Answer: seems to be yes! Can get AUCs of 0.98 with just 1x1 convolutions
- Also optimize some architecture choices here
- Also optimize which sets of layers to include
- EMB1 seems to be the most useful: always included in search
- EMB0 and 2 layers also important
- Tile2 also seems important
- Does it help in combinaiton with existing CNN network?
- Not so much ;(
- Seems like existing network has already learned similar levels of depth information
- or at least, the depth the 1x1 has learned
- Ben: what about 3x3? Or 3D cons? Could give it a try while we are exploring
- Wojtek: Take individual clusters where the architecture gets it right, and compare to the 1x1 getting it wrong: what is up in those images?
- Jonathan: graph neural network? deep sets?
- Ben agrees
- Put each cell in the 'real' position (could be relevant)
- But this is no longer 'transitionally invariant', so need more stats in training?
- Somethign to put on the list!
- simple code sample for a basic deepset model here: https://gitlab.cern.ch/jshlomi/GNN_Onnx_Example/-/blob/master/share/model.py
- example code for training with variable size input here: https://github.com/hadarser/SetToGraphPaper
- Sanmay: question on the rescaling: what level of detail?
- Scaling only takes into account volume
- Forward geometry is hard!
- Aaron figured out how to 'upscale' various endcap detectors to get the 'right' geometry to match our images
- But we could also use existing "tower" infrastructure in ATLAS to do this, it seems
- Continue with towers for now, fall back to manual method?
- Aaron: use differnet towers per layer, to get the full granularity per layer
- Sanmay: use upscaling in the NN?
- Haven't tried this yet: has seemed like expert knowledge (energy conservation, detector geometry expertise, etc.) is as good? but just intuition-- could be worth trying!
October 19, 2020
Agenda: https://indico.cern.ch/event/967235/
- MLTree can give both tracking information and calorimeter images
- But Tracking appears in a different tree: inconvenient :|
- Nicholas developed mltree2array_tracks.py
- Matching between clusters and tracks just done by run/event number, and currently doing first
- Feedback requested on some points
- Mark: did you apply track selections? number of hits, etc.
- Haven't done this-- probably explains the high track numbers
- Standard Tracking Recommendations should be looked up
- Athena Example (you can also use m_trackSelectorTool, provided by tracking CP, in your ntuple algorithm):
- https://acode-browser1.usatlas.bnl.gov/lxr/source/athena/Reconstruction/eflowRec/src/PFTrackSelector.cxx#0105
- If no tracks: what should things be?
- -999?
- Max: probably ok to go with single track for now
- Perfect is the enemy of good
- Will want to go back later and add more tracks later
- Mark: Also need to keep in mind that multiple clusters will have the same track sometimes
- For single pions: almost always 1, sometimes 0, rarely 2, rarely rarely 3
- David: outputs include eta/phi in each layer?
- Aaron: average energy expected lost?
- For down the line?
- Track pT/eta/phi available: should also add the eta/phi in each layer? asap
- Ben: also covariance information
- Marumi: depends on pion/electron, should double check
- Jonathan: truth matching? don't have much of this yet
- Marumi: really critical to have truth energy per particle for the future
- Mark: re truth matching athena has a tool that adds truth energies to CaloClusters (but we may in fact instead want to output per cell in the cluster?)
- Matching between clusters and tracks just done by run/event number, and currently doing first
- Max: did you try to see where each network is better?
- Sanmay: yes, graph and deep sets a bit more stable in e.g. energy
- Mark: does this rely on track to cluster matching?
- Not explicitly in the training, can be done generally
- Mark: in ATLAS we turn off PFlow in dense environments, and let "TCC" take over the splitting
- Would be interesting to see how that algorithm would work
- Marumi/David: TCC and UFO would be great
- Sounds great!
- Max to:
- Make egroup
- send out code
- send out existing samples
- Sanmay to:
- Send out code
- Will need to add per cell information in outputs: should be doable fairly easily, first place to collaborate
- Then start playing with models and discussing results :D
- Revisit meeting time (at least one of them?)
October 5, 2020
- Agenda: https://indico.cern.ch/event/962685/
- Intro:
- Job ad for TRIUMF posted: send those you think would be interested to apply!
- Ben: should we do a hackathon to get started on Pflow? yes
- Order one-month or so from now?
- Albert: learning pion resolutions
- Can use modified "genearlized absolute loss function" to learn quantiles: 86% and 14% allow for the 68% IQR to be measured
- The learned quantiles do seem to perform as expected!
- There's an unusual subset of events, that 'cross' quantiles
- But a low enough fraction that this isn't too much of a problem?
- David: what does it mean to learn a resolution per particle?
- Albert: we're essentially learning an 'uncertainty' per particle
- Ben: The average is also an 'average quanitty' over the training set
- Discussion on what it means to understand the resolution per particle
- Piyush: Do you train separate networks each time?
- Yes
- Piyush: What if you just calculate the IQR by hand?
- Get a resolution per particle, using the full cluster information: could do this 'by hand' but only in smpler cases
- Aaron: What does this actually tell us?
- Given a particular image, what is the resolution (not error) on that
- It's not an uncertainty: it's a resolution
- David; the jet resolution depends on the ensemble you used to measure it
- Same thing here: but we have a very precise statement, because we're using the image as input
- David: s5: why aysmmetric?
- Albert: going back to s3: these inputs do look to be aysmmetric on s3, so the network itself is learning something funny
- Wojtek: take denominator to be the 'directional' element of the IQR (depending on the over or undershoot)
- Wojtek: can we have one network to train all three?
- Ben: we can make 'q' random (uniformly sampled), then you can learn all the quantiles at once, but this is very tricky to do in practice
- Max: what will we use this for?
- Pflow: decide what to subtract
- Wojtek: build a likelihood for a full jet
- Piyush:
- Learning about out of cluster and dead material
- Using decision trees
- Can do a pretty good job of predicting the out of cluster energy!
- Some weird peak structures though?
- Dead material predictions are way better
- Also tests of using 'cheating' variables vs just more reasonable inputs
- Feature importance also studied
- Isolation variable not in the ntuples right now: should be added ACTION
- Will also add pre-sampler
- David: first features all seem to be energies? That's surprising that they're all so important, naively should be redundant?
- Some decision trees had restricted variables?
- Cluster eng tot used: just as a proxy for our outputs from NNs
- Some decision trees had restricted variables?
- Aaron: train in a variety of topologies?
- Makes sense! Out of cluster is kind of trivial in this context
- Max: comparison to LC OOC and DM calculations?
August 31, 2020
- Agenda: https://indico.cern.ch/event/950537/
- Nicholas working on Calo + track datasets
- Aaron working on close-by pion samples
- Albert working on learning resolution
- Sylvia working on updating 'no classification results': include also pi- now
- Seems quite dependent on composition! Perhaps not surprising
- Ben: maybe MAE is better? (Probably not)
- Ben: Perhaps not a problem, would have to be careful to get 'physical' composition, but could also try something like numerical inversion
- Ben: Could try simultaneous regression + classification with combined loss function?
- Piyush: Could also have just two outputs, two different loss's, should work this way too
August 14, 2020
- Agenda:
- Presentation from Sylvia on 'classification skip' regression
- Regression with EM Prob + energy already is able to do very well
- Regression with all images + eta + energy does super well without any classification
- Should test with different compositions: can just add the pi- samples to double charged pions
- Adding an auxiliary target (classification output?) might help training, might also be useful
- Piyush: questions on models ('simple flat DNN' for the images, and simple 3 layer DNNs for feature-based)
- Nicholas: working on tracking integration
- Software plans: Danush joining, welcome!
- ONNX might be something Danush could contribute to
- The rest of us might have to do the Calo stuff...
- Where does the code go? Calo or FlowElement?
- Scales may have been removed for run3 because they are not thread safe...
- Shallow copy may be the right solution for LC, could be what we do as well
- Could be done in FlowElement as well: https://gitlab.cern.ch/atlas-particleflow-software/athena/-/blob/mhodgkin_newPFlowEDM/Reconstruction/eflowRec/src/PFLCNeutralFlowElementCreatorAlgorithm.cxx as example
- Presentation from Sylvia on 'classification skip' regression