-
Notifications
You must be signed in to change notification settings - Fork 1
/
log.txt
199 lines (166 loc) · 8.04 KB
/
log.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
upcomming changes for next production run
-----------------------------------------
- Reduce float precision in event table where possible.
Especially the cherenkovsize and cherenkovpool tables
use float64 all over the place while float32 would be
sufficient. This will reduce sizes of the event_tables
significantly (approx 25%).
- Compress the event_table (snt.archive). Seems to reduce
the size down to 50%. As the compression is block wise,
selective reads are still possible.
- 'groundgrid_result/num_bins_above_threshold' must always be
populated in event-table if there is a 'groundgrid' entry.
- In the run's output zipfiles, the file:
'RRRRRR/plenoirf.production.draw_event_uids_for_debugging.json'
is actually compressed with gzip but misses the '.gz' extension.
- Rename the index_key in the event-tables to 'uid' because 'uid' is also used
in plenoirf.bookkeeping. Currently it is 'idx'.
log
---
2024-09-07
Improve corporate-identity, make darkmode figures for the
Gamma 2024 in Milano.
2024-08-22
After holiday brake: Adopt mayor changes in Sparse Numeric Table
which allow for faster selective reads from the large files.
2024-07-01
Reduce more items from the simulation runs.
Benchmarks, ground grid intensity, and event uids for debugging.
2024-06-29
Adding summary/analysis plots to show what was actually
thrown. Add plots which show statistics of the 'groundgrid'.
2024-06-27
Deploying the snt.archive for selective reads inside the
event tabeles. Also make more use of rename_after_writing
to ensure that files only have their final names when they
are complete and thus valid.
2024-06-26
A new fileformat for the sparse-numeric-table (SNT).
The SNT with its numpy based recarrays has proven to be
very fast and efficient. Now that statistics gets bigger,
we might benefit from a fileformat which allows selective
reading to not always read the entire table (several GB).
This will fix the current bottleneck of the summary/analysis
scripts and allow for quicker debugging iterations.
2024-06-25
Getting the summary/analysis scripts up and running again.
No more looping over sites inside an analysis script.
Need to define pointing ranges to visualize zenith
dependencies.
2024-06-20
Mass production is further running. About 1 permill of the
runs suffers from an exception in merlic_development_kit.
Beside this, there is stderr related to calling iminuit
with uncertainty estimates <= 0.0. For now, keep on producing.
The summary/analysis in plenoirf is coming up again.
The analysis scripts up to 0061 have been reworked to adopt
the full pointing capabilities of the new simulations.
2024-06-18
Mass production is running on comupe clusters. First attempts
made to reduce the results into event_tables.tar.
2024-06-17
Finalizing updating PyPi repositories.
New feature in ./install.py. Command ``pypi`` lists the local
and remote package versions and this indicates which PyPi
repos need an update.
2024-06-13
Updating PyPi repositories.
2024-06-07
Reducing the memory usage. So far 2/3 of the memory used by
plenopy.LightFieldGeometry was the 'lixel_polygon' which is
only used for plotting. The 'lixel_polygons' are now removed
from the default LightFieldGeometry and have to be computed
when needed.
2024-06-03
Adding benchmarks to investigate compute infrastructure.
2024-05-18
Finally! Full production of instrument response passed
first tests (several 10k) showers on local machine. Ready to
deploy on compute cluster. Lots of bookkeeping was added.
Massive performance optimization in ground grid to histogram
Cherenkov light into 3D grid on observation level.
All memory usage that scales with the number of photons in a
Cherenkov light pool have been removed and replaced with
block-by-block approaches. This makes the memory usage much
more predictable, and smaller.
2024-05-22
Prepare trajectory reconstruction. Explicit package names.
2024-05-13
The complexity in plenoirf.production is getting out of hands.
Repeatability of sections in the production turns out to be
very challeinging while managing random seeds and caching the
section's resutls. It is almost unusable complicated.
2024-04-16
Making plenoptics (the package which creates all the plots for
the optics paper) write less files. Reduction is about two orders
of magnitude. From now on, the files are collected in zipfiles.
2024-03-12
Fine tuning magnetic deflection. Ready to populate primary to
Cherenkov mappings for all sites and paricles.
2024-03-05
Massive progress with magnetic deflection. It seems that a magnetic
deflection mapping (cherenkov direction <-> primary direction) is
feasible with taking the full Cherenkov pool into account without
approximating the pool with its median direction. This will make
the scatter angles for primaries obsolete.
2024-02-29
I found a way to histogram directions very fast and efficient.
It might be possible to histogram the directions of every Cherenkov
photon in a shower to make the ``magnetic_deflection`` package no
longer depend on the median of the light's direction, but its
actual histogrammed distribution. To do this, I add a new package
``spherical_histogram``.
2024-02-26 v1.10.0
Definging the instrument's spherical coordiantes and CORSIKA's
spherical coordinates within the package spherical_coordiantes.
Adopting changes in CORSIKA. Preparing starter_kit to run the next
production with fixed cx/ux cy/vy pointing/direction fixes.
-Fixing import bugs in plenoptics.
2024-02-22
It seems we found the cause of the timing issue with the cherenkov light.
Both the corika_primary package and the merlict_development_kit
interpret the cherenkov bunches directions to be the momentum. U and V.
And this makes sense as it is also defined as such in the CORSIKA 7.56
manual in table 11 and figure 1 on page 108. This is why the simualtion
so far worked out fine.
BUTT, in between I forgot about this definition and interpreted the
cherenkov bunches direction as the incident direction. This mirrors the
direction. Based on this mistake, I created test and calibration sources
and implemented the transformation of cherenkov bunches which is crucial
to the pointing of the plenoscope. Since my mistake was implemented in
calibration sources and inside the tests, it looked that all
transformations run as expected when in reality it was different from
CORSIKA's definition. Anyhow, now I try to tidy up this mess by removing
the corsika_primary.I.BUNCH.CX_RAD and CY_RAD. This is plain wrong.
Actually it is CORSIKA's U and V defined in the manual in Table 11.
This effects vast parts of the code.
The corsika bunch in merlict-c89 functions like in corsika_primay and
the merlict-dev-kit. But I also renamed its cx_rad to ux and its cy_rad
to vy. The direction estimate in the grid algorthm was wrong though.
Now the grid algorithm adopts the corika bunches function to calculate
the direction of motion.
Get the unittests up and running again.
Make a dedicated repository for un_bound_histogram.
2024-02-21
Debugging and getting the plenoirf simualtion chain up and running again.
It seems something is wrong with the timing of the cherenkov-bunches
which are given to the merlict-ray-tracer. The signals are too spread
out in the light-field camera and thus no trigger occurs. Adding more
unittests to verify the transformation of Cherenkov bunches.
2024-02-19
Prepare storing paths to executables in user's configfiles. Fix bugs in
install.py. Include the phantom_source as a submodule. For some reason
this was never done despite the phantom_source having its own
repository. Make plenoirf, magnetic_deflection, and plenoptics adopt
the implicit lookup of executable paths.
2004-02-17 v1.9.2
Rename corsika_install to coriska_primary to be more consistent with the
python package name.
2024-02-14 v1.9.1
Overhaul 'install.py' Prepare to store executable paths in config files
inside the user's home. Change license to CC-BY-4.0.
2024-02-14 v1.9.0
Restructure 'starter_kit'. Put packes into seperate directory.
Remove obsolete examples. Move scripts i.e. code which is not yet a package
into packages/scripts. The fermi_lat/ directory is a collection of scripts
related to Fermi-Lat so we move it to packages/scripts