Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MalariaSummaryReport: Add control to use true density #12

Merged
merged 3 commits into from
Dec 19, 2024
Merged

MalariaSummaryReport: Add control to use true density #12

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
5e84553
MalariaSummaryReport: Add control to use true density
Bridenbecker Dec 4, 2024
21914ec
MSR - Added ability to add true and HRP2 prevalence channels
Bridenbecker Dec 16, 2024
2f6a23d
Update to EMOD 2.24
Bridenbecker Dec 19, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 28 additions & 8 deletions docs/emod/software-report-malaria-summary.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,13 @@ To generate this report, the following parameters must be configured in the cust
**Parasitemia_Bins**, float, -3.40E+38, 3.40E+38, "[50,500,5000,50000,FLT_MAX]", "Parasitemia Bins to aggregate within and report. A value greater than or equal to zero in the first bin indicates that the uninfected people should be added to this bin. The values must be in ascending order."
**Infectiousness_Bins**, float, -3.40E+38, 3.40E+38, "[20,40,60,80,100]", Infectiousness bins to aggregate within and report.
**Individual_Property_Filter**, string, NA, NA, (empty string), "The individual 'property:value' to filter on. The default of an empty string means the report is not filtered. For example: 'Risk:High'."

**Include_DataByTimeAndPfPRBinsAndAgeBins**, boolean, 0, 1, 1, "When set to true, the 'DataByTimeAndPfPRBinsAndAgeBins' element is included in the report. Default is true. You can save disk space by setting this to false."
**Include_DataByTimeAndInfectiousnessBinsAndPfPRBinsAndAgeBins**, boolean, 0, 1, 0, "When set to true, the 'DataByTimeAndInfectiousnessBinsAndPfPRBinsAndAgeBins' element is included in the report. Default is true. You can save disk space by setting this to false."
**Add_True_Density_Vs_Threshold**, boolean, 0, 1, 0, "If set to true, four new channels will be added to the report that use true density instead of measured. These additional channels are: 'PfPR_2to10-True', 'PfPR by Age Bin-True', 'Pf Gametocyte Prevalence by Age Bin-True', and 'Mean Log Parasite Density by Age Bin-True'. The true densities will be compared to thresholds: Detection_Threshold_True_Parasite_Density and Detection_Threshold_True_Gametocyte_Density."
**Detection_Threshold_True_Parasite_Density**, 0, 3.40282e+38, 0, "Used when 'Add_True_Density_Vs_Threshold' is true. The true parasite density is compared against this threshold. It impacts the 'PfPR_2to10-True', 'PfPR by Age Bin-True', and 'Mean Log Parasite Density by Age Bin-True' channels."
**Detection_Threshold_True_Gametocyte_Density**,0, 3.40282e+38, 0, "Used when 'Add_True_Density_Vs_Threshold' is true. The true gametocyte density is compared against this threshold. It impacts the 'Pf Gametocyte Prevalence by Age Bin-True' channel."
**Add_Prevalence_By_HRP2**, boolean, 0, 1, 0, "If true, the 'PfPR_2to10-HRP2' and the 'PfPR by Age Bin-HRP2' channels will be added. These channels will use 'Detection_Threshold_True_HRP2' to determine if person's HRP2 level counts towards prevalence."
**Detection_Threshold_True_HRP2**, 0, 3.40282e+38, 0, "Used when 'Add_Prevalence_By_HRP2' is true. If the true HRP2 value is greater than this threshold, the prevalence will be increased in the 'PfPR_2to10-HRP2' and the 'PfPR by Age Bin-HRP2' channels."


.. code-block:: json
Expand All @@ -42,6 +48,7 @@ To generate this report, the following parameters must be configured in the cust
"Filename_Suffix": "Node1",
"Start_Day": 365,
"End_Day": 465,
"Use_True_Density_Vs_Threshold" : 0,
"Node_IDs_Of_Interest": [ 1 ],
"Must_Have_IP_Key_Value": "Risk:LOW",
"Must_Have_Intervention": "UsageDependentBednet"
Expand Down Expand Up @@ -75,7 +82,14 @@ To generate this report, the following parameters must be configured in the cust
100
],
"Individual_Property_Filter": "Risk:High"
}
},
"Include_DataByTimeAndPfPRBinsAndAgeBins": 0,
"Include_DataByTimeAndInfectiousnessBinsAndPfPRBinsAndAgeBins": 0,
"Add_True_Density_Vs_Threshold": 1,
"Detection_Threshold_True_Parasite_Density": 40.0,
"Detection_Threshold_True_Gametocyte_Density":, 1.0,
"Add_Prevalence_By_HRP2": 1,
"Detection_Threshold_True_HRP2":, 1000000.0
],
"Use_Defaults": 1
}
Expand Down Expand Up @@ -127,7 +141,9 @@ The following statistics are collected:

Time Of Report, "Each entry is the final day of the reporting interval, in days."
Annual EIR, "The average Entomological Inoculation Rate (EIR) per year over the reporting interval."
PfPR_2to10, "The fraction of individuals whose age is 2 < age < 10 that would have been detected with the BLOOD_SMEAR_PARASITES diagnostic type of MalariaDiagnostic where the sensitivity is **Report_Parasites_Smear_Sensitivity** and the detection threshold is **Report_Detection_Threshold_Blood_Smear_Parasites**."
PfPR_2to10, "The fraction of individuals whose age is 2 < age < 10 that would have been detected with the BLOOD_SMEAR_PARASITES diagnostic type of MalariaDiagnostic where the sensitivity is **Report_Parasites_Smear_Sensitivity** and the detection threshold is zero. Please note that his measurement includes some random noise."
PfPR_2to10-True, "If Add_True_Density_Vs_Threshold is true, this chanel is added. It will contain the fraction of individuals whose age is 2 < age < 10 and whose true parasite density is greater than **Detection_Threshold_True_Parasite_Density**."
PfPR_2to10-HRP2, "If Add_Prevalence_By_HRP2 is true, this chanel is added. It will contain the fraction of individuals whose age is 2 < age < 10 and whose true HRP2 density is greater than **Detection_Threshold_True_HRP2**."
No Infection Streak, The maximum number of days without an infection during the interval.
Fraction Days Under 1pct Infected, The percentage of days during the interval in which the percentage of infected individuals was less than 1%.

Expand All @@ -142,9 +158,13 @@ Age Bin.
:header: Parameter, Description
:widths: 8, 20

PfPR by Age Bin, "The fraction of individuals in this age bin that would have been detected using the BLOOD_SMEAR_PARASITES diagnostic type of the MalariaDiagnostic intervention where the sensitivity is **Report_Parasites_Smear_Sensitivity** and the detection threshold is **Report_Detection_Threshold_Blood_Smear_Parasites.**"
pf Gametocyte Prevalence by Age Bin, "The fraction of individuals in this age bin that would have been detected using the BLOOD_SMEAR_GAMETOCYTES diagnostic type of the MalariaDiagnostic intervention where the sensitivity is **Report_Gametocyte_Smear_Sensitivity** and the detection threshold is **Report_Detection_Threshold_Blood_Smear_Gametocytes**."
Mean Log Parasite Density by Age Bin, The average Log10 parasite density of the population for that age bin based on the count of parasites using the BLOOD_SMEAR_PARASITES diagnostic type of MalariaDiagnostic where the sensitivity is **Report_Parasites_Smear_Sensitivity**.
PfPR by Age Bin, "The fraction of individuals in this age bin that would have been detected using the BLOOD_SMEAR_PARASITES diagnostic type of the MalariaDiagnostic intervention where the sensitivity is **Report_Parasites_Smear_Sensitivity** and the detection threshold is zero. Please note that his measurement includes some random noise."
PfPR by Age Bin-True, "If Add_True_Density_Vs_Threshold is true, this chanel is added. The fraction of individuals in this age bin whose true parasite density is greater than **Detection_Threshold_True_Parasite_Density**."
PfPR by Age Bin-HRP2, "If Add_Prevalence_By_HRP2 is true, this chanel is added. The fraction of individuals in this age bin whose true HRP2 density is greater than **Detection_Threshold_True_HRP2**."
Pf Gametocyte Prevalence by Age Bin, "The fraction of individuals in this age bin that would have been detected using the BLOOD_SMEAR_GAMETOCYTES diagnostic type of the MalariaDiagnostic intervention where the sensitivity is **Report_Gametocyte_Smear_Sensitivity** and the detection threshold is 0.02. Please note that his measurement includes some random noise."
Pf Gametocyte Prevalence by Age Bin-True, "If Add_True_Density_Vs_Threshold is true, this chanel is added. It will contain the fraction of individuals in this age bin whose true gametocyte density is greater than **Detection_Threshold_True_Gametocyte_Density**."
Mean Log Parasite Density by Age Bin, "The average Log10 parasite density of the population for that age bin based on the count of parasites using the BLOOD_SMEAR_PARASITES diagnostic type of MalariaDiagnostic where the sensitivity is **Report_Parasites_Smear_Sensitivity**. Please note that his measurement includes some random noise."
Mean Log Parasite Density by Age Bin-True, "If Add_True_Density_Vs_Threshold is true, this chanel is added. It will contain the average Log10 parasite density of the population for that age bin based on the count of true parasites."
New Infections by Age Bin, "The number of new infections during the reporting interval for each age bin."
Annual Clinical Incidence by Age Bin, "The number of new clinical symptoms per person per year. This channel is controlled by the **Clinical_Fever_Threshold_Low** and **Clinical_Fever_Threshold_High** parameters. The amount that an individual’s fever is above normal must be greater than both of these values to be considered clinical. This can also be influenced by the **Min_Days_Between_Clinical_Incidents** parameter."
Annual Severe Incidence by Age Bin, "The number of new severe symptoms per person per year. An individual is considered to be a severe case if the combined probability of anemia, parasite density, and fever is greater than a uniform random number. This combined probability is the combination of sigmoid using the following parameters: **Anemia_Severe_Threshold** and **Anemia_Severe_Inverse_Width**, **Parasite_Severe_Threshold** and **Parasite_Severe_Inverse_Width**, **Fever_Severe_Threshold** and **Fever_Severe_Inverse_Width**."
Expand All @@ -167,8 +187,8 @@ Parasitemia Bins, and Age Bins.
:header: Parameter, Description
:widths: 8, 20

PfPR by Parasitemia and Age Bin, "The fraction of individuals whose parasite density and age fall into this bin. The sum of the people whose true parasite density in the PfPRBin and age bin divided by the total number of people in the age bin."
PfPR by Gametocytemia and Age Bin, The fraction of individuals whose gametocyte density and age fall into this gametocyte bin.
PfPR by Parasitemia and Age Bin, "The fraction of individuals whose parasite density and age fall into this bin. The sum of the people whose true parasite density in the PfPRBin and age bin divided by the total number of people in the age bin. Please note that people with zero parasitemia (i.e. only gametocytes) are counted in the bin that includes zero."
PfPR by Gametocytemia and Age Bin, "The fraction of individuals whose gametocyte density and age fall into this gametocyte bin. Please note that people with zero gametocytes are counted in the bin that includes zero."
Smeared PfPR by Parasitemia and Age Bin, "The fraction of individuals in this age bin whose true parasite density when smeared by CountPositiveSlideFields falls into this parasitemia bin."
Smeared PfPR by Gametocytemia and Age Bin, "The fraction of individuals in this age bin whose true gametocyte density when smeared by CountPositiveSlideFields falls into this gametocyte bin."
Smeared True PfPR by Parasitemia and Age Bin, "The fraction of individuals in this age bin whose true parasite density when smeared by NASBADensityWithUncertainty falls into this parasitemia bin."
Expand Down
45 changes: 45 additions & 0 deletions emodpy_malaria/reporters/builtin.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -232,6 +232,13 @@ def add_malaria_summary_report(task, manifest,
reporting_interval: float = 365,
must_have_ip_key_value: str = "",
must_have_intervention: str = "",
include_time_age_pfpr_bins: bool = True,
include_time_age_pfpr_infectious_bins: bool = True,
add_true_density: bool = False,
parasite_detection_threshold: float = 0.0,
gametocyte_detection_threshold: float = 0.0,
add_hrp2_prevalence: bool = False,
hrp2_detection_threshold: float = 0.0,
age_bins: list = None,
infectiousness_bins: list = None,
max_number_reports: int = 100,
Expand All @@ -253,6 +260,31 @@ def add_malaria_summary_report(task, manifest,
means don't look at IPs (individual properties)
must_have_intervention: the name of the an intervention that the person must have in order to be included.
Empty string means don't look at the interventions
include_time_age_pfpr_bins: When set to true, the 'DataByTimeAndPfPRBinsAndAgeBins' element is included
in the report. Default is true. You can save disk space by setting this to false.
include_time_age_pfpr_infectious_bins: When set to true, the 'DataByTimeAndInfectiousnessBinsAndPfPRBinsAndAgeBins'
element is included in the report. Default is true. You can save disk space by setting this to false.
add_true_density: If set to true, four new channels will be added to the report that use true density instead
of measured. These additional channels are:
* 'PfPR_2to10-True',
* 'PfPR by Age Bin-True',
* 'Pf Gametocyte Prevalence by Age Bin-True', and
* 'Mean Log Parasite Density by Age Bin-True'.
The true densities will be compared to thresholds 'parasite_detection_threshold' and
'gametocyte_detection_threshold'. The Default is False.
parasite_detection_threshold: Used when 'add_true_density' is true. The true parasite density
is compared against this threshold. It impacts the:
* 'PfPR_2to10-True',
* 'PfPR by Age Bin-True', and
* 'Mean Log Parasite Density by Age Bin-True'
channels. Default is zero.
gametocyte_detection_threshold: Used when 'add_true_density' is true. The true gametocyte
density is compared against this threshold. It impacts the 'Pf Gametocyte Prevalence by Age Bin-True' channel.
Default is zero.
add_hrp2_prevalence: If true, the 'PfPR_2to10-HRP2' and the 'PfPR by Age Bin-HRP2' channels will be added.
These channels will use 'Detection_Threshold_True_HRP2' to determine if person's HRP2 level counts towards prevalence.
hrp2_detection_threshold: Used when 'add_hrp2_prevalence' is true. If the true HRP2 value is greater than this threshold,
the prevalence will be increased in the 'PfPR_2to10-HRP2' and the 'PfPR by Age Bin-HRP2' channels.
age_bins: The max age in years per bin, listed in ascending order. Use a large value for the last bin,
to collect all remaining individuals
infectiousness_bins: infectiousness Bins to aggregate within for the report
Expand All @@ -274,6 +306,19 @@ def rec_config_builder(params):
params.Start_Day = start_day
params.End_Day = end_day
params.Node_IDs_Of_Interest = node_ids if node_ids else []

params.Include_DataByTimeAndPfPRBinsAndAgeBins = 1 if include_time_age_pfpr_bins else 0
params.Include_DataByTimeAndInfectiousnessBinsAndPfPRBinsAndAgeBins = 1 if include_time_age_pfpr_infectious_bins else 0

params.Add_True_Density_Vs_Threshold = 1 if add_true_density else 0
if add_true_density:
params.Detection_Threshold_True_Parasite_Density = parasite_detection_threshold
params.Detection_Threshold_True_Gametocyte_Density = gametocyte_detection_threshold

params.Add_Prevalence_By_HRP2 = 1 if add_hrp2_prevalence else 0
if add_hrp2_prevalence:
params.Detection_Threshold_True_HRP2 = hrp2_detection_threshold

params.Age_Bins = age_bins if age_bins else []
params.Must_Have_IP_Key_Value = must_have_ip_key_value
params.Must_Have_Intervention = must_have_intervention
Expand Down
2 changes: 1 addition & 1 deletion requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
emodpy~=1.16
emod-malaria==0.0.72
emod-malaria==0.0.73
requests==2.29.0
24 changes: 24 additions & 0 deletions tests/unittests/test_malaria_reporters.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -221,6 +221,15 @@ def test_malaria_summary_report_default(self):
self.assertEqual(self.p_dict['Start_Day'], default_start_day)
self.assertEqual(self.p_dict['Must_Have_IP_Key_Value'], empty_string)
self.assertEqual(self.p_dict['Must_Have_Intervention'], empty_string)
self.assertEqual(self.p_dict['Include_DataByTimeAndPfPRBinsAndAgeBins'], 1)
self.assertEqual(self.p_dict['Include_DataByTimeAndInfectiousnessBinsAndPfPRBinsAndAgeBins'], 1)
self.assertEqual(self.p_dict['Add_True_Density_Vs_Threshold'], 0)
# not in the dict beccause Add_True_Density_Vs_Threshold is false
#self.assertEqual(self.p_dict['Detection_Threshold_True_Parasite_Density'], 0.0)
#self.assertEqual(self.p_dict['Detection_Threshold_True_Gametocyte_Density'], 0.0)
self.assertEqual(self.p_dict['Add_Prevalence_By_HRP2'], 0)
#self.assertEqual(self.p_dict['Detection_Threshold_True_HRP2'], 0.0)

pass

def test_malaria_summary_report_nodelist(self):
Expand All @@ -245,6 +254,13 @@ def test_malaria_summary_report_custom1(self):
def test_malaria_summary_report_custom2(self):
self.tmp_reporter = add_malaria_summary_report(None, schema_path_file,
must_have_ip_key_value="FavoriteCola:RC",
include_time_age_pfpr_bins=False,
include_time_age_pfpr_infectious_bins=False,
add_true_density=True,
parasite_detection_threshold=0.66,
gametocyte_detection_threshold=0.77,
add_hrp2_prevalence=True,
hrp2_detection_threshold=999.0,
infectiousness_bins=[-15, -5, 0, 3, 5, 8],
max_number_reports=63,
parasitemia_bins=[100, 500, 1500, 2345],
Expand All @@ -256,6 +272,14 @@ def test_malaria_summary_report_custom2(self):
self.assertEqual(self.p_dict['Infectiousness_Bins'], [-15, -5, 0, 3, 5, 8])
self.assertEqual(self.p_dict['Max_Number_Reports'], 63)
self.assertEqual(self.p_dict['Parasitemia_Bins'], [100, 500, 1500, 2345])
self.assertEqual(self.p_dict['Include_DataByTimeAndPfPRBinsAndAgeBins'], 0)
self.assertEqual(self.p_dict['Include_DataByTimeAndInfectiousnessBinsAndPfPRBinsAndAgeBins'], 0)
self.assertEqual(self.p_dict['Add_True_Density_Vs_Threshold'], 1)
self.assertEqual(self.p_dict['Detection_Threshold_True_Parasite_Density'], 0.66)
self.assertEqual(self.p_dict['Detection_Threshold_True_Gametocyte_Density'], 0.77)
self.assertEqual(self.p_dict['Add_Prevalence_By_HRP2'], 1)
self.assertEqual(self.p_dict['Detection_Threshold_True_HRP2'], 999.0)

pass

def test_malaria_summary_report_custom3(self):
Expand Down
Loading