Data quality assessment for manifests

[jmcook1186]

I've been thinking about whether a manifest can somehow capture a measure of the quality of the data it contains. The rationale is that it's often not going to be enough to simply say that a manifest has executed correctly - this tells you nothing about the granularity of the data, the coverage, the sources, the appropriateness of the methods employed, etc.

I saw that the Product Environmental Footprint has a formula for quantifying data quality:

Metric quality = (technological representativeness + geographical representativeness + Time representativeness + completeness + precision + methodological appropriateness)/6

There is also a rubric for quantifying each of the parameters in the formula - it looks as follows:

Quality level	Quality score	Definition	C	M	Ti	T	G	P
			To be judged with respect to the coverage for each EF impact category and in comparison to a hypothetical ideal data quality	The applied LCI methods and methodological choices (e.g. allocation, substitution, etc.) are in line with the goal and scope of the dataset, especially its intended applications as support to decisions. The methods have also been consistently applied across all data.	Degree to which the dataset reflects the specific conditions of the system being considered regarding the time / age of the data, and including background datasets, if any. Comment: i.e. of the given year (and, if applicable, of intraannual or intra-daily differences).	Degree to which the dataset reflects the true population of interest regarding technology, including for included background datasets, if any. Comment: i.e. of the technological characteristics including operating conditions.	Degree to which the dataset reflects the true population of interest regarding geography, including background datasets, if any. Comment: i.e. of the given location / site, region, country, market, continent, etc.	Qualitative expert judgement or relative standard deviation as a % if a Monte Carlo simulation is used. Comment: The uncertainty assessment is related to the resource use and emission data only; it does not cover the EF impact assessment
Very good	1	Meets the criterion to a high degree, with little significant need for improvement	Very good completeness (>90%)	Full compliance with all requirements of the PEF Guide	Context–specific	Context–specific	Context–specific	Very low uncertainty (<10%)
Good	2	Meets the criterion to an acceptable degree, but merits improvement.	Good completeness (80% to 90%)	Attributional process-based approach AND: Following three method requirements of the PEF Guide met: • Dealing with multifunctionality • End of life modelling • System boundary	Context–specific	Context–specific	Context–specific	Low uncertainty (10-20%)
Fair	3	Meets the criterion to an acceptable degree, but merits improvement.	Fair completeness (70% to 80%)	Attributional process-based approach AND: two of following three method requirements of the PEF Guide met: • Dealing with multifunctionality • End of life modelling • System boundary	Context–specific	Context–specific	Context–specific	Fair uncertainty Fair uncertainty (20% to 30%]
Poor	4	Does not meet the criterion to a sufficient degree. Requires improvement.	Poor completeness (50% to 70%)	Attributional processbased approach AND: One of the following three method requirements of the PEF Guide met: • Dealing with multifunctionality • End of life modelling • System boundary	Context–specific	Context–specific	Context–specific	High uncertainty High uncertainty (30% to 50%]
Very poor	5	Does not meet the criterion. Substantial improvement is necessary	Very poor completeness (<50%)	Attributional processbased approach BUT: None of the following three method requirements of the PEF Guide met: • Dealing with multifunctionality • End of life modelling • System boundary	Context–specific	Context–specific	Context–specific	Very high uncertainty Very high uncertainty (> 50%)

I'm interested to see if we can modify this table to be appropriate for calculating SCI or equivalent score using IF.

Here's an initial attempt as a discussion-starter!

Quality level	Quality score	Definition	C	M	Ti	T	G	P
			Completeness meaning the majority of system components, and upstream/downstream sources are accounted for in the manifest file. System boundary is well justified.	The applied methods and methodological choices (e.g. plugin choice, sources for coefficients, configuration is appropriate for the given purpose... Methods are applied consistently across components. Functional unit is appropriate to the specific use case. SCI is calculated according to ISO standard.	Timesteps are sufficiently granular to capture likely variations in the data. Aggregation is applied using appropriate methods. Time sync uses an appropriate range and resolution to accurately represent the data.	Impact metrics and measurement techniques are appropriate for the given system, e.g. the monitor API is used for azure vms, direct measurements are used in place of proxies or estimates, etc	Locations are as precise as possible. Each component has a location. Location-dependent values are as accurate as possible (e.g. using best in class services for grid intensity)	Qualitative expert judgement or relative standard deviation as a % if a Monte Carlo simulation is used or upper/lower bounds can be established.
Very good	1	Meets the criterion to a high degree, with little significant need for improvement	Very good completeness (>90%)	Full compliance with all requirements of IF best practise guide and SCI specification.	Context–specific	Context–specific	Context–specific	Very low uncertainty (<10%)
Good	2	Meets the criterion to an acceptable degree, but merits improvement.	Good completeness (80% to 90%)	Attributional process-based approach AND: Following three method requirements of the PEF Guide met: • Dealing with multifunctionality • End of life modelling • System boundary	Context–specific	Context–specific	Context–specific	Low uncertainty (10-20%)
Fair	3	Meets the criterion to an acceptable degree, but merits improvement.	Fair completeness (70% to 80%)	Attributional process-based approach AND: two of following three method requirements of the PEF Guide met: • Dealing with multifunctionality • End of life modelling • System boundary	Context–specific	Context–specific	Context–specific	Fair uncertainty Fair uncertainty (20% to 30%]
Poor	4	Does not meet the criterion to a sufficient degree. Requires improvement.	Poor completeness (50% to 70%)	Attributional processbased approach AND: One of the following three method requirements of the PEF Guide met: • Dealing with multifunctionality • End of life modelling • System boundary	Context–specific	Context–specific	Context–specific	High uncertainty High uncertainty (30% to 50%]
Very poor	5	Does not meet the criterion. Substantial improvement is necessary	Very poor completeness (<50%)	Attributional processbased approach BUT: None of the following three method requirements of the PEF Guide met: • Dealing with multifunctionality • End of life modelling • System boundary	Context–specific	Context–specific	Context–specific	Very high uncertainty Very high uncertainty (> 50%)

I'm very keen to understand which things can be quantified algorithmically and which require manual assessment, whether all these categories are really relevant, whether they capture what's needed to assess the quality of a manifest, and to what extent this can be made objective and therefore less vulnerable to being gamed.