Query stats followup #3255

awildturtok · 2023-12-20T15:28:46Z

No description provided.

…r.java, adds limit to Histogram, adds sum statistic

…stats rendering to backend

thoniTUB · 2024-02-27T09:43:35Z

backend/src/main/java/com/bakdata/conquery/models/query/statistics/ColumnStatsCollector.java

+			final int lowerPercentile = config.getVisualisationPercentiles().hasLowerBound() ? config.getVisualisationPercentiles().lowerEndpoint() : 0;
+			final int upperPercentile = config.getVisualisationPercentiles().hasUpperBound() ? config.getVisualisationPercentiles().upperEndpoint() : 100;


Wie validiert Range, dass die Range zwischen 0 und 100 liegt?

backend/src/main/java/com/bakdata/conquery/models/query/statistics/Histogram.java

thoniTUB · 2024-02-27T10:05:25Z

backend/src/main/java/com/bakdata/conquery/models/query/statistics/Histogram.java

+			);
+		}
+
+		final double width = roundWidth ? Math.max(1, Math.round((upper - lower) / expectedBins)) : (upper - lower) / expectedBins;


Warum wird beim Runden bei 1 geclippt?

Wir wollen beim runden mindestens eine binWidth von 1 haben. (Ich denke, mal du hast das als maximum darf das 1 sein gelesen? :D )

Über die Konsequenzen hatte ich mir noch nicht so gedanken gemacht. Wäre es ein problem für das Histogram, wenn die Werte nur zwischen zwei Ints liegen würden?

Du meinst zwischen zwei adjacent ints? rounding ist nur aktiv, wenn die bounds mehr als 1 auseinander liegen.

backend/src/main/java/com/bakdata/conquery/models/query/statistics/Histogram.java

...d/src/main/java/com/bakdata/conquery/models/query/statistics/NumberColumnStatsCollector.java

thoniTUB · 2024-02-27T11:32:01Z

...d/src/main/java/com/bakdata/conquery/models/query/statistics/NumberColumnStatsCollector.java


-	private final Comparator<TYPE> comparator;
+	public NumberColumnStatsCollector(String name, String label, String description, ResultType type, PrintSettings printSettings, int expectedBins, double lowerPercentile, double upperPercentile) {


Mich verwirrt, dass percentiles reingereicht werden und auch berechnet werden. Bitte einen anderen namen nehmen

Was genau verwirrt dich daran? Und was muss anders benannt werden? Die percentilen werden verwendet um das Histogramm auf einen relevanten Wertebereich zu fokussieren.

Mich hat verwirrt, dass die Percentilen durch den Constructor scheinbar von außen reingegeben werden, ob wohl sie nach meinem Verständnis erst aus dem collect dieser Klasse hervorgehen.

Ich schlage vor lowerPercentile -> lowerBoundPercentileHint zu nennen, damit klarer wird, dass auch die Daten einen Einfluss haben

Mich hat verwirrt, dass die Percentilen durch den Constructor scheinbar von außen reingegeben werden, ob wohl sie nach meinem Verständnis erst aus dem collect dieser Klasse hervorgehen.

Das hier reingegebene sind die percentilen die dann in describe verwendet werden, um dann die entsprechenden werte an der stelle auszuwerten. Sollten die Werte zu eng aneinander liegen werden sie expandiert.

...d/src/main/java/com/bakdata/conquery/models/query/statistics/NumberColumnStatsCollector.java

backend/src/main/java/com/bakdata/conquery/models/query/statistics/Histogram.java

…mber of lines

Implements limit for SingleTableResult, to only output a specified nu…

backend/src/main/java/com/bakdata/conquery/models/config/FrontendConfig.java

thoniTUB · 2024-02-29T13:24:53Z

backend/src/main/java/com/bakdata/conquery/models/query/statistics/ColumnStatsCollector.java

+			final int lowerPercentile = config.getVisualisationPercentiles().hasLowerBound() ? config.getVisualisationPercentiles().lowerEndpoint() : 0;
+			final int upperPercentile = config.getVisualisationPercentiles().hasUpperBound() ? config.getVisualisationPercentiles().upperEndpoint() : 100;


Die Validierung sollte den no bound-case schon abfangen

…to feature/query-stats-followup

awildturtok added 5 commits December 13, 2023 12:33

migrate BooleanColumnStatsCollector.java to StringColumnStatsCollecto…

dbc8ed8

…r.java, adds limit to Histogram, adds sum statistic

adds Histogram entry for tail values

42fdd13

fix EVENT_DATE access with non-standard queries

7b234f5

migrate NumberColumnStatsCollector to bin based histogram, also move …

d3a629f

…stats rendering to backend

cleanup

fe8c06f

awildturtok self-assigned this Dec 20, 2023

awildturtok requested a review from thoniTUB as a code owner December 20, 2023 15:28

awildturtok requested review from thoniTUB and removed request for thoniTUB December 20, 2023 15:40

awildturtok added 21 commits December 20, 2023 16:51

use LinkedHashMap to control sort order

8ff1c0b

uncomment tests and cleanup for compialtion

5e52666

fix layout for integers, also merge small bins

8450262

ensure width of at least 1

8185e0b

fixes copy paste typo

f20e713

fix tail null

f89f27b

fix sort order of postprocessed bins

1539496

use custom DynamicHistogram instead

c6a4a30

fixes min - value

03a97b1

fixes width calculation

b3962a7

adds null check

2bb084e

fixes balancing of histogram, also avoids Array bounds issues

6f3456c

fix not splitting very biased nodes

9f36dbb

adds stiffness parameter

740a110

adds initial test for Histogram

c7f1917

impl of different histogram modes

5b1c0fa

use plain histogram for now

f17fdb3

implement overflow bins, to get a better view on the data. focus on 2std

874a83c

use 80/20 percentile

0f26e43

cleanup

7a38036

extract HistogramColumnDescription

cd45987

dont use generics to make conceptnodes available

b17eabb

awildturtok requested a review from thoniTUB February 22, 2024 14:15

calculate descriptions in parallel

da796b9

thoniTUB requested changes Feb 27, 2024

View reviewed changes

awildturtok added 4 commits February 27, 2024 13:18

adds timing metrics for stats collection

f0874d3

adds validation for percentiles

9cd6e51

review feedback for readability

a1a51ba

fix zero comparison

9f294f6

awildturtok requested a review from thoniTUB February 27, 2024 13:32

awildturtok added 7 commits February 27, 2024 15:53

cleanup logging

6e7c26c

use more threads

8c15b19

Implements limit for SingleTableResult, to only output a specified nu…

6c0ea9b

…mber of lines

split code paths for zeroCentered and rounding

e68c05c

merge integer min/max if they represent only one integer.

e74faf5

adjusts bin labels and adds test for 0 adjacent labels

96c454d

Merge pull request #3315 from ingef/feature/limit-query-out

8eadea0

Implements limit for SingleTableResult, to only output a specified nu…

thoniTUB approved these changes Feb 29, 2024

View reviewed changes

awildturtok added 10 commits February 29, 2024 16:13

fixes missing l-adjust for unrounded histogram

35dd41c

enforce non-open sets

1aa34a6

Merge branch 'develop' into feature/query-stats-followup

f0e8a88

fixes Currency usage

08027fa

simplify bins, if integral

4b2edb7

Merge remote-tracking branch 'origin/feature/query-stats-followup' in…

ac7da74

…to feature/query-stats-followup

use em-dash and fix overflow bin-label

f1fe91d

fix missing overflow=true in rounded case

2964ab6

fix german label for std-dev

a566a7d

extract common string

d9e8b49

awildturtok changed the base branch from develop to release March 5, 2024 14:18

awildturtok merged commit 06042ec into release Mar 5, 2024

delete-merged-branch bot deleted the feature/query-stats-followup branch March 5, 2024 14:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Query stats followup #3255

Query stats followup #3255

awildturtok commented Dec 20, 2023

thoniTUB Feb 27, 2024

thoniTUB Feb 27, 2024

awildturtok Feb 27, 2024

thoniTUB Feb 27, 2024

awildturtok Feb 27, 2024

thoniTUB Feb 27, 2024

awildturtok Feb 27, 2024

thoniTUB Feb 27, 2024

awildturtok Feb 27, 2024

thoniTUB Feb 29, 2024

		final int lowerPercentile = config.getVisualisationPercentiles().hasLowerBound() ? config.getVisualisationPercentiles().lowerEndpoint() : 0;
		final int upperPercentile = config.getVisualisationPercentiles().hasUpperBound() ? config.getVisualisationPercentiles().upperEndpoint() : 100;


		private final Comparator<TYPE> comparator;
		public NumberColumnStatsCollector(String name, String label, String description, ResultType type, PrintSettings printSettings, int expectedBins, double lowerPercentile, double upperPercentile) {

Query stats followup #3255

Query stats followup #3255

Conversation

awildturtok commented Dec 20, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment