matching stats as join table by awildturtok · Pull Request #3835 · ingef/conquery

awildturtok · 2026-01-22T16:50:07Z

Eine Beispielhafte join-table könnte folgende Einträge enthalten:

                     resolved_id                      | col_val
------------------------------------------------------+----------
 database.icd.s00-t98.s00-s09.s00.s00_0.s00_00 | T612
 database.icd.s00-t98.s00-s09.s00.s00_0.s00_00 | T611
 database.icd.s00-t98.s00-s09.s00.s00_0.s00_00 | T610
 database.icd.j00-j99.j30-j39.j38.j38_0.j38_00 | J951

  select
    diag.id as "pid",
    least(
      'infinity'::date,
      lower("datum"),
      upper("datum")
    ) as "lower_bound",
    greatest(
      '-infinity'::date,
      lower("datum"),
      upper("datum")
    ) as "upper_bound",
    "resolved_id"
  from "diag"
    left outer join "icd_ids"
      on "diag"."icd_code" = "col_val"

im Falle von ColumnEquals wären dann noch weitere Spalten dabei. Bei PRESENT/EMPTY wären stattdessen true/false drin, die dann im join abgebildet werden.

# Conflicts: # backend/src/main/java/com/bakdata/conquery/mode/local/LocalStorageListener.java

… SqlMatchingStats easier

Also fixes a case, where missing entries were not registered to the root.

…ostgres

…tats-as-sql-function # Conflicts: # backend/src/main/java/com/bakdata/conquery/models/messages/namespaces/specific/UpdateMatchingStatsMessage.java

(experimental)

(this is a bit absurd tbh)

thoniTUB · 2026-01-23T07:21:32Z

+@Data
+public abstract class StorageListener<T extends Namespace>{


Ich hätte das Interface beibehalten und davon diese abstract Class abgeleitet.

Interfaces können auch praktisch sein, da man für diese Proxies implementieren kann

Ich hatte glaube ich gehofft mehr logik zu vereinheitlichen, bin auch nicht deiner Meinung. Aber hier ist es definitiv die klasse wert.

…tats-as-join-table # Conflicts: # backend/src/main/java/com/bakdata/conquery/mode/local/LocalManagerProvider.java # backend/src/main/java/com/bakdata/conquery/models/worker/LocalNamespace.java # backend/src/main/java/com/bakdata/conquery/sql/conversion/cqelement/concept/CQConceptConverter.java # backend/src/main/java/com/bakdata/conquery/sql/conversion/dialect/PostgreDialectBundle.java # backend/src/main/java/com/bakdata/conquery/sql/conversion/query/TableExportQueryConverter.java # backend/src/main/java/com/bakdata/conquery/util/TablePrimaryColumnUtil.java # backend/src/test/java/com/bakdata/conquery/integration/json/SqlTestDataImporter.java

awildturtok · 2026-04-09T12:14:48Z

+
+
+	@Override
+	public void execute() throws Exception {


Hier bin ich noch ziemlich unentschlossen ob das reicht. Ich denke ich möchte auf jeden Fall die HikariConfig exposen in der config.json. Dann können wir darüber das connection management machen anstatt Angst zu haben

awildturtok · 2026-04-09T12:15:16Z

+
+		Stopwatch stopwatch = Stopwatch.createStarted();
+
+		ExecutorService executorService = Executors.newSingleThreadExecutor();


die collectMatchingStatsForConcept hat ein async callable, das sollte also genau passen

awildturtok · 2026-04-09T12:16:02Z

+@Data
+public abstract class StorageListener<T extends Namespace>{


Ich hatte glaube ich gehofft mehr logik zu vereinheitlichen, bin auch nicht deiner Meinung. Aber hier ist es definitiv die klasse wert.

awildturtok · 2026-04-09T12:17:25Z

+
 	boolean matches(String value, CalculatedValue<Map<String, Object>> rowMap) throws ConceptConfigurationException;

+	//TODO implement using join-table


ist ein separates Projekt, also kann bleiben

awildturtok · 2026-04-09T12:39:37Z

-	protected static Field<Object> coalesceFields(List<Field<?>> fields) {
-		if (fields.size() == 1) {
-			return fields.get(0).coerce(Object.class);
+	protected static <T> Field<T> coalesceFields(List<? extends Field<?>> fields, Class<T> type) {


weiß leider nicht mehr wo aber hier gab es einen komischen Bug damit.

Ist es schlau hier die Argumente (also fields.get(0)) zu überschreiben? Willst du die Funktion nicht lieber idempotent machen.

Ich würde auch das coerce erst im Nachgang machen, wenn das geht.

das sieht tatsächlich besser aus, danke

thoniTUB

Ich muss nochmal weiter machen.

thoniTUB · 2026-04-10T06:49:29Z


 	@Setter @Getter @NotEmpty
 	private Set<String> values;
 	@NotEmpty @Setter @Getter


Suggested change

@NotBlank @Setter @Getter

thoniTUB · 2026-04-10T06:49:53Z

 public class ColumnEqualCondition implements CTCondition {

 	@Setter @Getter @NotEmpty
 	private Set<String> values;


Suggested change

private Set<@NotBlank String> values;

thoniTUB · 2026-04-10T07:08:11Z

+					fieldParams = otherParams;
+				}
+				else {
+					fieldParams = Sets.intersection(otherParams, myParams);


Warum ist das ein intersect und kein union?

weil die zwei elemente verwundet werden

🩸 🗡️

Okay muss ich nochmal überdenken, ich dachte hier werden Filter/Conditionen verundet und nicht die gefilterten Werte

hier werden Concept-Conditions verundet

thoniTUB · 2026-04-10T07:13:02Z

+	//TODO better name
+	record Expression(ConceptElement<?> conceptElement, Map<Field<?>, Set<Param<?>>> conditions) {


Es wird ja nur für matching stats genommen, oder? Dann würde ich das in den Namen einfließen lassen

thoniTUB · 2026-04-10T07:30:12Z

 /**
 * This condition requires that the selected Column has a value.
 */
 @CPSType(id="PRESENT", base=CTCondition.class)


Würde ich behalten

thoniTUB · 2026-04-10T12:03:42Z

+	/**
+	 * Drop the table, then recreate it.
+	 */
+	private void createConceptIdsTable(Name tableName, List<Field<?>> fields) {
+
+		log.debug("Creating table {} with fields {}", tableName, fields);
+
+		CreateTableElementListStep createTable =
+				dslContext.createTable(tableName)
+						  .columns(fields);
+
+		log.info("{}", createTable);


Ist das drop implizit? ich sehe es hier nicht

passiert im caller

Dann bitte die Doku anpassen

thoniTUB · 2026-04-14T09:29:36Z

-	protected static Field<Object> coalesceFields(List<Field<?>> fields) {
-		if (fields.size() == 1) {
-			return fields.get(0).coerce(Object.class);
+	protected static <T> Field<T> coalesceFields(List<? extends Field<?>> fields, Class<T> type) {


Ist es schlau hier die Argumente (also fields.get(0)) zu überschreiben? Willst du die Funktion nicht lieber idempotent machen.

Ich würde auch das coerce erst im Nachgang machen, wenn das geht.

thoniTUB · 2026-04-14T09:31:30Z

 {
  "type": "QUERY_TEST",
  "sqlSpec": {
-    "isEnabled": true


wieder enablen

thoniTUB · 2026-04-14T09:45:34Z

+			case MONEY -> true; // Not possible to introspect for
 			case DATE -> field.getDataType().isDate();
-			case DATE_RANGE -> field.getDataType().getTypeName().equals("daterange");
+			case DATE_RANGE -> true; // Not possible to introspect for


We wäre es hiermit?

Suggested change

case DATE_RANGE -> true; // Not possible to introspect for

case DATE_RANGE -> field.getDataType().getType().equals(org.jooq.postgres.extensions.types.DateRange.class);

ich kann mal schauen, aber iirc kommen die einfach als Object zurück

Zu mindest in den Tests würde habe ich gesehen, dass hier passen würde

Ich meine, dass ich es in den Tests auch mal drin hatte und dann deaktivieren musste, sobald ich es auf eine separate Instanz hab zeigen lassen. Ich habs mal zurückgedreht und schau es mir an.

awildturtok and others added 30 commits November 12, 2025 15:24

First draft of function to resolve concepts

74d9da4

# Conflicts: # backend/src/main/java/com/bakdata/conquery/mode/local/LocalStorageListener.java

remove qualification from CTConditionContext to make integration with…

e028841

… SqlMatchingStats easier

Collect Auxiliary columns to then generate proper signature

4cccd65

properly insert function (only postgres atm)

209d881

first draft towards applying matching stats

8620c65

Reworks registration of MatchingStats.

51f1aea

Also fixes a case, where missing entries were not registered to the root.

fixes usage in MatchingStatsTests.java

98b69c4

wip

f09cdaa

implements conversion and extraction of matching stats for Hana and P…

107407d

…ostgres

Merge remote-tracking branch 'origin/develop' into feature/matching-s…

1b4949f

…tats-as-sql-function # Conflicts: # backend/src/main/java/com/bakdata/conquery/models/messages/namespaces/specific/UpdateMatchingStatsMessage.java

disable daterange and money type compat check

3f0b1bb

adds missing coalesce for primaryColumn

ff18d93

don't remove connector column

b86af08

(experimental)

removes very verbose logging

7735740

use cursor to iterate results

ed98bd2

adds timing to SQL matching stats fetching

368b4f5

use transaction to disable autocommit

5ffdade

log select statement for debugging

0cb22e8

add PARALLEL SAFE marker to created functions

8e64d47

reworks SqlMatchingStats function as flattened table

8d5cd0a

(this is a bit absurd tbh)

adds some logging

44899ee

adds missing concept Id

e2df161

adds missing error handling in outer loop

750a03b

adds grouping by params to map into most specific child

152fc72

fixes wrong usage of immutable datasctructure

37009b0

try to insert the join-tables

67a0547

adds typing to fields

34b8bb5

remove primary key (nullability issue) and add index

4ff4776

fix index creation

11bd235

fix index creation #2

8e56eda

awildturtok added 3 commits January 22, 2026 16:21

more cleanup

53ba076

more cleanup

11c304f

fix union to intersection

3d02375

awildturtok requested a review from jnsrnhld January 22, 2026 16:50

awildturtok requested a review from thoniTUB as a code owner January 22, 2026 16:50

thoniTUB reviewed Jan 23, 2026

View reviewed changes

jnsrnhld reviewed Jan 30, 2026

View reviewed changes

cleanup

4366d85

awildturtok force-pushed the feature/matching-stats-as-join-table branch from 81dd8ff to 4366d85 Compare February 5, 2026 15:32

Cleanup of failing tests

1ef85b9

awildturtok force-pushed the feature/matching-stats-as-join-table branch from db2209e to 1ef85b9 Compare February 17, 2026 09:02

awildturtok added 2 commits February 17, 2026 11:45

Merge branch 'develop' into feature/matching-stats-as-join-table

e301226

fix hana insertion

14b72b5

awildturtok force-pushed the feature/matching-stats-as-join-table branch from 4a218ad to 14b72b5 Compare February 17, 2026 16:28

awildturtok and others added 2 commits February 18, 2026 14:54

Some more Hana fixes

7b9d34d

Merge branch 'develop' into feature/matching-stats-as-join-table

6e7d423

awildturtok enabled auto-merge (squash) March 5, 2026 15:06

awildturtok force-pushed the feature/matching-stats-as-join-table branch from 89dca10 to 8afe6c0 Compare April 8, 2026 13:32

Fixes computation of VARCHAR columns

9871265

awildturtok force-pushed the feature/matching-stats-as-join-table branch from 8afe6c0 to 9871265 Compare April 8, 2026 14:19

Fix quoting for hana

6aea1c9

awildturtok requested a review from thoniTUB April 9, 2026 10:00

awildturtok disabled auto-merge April 9, 2026 10:02

awildturtok commented Apr 9, 2026

View reviewed changes

Cleanup

bfaaba1

thoniTUB reviewed Apr 10, 2026

View reviewed changes

thoniTUB reviewed Apr 14, 2026

View reviewed changes

		@Data
		public abstract class StorageListener<T extends Namespace>{


		Stopwatch stopwatch = Stopwatch.createStarted();

		ExecutorService executorService = Executors.newSingleThreadExecutor();


		boolean matches(String value, CalculatedValue<Map<String, Object>> rowMap) throws ConceptConfigurationException;

		//TODO implement using join-table

		//TODO better name
		record Expression(ConceptElement<?> conceptElement, Map<Field<?>, Set<Param<?>>> conditions) {

	case DATE_RANGE -> true; // Not possible to introspect for
	case DATE_RANGE -> field.getDataType().getType().equals(org.jooq.postgres.extensions.types.DateRange.class);

Conversation

awildturtok commented Jan 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

thoniTUB left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

awildturtok commented Jan 22, 2026 •

edited

Loading