Development Release - dev

Bug Fixes

• Handled groupby with no dimensions fields (#68)
• first flag in CacheIterator was not set correctly (#101)
• CacheIterator descending from past db end (#105)
• Fix cache changes on update operation (#116)
• Benches report missing the comparison (#183)
• Fix schema validation when only some tables are used (#174)
• Wait until all schema initialized before start grpc server (#187)
• Secondary index key should never be empty (#200)
• Make dynamic server optional
• Redirect events to the right endpoint (#206)
• Db settings + schema checkpointing (#227)
• Remove all features flag in coverage (#255)
• Synchronize termination among multiple sources (#257)
• Remove array types (#259)
• Fix file path for snowflake benchmark (#297)
• Dont define snowflake as default feature for dozer-admin (#304)
• Rely on cache instead of events (#308)
• Use badge from main branch (#327)
• Simplify Aggregator test (#338)
• Update serial_test to 0.9.0 because the old version depended (#344)
• Use source based coverage and ignore ../* (#347)
• Reduce schema update calls for snowflake ingestion (#358)
• Numbers were serialized as strings in REST API (#369)
• Fix config folder location for rustdoc (#372)
• Allow conversion from null when field is nullable (#371)
• Seconary index was not calculated properly during querying (#366)
• Add param experimental_allow_proto3_optional as default when compiling protoc (#342)
• Implement proper timeout handling in sql tests (#378)
• Single order_by query didn't work (#375)
• order_by without range filter didn't work (#382)
• Handle NULL in cache (#391)
• Support for NULL and Decimal in SUM() aggregator (#394)
• Retrieve old record from db in sqlite tests (#392)
• Criterion build, remove build dependency (#356)
• Fix bench functions (#403)
• Implement NULL support for SQL expressions (#405)
• Reuse connection (#412)
• Update changes structure (#415)
• Fix coverage tests configs (#416)
• Remove conflicting parameters from tests sources config (#417)
• Bring back notifier which was wrongly removed in getdozer#407 (#424)
• Display error of dozer app (#429)
• Revert replication slots check query changes (#433)
• Readonly LmdbCache didn't open secondary index databases (#439)
• Handle eth data conversion for longer strings and fix a few orchestrator errors (#446)
• Handle event name conflicts (#449)
• Fix RDS user validation for postgres replication (#454)
• Cache benchmark was panicking because duplicated insertion (#461)
• Unwrap errors from pipeline (#460)
• Synchronize commits across multiple sources (#471)
• Fix schema initiation and port mapping in new source builder (#486)
• Hanging dag tests (#494)
• Increase max readers on lmdb cache (#497)
• Improve errors (#498)
• Readme (#502)
• Broken link in readme (#504)
• Synchronize sources startup (to pre-populate RecordReader) (#511)
• Propagate errors from connector threads. (#517)
• Oapi schema was not generated correctly (#545)
• Propagate nullables along SQL schema (#546)
• Update samples with public ecr image (#551)
• Fix postgres replication messages handling on update (#550)
• Id in LmdbCache can be duplicated for different records (#568)
• Remove duplicate dependencies and bump a few libs. (#571)
• Display user friendly postgres connection failure (#573)
• dozer-tests could not be started due to last renaming refactor (#579)
• Generate descriptor once per init (#591)
• Handle error return from thread when intialize grpc + rest (#581)
• Typos and links in proto files (#593)
• Fix eth e2e test. Update primary key mismatch error message. (#598)
• Examples and Readme (#617)
• Count APIs should default to no limit (#623)
• Discord notification blocked because PR is from public fork (#630)
• Dozer-api test utils was generating inconsistent schema and record (#635)
• Count API still used default limit when query string is empty (#634)
• Schema id was not copied to records before inserting (#636)
• Allow text fields being used in full text index (#639)
• Generate full text index for String fields (#638)
• Ignore unused sources for pipeline (#646)
• Copy schema id to records before letting cache handle it (#649)
• Fix sql parsing panic and statements being silently ignored (#642)
• Need ignore_pipeline_branch_filters to trigger when pushes are disabled (#662)
• OnEvent is not propagating data in some scenarios with postgres (#637)
• Don't pass commit message to BuildKite. (#665)
• Update issue template config with parent item (#677)
• Change mapping of sqlite data type to dozer data type (#689)
• CONCAT() to support multiple arguments (#690)
• Unify typed service query and on_event record representation (#647)
• Implement record versioning in RecordWriter and RecordReader (#682)
• Fix minor snowflake connector issues (#681)
• Print unknown command error and continue (#671)
• Forbid duplicated cte names (#703)
• Use context to solve alias properly (#708)
• Fix snowflake connector (#745)
• Unable clean untagged images for now (#752)
• Fix source with empty columns list validation (#714)
• serde_str is needed to bincode deserialize Decimal (#757)
• Don't commit when no source produces new op (#669)
• Fix stream exist check for tables (#754)
• Architecture diagram ([getdozer#758](https://github.com/...

Development Release - latest

Bug Fixes

• Handled groupby with no dimensions fields (#68)
• first flag in CacheIterator was not set correctly (#101)
• CacheIterator descending from past db end (#105)
• Fix cache changes on update operation (#116)
• Benches report missing the comparison (#183)
• Fix schema validation when only some tables are used (#174)
• Wait until all schema initialized before start grpc server (#187)
• Secondary index key should never be empty (#200)
• Make dynamic server optional
• Redirect events to the right endpoint (#206)
• Db settings + schema checkpointing (#227)
• Remove all features flag in coverage (#255)
• Synchronize termination among multiple sources (#257)
• Remove array types (#259)
• Fix file path for snowflake benchmark (#297)
• Dont define snowflake as default feature for dozer-admin (#304)
• Rely on cache instead of events (#308)
• Use badge from main branch (#327)
• Simplify Aggregator test (#338)
• Update serial_test to 0.9.0 because the old version depended (#344)
• Use source based coverage and ignore ../* (#347)
• Reduce schema update calls for snowflake ingestion (#358)
• Numbers were serialized as strings in REST API (#369)
• Fix config folder location for rustdoc (#372)
• Allow conversion from null when field is nullable (#371)
• Seconary index was not calculated properly during querying (#366)
• Add param experimental_allow_proto3_optional as default when compiling protoc (#342)
• Implement proper timeout handling in sql tests (#378)
• Single order_by query didn't work (#375)
• order_by without range filter didn't work (#382)
• Handle NULL in cache (#391)
• Support for NULL and Decimal in SUM() aggregator (#394)
• Retrieve old record from db in sqlite tests (#392)
• Criterion build, remove build dependency (#356)
• Fix bench functions (#403)
• Implement NULL support for SQL expressions (#405)
• Reuse connection (#412)
• Update changes structure (#415)
• Fix coverage tests configs (#416)
• Remove conflicting parameters from tests sources config (#417)
• Bring back notifier which was wrongly removed in getdozer#407 (#424)
• Display error of dozer app (#429)
• Revert replication slots check query changes (#433)
• Readonly LmdbCache didn't open secondary index databases (#439)
• Handle eth data conversion for longer strings and fix a few orchestrator errors (#446)
• Handle event name conflicts (#449)
• Fix RDS user validation for postgres replication (#454)
• Cache benchmark was panicking because duplicated insertion (#461)
• Unwrap errors from pipeline (#460)
• Synchronize commits across multiple sources (#471)
• Fix schema initiation and port mapping in new source builder (#486)
• Hanging dag tests (#494)
• Increase max readers on lmdb cache (#497)
• Improve errors (#498)
• Readme (#502)
• Broken link in readme (#504)
• Synchronize sources startup (to pre-populate RecordReader) (#511)
• Propagate errors from connector threads. (#517)
• Oapi schema was not generated correctly (#545)
• Propagate nullables along SQL schema (#546)
• Update samples with public ecr image (#551)
• Fix postgres replication messages handling on update (#550)
• Id in LmdbCache can be duplicated for different records (#568)
• Remove duplicate dependencies and bump a few libs. (#571)
• Display user friendly postgres connection failure (#573)
• dozer-tests could not be started due to last renaming refactor (#579)
• Generate descriptor once per init (#591)
• Handle error return from thread when intialize grpc + rest (#581)
• Typos and links in proto files (#593)
• Fix eth e2e test. Update primary key mismatch error message. (#598)
• Examples and Readme (#617)
• Count APIs should default to no limit (#623)
• Discord notification blocked because PR is from public fork (#630)
• Dozer-api test utils was generating inconsistent schema and record (#635)
• Count API still used default limit when query string is empty (#634)
• Schema id was not copied to records before inserting (#636)

Chore

• Add build folder to ignored list for coveralls (#296)

Documentation

• Add build dependences in README (#96)
• Update postgres requirements (#307)
• Include kill_cache=1 param (#370)
• Add gRPC proto file documentation (#477)
• Include contributing section (#609)
• Update logo (#633)

Features

• Init ingestion postgres
• Reorganise into a mono repo (#6)
• Merge pipeline code (#7)
• Merge api (#8)
• Implement first version of storage (#17)
• Refactor connector * iterator behavior (#24)
• Update issue templates
• Implement type conversion for postgresql (#25)
• Implement basic cache operations (#48)
• Implement secondary indexes (#49)
• Implement query bench (#50)
• Implement proper schema id usage in snapshotter and refactor mapping (#51)
• Implement simple orchestrator (#61)
• Init dozer ci (#77)
• Include concurrency and multiple restore keys (#82)
• Implement multi ingestion sources (#78)
• Implement continue of replication messages stream handling (#94)
• Dozer-api REST (#86)
• Allows to specify state store options (#112)
• Add FullText IndexType (#121)
• Multi-key indexer for state store (#127)
• Implement e2e tests ([getdozer#130](https://github.com/getdozer/dozer/is...