Marquez 0.49.0

Added

• API: Job-to-Job lineage #2752 @yanlibert
  Intended in part to spur a larger discussion of full parent/child hierarchy handling in Marquez. Changes only the backend API, adding the Job UUID along with the parent name to the Job metadata returned.

Fixed

• Web: security updates #2864 @phixMe
  Resolves critical security issues found using NPM's audit command.
• Web: encode Job name in API requests #2866 @dolfinus
  Urlencodes Job, Dataset, tag and field names while sending an API request.

Marquez 0.48.0

Added

• API: add endpoint method and path to metrics name #2850 @JDarDagran
  In the metrics endpoint, there was information gathered containing the SQL Object name and method name. This introduces labels (DAO name, DAO method, endpoint method, endpoint path) and adds more information about endpoints.
• API: add paging to dataset versions panel #2855 @davidsharp7
  Adds Datasets paging.
• API: add paging on Jobs panel #2852 @davidsharp7
  Adds Job-level paging of Runs.
• API: add Dataset schema versions #2763 @davidjgoss
  Adds Dataset schema versions to the model and enables writing to it.
• Docker: make db port configurable via POSTGRES_PORT #2751 @merobi-hub
  Adds support for easy db port reassignment.
• Java: allow customization of Apache HTTP in Java client #2822 @davidjgoss
  Allows customization of Apache HTTP in Java client.
• Web: add Job tagging to UI #2837 @davidsharp7
  Adds Job tagging to the UI.
• Web: source code facets #2833 @phixMe
  Adds typedef and rendering of the sourceCode facet for a Job if available.

Fixed

• API: Dataset query to get only the latest facet for each version #2859 @sophiely
  The facet partition is ranked by Dataset version and facet name so as we can take only the most recent facet for each Dataset UUID and type.
• API: optimize column lineage query performance #2821 @vinhnemo
  Adds a filter condition to the CTE dataset_fields_view in ColumnLineageDao.java.
• Web: deduplicate the versions displayed #2854 @namyyys
  Excludes the symlinks from the result of the query displaying the version history in order to exclude duplicate versions.
• Web: clean up issues highlighted by some Spark Integration Data #2856 @phixMe
  Fixes numerous issues in our interfaces related to some OpenLineage Spark events.
• Web: remove limit from assertion evaluation #2844 @phixMe
  Fixes bug where our status indicator was the wrong color.
• Web: bring Dataset tags into line with Job Tags #2841 @davidsharp7
  Brings Dataset tags into line with Job tags.
• Web: fix scroll issues for drawer and home pages #2820 @phixMe
  Scrolling improvements for drawer and home pages.
• Web: fix search endpoint parameters #2818 @Nisarg-Chokshi
  The search API parameters were not getting updated correctly on changing the filter and sort options.

Removed

• Web: DRY paging #2832 @phixMe
  Removes repeated code for paging on lineage events, jobs and datasets.

Marquez 0.47.0

Added

---

Data Quality and Job Status Display in Marquez Web

• Web: Data Quality #2810 #2785 #2812 @phixMe
  Adds Data Quality and Job Status display features.

---

• API: add job tagging to API #2774 @davidsharp7
  Adds support for job tagging to the API.
• Chart: add serviceAccountand extraContainers to helm chart values #2766 @kostas-theo
  To make the Kubernetes service account configurable, adds these values to the helm chart values with defaults set to maintain current functionality.
• Client/Java: add jobVersion field to Run in Java client #2808 @davidjgoss
  Adds jobVersion field to Run in Java client.
• Docker: improve down.sh script #2778 @dolfinus
  Adds new -v option and fixes down.sh script to rely on docker-compose down -v and make volume deletion optional.
• Web: tooltips and display updates #2809 @phixMe
  Updates tooltips to be more modernized and custom.
• Web: update JSON theme #2807 @phixMe
  Makes the JSON theme more in-line with the Marquez brand.
• Web: column lineage linking and sticky tab titles #2805 @phixMe
  Adds sticky Titles and moves column lineage links to the table definition.
• Web: refine panel feature set #2798 @phixMe
  Adds many refinements in response to user feedback.
• Web: update dataset/dataset field-tagging experience #2761 @davidsharp7
  Adds support for adding multiple tags at once, introduces a switch to allow field-level tags to be exposed, and fixes refresh for an improved field-tagging experience.
• Web: web refresh + loading states #2779 @phixMe
  Adds a refresh button for jobs, datasets, and lineage events pages. This also will work in empty states.

Removed

• Web: remove old files and dependencies #2801 @phixMe
  Drops deps and removes unused React components no longer required by the new lineage graph.

Fixed

• API: adapt column lineage query for symlink dataset #2775 @sophiely
  Changes the column lineage query in order to take only the 'main' dataset, not the dataset created via symlink.
• Web: resolve issue data quality assertion facet are not displayed #2528 @sophiely
  Fixes rendering of the DataQualityAssertion facet by adding support for dataset, unknown and input.
• Web: fix showTags refresh #2799 @davidsharp7
  Adds showTags to the dependencies of fetchDatasetVersions and disables the show tags toggle until the latest version has been pulled.
• Web: various dataset tags improvements #2813 @davidsharp7
  Various tag improvements including a carat for the dropdown.
• Web: use Webpack-bundled icon instead of GitHub-hosted content #2803 @dodo0822
  For compliance with a strict CSP, replaces the icon with an SVG bundled by Webpack instead of linking to raw.githubusercontent.com.

Marquez 0.46.0

Changed

• Web: various revisions #2770 @phixMe
  Includes clean up of issues in the UI and removal of non-useful elements.

Fixed

• Streaming API: fix behaviour for COMPLETE/FAIL events within streaming jobs #2768 @pawel-big-lebowski
  New job_version is not created for a streaming job terminal event with no dataset information and existing version is kept.

Marquez 0.45.0

Added

---

Redesigned Web UI Featuring Column Lineage

• Web: updates to Table and Column Lineage #2725 @phixMe
  A new page for column lineage and an updated view for lineage with a common set of shared principles.
• Web: quality of life updates for new lineage graph display #2750 @phixMe
  Visual updates from early feedback on lineage graph navigation, including a zoom button to center on the selected node.
• Web: improve visual display of lineage #2753 @phixMe
  Visual improvements to nodes including the addition of more detail and the ability to collapse dataset nodes manually.

---

• Web: add dataset field level tags to UI #2729 @davidsharp7.
  Updates to the DatasetTags component to allow for field-level tagging/deletion and addition of this to the DatasetInfo component.
• Web: update dataset tags to allow editing/addition of tags #2759 @davidsharp7
  Updates to DatasetTags to include a split button menu and a new dialog/reducer for adding new tags.
• Web: minor dataset tags revisions #2754 @phixMe
  Minor cleanup of the dataset tags feature including a pointer on the expandable row and a transition on row expansion, plus some new CSS elements.

Fixed

• Web: minor UI enhancements #2727 @phixMe
  Hygienic cleanup of project as a follow-up to #2725, including a fix for #2747.
• Web: fix symlink display #2736 @sophiely
  Changed behavior to display the symlink dataset in the previously empty namespace and link the symlink dataset lineage to the main dataset.

Marquez 0.45.0-rc.1

Added

• Web: updates for Table and Column Lineage #2725 @phixMe
  Creates a new page for column lineage and an updated view for lineage with a common set of shared principles.
• Web: add dataset field level tags to UI #2729 @davidsharp7
  Updates the DatasetTags component to allow for field-level tagging/deletion and adds this to the DatasetInfo component.

Fixed

• Web: minor UI enhancements #2727 @phixMe
  Hygienic cleanup of project as a follow-up to #2725, including a fix for #2747.
• API: fill data in column lineage input nodes #2742 @JDarDagran @wslulciuc
  Fixes the issue of null output nodes in the column lineage endpoint.

Marquez 0.44.0

Added

• Web: add dataset tags tabs for adding/deleting of tags #2714 @davidsharp7
  Adds a dataset tags component so that datasets can have tags added/deleted.
• API: Add endpoint to delete field-level tags #2705 @davidsharp7
  Adds delete endpoint to remove dataset field tags.

Fixed

• Web: fix dataset tag reducers bug #2716 @davidsharp7
  Removes result from dataset tags reducer to fix a sidebar bug.

Marquez 0.43.1

Fixed

• API: fix broken lineage graph for multiple runs of the same job #2710 @pawel-big-lebowski
  Problem: lineage graph was not available for jobs run multiple times of the same job as a result of bug introduced with recent release. In order to fix the inconsistent data, this UPDATE query should be run. This is not required when upgrading directly to 0.43.0.

Marquez 0.43.0

Added

• API: refactor the RunDao SQL query #2685 @sophiely
  Improves the performance of the SQL query used for listing all runs.
• API: refactor dataset version query #2683 @sophiely
  Improves the performance of the SQL query used for the dataset version.
• API: add support for a DatasetEvent #2641 #2654 @pawel-big-lebowski
  Adds a feature for saving into the Marquez model datasets sent via the DatasetEvent event type. Includes optimization of the lineage query.
• API: add support for a JobEvent #2661 @pawel-big-lebowski
  Adds a feature for saving into the Marquez model jobs and datasets sent via the JobEvent event type.
• API: add support for streaming jobs #2682 @pawel-big-lebowski
  Creates job version and reference rows at the beginning of the job instead of on complete. Updates the job version within the run if anything changes.
• API/spec: implement upstream run-level lineage #2658 @julienledem
  Returns the version of each job and dataset a run is depending on.
• API: add DELETE endpoint for dataset tags #2698 @davidsharp7
  Creates a new endpoint for removing the linkage between a dataset and a tag in datasets_tag_mapping to supply a way to delete a tag from a dataset via the API.
• Web: add a dataset drawer #2672 @davidsharp7
  Adds a drawer to the dataset column view in the GUI.

Fixed:

• Client/Java: change url path encoding to match jersey decoding #2693 @davidjgoss
  Swaps out the implementation of MarquezPathV1::encode to use the UrlEscapers path segment escaper, which does proper URI encoding.
• Web: fix pagination in the Jobs route #2655 @merobi-hub
  Hides job pagination in the case of no jobs.
• Web: fix empty search experience #2679 @phixMe
  Use of the previous search value was resulting in a bad request for the first character of a search.

Removed:

• Client/Java: remove maven-archiver dependency from the Java client #2695 @davidjgoss
  Removes a dependency from build.gradle that was bringing some transitive vulnerabilities.

Marquez 0.42.0

Added

• Client: add Java client method for dataset/job lineage #2623 @davidjgoss
  To add a method for the dataset/job-level endpoint (GET /lineage) to the Java SDK, this adds a new method to the MarquezClient for the endpoint, along with tests, and the necessary new subclasses of NodeData for datasets and jobs.
• Web: add IO tab #2613 @phixme
  Improves experience with large graphs by adding a new tab to move between graph elements without looking at the graph itself.
• Web: add hover-over Tag tooltip to datasets #2630 @davidsharp7
  For parity with columns in the GUI, this adds a Tag tooltip to datasets.

Changed

• Docker: upgrade to Docker Compose V2 #2644 @merobi-hub
  Docker Compose V1 has been at EOL since June, but docker/up.sh uses the V1 format. This upgrades the up command in up.sh to V2.

Removed

• API: drop table job_contexts and usage #2621 @wslulciuc
  Removes usage of job_contexts, which has been replaced by OpenLineage facets, and adds a migration to drop the table.
• API: remove usage of current_job_context_uuid column #2622 @wslulciuc
  Removes usage of job_context_uuid and current_job_context_uuid. Column to be removed in 0.43.0.

Fixed

• Web: fix Unix epoch time display for null endedAt values #2647 @merobi-hub
  Fixes the issue of the GUI displaying Unix epoch time (midnight on January 1, 1970) in the case of running jobs/null endedAt values.