v18.4.0

[v18.4.0] 2024-08-23

Migration Notes

Important
This release requires manual intervention before applying the Terraform changes. The two tickets below require manual database updates including adding indices, modifying table constraints, and updating types. Please read and complete the steps outlined in the following two tickets, CUMULUS-3320 and CUMULUS-3449 before deploying v18.4.0.

CUMULUS-3320 Update executions table

The work for CUMULUS-3320 required index updates as well as a modification of a
table constraint. To install the update containing these changes you should:

• Pre-generate the indexes on the execution table. This can be done via manual
  procedure prior to upgrading without downtime, or done more quickly before or
  during upgrade with downtime.
• Update the executions_parent_cumulus_id_foreign constraint. This will
  require downtime as updating the constraint requires a table write lock, and
  the update may take some time.

Deployments with low volume databases and low activity and/or test/development
environments should be able to install these updates via the normal automatic
Cumulus deployment process.

Please carefully review the migration process documentation. Failure to
make these updates properly will likely result in deployment failure and/or
degraded execution table operations.

CUMULUS-3449 Please follow the instructions before upgrading Cumulus

• The updates in CUMULUS-3449 requires manual update to postgres database in
  production environment. Please follow Update Cumulus_id Type and
  Indexes

Breaking Changes

Added

• CUMULUS-3320
  • Added endpoint /executions/bulkDeleteExecutionsByCollection to allow
    bulk deletion of executions from elasticsearch by collectionId
  • Added Bulk Execution Delete migration type to async operations types
• CUMULUS-3608
  • Exposes variables for sqs_message_consumer_watcher messageLimit and timeLimit configurations. Descriptions
    of the variables here include notes on usage and what users should
    consider if configuring something other than the default values.
• CUMULUS-3449
  • Updated the following database columns to BIGINT: executions.cumulus_id, executions.parent_cumulus_id,
    files.granule_cumulus_id, granules_executions.granule_cumulus_id, granules_executions.execution_cumulus_id
    and pdrs.execution_cumulus_id
  • Changed granules table unique constraint to granules_collection_cumulus_id_granule_id_unique
  • Added indexes granules_granule_id_index and granules_provider_collection_cumulus_id_granule_id_index
    to granules table

Changed

• CUMULUS-3320
  • Updated executions table (please see Migration section and Upgrade
    Instructions for more information) to:
    • Add index on collection_cumulus_id
    • Add index on parent_cumulus_id
    • Update executions_parent_cumulus_id_foreign constraint to add ON DELETE SET NULL. This change will cause deletions in the execution table to
      allow deletion of parent executions, when this occurs the child will have
      it's parent reference set to NULL as part of the deletion operations.
• CUMULUS-3449
  • Updated @cumulus/db package and configure knex hook postProcessResponse to convert the return string
    from columns ending with "cumulus_id" to number.
• CUMULUS-3841
  • Increased fetchRules page size to default to 100 instead of 10. This improves overall query time when
    fetching all rules such as in sqsMessageConsumer.

Fixed

• CUMULUS-3817
  • updated applicable @AWS-SDK dependencies to 3.621.0 to remove inherited vulnerability from fast-xml-parser
• CUMULUS-3320
  • Execution database deletions by cumulus_id should have greatly improved
    performance as a table scan will no longer be required for each record
    deletion to validate parent-child relationships
• CUMULUS-3818
  • Fixes default value (updated to tag 52) for async-operation-image in tf-modules/cumulus.
• CUMULUS-3840
  • Fixed @cumulus/api/bin/serve to correctly use EsClient.