Releases: nasa/cumulus
v16.1.1
Release 16.1.1
Compatible Orca Version
- v6.0.3 or v6.04. The orca related ticket work is for future orca releases.
Notable Changes
- The async_operation_image property of cumulus module should be updated to pull
the ECR image for cumuluss/async-operation:47
Added
- CUMULUS-3298
- Added extra time to the buffer for replacing the launchpad token before it expires to alleviate CMR error messages
- CUMULUS-3220
- Created a new send-pan task
- CUMULUS-3287
- Added variable to allow the aws_ecs_task_definition health check to be configurable.
- Added clarity to how the bucket field needs to be configured for the move-granules task definition
Changed
- Security upgrade node from 14.19.3-buster to 14.21.1-buster
- CUMULUS-2985
- Changed
onetime
rules RuleTrigger to only execute when the state isENABLED
and updated documentation to reflect the change - Changed the
invokeRerun
function to only re-run enabled rules
- Changed
- CUMULUS-3188
- Updated QueueGranules to support queueing granules that meet the required API granule schema.
- Added optional additional properties to queue-granules input schema
- CUMULUS-3252
- Updated example/cumulus-tf/orca.tf to use orca v8.0.1
- Added cumulus task
@cumulus/orca-copy-to-archive-adapter
, and add the task totf-modules/ingest
- Updated
tf-modules/cumulus
module to take variableorca_lambda_copy_to_archive_arn
and pass totf-modules/ingest
- Updated
example/cumulus-tf/ingest_and_publish_granule_with_orca_workflow.tf
CopyToGlacier
(renamed toCopyToArchive
) step to call
orca_copy_to_archive_adapter_task
- CUMULUS-3253
- Added cumulus task
@cumulus/orca-recovery-adapter
, and add the task totf-modules/ingest
- Updated
tf-modules/cumulus
module to take variableorca_sfn_recovery_workflow_arn
and pass totf-modules/ingest
- Added
example/cumulus-tf/orca_recovery_adapter_workflow.tf
,OrcaRecoveryAdapterWorkflow
workflow hasOrcaRecoveryAdapter
task
to call the ORCA recovery step-function. - Updated
example/data/collections/
collection configurationmeta.granuleRecoveryWorkflow
to useOrcaRecoveryAdapterWorkflow
- Added cumulus task
- CUMULUS-3215
- Create reconciliation reports will properly throw errors and set the async
operation status correctly to failed if there is an error. - Knex calls relating to reconciliation reports will retry if there is a
connection terminated unexpectedly error - Improved logging for async operation
- Set default async_operation_image_version to 47
- Create reconciliation reports will properly throw errors and set the async
- CUMULUS-3024
- Combined unit testing of @cumulus/api/lib/rulesHelpers to a single test file
api/tests/lib/test-rulesHelpers
and removed extraneous test files.
- Combined unit testing of @cumulus/api/lib/rulesHelpers to a single test file
- CUMULUS-3209
- Apply brand color with high contrast settings for both (light and dark) themes.
- Cumulus logo can be seen when scrolling down.
- "Back to Top" button matches the brand color for both themes.
- Update "note", "info", "tip", "caution", and "warning" components to new admonition styling.
- Add updated arch diagram for both themes.
- CUMULUS-3203
- Removed ACL setting of private on S3.multipartCopyObject() call
- Removed ACL setting of private for s3PutObject()
- Removed ACL confguration on sync-granules task
- Update documentation on dashboard deployment to exclude ACL public-read setting
- CUMULUS-3245
- Update SQS consumer logic to catch ExecutionAlreadyExists error and
delete SQS message accordingly. - Add ReportBatchItemFailures to event source mapping start_sf_mapping
- Update SQS consumer logic to catch ExecutionAlreadyExists error and
- Added missing name to throttle_queue_watcher Cloudwatch event in
throttled-queue.tf
Fixed
- CUMULUS-2625
- Optimized heap memory and api load in queue-granules task to scale to larger workloads.
- CUMULUS-3265
- Fixed
@cumulus/api
getGranulesForPayload
function to query cloud metrics es when needed.
- Fixed
v16.0.0
Release v16.0.0
Dashboard Requirement
Note that this release contains breaking changes for the Cumulus Dashboard. You must use the Cumulus Dashboard v12.0.1 https://github.com/nasa/cumulus-dashboard/releases/tag/v12.0.1
Notable Changes
- The async_operation_image property of cumulus module should be updated to pull
the ECR image for cumuluss/async-operation:46
MIGRATION notes
PI release version
When updating directly to v16 from prior releases older that V15, please make sure to
read through all prior release notes.
Notable migration concerns since the last PI release version (11.1.x):
- [v14.1.0] - Postgres compatibility update to Aurora PostgreSQL 11.13.
- [v13.1.0] - Postgres update to add
files_granule_cumulus_id_index
to the
files
table may require manual steps depending on load.
RDS Phase 3 migration notes
This release includes updates that remove existing DynamoDB tables as part of
release deployment process. This release cannot be properly rolled back in
production as redeploying a prior version of Cumulus will not recover the
associated Dynamo tables.
Please read the full change log for RDS Phase 3 and consult the RDS Phase 3 update
documentation
API Endpoint Versioning
As part of the work on CUMULUS-3072, we have added a required header for the
granule PUT/PATCH endpoints -- to ensure that older clients/utilities do not
unexpectedly make destructive use of those endpoints, a validation check of a
header value against supported versions has been implemented.
Moving forward, if a breaking change is made to an existing endpoint that
requires user updates, as part of that update we will set the current version of
the core API and require a header that confirms the client is compatible with
the version required or greater.
In this instance, the granule PUT/PATCH
endpoints will require a Cumulus-API-Version
value of at least 2
.
curl --request PUT https://example.com/granules/granuleId.A19990103.006.1000\
--header 'Cumulus-API-Version: 2'\
--header 'Authorization: Bearer ReplaceWithToken'\
--data ...
Users/clients that do not make use of these endpoints will not be impacted.
RDS Phase 3
Breaking Changes
- CUMULUS-2688
- Updated bulk operation logic to use collectionId in addition to granuleId to fetch granules.
- Tasks using the
bulk-operation
Lambda should provide collectionId and granuleId e.g. { granuleId: xxx, collectionId: xxx }
- CUMULUS-2856
- Update execution PUT endpoint to no longer respect message write constraints and update all values passed in
Changed
- CUMULUS-3282
- Updated internal granule endpoint parameters from :granuleName to :granuleId
for maintenance/consistency reasons
- Updated internal granule endpoint parameters from :granuleName to :granuleId
- CUMULUS-2312 - RDS Migration Epic Phase 3
- CUMULUS-2645
- Removed unused index functionality for all tables other than
ReconciliationReportsTable
fromdbIndexer
lambda
- Removed unused index functionality for all tables other than
- CUMULUS-2398
- Remove all dynamoDB updates for
@cumulus/api/ecs/async-operation/*
- Updates all api endpoints with updated signature for
asyncOperationsStart
calls - Remove all dynamoDB models calls from async-operations api endpoints
- Remove all dynamoDB updates for
- CUMULUS-2801
- Move
getFilesExistingAtLocation
from api granules model to api/lib, update granules put
endpoint to remove model references
- Move
- CUMULUS-2804
- Updates api/lib/granule-delete.deleteGranuleAndFiles:
- Updates dynamoGranule -> apiGranule in the signature and throughout the dependent code
- Updates logic to make apiGranule optional, but pgGranule required, and
all lookups use postgres instead of ES/implied apiGranule values - Updates logic to make pgGranule optional - in this case the logic removes the entry from ES only
- Removes all dynamo model logic from api/endpoints/granules
- Removes dynamo write logic from api/lib/writeRecords.*
- Removes dynamo write logic from api/lib/ingest.*
- Removes all granule model calls from api/lambdas/bulk-operations and any dependencies
- Removes dynamo model calls from api/lib/granule-remove-from-cmr.unpublishGranule
- Removes Post Deployment execution check from sf-event-sqs-to-db-records
- Moves describeGranuleExecution from api granule model to api/lib/executions.js
- Updates api/lib/granule-delete.deleteGranuleAndFiles:
- CUMULUS-2806
- Remove DynamoDB logic from executions
POST
endpoint - Remove DynamoDB logic from sf-event-sqs-to-db-records lambda execution writes.
- Remove DynamoDB logic from executions
PUT
endpoint
- Remove DynamoDB logic from executions
- CUMULUS-2808
- Remove DynamoDB logic from executions
DELETE
endpoint
- Remove DynamoDB logic from executions
- CUMULUS-2809
- Remove DynamoDB logic from providers
PUT
endpoint - Updates DB models asyncOperation, provider and rule to return all fields on upsert.
- Remove DynamoDB logic from providers
- CUMULUS-2810
- Removes addition of DynamoDB record from API endpoint POST /provider/
- CUMULUS-2811
- Removes deletion of DynamoDB record from API endpoint DELETE /provider/
- CUMULUS-2817
- Removes deletion of DynamoDB record from API endpoint DELETE /collection//
- CUMULUS-2814
- Move event resources deletion logic from
rulesModel
torulesHelper
- Move event resources deletion logic from
- CUMULUS-2815
- Move File Config and Core Config validation logic for Postgres Collections from
api/models/collections.js
toapi/lib/utils.js
- Move File Config and Core Config validation logic for Postgres Collections from
- CUMULUS-2813
- Removes creation and deletion of DynamoDB record from API endpoint POST /rules/
- CUMULUS-2816
- Removes addition of DynamoDB record from API endpoint POST /collections
- CUMULUS-2797
- Move rule helper functions to separate rulesHelpers file
- CUMULUS-2821
- Remove DynamoDB logic from
sfEventSqsToDbRecords
lambda
- Remove DynamoDB logic from
- CUMULUS-2856
- Update API/Message write logic to handle nulls as deletion in execution PUT/message write logic
- CUMULUS-2645
Added
- CUMULUS-2312 - RDS Migration Epic Phase 3
- CUMULUS-2813
- Added function
create
in thedb
model for Rules
to return an array of objects containing all columns of the created record.
- Added function
- CUMULUS-2812
- Move event resources logic from
rulesModel
torulesHelper
- Move event resources logic from
- CUMULUS-2820
- Remove deletion of DynamoDB record from API endpoint DELETE /pdr/
- CUMULUS-2688
- Add new endpoint to fetch granules by collectionId as well as granuleId: GET /collectionId/granuleId
- Add new endpoints to update and delete granules by collectionId as well as
granuleId
- CUMULUS-2813
Removed
- CUMULUS-2994
- Delete code/lambdas that publish DynamoDB stream events to SNS
- CUMULUS-3226
- Removed Dynamo Async Operations table
- CUMULUS-3199
- Removed DbIndexer lambda and all associated terraform resources
- CUMULUS-3009
- Removed Dynamo PDRs table
- CUMULUS-3008
- Removed DynamoDB Collections table
- CUMULUS-2815
- Remove update of DynamoDB record from API endpoint PUT /collections//
- CUMULUS-2814
- Remove DynamoDB logic from rules
DELETE
endpoint
- Remove DynamoDB logic from rules
- CUMULUS-2812
- Remove DynamoDB logic from rules
PUT
endpoint
- Remove DynamoDB logic from rules
- CUMULUS-2798
- Removed AsyncOperations model
- CUMULUS-2797
- CUMULUS-2795
- Removed API executions model
- CUMULUS-2796
- Remove API pdrs model and all related test code
- Remove API Rules model and all related test code
- CUMULUS-2794
- Remove API Collections model and all related test code
- Remove lambdas/postgres-migration-count-tool, api/endpoints/migrationCounts and api-client/migrationCounts
- Remove lambdas/data-migration1 tool
- Remove lambdas/data-migration2 and
lambdas/postgres-migration-async-operation
- CUMULUS-2793
- Removed Provider Dynamo model and related test code
- CUMULUS-2792
- Remove API Granule model and all related test code
- Remove granule-csv endpoint
- CUMULUS-2645
- Removed dynamo structural migrations and related code from
@cumulus/api
- Removed
executeMigrations
lambda - Removed
granuleFilesCacheUpdater
lambda - Removed dynamo files table from
data-persistence
module. This table and
all of its data will be removed on deployment.
- Removed dynamo structural migrations and related code from
Added
- CUMULUS-3072
- Added
replaceGranule
to@cumulus/api-client/granules
to add usage of the
updated RESTful PUT logic
- Added
- CUMULUS-3121
- Added a map of variables for the cloud_watch_log retention_in_days for the various cloudwatch_log_groups, as opposed to keeping them hardcoded at 30 days. Can be configured by adding the _<cloudwatch_log_group_name>_log_retention value in days to the cloudwatch_log_retention_groups map variable
- CUMULUS-3201
- Added support for sha512 as checksumType for LZARDs backup task.
Changed
- CUMULUS-3315
- Updated
@cumulus/api-client/granules.bulkOperation
to removeids
parameter in favor ofgranules
parameter, in the form of a
@cumulus/types/ApiGranule
that requires the following keys:[granuleId, collectionId]
- Updated
- CUMULUS-3307
- Pinned cumulus dependency on
pg
tov8.10.x
- Pinned cumulus dependency on
- CUMULUS-3279
- Updated core dependencies on
xml2js
tov0.5.0
- Forcibly updated downstream dependency for
xml2js
insaml2-js
to
v0.5.0
- Added audit-ci CVE override until July 1 to allow for Core package releases
- Updated core dependencies on
- CUMULUS-3106
- Updated localstack version to 1.4.0 and removed 'skip' from all skipped tests
- CUMULUS-3115
- Fixed DiscoverGranules' workflow's duplicateHandling when set to
skip
orerror
to stop retrying
after receiving a 404 Not Found Response Error from thecumulus-api
.
- Fixed DiscoverGranules' workflow's duplicateHandling when set to
- CUMULUS-3165
- Update example/cumulus-tf/orca.tf to use orca v6...
v16.1.0-alpha.0
Targeted alpha release from jk/terraform-provider-with-rel-16 for partner use. NON-PRODUCTION RELEASE.
v15.0.4
Changed
- CUMULUS-3307
- Pinned cumulus dependency on
pg
tov8.10.x
- Pinned cumulus dependency on
Fixed
- CUMULUS-3115
- Fixed DiscoverGranules' workflow's duplicateHandling when set to
skip
orerror
to stop retrying
after receiving a 404 Not Found Response Error from thecumulus-api
.
- Fixed DiscoverGranules' workflow's duplicateHandling when set to
- CUMULUS-3315
- Update CI scripts to use shell logic/GNU timeout to bound test timeouts
instead of NPMparallel
package, as timeouts were not resulting in
integration test failure
- Update CI scripts to use shell logic/GNU timeout to bound test timeouts
- CUMULUS-3223
- Update
@cumulus/cmrjs/cmr-utils.getGranuleTemporalInfo
to handle the error when the cmr file s3url is not available - Update
sfEventSqsToDbRecords
lambda to return partial batch failure,
and only reprocess messages when cumulus message can't be retrieved from the execution events. - Update
@cumulus/cumulus-message-adapter-js
to2.0.5
for all cumulus tasks
- Update
v15.0.3
v15.0.2
Release v15.0.2
Fixed
- CUMULUS-3120
- Fixed a bug by adding in
default_log_retention_periods
andcloudwatch_log_retention_periods
to Cumulus modules so they can be used during deployment for configuring cloudwatch retention periods, for more information check here: retention document - Updated cloudwatch retention documentation to reflect the bugfix changes
- Fixed a bug by adding in
v15.0.1
Changed
- CUMULUS-3279
- Updated core dependencies on
xml2js
tov0.5.0
- Forcibly updated downstream dependency for
xml2js
insaml2-js
to
v0.5.0
- Added audit-ci CVE override until July 1 to allow for Core package releases
- Updated core dependencies on
Fixed
- CUMULUS-3285
- Updated
api/lib/distribution.js isAuthBearTokenRequest
to handle non-Bearer authorization header
- Updated
v15.0.0
Breaking Changes
- CUMULUS-3147
- The minimum supported version for all published Cumulus Core npm packages is now Node 16.19.0
- Tasks using the
cumuluss/cumulus-ecs-task
Docker image must be updated tocumuluss/cumulus-ecs-task:1.9.0.
which is built with node:16.19.0-alpine. This can be done by updating theimage
property of any tasks defined using thecumulus_ecs_service
Terraform module. - Updated Dockerfile of async operation docker image to build from node:16.19.0-buster
- Published new tag
44
ofcumuluss/async-operation
to Docker Hub. - The
async_operation_image
property ofcumulus
module must be updated to pull the ECR image forcumuluss/async-operation:44
.
Changed
- CUMULUS-2997
- Migrate Cumulus Docs to Docusaurus v2 and DocSearch v3.
- CUMULUS-3147
- Set example/cumulus-tf default async_operation_image_version to 44.
- Set example/cumulus-tf default ecs_task_image_version to 1.9.0.
- CUMULUS-3166
- Updated example/cumulus-tf/thin_egress_app.tf to use tea 1.3.2
Fixed
- CUMULUS-3187
- Restructured Earthdata Login class to be individual methods as opposed to a Class Object
- Removed typescript no-checks and reformatted EarthdataLogin code to be more type friendly
v14.1.0
MIGRATION notes
From this release forward Core will be tested against PostgreSQL 11 Existing
release compatibility testing was done for release 11.1.8/14.0.0+. Users
should migrate their datastores to Aurora PostgreSQL 11.13+ compatible data stores
as soon as possible.
Users utilizing the cumulus-rds-tf
module will have upgraded/had their
database clusters forcibly upgraded at the next maintenance window after 31 Jan
2023. Our guidance to mitigate this issue is to do a manual (outside of
terraform) upgrade. This will result in the cluster being upgraded with a
manually set parameter group not managed by terraform.
If you manually upgraded and the cluster is now on version 11.13, to continue
using the cumulus-rds-tf
module once upgraded update following module
configuration values if set, or allow their defaults to be utilized:
parameter_group_family = "aurora-postgresql11"
engine_version = 11.13
When you apply this update, the original PostgreSQL v10 parameter group will be
removed, and recreated using PG11 defaults/configured terraform values and
update the database cluster to use the new configuration.
- CUMULUS-3121
- Added a map of variables for the cloud_watch_log retention_in_days for the various cloudwatch_log_groups, as opposed to keeping them hardcoded at 30 days. Can be configured by adding the _<cloudwatch_log_group_name>_log_retention value in days to the cloudwatch_log_retention_groups map variable
Added
- CUMULUS-3193
- Add a Python version file
- CUMULUS-3121
- Added a map of variables for the cloud_watch_log retention_in_days for the various cloudwatch_log_groups, as opposed to keeping them hardcoded at 30 days. Can be configured by adding the _<cloudwatch_log_group_name>_log_retention value in days to the cloudwatch_log_retention_groups map variable
- CUMULUS-3071
- Added 'PATCH' granules endpoint as an exact duplicate of the existing
PUT
endpoint. In future releases thePUT
endpoint will be replaced with valid PUT logic
behavior (complete overwrite) in a future release. The existing PUT
implementation is deprecated and users should move all existing usage of
PUT
toPATCH
before upgrading to a release withCUMULUS-3072
.
- Added 'PATCH' granules endpoint as an exact duplicate of the existing
Removed
- Removed a few tests that were disabled 3-4 years ago
Fixed
-
CUMULUS-3033
- Fixed
granuleEsQuery
to properly terminate ifbody.hit.total.value
is 0.
- Fixed
-
CUMULUS-3072
- Fixed issue introduced in CUMULUS-3070 where new granules incorrectly write
a value forfiles
as[]
to elasticsearch instead of undefined in cases
where[]
is specified in the new granule. - Fixed issue introduced in CUMULUS-3070 where DynamoDB granules with a value
files
as[]
when the granule does not have the files value set as
mutable (e.g. in arunning
state) from a framework message write and
files was not previously defined will write[]
instead of leaving the value
undefined.
- Fixed issue introduced in CUMULUS-3070 where new granules incorrectly write
-
The
getLambdaAliases
function has been removed from the@cumulus/integration-tests
package -
The
getLambdaVersions
function has been removed from the@cumulus/integration-tests
package -
CUMULUS-3117
- Update
@cumulus/es-client/indexer.js
to properly handle framework write
constraints for queued granules. Queued writes will now be properly
dropped from elasticsearch writes along with the primary datastore(s) when
write constraints apply
- Update
-
CUMULUS-3134
- Get tests working on M1 Macs
-
CUMULUS-3148:
- Updates cumulus-rds-tf to use defaults for PostgreSQL 11.13
- Update IngestGranuleSuccessSpec as test was dependant on file ordering and
PostgreSQL 11 upgrade exposed dependency on database results in the API return - Update unit test container to utilize PostgreSQL 11.13 container
-
CUMULUS-3149
- Updates the api
/granules/bulkDelete
endpoint to take the
following configuration keys for the bulkDelete:- concurrency - Number of concurrent bulk deletions to process at a time.
Defaults to 10, increasing this value may improve throughput at the cost
of additional database/CMR/etc load. - maxDbConnections - Defaults to
concurrency
, and generally should not be
changed unless troubleshooting performance concerns.
- concurrency - Number of concurrent bulk deletions to process at a time.
- Updates all bulk api endpoints to add knexDebug boolean query parameter to
allow for debugging of database connection issues in the future. Defaults
to false. - Fixed logic defect in bulk deletion logic where an information query was
nested in a transaction call, resulting in transactions holding knex
connection pool connections in a blocking way that would not resolve,
resulting in deletion failures.
- Updates the api
-
CUMULUS-3142
- Fix issue from CUMULUS-3070 where undefined values for status results in
unexpected insertion failure on PATCH.
- Fix issue from CUMULUS-3070 where undefined values for status results in
-
CUMULUS-3181
- Fixed
sqsMessageConsumer
lambda to correctly retrieve ENABLED sqs rules.
- Fixed
-
CUMULUS-3189
- Upgraded
cumulus-process
andcumulus-message-adapter-python
versions to
support pip 23.0
- Upgraded
-
CUMULUS-3196
- Moved
createServer
initialization outside thes3-credentials-endpoint
lambda
handler to reduce file descriptor usage
- Moved
-
README shell snippets better support copying
-
CUMULUS-3111
- Fix issue where if granule update dropped due to write constraints for writeGranuleFromMessage, still possible for granule files to be written
- Fix issue where if granule update is limited to status and timestamp values due to write constraints for writeGranuleFromMessage, Dynamo or ES granules could be out of sync with PG
Changed
- Snyk Security
- Upgraded jsonwebtoken from 8.5.1 to 9.0.0
- CUMULUS-3160: Upgrade knex from 0.95.15 to 2.4.1
- Upgraded got from 11.8.3 to ^11.8.5
- Dependabot Security
- Upgraded the python package dependencies of the example lambdas
- CUMULUS-3043
- Organize & link Getting Started public docs for better user guidance
- Update Getting Started sections with current content
- CUMULUS-3046
- Update 'Deployment' public docs
- Apply grammar, link fixes, and continuity/taxonomy standards
- CUMULUS-3071
- Updated
@cumulus/api-client
packages to usePATCH
protocol for existing
granulePUT
calls, this change should not require user updates for
api-client
users.@cumulus/api-client/granules.updateGranule
@cumulus/api-client/granules.moveGranule
@cumulus/api-client/granules.updateGranule
@cumulus/api-client/granules.reingestGranule
@cumulus/api-client/granules.removeFromCMR
@cumulus/api-client/granules.applyWorkflow
- Updated
- CUMULUS-3097
- Changed
@cumulus/cmr-client
package's token from Echo-Token to Earthdata Login (EDL) token in updateToken method - Updated CMR header and token tests to reflect the Earthdata Login changes
- Changed
- CUMULUS-3144
- Increased the memory of API lambda to 1280MB
- CUMULUS-3140
- Update release note to include cumulus-api release
- CUMULUS-3193
- Update eslint config to better support typing
- Improve linting of TS files
v14.0.0
Breaking Changes
- Cumulus dashboard upgrade is required, dashboard release v12.0.0
- CUMULUS-2915
- API endpoint GET
/executions/status/${executionArn}
returnspresignedS3Url
anddata
- The user (dashboard) must read the
s3SignedURL
anddata
from the return
- API endpoint GET
- CUMULUS-3070/3074
- Updated granule PUT/POST endpoints to no longer respect message write
constraints. Functionally this means that:- Granules with older createdAt values will replace newer ones, instead of
ignoring the write request - Granules that attempt to set a non-complete state (e.g. 'queued' and
'running') will now ignore execution state/state change and always write - Granules being set to non-complete state will update all values passed in,
instead of being restricted to['createdAt', 'updatedAt', 'timestamp', 'status', 'execution']
- Granules with older createdAt values will replace newer ones, instead of
- Updated granule PUT/POST endpoints to no longer respect message write
Note
- If you upgrade from cumulus version prior to v11.1.8, read release note in v11.1.8 about node14.19.1 upgrade.
Added
- CUMULUS-3070
- Remove granules dynamoDb model logic that sets default publish value on record
validation - Update API granule write logic to not set default publish value on record
updates to avoid overwrite (PATCH behavior) - Update API granule write logic to publish to false on record
creation if not specified - Update message granule write logic to set default publish value on record
creation update. - Update granule write logic to set published to default value of
false
if
null
is explicitly set with intention to delete the value. - Removed dataType/version from api granule schema
- Added
@cumulus/api/endpoints/granules
unit to cover duration overwrite
logic for PUT/PATCH endpoint.
- Remove granules dynamoDb model logic that sets default publish value on record
- CUMULUS-3098
- Added task configuration setting named
failTaskWhenFileBackupFail
to the
lzards-backup
task. This setting isfalse
by default, but when set to
true
, task will fail if one of the file backup request fails.
- Added task configuration setting named
Changed
- Updated CI deploy process to utilize the distribution module in the published zip file which
will be run against for the integration tests - CUMULUS-2915
- Updated API endpoint GET
/executions/status/${executionArn}
to return the
pre-signed s3 URL in addition to execution status data
- Updated API endpoint GET
- CUMULUS-3045
- Update GitHub FAQs:
- Add new and refreshed content for previous sections
- Add new dedicated Workflows section
- Update GitHub FAQs:
- CUMULUS-3070
- Updated API granule write logic to no longer require createdAt value in
dynamo/API granule validation. Write-time createdAt defaults will be set in the case
of new API granule writes without the value set, and createdAt will be
overwritten if it already exists. - Refactored granule write logic to allow PATCH behavior on API granule update
such that existing createdAt values will be retained in case of overwrite
across all API granule writes. - Updated granule write code to validate written createdAt is synced between
datastores in cases where granule.createdAt is not provided for a new
granule. - Updated @cumulus/db/translate/granules.translateApiGranuleToPostgresGranuleWithoutNilsRemoved to validate incoming values to ensure values that can't be set to null are not
- Updated @cumulus/db/translate/granules.translateApiGranuleToPostgresGranuleWithoutNilsRemoved to handle null values in incoming ApiGranule
- Updated @cumulus/db/types/granules.PostgresGranule typings to allow for null values
- Added ApiGranuleRecord to @cumulus/api/granule type to represent a written/retrieved from datastore API granule record.
- Update API/Message write logic to handle nulls as deletion in granule PUT/message write logic
- Updated API granule write logic to no longer require createdAt value in
- CUMULUS-3075
- Changed the API endpoint return value for a granule with no files. When a granule has no files, the return value beforehand for
the translatePostgresGranuletoApiGranule, the function which does the translation of a Postgres granule to an API granule, was
undefined, now changed to an empty array. - Existing behavior which relied on the pre-disposed undefined value was changed to instead accept the empty array.
- Standardized tests in order to expect an empty array for a granule with no files files' object instead of undefined.
- Changed the API endpoint return value for a granule with no files. When a granule has no files, the return value beforehand for
- CUMULUS-3077
- Updated
lambdas/data-migration2
granule and files migration to have aremoveExcessFiles
function like in write-granules that will remove file records no longer associated with a granule being migrated
- Updated
- CUMULUS-3080
- Changed the retention period in days from 14 to 30 for CloudWatch logs for NIST-5 compliance
- CUMULUS-3100
- Updated
POST
granules endpoint to check if granuleId exists across all collections rather than a single collection. - Updated
PUT
granules endpoint to check if granuleId exists across a different collection and throw conflict error if so. - Updated logic for writing granules from a message to check if granuleId exists across a different collection and throw conflict error if so.
- Updated
Fixed
- CUMULUS-3070
- Fixed inaccurate typings for PostgresGranule in @cumulus/db/types/granule
- Fixed inaccurate typings for @cumulus/api/granules.ApiGranule and updated to
allow null
- CUMULUS-3104
- Fixed TS compilation error on aws-client package caused by @aws-sdk/client-s3 3.202.0 upgrade
- CUMULUS-3116
- Reverted the default ElasticSearch sorting behavior to the pre-13.3.0 configuration
- Results from ElasticSearch are sorted by default by the
timestamp
field. This means that the order
is not guaranteed if two or more records have identical timestamps as there is no secondary sort/tie-breaker.