Releases: nasa/cumulus
v1.14.2
Migration Instructions
Your Cumulus Message Adapter version should be pinned to v1.0.13
or lower in your app/config.yml
using message_adapter_version: v1.0.13
OR
you should use the workflow migration steps below to work with CMA v1.1.1+.
The newest release of the Cumulus Message Adapter (v1.1.1) requires that parameterized configuration be used for remote message functionality. Once released, Kes will automatically bring in CMA v1.1.1 without additional configuration.
Migration instructions for CMA 1.1.1+
Oversized messages are no longer written to S3 automatically. In order to utilize remote messaging functionality, configure a ReplaceConfig
AWS Step Function parameter on your CMA task:
ParsePdr:
Parameters:
cma:
event.$: '$'
ReplaceConfig:
FullMessage: true
Accepted fields in ReplaceConfig
include MaxSize
, FullMessage
, Path
and TargetPath
.
See https://github.com/nasa/cumulus-message-adapter/blob/master/CONTRACT.md#remote-message-configuration for full details.
As this change is backward compatible in Cumulus Core, users wishing to utilize the previous version of the CMA may opt to transition to using a CMA lambda layer, or set message_adapter_version
in their configuration to a version prior to v1.1.0.
Workflow reporting - CUMULUS-1394
The implementation of the SfSnsReport
Lambda requires additional environment variables for integration with the new ingest notification SNS topics. Therefore, you must update the definition of SfSnsReport
in your lambdas.yml
like so:
SfSnsReport:
handler: index.handler
timeout: 300
source: node_modules/@cumulus/sf-sns-report/dist
tables:
- ExecutionsTable
envs:
execution_sns_topic_arn:
function: Ref
value: reportExecutionsSns
granule_sns_topic_arn:
function: Ref
value: reportGranulesSns
pdr_sns_topic_arn:
function: Ref
value: reportPdrsSns
Notable Changes
-
NASA Launchpad is an option for authentication for CMR, the API, and dashboard. This must be set up and configured. NASA users can find Launchpad configuration documentation here for CMR and here for the API. Please note you will have to get the latest version of the Cumulus dashboard as well.
-
Ingest notifications are now provided via 3 separate SNS topics for executions, granules, and PDRs, instead of a single
sftracker
SNS topic. Whereas thesftracker
SNS topic received a full Cumulus execution message, the new topics all receive generated records for the given object. The new topics are only published to if the given object exists for the current execution. For a given execution/granule/PDR, two messages will be received by each topic: one message indicating that ingest is running and another message indicating that ingest has completed or failed. The new SNS topics are:reportExecutions
- Receives 1 message per executionreportGranules
- Receives 1 message per granule in an executionreportPdrs
- Receives 1 message per PDR
Added
-
CUMULUS-1394
- Added
Granule.generateGranuleRecord()
method to granules model to generate a granule database record from a Cumulus execution message - Added
Pdr.generatePdrRecord()
method to PDRs model to generate a granule database record from a Cumulus execution message - Added helpers to
@cumulus/common/message
:getMessageExecutionName()
- Get the execution name from a Cumulus execution messagegetMessageStateMachineArn()
- Get the state machine ARN from a Cumulus execution messagegetMessageExecutionArn()
- Get the execution ARN for a Cumulus execution messagegetMessageGranules()
- Get the granules from a Cumulus execution message, if any.
- Added
@cumulus/common/cloudwatch-event/isFailedSfStatus()
to determine if a Step Function status from a Cloudwatch event is a failed status
- Added
-
CUMULUS-639
- Adds SAML JWT and launchpad token authentication to Cumulus API (configurable)
- NOTE to authenticate with Launchpad ensure your launchpad user_id is in the
<prefix>-UsersTable
- when Cumulus configured to protect API via Launchpad:
- New endpoints
GET /saml/login
- starting point for SAML SSO creates the login request url and redirects to the SAML Identity Provider Service (IDP)POST /saml/auth
- SAML Assertion Consumer Service. POST receiver from SAML IDP. Validates response, logs the user in, and returnes a SAML-based JWT.
- Disabled endpoints
GET /token
POST /refresh
- Changes authorization worklow:ensureAuthorized
now presumes the bearer token is a JWT and tries to validate. If the token is malformed, it attempts to validate the token against Launchpad. This allows users to bring their own token as described here https://wiki.earthdata.nasa.gov/display/CUMULUS/Cumulus+API+with+Launchpad+Authentication. But it also allows dashboard users to manually authenticate via Launchpad SAML to receive a Launchpad-based JWT.
- New endpoints
- NOTE to authenticate with Launchpad ensure your launchpad user_id is in the
- Adds SAML JWT and launchpad token authentication to Cumulus API (configurable)
Changed
-
CUMULUS-1485 Update
@cumulus/cmr-client
to return error message from CMR for validation failures. -
CUMULUS-1394
- Renamed
Execution.generateDocFromPayload()
toExecution.generateRecord()
on executions model. The method generates an execution database record from a Cumulus execution message.
- Renamed
-
CUMULUS-1432
logs
endpoint takes the level parameter as a string and not a number- Elasticsearch term query generation no longer converts numbers to boolean
-
CUMULUS-1447
- Consolidated all remote message handling code into @common/aws
- Update remote message code to handle updated CMA remote message flags
- Update example SIPS workflows to utilize Parameterized CMA configuration
-
CUMULUS-1448 Refactor workflows that are mutating cumulus_meta to utilize meta field
-
CUMULUS-1375
- Migrate Cumulus from deprecated Elasticsearch JS client to new, supported one in
@cumulus/api
- Migrate Cumulus from deprecated Elasticsearch JS client to new, supported one in
-
CUMULUS-1451
- Elasticsearch cluster setting
auto_create_index
will be set to false. This had been causing issues in the bootstrap lambda on deploy.
- Elasticsearch cluster setting
-
CUMULUS-1456
@cumulus/api
endpoints default error handler usesboom
package to format errors, which is consistent with other API endpoint errors.
Fixed
- CUMULUS-1432
logs
endpoint filter correctly filters logs by level - CUMULUS-1484
useMessageAdapter
now does not set CUMULUS_MESSAGE_ADAPTER_DIR whentrue
Removed
- CUMULUS-1394
- Removed
sfTracker
SNS topic. Replaced by three new SNS topics for granule, execution, and PDR ingest notifications. - Removed unused functions from
@cumulus/common/aws
:getGranuleS3Params()
setGranuleStatus()
- Removed
v1.14.1
Fixed
-
CUMULUS-1455
- CMR token links updated to point to CMR legacy services rather than echo
-
CUMULUS-1211
- Errors thrown during granule discovery are no longer swallowed and ignored.
Rather, errors are propagated to allow for proper error-handling and
meaningful messaging.
- Errors thrown during granule discovery are no longer swallowed and ignored.
v1.13.5
Fixed
- CUMULUS-1455
- CMR token links updated to point to CMR legacy services rather than echo
v1.14.0
Migration Instructions
-
Redeploy your
iam
stack. -
Delete existing API Gateway stages: To allow enabling of API Gateway logging, Cumulus now creates and manages a Stage resource during deployment. Before upgrading Cumulus, it is necessary to delete the API Gateway stages on both the Backend API and the Distribution API. Instructions are included in the documenation under Delete API Gateway Stages.
-
Configure a distribution deployment
The Cumulus distribution API has been deprecated and is being replaced by a Terraform distribution app containing ASF's Thin Egress App + the direct S3 credentials app. By default, the distribution API will not deploy. Please follow the instructions for deploying and configuring the Thin Egress distribution app.
To instead continue to deploy and use the legacy Cumulus distribution app, add the following to your config.yml
:
deployDistributionApi: true
- Update any tasks manually packaging the CMA (required) and configure lambda layers (optional)
Adding a defaultcmaDir
key to configuration will causeCUMULUS_MESSAGE_ADAPTER_DIR
to be set by default to/opt
for any Lambda not settinguseCma
to true, or explicitly setting the CMA environment variable. In lambdas that package the CMA independently of the Cumulus packaging. Lambdas manually packaging the CMA should have their Lambda configuration updated to set the CMA path, or alternately if not using the CMA as a Lambda layer in this deployment setcmaDir
to./cumulus-message-adapter
.
To create a CMA as a lambda layer, you can do it through the console or using this Terraform module. CMA releases
In lambdas.yml
, the configuration looks like this:
layers:
- '{{cma_layer}}'
where cma_layer
is the ARN of the lambda layer.
Real life example. Note there is currently an issue using this with the hello world lambda.
-
Configure metrics logging (optional)
See steps below. -
Configure CMR Launchpad authentication (optional)
-
Configure API Launchpad authentication (optional)
-
Configure EMS submission (optional)
Follow the instructions here and populate theems
parameters in yourapp/config.yml
as described here
Configure metrics logging
-
Set up account permissions for API Gateway to write to CloudWatch: In a one time operation for your AWS account, to enable CloudWatch Logs for API Gateway, you must first grant the API Gateway permission to read and write logs to CloudWatch for your account. The
AmazonAPIGatewayPushToCloudWatchLogs
managed policy (with an ARN ofarn:aws:iam::aws:policy/service-role/AmazonAPIGatewayPushToCloudWatchLogs
) has all the required permissions. You can find a simple how to in the documentation under Enable API Gateway Logging. -
Configure API Gateway to write logs to CloudWatch To enable execution logging for the distribution API set
config.yaml
apiConfigs.distribution.logApigatewayToCloudwatch
value totrue
. More information Enable API Gateway Logs -
Configure CloudWatch log delivery: It is possible to deliver CloudWatch API execution and access logs to a cross-account shared AWS::Logs::Destination. An operator does this by adding the key
logToSharedDestination
to theconfig.yml
at the default level with a value of a writable log destination. More information in the documenation under Configure CloudWatch Logs Delivery. -
Additional Lambda Logging: It is now possible to configure any lambda to deliver logs to a shared subscriptions by setting
logToSharedDestination
to the ARN of a writable location (either an AWS::Logs::Destination or a Kinesis Stream) on any lambda config. Documentation for Lambda Log Subscriptions -
Configure S3 Server Access Logs: If you are running Cumulus in an NGAP environment you may configure S3 Server Access Logs to be delivered to a shared bucket where the Metrics Team will ingest the logs into their ELK stack. Contact the Metrics team for permission and location.
-
Configure dashboard to show metrics Dashboard release 1.5.0 contains updates to add distribution metrics to the dashboard home page. Use the release notes to configure your dashboard to display metrics.
Breaking Changes
-
Cumulus distribution will not deploy by default. Please see migration instructions above.
-
For tasks manually injecting the CMA:
Adding a defaultcmaDir
key to configuration will causeCUMULUS_MESSAGE_ADAPTER_DIR
to be set by default to/opt
for any Lambda not settinguseCma
to true, or explicitly setting the CMA environment variable. In lambdas that package the CMA independently of the Cumulus packaging. Lambdas manually packaging the CMA should have their Lambda configuration updated to set the CMA path, or alternately if not using the CMA as a Lambda layer in this deployment setcmaDir
to./cumulus-message-adapter
. -
To allow enabling of API Gateway logging, Cumulus now creates and manages a Stage resource during deployment. Before upgrading Cumulus, it is necessary to delete the API Gateway stages on both the Backend API and the Distribution API. Instructions are included in the documenation under Delete API Gateway Stages.
Important Notes
- We have encountered transient lambda service errors in our integration testing. Please handle transient service errors following these guidelines. The workflows in the
example/workflows
folder have been updated with retries configured for these errors.
Notable Changes
-
This release completes features for SWOT technical qualifications
-
CMR and Cumulus API Launchpad authentication is implemented in this release
-
Added support for limiting the number of concurrent executions started from a queue. See the data cookbook for more information.
-
Support for lambda layers added to lambdas and for using the CMA as a lambda layer
-
Added
cmrGranuleUrlType
to the@cumulus/move-granules
task. This determines what kind of links go in the CMR files. The options aredistribution
,s3
, ornone
, with the default being distribution. If there is no distribution API being used with Cumulus, you must set the value tos3
ornone
.
Full Changelog
PLEASE NOTE
-
CUMULUS-799 added additional IAM permissions to support reading CloudWatch and API Gateway, so you will have to redeploy your IAM stack.
-
CUMULUS-800 Several items:
-
Delete existing API Gateway stages: To allow enabling of API Gateway logging, Cumulus now creates and manages a Stage resource during deployment. Before upgrading Cumulus, it is necessary to delete the API Gateway stages on both the Backend API and the Distribution API. Instructions are included in the documentation under Delete API Gateway Stages.
-
Set up account permissions for API Gateway to write to CloudWatch: In a one time operation for your AWS account, to enable CloudWatch Logs for API Gateway, you must first grant the API Gateway permission to read and write logs to CloudWatch for your account. The
AmazonAPIGatewayPushToCloudWatchLogs
managed policy (with an ARN ofarn:aws:iam::aws:policy/service-role/AmazonAPIGatewayPushToCloudWatchLogs
) has all the required permissions. You can find a simple how to in the documentation under Enable API Gateway Logging. -
Configure API Gateway to write logs to CloudWatch To enable execution logging for the distribution API set
config.yaml
apiConfigs.distribution.logApigatewayToCloudwatch
value totrue
. More information Enable API Gateway Logs -
Configure CloudWatch log delivery: It is possible to deliver CloudWatch API execution and access logs to a cross-account shared AWS::Logs::Destination. An operator does this by adding the key
logToSharedDestination
to theconfig.yml
at the default level with a value of a writable log destination. More information in the documenation under [Configure CloudWatch Logs Delivery.](https://nasa.github.io/cumulus/docs/additional-deployment-op...
-
v1.13.4
Note: Please skip versions 1.13.1
-1.13.3
and upgrade directly to 1.13.4
Please configure the following in your app/config.yml
:
cmr:
cmrEnvironment: UAT
cmrLimit: 100
cmrPageSize: 50
Fixed
- CUMULUS-1374 - Resolve audit compliance with lodash version for api package subdependency
- CUMULUS-1412 - Resolve audit compliance with googleapi package
- CUMULUS-1345 Full backport of CUMULUS-1345 features
- Adds new variables to the app deployment undercmr
.cmrEnvironment
values areSIT
,UAT
, orOPS
withUAT
as the default.cmrLimit
andcmrPageSize
have been added as configurable options.
- CUMULUS-1411 - Fix deployment issue when using a template override
v1.13.1
Due to npm publishing issues, please skip this release and upgrade directly to 1.13.4
v1.13.0
Notable Changes
- Separate DB stack for the data layer (Dynamo and Elasticsearch)
- All stacks use the
app/config.yml
- ECS Autoscaling
- Default AMI is NASA NGAP AMI
- Elasticsearch is deployed behind the VPC
- Support for configuring private API Gateways
Migration Steps
- Read through the notes below, particularly noting the breaking changes
- Delete your Cumulus
app
stack in Cloudformation.- If running Cumulus within a VPC and extended downtime is acceptable, we recommend doing this at the end of the day to allow AWS backend resources and network interfaces to be cleaned up overnight.
- Setup a Service Linked Role for Elasticsearch
- This is a one-time setup for the account. You can follow the instructions to use the AWS console or AWS CLI here or use the following AWS CLI command:
aws iam create-service-linked-role --aws-service-name es.amazonaws.com
- This is a one-time setup for the account. You can follow the instructions to use the AWS console or AWS CLI here or use the following AWS CLI command:
- Delete your
iam/config.yml
and move your configurations toapp/config.yml
See notes below for CUMULUS-1310. - Redeploy your
iam
stack - Configure your desired security group using an existing security group (In an NGAP environment, this would be the default application security group) in your
app/config.yml
, populatevpc.securityGroup
- Configure your API to be private if desired using the
apiConfigs
configuration in yourapp/config.yml
- This is needed for NGAP compliance. Please read this wiki page before deploying to make sure you have the prerequisites. This can be done at a later time.
- If not in an NGAP environment, configure your
ecs.amiid
in yourapp/config.yml
to point to an AWS AMI. - Deploy your
db
stack - Deploy your
app
stack
PLEASE NOTE
CUMULUS-802 added some additional IAM permissions to support ECS autoscaling and changes were needed to run all lambdas in the VPC, so you will have to redeploy your IAM stack.
As a result of the changes for CUMULUS-1193, CUMULUS-1264, and CUMULUS-1310, you must delete your existing stacks (except IAM) before deploying this version of Cumulus.
If running Cumulus within a VPC and extended downtime is acceptable, we recommend doing this at the end of the day to allow AWS backend resources and network interfaces to be cleaned up overnight.
BREAKING CHANGES
-
CUMULUS-1228
- The default AMI used by ECS instances is now an NGAP-compliant AMI. This
will be a breaking change for non-NGAP deployments. If you do not deploy to
NGAP, you will need to find the AMI ID of the
most recent Amazon ECS-optimized AMI,
and set theecs.amiid
property in your config. Instructions for finding
the most recent NGAP AMI can be found using
these instructions.
- The default AMI used by ECS instances is now an NGAP-compliant AMI. This
-
CUMULUS-1310
- Database resources (DynamoDB, ElasticSearch) have been moved to an independent
db
stack.
Migrations for this version will need to be user-managed. (e.g. elasticsearch and dynamoDB).
Order of stack deployment isiam
->db
->app
. - All stacks can now be deployed using a single
config.yml
file, i.e.:kes cf deploy --kes-folder app --template node_modules/@cumulus/deployment/[iam|db|app] [...]
Backwards-compatible. For development, please re-runnpm run bootstrap
to build newkes
overrides.
Deployment docs have been updated to show how to deploy a single-config Cumulus instance. params
have been moved: Nestparams
fields underapp
,db
oriam
to override all Parameters for a particular stack's cloudformation template. Backwards-compatible with multi-config setups.stackName
andstackNameNoDash
have been retired. Useprefix
andprefixNoDash
instead.- The
iams
section inapp/config.yml
IAM roles has been deprecated as a user-facing parameter,
unless your IAM role ARNs do not match the convention shown in@cumulus/deployment/app/config.yml
- The
vpc.securityGroup
will need to be set with a pre-existing security group ID to use Cumulus in a VPC. Must allow inbound HTTP(S) (Port 443).
- Database resources (DynamoDB, ElasticSearch) have been moved to an independent
-
CUMULUS-1212
@cumulus/post-to-cmr
will now fail if any granules being processed are missing a metadata file. You can set the new config optionskipMetaCheck
totrue
to pass post-to-cmr without a metadata file.
-
CUMULUS-1232
@cumulus/sync-granule
will no longer silently pass if no checksum data is provided. It will use input
from the granule object to:- Verify checksum if
checksumType
andchecksumValue
are in the file record OR a checksum file is provided
(throwsInvalidChecksum
on fail), else log warning that no checksum is available. - Then, verify synced S3 file size if
file.size
is in the file record (throwsUnexpectedFileSize
on fail),
else log warning that no file size is available. - Pass the step.
- Verify checksum if
-
CUMULUS-1264
- The Cloudformation templating and deployment configuration has been substantially refactored.
CumulusApiDefault
nested stack resource has been renamed toCumulusApiDistribution
CumulusApiV1
nested stack resource has been renamed toCumulusApiBackend
- The
urs: true
config option for when defining your lambdas (e.g. inlambdas.yml
) has been deprecated. There are two new options to replace it:urs_redirect: 'token'
: This will expose aTOKEN_REDIRECT_ENDPOINT
environment variable to your lambda that references the/token
endpoint on the Cumulus backend APIurs_redirect: 'distribution'
: This will expose aDISTRIBUTION_REDIRECT_ENDPOINT
environment variable to your lambda that references the/redirect
endpoint on the Cumulus distribution API
- The Cloudformation templating and deployment configuration has been substantially refactored.
-
CUMULUS-1193
- The elasticsearch instance is moved behind the VPC.
- Your account will need an Elasticsearch Service Linked role. This is a one-time setup for the account. You can follow the instructions to use the AWS console or AWS CLI here or use the following AWS CLI command:
aws iam create-service-linked-role --aws-service-name es.amazonaws.com
-
CUMULUS-802
- ECS
maxInstances
must be greater thanminInstances
. If you use defaults, no change is required.
- ECS
-
CUMULUS-1269
- Brought Cumulus data models in line with CNM JSON schema:
- Renamed file object
fileType
field totype
- Renamed file object
fileSize
field tosize
- Renamed file object
checksumValue
field tochecksum
where not already done. - Added
ancillary
andlinkage
type support to file objects.
- Renamed file object
- Brought Cumulus data models in line with CNM JSON schema:
Added
-
CUMULUS-1242 - Added
sqs2sfThrottle
lambda. The lambda reads SQS messages for queued executions and uses semaphores to only start new executions if the maximum number of executions defined for the priority key (cumulus_meta.priorityKey
) has not been reached. Any SQS messages that are read but not used to start executions remain in the queue. -
CUMULUS-1240
- Added
sfSemaphoreDown
lambda. This lambda receives SNS messages and for each message it decrements the semaphore used to track the number of running executions if:- the message is for a completed/failed workflow AND
- the message contains a level of priority (
cumulus_meta.priorityKey
)
- Added
sfSemaphoreDown
lambda as a subscriber to thesfTracker
SNS topic
- Added
-
CUMULUS-1265
- Added
apiConfigs
configuration option to configure API Gateway to be private - All internal lambdas configured to run inside the VPC by default
- Removed references to
NoVpc
lambdas from documentation andexample
folder.
- Added
-
CUMULUS-802
- Adds autoscaling of ECS clusters
- Adds autoscaling of ECS services that are handling StepFunction activities
- See ECS deployment configuration options
Changed
-
Updated
@cumulus/ingest/http/httpMixin.list()
to trim trailing spaces on discovered filenames -
CUMULUS-1310
- Database resources (DynamoDB, ElasticSearch) have been moved to an independent
db
stack.
This will enable future updates to avoid affecting database resources or requiring migrations.
Migrations for this version will need to be user-managed.
(e.g. elasticsearch and dynamoDB).
Order of stack deployment isiam
->db
->app
. - All stacks can now be deployed using a single
config.yml
file, i.e.:kes cf deploy --kes-folder app --template node_modules/@cumulus/deployment/[iam|db|app] [...]
Backwards-compatible. Please re-runnpm run bootstrap
to build newkes
overrides.
Deployment docs have been updated to show how to deploy a single-config Cumulus instance. params
fields should now be nested under the stack key (i.e.app
,db
oriam
) to provide Parameters for a particular stack's cloudformation template,
for use with single-config instances. Keys must match the name of the deployment package folder (app
,db
, or `iam...
- Database resources (DynamoDB, ElasticSearch) have been moved to an independent
v1.12.1
This release includes all of SWOT Compatibility test and the following new features:
- CMR Ingest of UMM-G (all versions)
- Correct categorization of browse, QA, and metadata files in CMR metadata
s3Credentials
endpoint on the distribution API for in-region direct S3 access (NGAP AWS account required)- Reconciliation reports include differences between CMR and Cumulus
- CloudWatch alarms for Elasticsearch and ECS, dashboard for both. Found in CloudWatch --> Dashboards in your AWS console.
Please note breaking changes below and migrate accordingly.
BREAKING CHANGES
-
CUMULUS-1171
- Fixed provider handling in the API to make it consistent between protocols.
NOTE: This is a breaking change. When applying this upgrade, users will need to:- Disable all workflow rules
- Update any
http
orhttps
providers so that the host field only
contains a valid hostname or IP address, and the port field contains the
provider port. - Perform the deployment
- Re-enable workflow rules
- Fixed provider handling in the API to make it consistent between protocols.
-
CUMULUS-1176:
@cumulus/move-granules
input expectations have changed.@cumulus/files-to-granules
is a new intermediate task to perform input translation in the old style.
See the Added and Changed sections of this release changelog for more information.
Example use in workflow here
-
CUMULUS-670
- The behavior of ParsePDR and related code has changed in this release. PDRs with FILE_TYPEs that do not conform to the PDR ICD (+ TGZ) (https://cdn.earthdata.nasa.gov/conduit/upload/6376/ESDS-RFC-030v1.0.pdf) will fail to parse.
-
CUMULUS-1139
granule.applyWorkflow
uses the new-style granule record as input to workflows. The granule and file schemas can be found in schemas.js
-
CUMULUS-1208
- The granule object input to
@cumulus/queue-granules
will now be added to ingest workflow messages as is. In practice, this means that if you are using@cumulus/queue-granules
to trigger ingest workflows and your granule objects input have invalid properties, then your ingest workflows will fail due to schema validation errors.
- The granule object input to
-
Specify
AWS_REGION
in.env
to be used by deployment script and make sureAWS_ACCOUNT_ID
is specified in there as well.
Added
-
CUMULUS-805
- Added a CloudWatch alarm to check running ElasticSearch instances, and a CloudWatch dashboard to view the health of ElasticSearch
- Specify
AWS_REGION
in.env
to be used by deployment script
-
CUMULUS-803
- Added CloudWatch alarms to check running tasks of each ECS service, and add the alarms to CloudWatch dashboard
-
CUMULUS-670
- Added Ancillary Metadata Export feature (see https://nasa.github.io/cumulus/docs/features/ancillary_metadata for more information)
- Added new Collection file parameter "fileType" that allows configuration of workflow granule file fileType
-
CUMULUS-1184 - Added kes logging output to ensure we always see the state machine reference before failures due to configuration
-
CUMULUS-1105 - Added a dashboard endpoint to serve the dashboard from an S3 bucket
-
CUMULUS-666
- Added
@api/endpoints/s3credentials
to allow EarthData Login authorized users to retrieve temporary security credentials for same-region direct S3 access. - CUMULUS-1199 - Moves
s3credentials
endpoint from the backend to the distribution API.
- Added
-
CUMULUS-671
- Added
@packages/integration-tests/api/distribution/getDistributionApiS3SignedUrl()
to return the S3 signed URL for a file protected by the distribution API
- Added
-
CUMULUS-672
- Added
cmrMetadataFormat
andcmrConceptId
to output for individual granules from@cumulus/post-to-cmr
.cmrMetadataFormat
will be read from thecmrMetadataFormat
generated for each granule in@cumulus/cmrjs/publish2CMR()
- Added helpers to
@packages/integration-tests/api/distribution
:getDistributionApiFileStream()
returns a stream to download files protected by the distribution APIgetDistributionFileUrl()
constructs URLs for requesting files from the distribution API
- Added
-
CUMULUS-1185
@cumulus/api/models/Granule.removeGranuleFromCmrByGranule
to replace@cumulus/api/models/Granule.removeGranuleFromCmr
and use the Granule UR from the CMR metadata to remove the granule from CMR -
CUMULUS-1101
- Added new
@cumulus/checksum
package. This package provides functions to calculate and validate checksums. - Added new checksumming functions to
@cumulus/common/aws
:calculateS3ObjectChecksum
andvalidateS3ObjectChecksum
, which depend on thechecksum
package.
- Added new
-
CUMULUS-1171
- Added
@cumulus/common
API documentation topackages/common/docs/API.md
- Added an
npm run build-docs
task to@cumulus/common
- Added
@cumulus/common/string#isValidHostname()
- Added
@cumulus/common/string#match()
- Added
@cumulus/common/string#matches()
- Added
@cumulus/common/string#toLower()
- Added
@cumulus/common/string#toUpper()
- Added
@cumulus/common/URLUtils#buildURL()
- Added
@cumulus/common/util#isNil()
- Added
@cumulus/common/util#isNull()
- Added
@cumulus/common/util#isUndefined()
- Added
@cumulus/common/util#negate()
- Added
-
CUMULUS-1176
- Added new
@cumulus/files-to-granules
task to handle converting file array output fromcumulus-process
tasks into granule objects.
Allows simplification of@cumulus/move-granules
and@cumulus/post-to-cmr
, see Changed section for more details.
- Added new
-
CUMULUS-1151
- Compare the granule holdings in CMR with Cumulus' internal data store
-
CUMULUS-1152
- Compare the granule file holdings in CMR with Cumulus' internal data store
-
CUMULUS-777
- Added new cookbook entry on configuring Cumulus to track ancillary files.
-
CUMULUS-1223
- Adds convenience function
@cumulus/common/bucketsConfigJsonObject
for fetching stack's bucket configuration as an object .
- Adds convenience function
-
CUMULUS-853
- Updated FakeProcessing example lambda to include option to generate fake browse
- Added feature documentation for ancillary metadata export, a new cookbook entry describing a workflow with ancillary metadata generation(browse), and related task definition documentation
Changed
- CUMULUS-1208 - Updated
@cumulus/ingest/queue/enqueueGranuleIngestMessage()
to not transform granule object passed to it when building an ingest message - CUMULUS-1198 -
@cumulus/ingest
no longer enforces any expectations about whetherprovider_path
contains a leading slash or not. - CUMULUS-1170
- Update scripts and docs to use
npm
instead ofyarn
- Use
package-lock.json
files to ensure matching versions of npm packages - Update CI builds to use
npm ci
instead ofnpm install
- Update scripts and docs to use
- CUMULUS-670
- Updated ParsePDR task to read standard PDR types+ (+ tgz as an external customer requirement) and add a fileType to granule-files on Granule discovery
- Updated ParsePDR to fail if unrecognized type is used
- Updated all relevant task schemas to include granule->files->filetype as a string value
- Updated tests/test fixtures to include the fileType in the step function/task inputs and output validations as needed
- Updated MoveGranules task to handle incoming configuration with new "fileType" values and to add them as appropriate to the lambda output.
- Updated DiscoverGranules step/related workflows to read new Collection file parameter fileType that will map a discovered file to a workflow fileType
- Updated CNM parser to add the fileType to the defined granule file fileType on ingest and updated integration tests to verify/validate that behavior
- Updated generateEcho10XMLString in cmr-utils.js to use a map/related library to ensure order as CMR requires ordering for their online resources.
- Updated post-to-cmr task to appropriately export CNM filetypes to CMR in echo10/UMM exports
- CUMULUS-1139 - Granules stored in the API contain a
files
property. That schema has been greatly
simplified and now better matches the CNM format.- The
name
property has been renamed tofileName
. - The
filepath
property has been renamed tokey
. - The
checksumValue
property has been renamed tochecksum
. - The
path
property has been removed. - The
url_path
property has been removed. - The
filename
property (which contained ans3://
URL) has been removed, and thebucket
andkey
properties should be used instead. Any requests sent to the API containing agranule.files[].filename
property will be rejected, and any responses coming back from the API will not contain that
filename
property. - A
source
property has been added, which is a URL indicating the original source of the file. @cumulus/ingest/granule.moveGranuleFiles()
no longer includes afilename
field in its
output. Thebucket
andkey
fields should be used instead.
- The
- CUMULUS-672
- Changed
@cumulus/integration-tests/api/EarthdataLogin.getEarthdataLoginRedirectResponse
to@cumulus/integration-tests/api/EarthdataLogin.getEarthdataAccessToken
. The new function returns an access response from Earthdata login, if successful. @cumulus/integration-tests/cmr/getOnlineResources
now accepts an object of options, includingcmrMetadataFormat
. Based on thecmrMetadataFormat
, the function will correctly retrieve the online resources for each metadata format (ECHO10, UMM-G)
- Changed
- CUMULUS-1183
...
v1.12.0
There was an issue with the publish of this release. Please upgrade to 1.12.1.
v1.11.3
Added
- CUMULUS-1187 - Added
@cumulus/ingest/granule/duplicateHandlingType()
to determine how duplicate files should be handled in an ingest workflow
Fixed
- CUMULUS-1187 - workflows not respecting the duplicate handling value specified in the collection
- Removed refreshToken schema requirement for OAuth