Modifying Existing Resources - Adding APIs, Event Sources etc

[sanathkr]

Parent tracking Issue for a large class of questions - "Can SAM modify existing resources?". More specifically:

• Can SAM add API path to an existing API Gateway resource? (Use existing API Gateway API when adding Lambda Api Trigger #232)
• Can SAM add trigger to an existing S3 bucket? (How to reference an existing s3 bucket & prefix #124)
• Can SAM add trigger to existing IoT Topic Rule? (Allow Naming IoTRule Event for better management or point to existing rule #241)

The answer for most of them is NO. SAM templates are deployed using AWS CloudFormation which does not support modifying existing resources created "outside" of this CloudFormation stack. For example, adding S3 trigger to an existing bucket will require SAM to add a NotificationConfiguration property to the bucket. Since CloudFormation can't modify resources created outside of stack, this is a no go with SAM. If your requirement falls under this umbrella, then it is not possible to support it in SAM today.

Technically speaking, this behavior is out of bounds for native CloudFormation resources, but we can always implement "custom resources" which do this for you. Custom Resources are Lambda functions that can make relevant API calls without being limited by CloudFormation's native constraints.

Keeping this tracking issue open to collect all requirements for modifying existing resources. If this gathers enough steam, SAM can automatically create and maintain these custom resources to do the job for you.

[rmmeans]

I support the idea of SAM going around CloudFormation's native constraints by using custom resources where those are managed by the SAM team instead of leaving it to the developers using SAM to deal with these CloudFormation shortcomings. 👍

On the implementation of this - I have several questions - mostly around who's lambda's those are and how they are invoked. I'm presuming it would be Lambdas that AWS would own, operate, and keep up to date as AWS API's evolve? (e.g. this is not something that is happening at packaging via a specific version of the AWS CLI - it is something where the CloudFormation service automatically creates the custom resource types needed by stack?)

Honestly, many times I have been tempted to write custom resource types, but it feels like a lot of overhead to solve problems for just me when these problems are generally broader problems across a multitude of customers that I wish CloudFormation or SAM would just solve themselves without me knowing how my template is actually turned into resources under the covers.

[rzijp]

@sanathkr, does this limitation also apply to subscribing a lambda to an SNS topic? I'm considering it as a workaround for triggering lambdas by events on externally created S3 buckets.
#105 and your comment seem to suggest it is not possible, but that might also be due to the context of different regions.
And this comment by @sanyer mentions automation by using SNS, but does not mention if this is automated via SAM or other means.

[smurfpandey]

@rzijp This limitation does not apply to SNS, I did a small POC implementation few days back and it worked. You just need to specify the ARN of your SNS topic in the template.yml and it will work. Just make sure the IAM role doing the deployment has permission to subscribe to that(or any) topic.

[triplewy]

If I created an API within a SAM template, can I access it from another SAM template? Technically the API is "inside" the cloudformation stack. Is it possible to do this?

[keetonian]

Yes, you can access it from another template (either by exporting values or referencing outputs from nested stacks). However, you can't use any SAM features this way, so API events or swagger generation and other SAM features don't work across stacks.

[dalumiller]

Edit: I was mistaken. I did not create a new lambda referencing a pre-existing IoT rule. I reused a IoT topic, but now wonder why I can't use pre-existing IoT rules if I can use pre-existing SNS topics and SQS Queues.

@sanathkr, your initial comment in this thread and response to #241 makes it sound like you cannot publish a new SAM template with a new lambda referencing a pre-existing IoT rule. I just did this in my own stack and assume IoT rules and SNS topics behave similarly in this manner since @smurfpandey mentioned that this limitation does not apply to SNS.

[mrbalkan]

I think this is a CF related discussion rather than SAM. And personally, the current design makes perfectly sense to me. Let me explain why:
In a complete automated environment, there is no room for manual infrastructure configurations via console/cli.
If the stack was able to change existing resources outside it’s control, there will be a mess in a multi-stack/account/microservices environment. That means as a bounded context owner, I am giving away the control to change “my” infrastructure resources to another entity which is completely outside my control. Events are “owned” by the S3 bucket. And that bucket is a repository of an application under a different bounded context/responsibility.
When you enable S3 events and provide an S3 Event destination, it is a property/configuration of a resource (Bucket). If that resource is not under your control, you are not supposed to change it. You can subscribe to it, if the resource (like message brokers, SNS etc) allows.

[dalumiller]

@mrbalkan I agree regarding the logic of the current design making sense. You're right too that this is likely more related to CF than SAM specifically. I'm still left wondering what the difference is between and IoT Rule which is just a collection of topics and triggers and consumers, and an SNS Topic or SQS Queue which have a list of consumers too. We are allowed to reference pre-existing SNS and SQS, but not pre-existing IoT rules, and this seems odd since they feel like the similar services providing similar triggering frameworks.

[keetonian]

CloudFormation released support for importing resources not in a stack into a stack, which could alleviate some (but definitely not all) of the issues discussed here.

I tested out this feature and confirmed that a resource can only be in one stack at a time, and must be deleted from a stack before it can be added to another stack.

For more information about this feature, see the blog post.

[pathikrit-repo]

What is the status if anyone can share. I see it is in stage pm-review state but when can we expect it to be live?

[VincentGaoHJ]

For me, this is an issue related to working with Amplify and SAM template like multiple stacks sharing one Appsync backend env, so I follow the same logic to create S3 resources in Amplify and try to use it in SAM stack. For now, I can access features from another template by exporting values or referencing, why can not add a trigger to a referenced S3 bucket?