Exactly-once support for Pub/Sub

We've been hard at work for a little bit on this one, and it's finally here!
Now when you create a Pub/Sub topic, you can specify an exactly-once delivery guarantee:

var Signups = pubsub.NewTopic[*SignupEvent]("signups", pubsub.TopicConfig{
    DeliveryGuarantee: pubsub.ExactlyOnce,
})

This provides a much stronger guarantee, backed by the cloud infrastructure used to run your app, that a message published will not be delivered multiple times to your application's subscriptions if processed without an error.
For more information checkout the docs.

Remember to first update to the latest version of Encore using encore version update, and update your dependency on encore.dev within your app so your IDE sees the new option using go get -u encore.dev@latest.

What's next for Pub/Sub

We plan to add support for ordered topics in the coming releases, which will allow you to process events in the order they were published based on a grouping key (e.g. user ID or order ID).

Full Changelog: v1.17.0...v1.18.0

Outboxes, Dashboards & Databases

May is here and here in Sweden we survived the yearly ritual bonfire night unscathed. We're excited to announce Encore v1.17.0 which brings even more features! As always, update with encore version update, or head over to the install docs to install Encore.

Transactional Outbox

One of the hardest parts of building an event-driven application is ensuring consistency between services. A common pattern is for each service to have its own database and use Pub/Sub to notify other systems of business events. Inevitably this leads to inconsistencies since the Pub/Sub publishing is not transactional with the database writes.

To better handle such use cases, Encore now provides a utility package for implementing the Transactional Outbox pattern. To use it make sure you're on Encore v1.17.0 (this release), then read up on the docs.

Code viewer 2.0

We've spruced up the code viewer over on the Cloud Dashboard. With this update, you now get cross-links to jump between code and the Service Catalog and Tracing. This makes it super quick to navigate between code and API Docs / Traces to see how everything connects. Check out the video below, or head over to the Uptime demo app to see it for yourself.

codeviewer.mp4

Improved database management

Encore now supports creating databases using the new sqldb.NewDatabase, bringing the encore.dev/storage/sqldb package in line with the other infrastructure resources. This approach to creating databases is more explicit and better allows for future extensibility than the previous behavior (which implicitly created a database based on the presence of a migrations folder).

Test-only infrastructure

Encore now supports defining most infrastructure resources in _test.go files, allowing tests to declare their own infrastructure resources. This allows library code that operates on infrastructure resources to be more easily tested (normal infrastructure resources can only be defined within Encore services). For an example of such a use case see the new transactional outbox tests.

On a related note, most encore db commands now take an optional --test flag to connect to test databases.

Client generation service filter

The encore gen client command now supports filtering which services to be included in the output with the new --services flag. Thanks @jdbann for the suggestion!

Thanks to all contributors!

We rely on the community to improve Encore, and we continue to be humbled by your incredible support and encouragement. We've had a huge influx of suggestions and feedback recently, so a huge thank you to everyone. In particular, thanks to John Bannister, Juha Ristolainen, Che Sampat, Vilhelm Melkstam, Steven Edwards, Rafal Glowka, George Arscott, and George Antoniadis!

What’s next

We're working on adding more flexibility in controlling underlying infra for individual services, deploying to Kubernetes, and much more. We're also working on bringing a bunch of improvements to the local development dashboard.
Stay tuned and please vote on your favorite feature on the roadmap!

As always, if you have questions or feedback, tell us on Slack or post on the Community Forum.

We’re excited to hear your feedback! ❤️

Cookies, Pub/Sub, test execution speed, and more!

Spring is here and you know what that means! We're excited to announce a series of quality-of-life improvements to the Encore developer experience!

🍪 Cookie support

Cookies can now be used in auth handlers, offering an easy way to authenticate requests via cookies:

type MyAuthParams struct {
	// SessionCookie is set to the value of the "session" cookie.
	// If the cookie is not set it's nil.
	SessionCookie string `cookie:"session"`
}

//encore:authhandler
func AuthHandler(ctx context.Context, p *MyAuthParams) (auth.UID, error) {
    // ...
}

To set and read cookies in Encore APIs, use a raw endpoint.

Pub/Sub Topic references

Encore uses static analysis to determine which services are publishing messages to what topics. That information is used to provision infrastructure correctly, render architecture diagrams, and configure IAM permissions. This means that *pubsub.Topic objects can't be passed around however you'd like, as it makes static analysis impossible in many cases.

To better support use cases like passing topics to library code or using them with dependency injection, Encore now supports the ability
to get a Topic Reference that can be passed around any way you want:

var Signups = pubsub.NewTopic[*SignupEvent]("signups", pubsub.TopicConfig{
    DeliveryGuarantee: pubsub.AtLeastOnce,
})

// Create a topic reference that can be passed around arbitrarily
ref := pubsub.TopicRef[pubsub.Publisher[*SignupEvent]](Signups)

For more information see the docs

Fallback routes

Encore now supports defining fallback routes. Fallback routes are similar to wildcard routes,
but they are only matched if no other route matches the request. The primary use case for
fallback routes is to simplify migrating existing backends over to Encore.

With fallback routes you can define a catch-all endpoint that forwards requests to your existing
HTTP router with Encore, and then gradually move over endpoints one by one while preserving
the behavior for the remaining routes. For example:

//encore:api public raw path=/!fallback
func FallbackRoute(w http.ResponseWriter, req *http.Request) {
    // Forward the request to the existing router
    existingRouter.ServeHTTP(w, req)
}

Test speed

As a side effect of how Encore generated code for running tests the Go compiler ended up needing
to re-link binaries for every package being tested. This caused slow compilation times and slow tests.

We've tweaked Encore's code generation to handle this better, resulting in as much as a 2-3x improvement in test execution speed in many cases!

Panic stack traces

When your API handler panics Encore now displays a much nicer stack trace compared to before, allowing you to more easily pinpoint the issue. This works both in local development (encore run as well as the local trace viewer) as well as traces in cloud environments.

OpenAPI generator (experimental)

Encore now supports generating an OpenAPI specification for an API, using encore gen client --lang openapi <app-id>. Let us know what you think about the output and if you'd like to see something improved!

Bugfixes & other improvements

• sqldb.DriverConn now offers a way to get the underlying *pgx.Conn from a standard library *database/sql.Conn.
• The new Encore parser and runtime is now used everywhere and the old parser and runtime have been deleted, offering better performance and powering many of the features listed above

Thanks to all contributors!

We rely on the community to improve Encore, and we continue to be humbled by your incredible support and encouragement. A massive thank you to Brent Dillingham, Luka Giorgadze, Simon Vans-Colina, Otar Adamia, Phakorn Kiong, George Antoniadis, Jan Brucek, and more!

What’s next

We're working on adding more flexibility in controlling underlying infra for individual services, deploying to Kubernetes, and much more. We're also working on bringing a bunch of improvements to the local development dashboard.
Stay tuned and please vote on your favorite feature on the roadmap!

As always, if you have questions or feedback, tell us on Slack or post on the Community Forum.

We’re excited to hear your feedback! ❤️

Introducing API Explorer 2.0

apiexplorer.mp4

We're excited to announce a significant upgrade to the API Explorer in the Encore Platform!

Thanks to your feedback, we've made these changes for a more seamless workflow:

• Fuzzy search endpoints with ⌘ + K - find what you need, fast!
• Enhanced sidebar navigation - get a better overview at a glance
• Upgraded endpoint testing - save authentication data with ease
• Handy cross-links - view pre-filtered traces & flow diagrams per service/endpoint
• UI facelift & bugfixes - enjoy a cleaner look and smoother experience
• Services & endpoints now get dedicated urls - preparing for future content

Here's some of what we're considering next — let us know if it sounds valuable (or not):

• Cross-linking to code for easy lookups
• A widget displaying the last few traces per endpoint
• Stats & metrics per service/endpoint

Bug-fixes and other improvements

• schemautil: fix infinite recursion in #659
• runtime: fix leveled logging for encore cloud in #660
• Increase test timeout for nightly test that runs multiple times in #661
• builderimpl: always use v2 in #662
• experiment: add beta-runtime experiment in #663

Thanks to all contributors!

We rely on the community to improve Encore, and we continue to be humbled by your incredible support and encouragement. A massive thank you to Patryk Siemiński, Paweł Słomka, Brent Dillingham, Emily Rymer, John Bannister, Mason Stewart, Simon Vans-Colina, Ohm Patel, Jakob Nordfeldt, and many more!

What’s next

We're very close to releasing Encore's V2 parser and major speed improvements to running tests.
We're also working on adding more flexibility in controlling underlying infra for individual services, deploying to Kubernetes, and much more. Stay tuned and please vote on your favorite feature on the roadmap!

As always, if you have questions or feedback, tell us on Slack or post on the Community Forum.

We’re excited to hear your feedback! ❤️

Full Changelog: v1.14.5...v1.15.0

Introducing: Metrics

Having easy access to metrics is a critical part of application observability. Encore now automatically provides metrics for your application for everyday use cases such as Request rate, Memory usage, and Goroutines. These work in all environments and are accessible from the Encore Platform.

By default, Encore also exports metrics data to your cloud provider's built-in monitoring service, so you can easily use your preferred observability tools.

Encore also makes it simple to define custom metrics for your application, which automatically display in the built-in metrics dashboard. Learn more in the docs.

Grafana integration

Many teams already use observability tools like Grafana, so we've added an easy integration to help you get all your metrics in one place. (For fans of Datadog, an integration is coming very soon!)

Other improvements

There's a huge list of improvements since the last release announcement. Here are some highlights:

• Added support for CORS Expose-Headers
• Trace Logs now display as JSON
• Local Dev Dash alerts if the app is not running
• And many smaller improvements and bug-fixes

Full Changelog: v1.13.0...v1.14.4

Secrets 2.0

Introducing Secrets 2.0! 🎉

Encore has had built-in secrets management from day one, but it was quite limited: each secret value only applied to either {production} environments or to {development,preview,local} environments. This worked fine for basic use cases but was insufficient for many other use cases. For example, it's common to want a different secret value for local development specifically. In other use cases it was common to want to use the same secret value for all environments, which previously required setting the same value twice.

Today we're excited to announce a much more flexible secrets management experience that better addresses these use cases! Starting today, each secret value can be individually configured to apply to exactly the environment(s) you want.

For example:

# Set for local only
$ encore secret set --type local MySecret

# Set for all environments
$ encore secret set --type prod,dev,pr,local SomeOtherSecret

# Set for all preview environments and the environment named 'staging'
$ encore secret set --type pr --env staging SecretPassword

Read the docs for more information.

There's also a brand new secrets management UI on the Encore Platform:

Redacting sensitive data

Encore now supports redacting sensitive data from traces. Fields marked with the struct tag encore:"sensitive" (in request/response structs) are automatically redacted. This works for both individual values as well as nested fields.

Note that inputs to auth handlers are automatically marked as sensitive and are always redacted.

Read the docs for more information.

Database driver

The encore.dev/storage/sqldb package now supports accessing the underlying connection pool using sqldb.Driver.
See the package documentation for more information.

Bugfixes and other improvements

• Improved API Explorer to use JSON fields instead of struct fields (#577)
• Fixed trace id lookups on the Encore Platform

Thanks to our contributors!

We rely on the community to improve Encore, and we're always amazed by your incredible support and encouragement. A massive thank you to Patryk Siemiński, Mason Stewart, Sami Moustachir, Alex Yakubenko, Sean Knowles, David Mytton, Prasanna Balaji, Gabriel Bellon, and others for your contributions and feedback!

What’s next

We're very close to releasing support for metrics! We're also working on Kubernetes support, using existing infrastructure, and more.
– Remember to vote on your favorite feature on the roadmap!

As always, if you have questions or feedback, tell us on Slack or post on the Community Forum.

We’re excited to hear your feedback! ❤️

Full Changelog: v1.12.1...v1.13.0

Trace filters & loads of improvements

Trace filtering is here!

We've finally added filtering to the tracing dashboard, making it easier to find exactly what you're looking for when working on a big system.

• You can now filter on duration, User ID, Trace ID, and X-Request-ID
• Check it out for yourself in the web platform

Bugfixes and other improvements

Thank you for all your bug reports and feedback! Here are some improvements we've made lately:

• Upgraded Encore to Go 1.19.4
• Log subscription ID's on Push Errors
• Fix PubSub traces
• Fix log streaming from GCP enviroments
• Prevent conversions to errs.Error on handler errors
• Explicitly set content types
• Payload tracing fixes
• Add support for Private Network Access in CORS
• Fix GCP log format
• Started using FSNotify for file system watching
• Automatically add headers being used in types to CORS config
• Added *sqldb.Error type to expose information about underlying database errors

Thanks to our contributors!

We rely on the community to improve Encore, and we're always amazed by your incredible support and encouragement. A massive thank you to Patryk Siemiński, Amr, Daniel Stocks, @Willyham, @klaidliadon, MaxD, @melkstam, Jakob Nordfeldt, Prasanna Balaji for your contributions and feedback!

What’s next

We're very close to releasing a huge upgrade to Encore's infrastructure provisioning system. This will give you more flexibility, and it will be simple to add specific requirements, e.g. deploying to existing Kubernetes clusters. We're also about to release built-in metrics dashboards and an upgraded secrets manager!
– Remember to vote on your favorite feature on the roadmap!

As always, if you have questions or feedback, tell us on Slack or post on the Community Forum.

We’re excited to hear your feedback! ❤️

Full Changelog: v1.10.1...v1.11.0

Tracing improvements & Performance upgrade

Tracing is now sooo much better

We've just released a much-improved trace view. Here's what you need to know:

• Raw endpoints are now traced – including HTTP path, method, request/response bodies and headers.
• Traces now come with a JSON viewer, letting you expand/collapse parts of the payload if it's unwieldy to view.
• Remember to update before trying it out: encore version update

Speed improvements to parser and tests

To make Encore better for everyone with a very large code base, we've improved Encore's static analysis performance. The parser is now 5x faster!

With Encore you run tests with encore test instead of go test. This is a wrapper that compiles the Encore app and then runs go test. It supports all the same flags that the go test command does. For bigger projects, we want to make sure this is snappy! – In the latest release, we've improved test speed 2x, both for local development as well as for CI/CD

Bugfixes and other improvements

We're always thankful when we get feedback and bug reports from anyone using Encore. Here are some improvements we've made lately, thanks to your input:

• The metadata APIs that provide information about the current request also contains information about PubSub messages and Cron Job executions.
• You can now override auth information for the current request inside tests.
• Improved documentation for how to structure your Encore app (thanks to @Minivera!).

Thanks to our contributors
We rely on the community to improve Encore, and we continue to be surprised and humbled by your incredible support and encouragement. A massive thank you to @Minivera , Patryk Siemiński, Amr, Daniel Stocks, @Willyham, MaxD, @melkstam, Jakob Nordfeldt, Prasanna Balaji, and Juan Álvarez for your contributions and feedback!

What’s next

We're close to releasing a big upgrade to infrastructure provisioning, making it more flexible and easier for you to add specific requirements like deploying to existing Kubernetes clusters. We'll also soon release new observability and metrics capabilities.
– Remember to vote on your favorite feature on the roadmap!

As always, if you have questions or feedback, tell us on Slack or post on the Community Forum.

We’re excited to hear your feedback! ❤️

Full Changelog: v1.9.3...v1.10.1

Configure the World!

Or maybe just your application to start with.

In this release, we're introducing a new config package that combines the power of CUE lang with the smooth Encore developer experience. It gives you both the type safety of Go, and the powerful flexibility of CUE. This lets you focus on writing your business logic, safe in the knowledge that Encore's compiler will prevent invalid configurations from ever reaching your production environment.

To use it, you simply call config.Load[*MyConfigType]() and assign the returned instance of your config type to a package-level variable (Remember to first update with encore version update.)

As with many of Encore's features, config comes with isolated test support. This allows you to override configuration values on inside tests, without impacting any other concurrently running tests.

For more information about Encore native config support, check out the docs.

Bugfixes and other improvements

We're always thankful when we get feedback and bug reports from anyone using Encore!
Here are some improvements we've made lately, thanks to your input:

• Added documentation to the encore.dev Go package to explain why those functions all panic, as well as including links to the underlying implementation (encoredev/encore.dev#11)
• Fixed an issue with walking recursive types (#439)
• Fixed an issue with the rendering of pointer types in the development dashboard (#440)
• Improved the documentation by improving clarity on some pages, rewriting others and fixing various typos.

Thanks to our contributors

We rely on the community to improve Encore, and we're overwhelmed by your support and encouragement. A huge thank you to @SarojWasti & @ArjunSharda for your contributions to this release!

What's next

Over the next couple of weeks, we're going to release a next-generation cloud provisioning system for Encore. Check out this blog post to get a sneak preview. – Remember to vote on your favorite feature on the roadmap!

As always, if you have questions or feedback, tell us on Slack or our Community Forums.

We're excited to hear your feedback! ❤️

Catch you in the cloud,
Dom & the Encore team

Full Changelog: v1.8.0...v1.9.0

Introducing next-generation error reports

If there's an error, at least it's a nice one now

Compiler errors are a source of frustration for most developers, as they often force us to decipher confusing or misleading error messages. To make this a better experience we've taken inspiration from Elm and Rust, and we've modified how Encore reports into human-readable errors.

Instead of simply reporting a filename, line number, and column number with a single line of text describing the error, Encore now shows a summary of the error, the line(s) of code which caused the error, and also show other lines of code to provide additional context.

All errors returned by Encore will now render in this format. Over the coming months, we plan to improve the readability of all errors by providing additional context or help text within the error itself.

GoLand Plugin

We are happy to announce that Encore now has an official plugin for GoLand and IntelliJ. The plugin allows you to run unit tests on Encore applications from the comfort of your IDE, as well as allowing you to run those same tests in debug mode with breakpoint support!

As with Encore itself, the plugin is open source. Bug reports and contributions are welcome at encoredev/intellij-plugin.

Download the plugin today: https://plugins.jetbrains.com/plugin/20010-encore/

Bugfixes and other improvements

We're always thankful when we get feedback and bug reports from anyone using Encore. Here are some improvements we've made lately, thanks to your input:

• We've made multiple improvements to the local dashboard from fixing the font loading to adding scrolling to the logs shown in the trace view.
• When you use the encore logs command to stream logs, we show a "waiting for logs" message to indicate that the stream is connected successfully.
• We've fixed an issue with authorization tokens to the Encore platform becoming corrupted and not refreshing correctly.
• Fixed a bug where if a DI-based API was used in a cronjob, Encore was unable to parse the application if the encore.gen.go file was missing. (Thanks @Willyham for the bug report)
• Various improvements to the documentation.

Thanks to our contributors

We rely on the community to improve Encore, and we're overwhelmed by your support and encouragement. A huge thank you to @kVarunkk, @eddy-geek, @AM1TB, @Minivera, @michizhou for your contributions to this release!

What's next

Over the next couple of weeks, we're super excited to announce we'll be releasing Encore native support for per-service configuration, powered by the CUE language. Remember to vote on your favourite feature on the roadmap!

As always, if you have questions or feedback, tell us on Slack or our Community Forums.

We're excited to hear your feedback! ❤️

Catch you in the cloud,
Dom & the Encore team

Full Changelog: v1.7.0...v1.8.0