feat: Rust cargo lambda workflow (#350)

* feat: support rust via cargo * make black reformat * use os.path.join in path assertion tests to address windows paths * address pylint ci errors * fix reformatted list comprehensions * try passing rust-lld linker on windows * update rust cargo design doc. windows support was added and tested * add test for cargo workspaces project * reformat test sources * build with musl on linux because glibc may differ on lambda * update linux copy and bin paths * update make test again, use the version appveyor is complaining about * update integration tests for rust cargo with latest aws rust runtime interfaces * align TestCustomMakeWorkflow integ test assumptions with appveyor reality * make x86_64-unknown-linux-musl a const for the rust cargo workflow * add integ test for failing cargo rust build * Update Rust workflow to use cargo-lambda Cargo-lambda takes care of cross compiling using Zig as linker. This works on Windows, Linux, and MacOS natively. Signed-off-by: David Calavera <david.calavera@gmail.com> * Fix deprecation warnings. Signed-off-by: David Calavera <david.calavera@gmail.com> * Install Zig on Windows manually Signed-off-by: David Calavera <david.calavera@gmail.com> * Print Zig version on Windows Signed-off-by: David Calavera <david.calavera@gmail.com> * Update version of cargo-lambda Signed-off-by: David Calavera <david.calavera@gmail.com> * Run windows tests in powershell Signed-off-by: David Calavera <david.calavera@gmail.com> * Fix powershell env notation Signed-off-by: David Calavera <david.calavera@gmail.com> * Print clang version on windows Signed-off-by: David Calavera <david.calavera@gmail.com> * Fix env variable name Signed-off-by: David Calavera <david.calavera@gmail.com> * Try new version of zigbuild that fixes some linker issues on Windows. Signed-off-by: David Calavera <david.calavera@gmail.com> * Update releases URL. Signed-off-by: David Calavera <david.calavera@gmail.com> * Upgrade LLVM and clang Signed-off-by: David Calavera <david.calavera@gmail.com> * Update visual studio image To check if that makes any difference. Signed-off-by: David Calavera <david.calavera@gmail.com> * Change the visual studio image everywhere. Signed-off-by: David Calavera <david.calavera@gmail.com> * Revert upgrade changes They didn't fix the problem Signed-off-by: David Calavera <david.calavera@gmail.com> * Print environment Signed-off-by: David Calavera <david.calavera@gmail.com> * Update to Visual Studio 2022 Signed-off-by: David Calavera <david.calavera@gmail.com> * Fix python variable Signed-off-by: David Calavera <david.calavera@gmail.com> * Fix package urls Signed-off-by: David Calavera <david.calavera@gmail.com> * Update missing vs 2019 reference. Signed-off-by: David Calavera <david.calavera@gmail.com> * Change windows package suffix Signed-off-by: David Calavera <david.calavera@gmail.com> * Update cargo-lambda to version 0.9.0 Signed-off-by: David Calavera <david.calavera@gmail.com> * Go back to the original VS version. Signed-off-by: David Calavera <david.calavera@gmail.com> * Cleanup options - Use architecture to setup the right build target. - Don't require `--bin` flag, the default behaviour should work for the majority of functions. - Add flags option to provide a list of additional flags for projects that need extra configuration, like projects within a workspace. Signed-off-by: David Calavera <david.calavera@gmail.com> * Add integration test with cargo_lambda_flags Signed-off-by: David Calavera <david.calavera@gmail.com> * Detect binaries when the project only includes one function. Signed-off-by: David Calavera <david.calavera@gmail.com> * Bring back handler as an optional argument. It saves some duplicated flags for working with workspaces. Signed-off-by: David Calavera <david.calavera@gmail.com> * Ignore errors if directory doesn't exist. Signed-off-by: David Calavera <david.calavera@gmail.com> * Add debug logs Set RUST_LOG=debug when debug is enabled. Signed-off-by: David Calavera <david.calavera@gmail.com> * Log the artifact destination path. Signed-off-by: David Calavera <david.calavera@gmail.com> * Add missing comma Signed-off-by: David Calavera <david.calavera@gmail.com> * Print out and err in the log when debug is enabled Signed-off-by: David Calavera <david.calavera@gmail.com> * Only set RUST_LOG when it's not already set Log it's value, so users know what's set at. Signed-off-by: David Calavera <david.calavera@gmail.com> * Add experimentalCargoLambda feature flag. Signed-off-by: David Calavera <david.calavera@gmail.com> * Add integration test for multi-function projects. Signed-off-by: David Calavera <david.calavera@gmail.com> * Remove type hints They're causing false positives in Python 3.9 with pylint. Signed-off-by: David Calavera <david.calavera@gmail.com> * Add new CI steps for GHA Signed-off-by: David Calavera <david.calavera@gmail.com> * Add build_in_source_support Signed-off-by: David Calavera <david.calavera@gmail.com> * Test rust logger Signed-off-by: David Calavera <david.calavera@gmail.com> * Fix assertion Signed-off-by: David Calavera <david.calavera@gmail.com> * Don't fail fast Signed-off-by: David Calavera <david.calavera@gmail.com> * fix: Fix failing esbuild integration tests (#423) * fix: Fix failing esbuild integration tests * Replace npm ci with npm install * Remove default shell Signed-off-by: David Calavera <david.calavera@gmail.com> * Revert "Remove default shell" This reverts commit 478c3b6. * Add check to ensure that Cargo Lambda is installed. Include a link to the gettings started guide that gives direct installation instructions based on the platform. Signed-off-by: David Calavera <david.calavera@gmail.com> * feat: Add support for mjs files with esbuild (#427) * feat: Add support for mjs files with esbuild * Black reformat * Test Cargo Lambda check Signed-off-by: David Calavera <david.calavera@gmail.com> * Don't redefine which Signed-off-by: David Calavera <david.calavera@gmail.com> * Organize code in more modules This structure follows other workflows, and provides better testeability. Signed-off-by: David Calavera <david.calavera@gmail.com> * Add more documentation Signed-off-by: David Calavera <david.calavera@gmail.com> * Format code Signed-off-by: David Calavera <david.calavera@gmail.com> * Capture exception Signed-off-by: David Calavera <david.calavera@gmail.com> * chore: Remove type/bug label for Bug Issue Template (#425) Co-authored-by: Jacob Fuss <jfuss@users.noreply.github.com> Co-authored-by: Mehmet Nuri Deveci <5735811+mndeveci@users.noreply.github.com> * Clean design doc and tests Signed-off-by: David Calavera <david.calavera@gmail.com> * chore: Version bump to 1.25.0 (#429) * refactor: assign each workflow a default build directory (#428) * fix: remove unused symlinking (#432) * chore: Move to ruff from pylint (#435) When we started the project we defaulted to use pylint. Pylint has served it's purpose but it pretty slow. Ruff is a newer linter in the python ecosystem but is written in Rust. This makes Ruff was faster than pylint. On my machine (while testing in SAM CLI), pylint took about 70s to lint the repo but with ruff it took .04s. Co-authored-by: Jacob Fuss <jfuss@users.noreply.github.com> * chore: Enable pylint within ruff (#436) Co-authored-by: Jacob Fuss <jfuss@users.noreply.github.com> * refactor: esbuild refactor for readability (#433) * feat: use build_dir in esbuild workflow to support building in source (#437) * Fix formatting Signed-off-by: David Calavera <david.calavera@gmail.com> * Update build in source settings Signed-off-by: David Calavera <david.calavera@gmail.com> * fix: remove python3.6 support (#434) * chore: bump version to 1.26.0 (#441) * Remove default value for subprocess_cargo_lambda Signed-off-by: David Calavera <david.calavera@gmail.com> * feat: Pin ruff version, add dependabot config (#442) * feat: Pin ruff version, add dependabot config to keep our dependencies up to date * Exlude isort and flake8 from dependabot updates * feat: Add sources content flag to supported esbuild options (#439) * Better process management Signed-off-by: David Calavera <david.calavera@gmail.com> * Make release mode the default Signed-off-by: David Calavera <david.calavera@gmail.com> * Remove already default None Signed-off-by: David Calavera <david.calavera@gmail.com> * Turn debug message into warning Signed-off-by: David Calavera <david.calavera@gmail.com> * Update documentation format Signed-off-by: David Calavera <david.calavera@gmail.com> * Fix formatting Signed-off-by: David Calavera <david.calavera@gmail.com> * Preserve binary permissions Use copy2 instead of copyfile Signed-off-by: David Calavera <david.calavera@gmail.com> --------- Signed-off-by: David Calavera <david.calavera@gmail.com> Co-authored-by: softprops <d.tangren@gmail.com> Co-authored-by: Daniel Mil <84205762+mildaniel@users.noreply.github.com> Co-authored-by: Jacob Fuss <32497805+jfuss@users.noreply.github.com> Co-authored-by: Jacob Fuss <jfuss@users.noreply.github.com> Co-authored-by: Mehmet Nuri Deveci <5735811+mndeveci@users.noreply.github.com> Co-authored-by: Ruperto Torres <86501267+torresxb1@users.noreply.github.com>
aws · Feb 9, 2023 · b7ec47b · b7ec47b
1 parent 98ed70b
commit b7ec47b
Show file tree

Hide file tree

Showing 38 changed files with 4,013 additions and 2 deletions.
diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
@@ -21,6 +21,7 @@ jobs:
       - python-integration
       - ruby-integration
       - dotnet-integration
+      - rust-cargo-lambda-integration
     steps:
       - name: report-failure
         if: |
@@ -33,7 +34,8 @@ jobs:
           needs.custom-make-integration.result != 'success' ||
           needs.python-integration.result != 'success' ||
           needs.ruby-integration.result != 'success' ||
-          needs.dotnet-integration.result != 'success'
+          needs.dotnet-integration.result != 'success' ||
+          needs.rust-cargo-lambda-integration.result != 'success'
         run: exit 1
       - name: report-success
         run: exit 0
@@ -296,3 +298,57 @@ jobs:
           python-version: ${{ matrix.python }}
       - run: make init
       - run: pytest -vv tests/integration/workflows/dotnet_clipackage
+
+  rust-cargo-lambda-integration:
+    name: ${{ matrix.os }} / ${{ matrix.python }} / rust-cargo-lambda
+    if: github.repository_owner == 'aws'
+    runs-on: ${{ matrix.os }}
+    env:
+      CARGO_LAMBDA_VERSION: 0.15.0
+    defaults:
+      run:
+        shell: bash
+    strategy:
+      fail-fast: false
+      matrix:
+        os:
+          - ubuntu-latest
+          - windows-latest
+        python:
+          - "3.9"
+          - "3.8"
+          - "3.7"
+        rust:
+          - stable
+    steps:
+      - uses: actions/checkout@v3
+      - uses: actions/setup-python@v4
+        with:
+          python-version: ${{ matrix.python }}
+
+      # Install and configure Rust
+      - name: Install rustup
+        run: |
+          : install rustup if needed
+          if ! command -v rustup &> /dev/null ; then
+            curl --proto '=https' --tlsv1.2 --retry 10 --retry-connrefused -fsSL "https://sh.rustup.rs" | sh -s -- --default-toolchain none -y
+            echo "${CARGO_HOME:-$HOME/.cargo}/bin" >> $GITHUB_PATH
+          fi
+        if: ${{ matrix.os }} == 'ubuntu-latest'
+      - name: rustup toolchain install ${{ matrix.rust }}
+        run: rustup toolchain install ${{ matrix.rust }} --profile minimal --no-self-update
+      - run: rustup default ${{ matrix.rust }}
+      - run: |
+          : disable incremental compilation
+          echo CARGO_INCREMENTAL=0 >> $GITHUB_ENV
+      - run: |
+          : enable colors in Cargo output
+          echo CARGO_TERM_COLOR=always >> $GITHUB_ENV
+
+      # Install and configure Cargo Lambda
+      - name: Install Cargo Lambda
+        run: pip install cargo-lambda==$CARGO_LAMBDA_VERSION
+      - run: echo "$HOME/.local/bin" >> $GITHUB_PATH
+
+      - run: make init
+      - run: pytest -vv tests/integration/workflows/rust_cargo
diff --git a/Makefile b/Makefile
@@ -2,10 +2,13 @@ init:
 	LAMBDA_BUILDERS_DEV=1 pip install -e '.[dev]'
 
 test:
-	# Run unit tests
+	# Run unit and functional tests
 	# Fail if coverage falls below 94%
 	LAMBDA_BUILDERS_DEV=1 pytest -vv --cov aws_lambda_builders --cov-report term-missing --cov-fail-under 94 tests/unit tests/functional
 
+unit-test:
+	LAMBDA_BUILDERS_DEV=1 pytest tests/unit
+
 func-test:
 	LAMBDA_BUILDERS_DEV=1 pytest tests/functional
 

diff --git a/README.md b/README.md
@@ -18,6 +18,7 @@ Lambda Builders currently contains the following workflows
 * Typescript with esbuild
 * Ruby with Bundler
 * Go with Mod
+* Rust with Cargo
 
 In Addition to above workflows, AWS Lambda Builders also supports *Custom Workflows* through a Makefile.
 

diff --git a/aws_lambda_builders/workflows/__init__.py b/aws_lambda_builders/workflows/__init__.py
@@ -11,3 +11,4 @@
 import aws_lambda_builders.workflows.dotnet_clipackage
 import aws_lambda_builders.workflows.custom_make
 import aws_lambda_builders.workflows.nodejs_npm_esbuild
+import aws_lambda_builders.workflows.rust_cargo
diff --git a/aws_lambda_builders/workflows/rust_cargo/DESIGN.md b/aws_lambda_builders/workflows/rust_cargo/DESIGN.md
@@ -0,0 +1,28 @@
+# Rust Cargo Builder
+
+## Scope
+
+This package enables the creation of a Lambda deployment package for Rust projects managed using the [cargo](https://doc.rust-lang.org/cargo/) build tool targeting Lambda's "provided" runtime. Rust support for the provided runtime is bundled as a compilation dependency of these projects, provided by the [lambda](https://github.com/awslabs/aws-lambda-rust-runtime) crate.
+
+## Implementation
+
+This package uses [Cargo Lambda](https://www.cargo-lambda.info) to do all the heavy lifting for cross compilation, target validation, and other executable optimizations.
+
+It supports X86-64 architectures with the target `x86_64-unknown-linux-gnu` by default. It also supports ARM architectures with the target option `aarch64-unknown-linux-gnu`. Those are the only two valid targets. The target is automatically configured based on the `architecture` option in the `RustCargoLambdaWorkflow`.
+
+The general algorithm for preparing a rust executable for use on AWS Lambda is as follows.
+
+### Build
+
+It builds a binary in the standard cargo target directory. The binary's name is always `bootstrap`, and it's always located under `target/lambda/HANDLER_NAME/bootstrap`.
+
+### Copy and Rename executable
+
+It then copies the executable to the target directory honoring the provided runtime's [expectation on executable names](https://docs.aws.amazon.com/lambda/latest/dg/runtimes-custom.html).
+
+## Notes
+
+Like the go builders, the workflow argument `options.artifact_executable_name`
+interface can used to provide a handler name that resolves to an executable. This
+enables sam support for cargo workspaces allowing for one rust project to have multiple lambdas. Cargo workspaces have a notion of a `package` and `bin`. A `package` can have
+multiple bins but typically `packages` have a 1-to-1 relationship with a default `bin`: `main.rs`. The handler names must be uniques across a Rust project, regardless of how many packages and binaries that project includes.
diff --git a/aws_lambda_builders/workflows/rust_cargo/__init__.py b/aws_lambda_builders/workflows/rust_cargo/__init__.py
@@ -0,0 +1,5 @@
+"""
+Builds Rust Lambda functions using Cargo Lambda
+"""
+
+from .workflow import RustCargoLambdaWorkflow
diff --git a/aws_lambda_builders/workflows/rust_cargo/actions.py b/aws_lambda_builders/workflows/rust_cargo/actions.py
@@ -0,0 +1,146 @@
+"""
+Rust Cargo build actions
+"""
+
+import logging
+import os
+
+from aws_lambda_builders.workflow import BuildMode
+from aws_lambda_builders.actions import ActionFailedError, BaseAction, Purpose
+from aws_lambda_builders.architecture import X86_64, ARM64
+from .exceptions import CargoLambdaExecutionException
+from .utils import OSUtils
+
+
+LOG = logging.getLogger(__name__)
+
+
+class RustCargoLambdaBuildAction(BaseAction):
+    NAME = "CargoLambdaBuild"
+    DESCRIPTION = "Building the project using Cargo Lambda"
+    PURPOSE = Purpose.COMPILE_SOURCE
+
+    def __init__(
+        self,
+        source_dir,
+        binaries,
+        mode,
+        subprocess_cargo_lambda,
+        architecture=X86_64,
+        handler=None,
+        flags=None,
+    ):
+        """
+        Build the a Rust executable
+
+        :type source_dir: str
+        :param source_dir:
+            Path to a folder containing the source code
+
+        :type binaries: dict
+        :param binaries:
+            Resolved path dependencies
+
+        :type mode: str
+        :param mode:
+            Mode the build should produce
+
+        :type architecture: str, optional
+        :param architecture:
+            Target architecture to build the binary, either arm64 or x86_64
+
+        :type handler: str, optional
+        :param handler:
+            Handler name in `bin_name` format
+
+        :type flags: list, optional
+        :param flags:
+            Extra list of flags to pass to `cargo lambda build`
+
+        :type subprocess_cargo_lambda: aws_lambda_builders.workflows.rust_cargo.cargo_lambda.SubprocessCargoLambda
+        :param subprocess_cargo_lambda: An instance of the Cargo Lambda process wrapper
+        """
+
+        self._source_dir = source_dir
+        self._mode = mode
+        self._binaries = binaries
+        self._handler = handler
+        self._flags = flags
+        self._architecture = architecture
+        self._subprocess_cargo_lambda = subprocess_cargo_lambda
+
+    def build_command(self):
+        cmd = [self._binaries["cargo"].binary_path, "lambda", "build"]
+        if self._mode != BuildMode.DEBUG:
+            cmd.append("--release")
+        if self._architecture == ARM64:
+            cmd.append("--arm64")
+        if self._handler:
+            cmd.extend(["--bin", self._handler])
+        if self._flags:
+            cmd.extend(self._flags)
+
+        return cmd
+
+    def execute(self):
+        try:
+            return self._subprocess_cargo_lambda.run(command=self.build_command(), cwd=self._source_dir)
+        except CargoLambdaExecutionException as ex:
+            raise ActionFailedError(str(ex))
+
+
+class RustCopyAndRenameAction(BaseAction):
+    NAME = "RustCopyAndRename"
+    DESCRIPTION = "Copy Rust executable, renaming if needed"
+    PURPOSE = Purpose.COPY_SOURCE
+
+    def __init__(self, source_dir, artifacts_dir, handler=None, osutils=OSUtils()):
+        """
+        Copy and rename Rust executable
+
+        Parameters
+        ----------
+        source_dir : str
+            Path to a folder containing the source code
+
+        artifacts_dir : str
+            Path to a folder containing the deployable artifacts
+
+        handler : str, optional
+            Handler name in `package.bin_name` or `bin_name` format
+
+        osutils : aws_lambda_builders.workflows.rust_cargo.utils.OSUtils, optional
+            Optional, External IO utils
+        """
+        self._source_dir = source_dir
+        self._handler = handler
+        self._artifacts_dir = artifacts_dir
+        self._osutils = osutils
+
+    def base_path(self):
+        return os.path.join(self._source_dir, "target", "lambda")
+
+    def binary_path(self):
+        base = self.base_path()
+        if self._handler:
+            binary_path = os.path.join(base, self._handler, "bootstrap")
+            LOG.debug("copying function binary from %s", binary_path)
+            return binary_path
+
+        output = os.listdir(base)
+        if len(output) == 1:
+            binary_path = os.path.join(base, output[0], "bootstrap")
+            LOG.debug("copying function binary from %s", binary_path)
+            return binary_path
+
+        LOG.warning("unexpected list of binary directories: [%s]", ", ".join(output))
+        raise CargoLambdaExecutionException(
+            message="unable to find function binary, use the option `artifact_executable_name` to specify the name"
+        )
+
+    def execute(self):
+        self._osutils.makedirs(self._artifacts_dir)
+        binary_path = self.binary_path()
+        destination_path = os.path.join(self._artifacts_dir, "bootstrap")
+        LOG.debug("copying function binary from %s to %s", binary_path, destination_path)
+        self._osutils.copyfile(binary_path, destination_path)