Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: improve dependency installation in rust builder #232

Merged
merged 36 commits into from
May 13, 2024
Merged
Show file tree
Hide file tree
Changes from 16 commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
af424df
fix: update rust-base
bkioshn Apr 12, 2024
0f7f1d5
Merge branch 'master' into fix/improve-rust-builder
bkioshn Apr 12, 2024
7bb4b63
fix: revert back
bkioshn Apr 12, 2024
b65acdd
fix: update rust builder
bkioshn Apr 13, 2024
96a7ab6
fix: spelling
bkioshn Apr 13, 2024
1f63d83
fix: rust base
bkioshn Apr 14, 2024
53595a3
fix: refinery folder
bkioshn Apr 14, 2024
854bb31
fix: target
bkioshn Apr 14, 2024
fa8e001
fix: update wasm c earthfile
bkioshn Apr 14, 2024
29a4800
fix: cargo path name
bkioshn Apr 14, 2024
5a74c5f
fix: cleanup
bkioshn Apr 14, 2024
3dd886e
fix: update earthly version in rust example
bkioshn Apr 15, 2024
28d7c80
fix: cleanup
bkioshn Apr 15, 2024
6e26955
chore: fix rust docs
bkioshn Apr 15, 2024
4410cb5
fix: update rust installer
bkioshn Apr 15, 2024
e885373
fix: update earthly
bkioshn Apr 15, 2024
1b2717e
fix: remove name
bkioshn Apr 19, 2024
33761f5
Merge branch 'master' into fix/improve-rust-builder
bkioshn Apr 19, 2024
50f05c1
feat: add wasm-pack
bkioshn Apr 19, 2024
22177f7
fix: typo
bkioshn Apr 19, 2024
fc264ba
test: try lib/rust earthly
bkioshn Apr 29, 2024
6b58d94
fix: typo
bkioshn Apr 29, 2024
68ff8bb
fix: remove artifact name
bkioshn Apr 30, 2024
f6269c6
fix(rust-builders): Improve rust builder (#236)
stevenj May 3, 2024
62e2c54
fix: update docs
bkioshn May 3, 2024
fcce605
Revert docs changes
bkioshn May 4, 2024
18cb3cb
doc: update style doc
bkioshn May 4, 2024
d2e2449
doc: update rust doc
bkioshn May 5, 2024
c104c9a
doc: fix md style
bkioshn May 5, 2024
051715a
chore: fix spelling
bkioshn May 5, 2024
aec6a22
feat(rust): Add parallel building of Tooling and Rust code (#240)
stevenj May 6, 2024
3c83ff4
doc: update rust doc
bkioshn May 6, 2024
af861b6
fix: update project dic
bkioshn May 6, 2024
54a0251
fix(rust): Fix/improve rust builder (#243)
stevenj May 10, 2024
03d1d05
fix(rust): Remove debug code in Earthfile
stevenj May 13, 2024
8744282
fix(cspell): Remove redundant word from dictionary
stevenj May 13, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .config/dictionaries/project.dic
Original file line number Diff line number Diff line change
@@ -1,10 +1,12 @@
aarch
bindgen
buildkit
camelcase
codegen
colordiff
cowsay
cytopia
depgraph
devenv
dind
dockerhub
Expand Down Expand Up @@ -85,6 +87,7 @@ UDCs
uniseg
voteplan
wasi
wasmtime
webkitallowfullscreen
WORKDIR
xerrors
Expand Down
29 changes: 15 additions & 14 deletions docs/src/guides/languages/rust.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ Also we will take a look how we are setup Rust projects and what configuration i
### Prepare base builder

```Earthfile
VERSION --global-cache 0.7
VERSION 0.8

# Set up our target toolchains, and copy our files.
builder:
Expand All @@ -60,10 +60,11 @@ builder:
COPY clippy.toml deny.toml rustfmt.toml .
```

The first target `builder` is responsible for preparing an already configured Rust environment,
instal all needed tools and dependencies.
The first target `builder` is responsible for preparing configured Rust environments and,
install all needed tools and dependencies.

The fist step of the `builder` target is to prepare a Rust environment via `+rust-base` target.
The fist step of the `builder` target is to prepare a Rust environment via `+rust-base` target,
which is called in `SETUP` Function.
Next step is to copy source code of the project.
Note that you need to copy only needed files for Rust build process,
any other irrelevant stuff should omitted.
Expand Down Expand Up @@ -92,8 +93,8 @@ all-hosts-check:
```

With prepared environment and all data, we're now ready to start operating with the source code and configuration files.
The `check` target which actually performs all checks and validation
with the help of `std_checks.py` script.
The `check` target performs all checks and validation procedures
using the help of `std_checks.py` script.
This script performs static checks of the Rust project as
`cargo fmt`, `cargo machete`, `cargo deny` which will validate formatting,
find unused dependencies and any supply chain issues with dependencies.
Expand All @@ -105,18 +106,18 @@ look at `./earthly/rust/stdcfgs/cargo_config.toml`)Checking Rust Code Format.
3. `cargo machete` - Checking for Unused Dependencies.
4. `cargo deny check` - Checking for Supply Chain Issues.

As it was mentioned above it validates configuration files as
As it was mentioned above, it validates configuration files as
`.cargo/config.toml`, `rustfmt.toml`, `.config/nextest.toml`, `clippy.toml`, `deny.toml`
to be the same as defined in `earthly/rust/stdcfgs` directory of the `catalyst-ci` repo.
So when you are going to setup a new Rust project copy these configuration files
So when you are going to setup a new Rust project, copy these configuration files
described above to the appropriate location of your Rust project.

Another target as `all-hosts-check` just invokes `check` with the specified `--platform`.
It is needed for the local development to double check that everything is works for different platforms.
It is important to define a `linux` target platform with a proper cpu architecture
It is needed for the local development to double check that everything works for different platforms.
It is important to define a `linux` target platform with a proper CPU architecture
for the Rust project when you are building it inside Docker
and check the build process with different scenarios.
The same approach we will see for the another targets of this guide.
The same approach will be seen in other targets throughout this guide.

### Build

Expand Down Expand Up @@ -148,10 +149,10 @@ all-hosts-build:
```

After successful performing checks of the Rust project we can finally `build` artifacts.
Obviously it inherits `builder` target environment and than performs build of the binary.
Obviously it inherits `builder` target environment and then performs build of the binary.
Important to note that in this particular example we are dealing with the executable Rust project,
so it produces binary as a final artifact.
Another case of the building Rust library we will consider later.
We will discuss another scenario of building a Rust library later.
Actual build process is done with the `std_build.py` script.
Here is the full list of configuration of this script:

Expand Down Expand Up @@ -230,7 +231,7 @@ Final step is to provide desired artifacts: docs and binary.
### Test

As you already mentioned that running of unit tests is done during the `build` process,
but if you need some integration tests pls follow how it is done for [PostgreSQL builder](./postgresql.md),
but if you need some integration tests please follow this [PostgreSQL builder](./postgresql.md),
Rust will have the same approach.

### Release and publish
Expand Down
2 changes: 1 addition & 1 deletion earthly/postgresql/Earthfile
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ postgres-base:
RUN sqlfluff version

# Get refinery
COPY ../rust+rust-base/refinery /bin
COPY ../rust/tools+tool-refinery/refinery /bin

# Get dbviz
COPY ../../utilities/dbviz+build/dbviz /bin
Expand Down
47 changes: 21 additions & 26 deletions earthly/rust/Earthfile
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
# Common Rust UDCs and Builders.
VERSION --global-cache --use-function-keyword 0.7
VERSION 0.8

# cspell: words rustup miri ripgrep stdcfgs toolset depgraph lcov psycopg bindgen
IMPORT ./tools AS tools

# cspell: words rustup miri ripgrep stdcfgs toolset lcov psycopg
# cspell: words TARGETPLATFORM TARGETOS TARGETARCH TARGETVARIANT USERPLATFORM USEROS USERARCH USERVARIANT

# Base Rustup build container.
Expand Down Expand Up @@ -37,7 +38,6 @@ rust-base:
echo "USEROS = $USEROS"; \
echo "USERARCH = $USERARCH"; \
echo "USERVARIANT = $USERVARIANT";

WORKDIR /root

# Install necessary packages
Expand Down Expand Up @@ -71,26 +71,21 @@ rust-base:
# Install the default cargo config.
COPY stdcfgs/cargo_config.toml $CARGO_HOME/config.toml

# Install rust based tooling
# Install tools we use commonly with `cargo`.
# Note, we disable static compiles for tools, specifically, as its not required.
# These tools are not artifacts and we do not use them in production.
RUN cargo install cargo-nextest --version=0.9.68 --locked
RUN cargo install cargo-machete --version=0.6.1 --locked
RUN cargo install refinery_cli --version=0.8.13 --locked
# TODO: https://github.com/input-output-hk/catalyst-ci/issues/214
RUN cargo install cargo-deny --version=0.14.10 --locked
RUN cargo install cargo-modules --version=0.14.0 --locked
RUN cargo install cargo-depgraph --version=1.6.0 --locked
RUN cargo install cargo-llvm-cov --version=0.6.8 --locked
RUN cargo install wasm-tools --version=1.201.0 --locked
RUN cargo install cargo-expand --version=1.0.79 --locked
RUN cargo install wit-bindgen-cli --version=0.24.0 --locked
RUN cargo install --git https://github.com/bytecodealliance/wasmtime --tag v17.0.0 verify-component-adapter --locked

SAVE ARTIFACT $CARGO_HOME/bin/refinery refinery
SAVE ARTIFACT $CARGO_HOME/bin/wasm-tools wasm-tools
SAVE ARTIFACT $CARGO_HOME/bin/wit-bindgen wit-bindgen
# installer - Install necessary tools for the rust-base.
installer:
FROM +rust-base

COPY tools+tool-cargo-nextest/cargo-nextest $CARGO_HOME/bin/cargo-nextest
COPY tools+tool-cargo-machete/cargo-machete $CARGO_HOME/bin/cargo-machete
COPY tools+tool-refinery/refinery $CARGO_HOME/bin/refinery
COPY tools+tool-cargo-deny/cargo-deny $CARGO_HOME/bin/cargo-deny
COPY tools+tool-cargo-modules/cargo-modules $CARGO_HOME/bin/cargo-modules
COPY tools+tool-cargo-depgraph/cargo-depgraph $CARGO_HOME/bin/cargo-depgraph
COPY tools+tool-cargo-llvm-cov/cargo-llvm-cov $CARGO_HOME/bin/cargo-llvm-cov
COPY tools+tool-wasm-tools/wasm-tools $CARGO_HOME/bin/wasm-tools
COPY tools+tool-cargo-expand/cargo-expand $CARGO_HOME/bin/cargo-expand
COPY tools+tool-wit-bindgen-cli/wit-bindgen $CARGO_HOME/bin/wit-bindgen
COPY tools+tool-wasmtime/verify-component-adapter $CARGO_HOME/bin/verify-component-adapter

# Universal build scripts we will always need and are not target dependent.
COPY --dir scripts /scripts
Expand All @@ -109,14 +104,14 @@ rust-base:
# docker enabled on Mac.
# Again, this is just a test target, and not for general use.
rust-base-all-hosts:
BUILD --platform=linux/amd64 --platform=linux/arm64 +rust-base
BUILD --platform=linux/amd64 --platform=linux/arm64 +installer

# Common Rust setup.
# Parameters:
# * toolchain : The `rust-toolchain` toml file.
SETUP:
FUNCTION
FROM +rust-base
FROM +installer

ARG toolchain=./rust-toolchain.toml

Expand Down
57 changes: 57 additions & 0 deletions earthly/rust/tools/Earthfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
VERSION 0.8

tool-cargo-nextest:
FROM ../+rust-base
RUN cargo install cargo-nextest --version=0.9.68 --locked
SAVE ARTIFACT /usr/local/cargo/bin/cargo-nextest cargo-nextest
bkioshn marked this conversation as resolved.
Show resolved Hide resolved


tool-cargo-machete:
FROM ../+rust-base
RUN cargo install cargo-machete --version=0.6.2 --locked
SAVE ARTIFACT /usr/local/cargo/bin/cargo-machete cargo-machete

tool-refinery:
FROM ../+rust-base
RUN cargo install refinery_cli --version=0.8.14 --locked
SAVE ARTIFACT /usr/local/cargo/bin/refinery refinery

tool-cargo-deny:
FROM ../+rust-base
RUN cargo install cargo-deny --version=0.14.10 --locked
SAVE ARTIFACT /usr/local/cargo/bin/cargo-deny cargo-deny

tool-cargo-modules:
FROM ../+rust-base
RUN cargo install cargo-modules --version=0.14.0 --locked
SAVE ARTIFACT /usr/local/cargo/bin/cargo-modules cargo-modules

tool-cargo-depgraph:
FROM ../+rust-base
RUN cargo install cargo-depgraph --version=1.6.0 --locked
SAVE ARTIFACT /usr/local/cargo/bin/cargo-depgraph cargo-depgraph

tool-cargo-llvm-cov:
FROM ../+rust-base
RUN cargo install cargo-llvm-cov --version=0.6.9 --locked
SAVE ARTIFACT /usr/local/cargo/bin/cargo-llvm-cov cargo-llvm-cov

tool-wasm-tools:
FROM ../+rust-base
RUN cargo install wasm-tools --version=1.203.0 --locked
SAVE ARTIFACT /usr/local/cargo/bin/wasm-tools wasm-tools

tool-cargo-expand:
FROM ../+rust-base
RUN cargo install cargo-expand --version=1.0.82 --locked
SAVE ARTIFACT /usr/local/cargo/bin/cargo-expand cargo-expand

tool-wit-bindgen-cli:
FROM ../+rust-base
RUN cargo install wit-bindgen-cli --version=0.24.0 --locked
SAVE ARTIFACT /usr/local/cargo/bin/wit-bindgen wit-bindgen

tool-wasmtime:
FROM ../+rust-base
RUN cargo install --git https://github.com/bytecodealliance/wasmtime --tag v17.0.0 verify-component-adapter --locked
SAVE ARTIFACT /usr/local/cargo/bin/verify-component-adapter verify-component-adapter
4 changes: 2 additions & 2 deletions earthly/wasm/c/Earthfile
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,8 @@ wasm-c-base:
lld=17.0.5-r0 \
wasi-sdk=20-r3

COPY ./../../rust+rust-base/wasm-tools /bin
COPY ./../../rust+rust-base/wit-bindgen /bin
COPY ./../../rust/tools+tool-wasm-tools/wasm-tools /bin
COPY ./../../rust/tools+tool-wit-bindgen-cli/wit-bindgen /bin

# Universal build scripts.
COPY --dir scripts /scripts
Expand Down
2 changes: 1 addition & 1 deletion examples/rust/Earthfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
VERSION --try --global-cache 0.7
VERSION --try 0.8

# cspell: words USERARCH toolsets

Expand Down
Loading