fix(postgresql): various fixes/improvements related to using the post…

…gres builders in catalyst voices. (#108)

* fix(postgresql): Not permitted to push images to a registry in the build step.

* fix(postgresql): Remove necessity to manually build the sqlfluff-image when locally formatting sql.

* fix(postgresql): Don't overwrite .sqlfluff file if it is provided already.

* fix(postgresql): Globally disable `format` as its corrupting file ownership

* fix(postgresql): Test if behavior changed from v2.3.2 and 2.3.5

* fix(postgresql): Fix sqlfluff version back to latest after lint discrepancy found

* feat(postgresql): Add utility we need to generate postgresql schema diagrams

* fix(postgresql): fix format of dbviz code

* feat(bash): Add a bash duplicate and lint checker, as we are starting to accumulate a lot of bash scripts.

* docs(bash): fix spelling errors

* fix(docs): Fix script includes copy path

* fix(postgresql): check the schema applies cleanly during the build, so that containers won't be created if they don't.

* fix(postgresql): We probably don't want to dump the password to logs.

* feat(postgresql): Add new file of bash functions to do the various steps we execute on the database.

* feat(bash): add bash utility function to make handling parameters easier.

* feat(postgresql): Build now ensures the schema can apply cleanly, and that all seed data is valid,

* refactor(postgresql): Move seed ata from data to seed directory in the example

* fix(bash): Fix bash scripts finding relative references (as good as bash can)

* feat(bash): Add bash format checks

* fix(bash): Fix bash script formatting

* fix(bash): Fix bash format check not running on correct src

* fix(bash): Fix spelling in bash files

* fix(postgresql): Fix global .sql linting in the CI repo

* fix(bash): Fix sourced bash script references because bash is unreliable at importing with relative references.

* refactor(postgresql): Rework how postgresql integration tests could work in the example

* fix(cat-ci): Add --global-cache option to cat-ci to improve cache locality and hits across multiple earthfiles

* fix(rust): Don't build nushell in the rust builder

* feat(postgresql): Add nushell to postgres integration test container

* feat(postgresql): Refactored sql builders and nushell utility

* Add dbviz artifacts for postgres builder to use

* feat(postgresql): Add python scripting for the postgres builders

* fix(cat-ci): Ignore python cruft

* feat(postgresql): Add initial docs builders for postgresql

* fix(bash): Fix bash linter to be multi-check target compliant

* feat(postgresql): First auto postgres db docs building running

* fix(bash): Fix shell format errors that exist in master

* fix(bash): bash lint issues

* fix(postgresql): Use latest rust configs for dbviz

* fix(postgresql): update cargo.toml for dbviz

* docs(postgresql): Try and add generated docs for the postgres example, to our example docs

* fix(docs): Put postgres docs in the correct place and include the markdown file.

* fix(postgresql): Migrations need to use the user role, not super user.

* fix(postgresql): Use the correct role when generating db diagrams

* fix(docs): Seems graphviz needs fonts installed to generate svg reliably

* fix(postgresql): Make sure schema definitions in docs don;t accumulate

* fix(docs): spelling correction

* fix(docs): Fix included tables in autogenerated diagrams

* fix(postgresql): can not pass targets as arguments, because the target is relative to cat-ci not the caller.

* fix(postgresql): Allow lines up to 120 chars, 80 is a bit short

* fix(postgresql): allow functions to be used by the normal user

* fix(postgresql): try to grant function privileges again

* fix(docs): dbdocs set comments to empty string if no comments to show.

* fix(docs): dbdocs generation failure needs to result in an error

* docs(postgresql): test

* docs(postgresql): make docs results check for any error properly

* docs(postgresql): Fix finding the optional create table syntax plus add some debug logic

* docs(postgresql): make comments optional at sub diagram level, and add debug logic

* docs(postgresql): more debug

* docs(postgresql): only print debug if match found

* docs(postgresql): more debug

* docs(postgresql): more debug

* docs(postgresql): Make type of match capture

* fix(docs): fixed diagram option reading, remove debug code

* docs(postgresql): debug docs that dont generate

* fix(docs): Fix removing excluded tables from includes

* docs(postgresql): Includes/excludes lists still not calculated correctly

* docs(postgresql): Make sure we auto include tables defined in a migration

* docs(postgresql): fix excluded tables calculation

* docs(postgresql): cleanup title generation

* docs(postgresql): add debug to include generation

* docs(postgresql): need schema wide exclude to generate include list properly

* docs(postgresql): specify option flag multiple times when needed

* fix(docs): missing comma

* docs(postgresql): still fixing the include/exclude formatting

* fix(docs): misnamed Readme.md

* docs(postgresql): Fix comment embedding option for migration diagrams

* fix(postgresql): Can't cleanup data when applying seed data during build

* fix(postgresql): lint issues

* fix(postgresql): dropping database between seed data tests

* docs(bash): Remove duplicated comment

* fix(cspell): Fix typo

* docs(postgresql): doc cleanups

* chore(rust): Match rust cargo config

.config/dictionaries/project.dic

@@ -11,6 +11,7 @@ Earthfiles
 errchkjson
 extldflags
 fontawesome
+fontconfig
 ginkgolinter
 gitops
 glightbox

.editorconfig

@@ -0,0 +1,14 @@
+[*.sh]
+# like -i=4
+indent_style = space
+indent_size = 4
+
+# --language-variant
+shell_variant      = bash
+binary_next_line   = true
+# --case-indent
+switch_case_indent = true
+space_redirects    = true
+keep_padding       = false
+# --func-next-line
+function_next_line = false

.gitignore

@@ -17,4 +17,7 @@ node_modules
 /earthly/docs/local
 
 # Local Build Artefacts
-target/
+target/
+
+# Python junk
+**/*.pyc

earthly/postgresql/.sqlfluff → .sqlfluff

@@ -3,6 +3,7 @@
 [sqlfluff]
 dialect = postgres
 large_file_skip_char_limit = 0
+max_line_length = 120
 
 [sqlfluff:indentation]
 tab_space_size = 2

Earthfile

@@ -1,6 +1,5 @@
 # Set the Earthly version to 0.7
-VERSION 0.7
-FROM debian:stable-slim
+VERSION --global-cache 0.7
 
 # cspell: words livedocs sitedocs
 
@@ -18,6 +17,13 @@ markdown-check-fix:
 check-spelling:
     DO ./earthly/cspell+CHECK
 
+# check-bash - test all bash files lint properly according to shellcheck.
+check-bash:
+    FROM alpine:3.18
+
+    DO ./earthly/bash+SHELLCHECK --src=.
+
+
 ## -----------------------------------------------------------------------------
 ##
 ## Standard CI targets.
@@ -28,14 +34,24 @@ check-spelling:
 check:
     BUILD +check-spelling
     BUILD +check-markdown
+    BUILD +check-bash
 
+# Internal: Reference to our repo root documentation used by docs builder.
 repo-docs:
     # Create artifacts of extra files we embed inside the documentation when its built.
     FROM scratch
     #FROM alpine:3.18
 
     WORKDIR /repo
     COPY --dir *.md LICENSE-APACHE LICENSE-MIT .
-    #RUN ls -al /repo
+
+    SAVE ARTIFACT /repo repo
+
+repo-config:
+    # Create artifacts of config file we need to refer to in builders.
+    FROM scratch
+
+    WORKDIR /repo
+    COPY --dir .sqlfluff .
 
     SAVE ARTIFACT /repo repo

actions/configure-runner/Earthfile

@@ -1,4 +1,4 @@
-VERSION 0.7
+VERSION --global-cache 0.7
 
 deps:
     FROM node:20-bookworm

actions/discover/Earthfile

@@ -1,4 +1,4 @@
-VERSION 0.7
+VERSION --global-cache 0.7
 
 deps:
     FROM node:20-bookworm

actions/install/Earthfile

@@ -1,4 +1,4 @@
-VERSION 0.7
+VERSION --global-cache 0.7
 
 deps:
     FROM node:20-bookworm

actions/merge/Earthfile

@@ -1,4 +1,4 @@
-VERSION 0.7
+VERSION --global-cache 0.7
 
 deps:
     FROM node:20-bookworm

actions/push/Earthfile

@@ -1,4 +1,4 @@
-VERSION 0.7
+VERSION --global-cache 0.7
 
 deps:
     FROM node:20-bookworm

actions/run/Earthfile

@@ -1,4 +1,4 @@
-VERSION 0.7
+VERSION --global-cache 0.7
 
 deps:
     FROM node:20-bookworm

cli/Earthfile

@@ -1,4 +1,4 @@
-VERSION 0.7
+VERSION --global-cache 0.7
 FROM golang:1.20-alpine3.18
 
 # cspell: words onsi ldflags extldflags

docs/Earthfile

@@ -1,4 +1,4 @@
-VERSION 0.7
+VERSION --global-cache 0.7
 
 # Copy all the source we need to build the docs
 src:
@@ -8,6 +8,8 @@ src:
     # Now copy into that any artifacts we pull from the builds.
     COPY --dir ../+repo-docs/repo includes
 
+    # Copy docs we build in the postgres example.
+    COPY --dir ../examples/postgresql+build/docs src/appendix/examples/built_docs/postgresql
 
 # Build the docs here.
 docs:

docs/src/appendix/earthly.md

@@ -75,7 +75,7 @@ Like a `Dockerfile`, only a single `Earthfile` can exist per directory and it *m
 #### Sample Structure
 
 ```Earthfile
-VERSION 0.7  # This defines the "schema" that this Earthfile satisfies, much like the version of a Docker Compose file
+VERSION --global-cache 0.7  # This defines the "schema version" that this Earthfile satisfies
 
 # A target, which is functionally equivalent to a `makefile` target.
 deps:
@@ -104,7 +104,7 @@ A target can be thought of as a grouping of image layers, similar to the way mul
 Each target then specifies one or more commands that create the image layers associated with that target.
 
 ```Earthfile
-VERSION 0.7
+VERSION --global-cache 0.7
 
 deps:
     FROM golang:1.20-alpine3.18

docs/src/guides/docs.md

@@ -39,7 +39,7 @@ This folder already has an `Earthfile` in it, which contains all build process.
 <!-- markdownlint-enable max-one-sentence-per-line -->
 
 ```Earthfile
-VERSION 0.7
+VERSION --global-cache 0.7
 
 # Copy all the source we need to build the docs
 src:

docs/src/guides/languages/bash.md

@@ -0,0 +1,123 @@
+---
+icon: material/bash
+title: Bash Scripts
+tags:
+    - Bash
+---
+
+<!-- markdownlint-disable single-h1 -->
+# :material-bash: Bash Scripts
+<!-- markdownlint-enable single-h1 -->
+
+## Introduction
+
+<!-- markdownlint-disable max-one-sentence-per-line -->
+!!! Tip
+    If you're just looking for a complete example,
+    [click here](https://github.com/input-output-hk/catalyst-ci/blob/master/Earthfile).
+    This guide will provide detailed instructions for how the example was built.
+<!-- markdownlint-enable max-one-sentence-per-line -->
+
+This guide will get you started with using the Catalyst CI to build projects that include Bash scripts.
+By the end of the guide, we'll have an `Earthfile` that utilizes the Catalyst CI that can check your Bash scripts.
+
+Bash is not considered a stand alone target,  although bash scripts are used extensively across multiple targets.
+The Bash support consists solely of a single repo wide `check` target which validates:
+
+* Are any of the `bash` shell scripts redundant.
+  * This prevent maintenance issues where common scripts are copy/pasted rather than being properly organized.
+* Do the bash scripts pass `shellcheck` lints.
+  * This forces us to follow a consistent style guide, and also checks for problematic Bash syntax.
+
+To begin, clone the Catalyst CI repository:
+
+## Adding Bash checks to your Repo that is already using Catalyst-CI
+
+Bash script checking is to be added to a repo that is already using Catalyst CI.
+
+All that needs to happen is the following be added to the `Earthfile` in the root of the repo.
+
+```Earthfile
+# Internal: shell-check - test all bash files against our shell check rules.
+shell-check:
+    FROM alpine:3.18
+
+    DO github.com/input-output-hk/catalyst-ci/earthly/bash:vx.y.z+SHELLCHECK --src=.
+
+# check all repo wide checks are run from here
+check:
+    FROM alpine:3.18
+
+    # Lint all bash files.
+    BUILD +shell-check
+
+```
+
+<!-- markdownlint-disable max-one-sentence-per-line -->
+!!! Note
+    It is expected that there may be multiple repo level `checks`.
+    This pattern shown above allows for this by separating the individual checks into their own targets.
+    The `check` target then just executed `BUILD` once for each check.
+<!-- markdownlint-enable max-one-sentence-per-line -->
+
+### Common Scripts
+
+It is not a good practice to copy bash scripts with common functionality.
+Accordingly, the *Utility* target `./utilities/scripts+bash-scripts` exists to provide a central location for common scripts.
+These are used locally to this repo and may be used by other repos using catalyst-ci.
+
+These scripts are intended to be used inside Earthly builds, and not locally.
+
+A common pattern to include these common scripts is the following:
+
+```Earthfile
+    # Copy our target specific scripts
+    COPY --dir scripts /scripts
+
+    # Copy our common scripts so we can use them inside the container.
+    DO ../../utilities/scripts+ADD_BASH_SCRIPTS
+```
+
+<!-- markdownlint-disable max-one-sentence-per-line -->
+!!! Note
+    Always source scripts using `source "/scripts/include/something.sh"`.
+    This will ensure the scripts are properly located.
+    bash has no concept of the directory a script is located and so relative
+    source commands are unreliable.
+<!-- markdownlint-enable max-one-sentence-per-line -->
+
+<!-- markdownlint-disable max-one-sentence-per-line -->
+!!! Note
+    This is just an example, and you would adapt it to your specific requirements.
+<!-- markdownlint-enable max-one-sentence-per-line -->
+
+### Running checks
+
+From the root of the repo, you can check all bash scripts within the repo by running:
+
+```sh
+earthly +check
+```
+
+This will also run all other repo-wide checks that are in use.
+
+### Build and test
+
+Bash scripts should not have a `build` target.
+They can form part of the Build of other targets.
+
+### Releasing
+
+Bash scripts should not have a discreet `release` target.
+They can form part of the `release` of other targets.
+
+### Publishing
+
+Bash scripts should not have a discreet `publish` target.
+They can form part of the `publish` of other targets.
+
+## Conclusion
+
+You can see the final `Earthfile` [here](https://github.com/input-output-hk/catalyst-ci/blob/master/Earthfile).
+
+This `Earthfile` will check the quality of the Bash files within the Catalyst-CI repo.

docs/src/guides/languages/go.md

@@ -46,7 +46,7 @@ You can choose to either delete the file and start from scratch, or read the gui
 ### Installing dependencies
 
 ```Earthfile
-VERSION 0.7
+VERSION --global-cache 0.7
 
 deps:
     # This target is used to install external Go dependencies.

docs/src/guides/languages/postgresql.md

@@ -49,7 +49,7 @@ You can choose to either delete the file and start from scratch, or read the gui
 ### Prepare base builder
 
 ```Earthfile
-VERSION 0.7
+VERSION --global-cache 0.7
 
 builder:
     FROM ./../../earthly/postgresql+postgres-base
@@ -75,9 +75,14 @@ check:
 
     DO ./../../earthly/postgresql+CHECK
 
+build-sqlfluff:
+    BUILD ./../../earthly/postgresql+sqlfluff-image   
+
 format:
     LOCALLY
 
+    RUN earthly +build-sqlfluff
+
     DO ./../../earthly/postgresql+FORMAT --src=$(echo ${PWD})
 ```
 
@@ -86,12 +91,6 @@ At this step we can begin performing static checks against `*.sql` files.
 These checks are intended to verify the code is healthy and well formatted to a certain standard
 and done with the help of the `sqlfluff` tool which is already configured during the `+postgres-base` target.
 
-<!-- markdownlint-disable max-one-sentence-per-line -->
-!!! Note
-    Before perform formatting it is needed to build `sqlfluff-image` docker image using following command
-    `earthly github.com/input-output-hk/catalyst-ci/earthly/postgresql+sqlfluff-image`
-<!-- markdownlint-enable max-one-sentence-per-line -->
-
 To apply and fix some formatting issues you can run `+format` target which will picks up directory
 where your Earthly file lies in as a source dir for formatting and run `+FORMAT` UDC target.
 Under the hood `+FORMAT` UDC target runs `sqlfluff-image` docker image,

docs/src/guides/languages/python.md

@@ -45,7 +45,7 @@ or read the guide and follow along in the file.
 ### Prepare base builder
 
 ```Earthfile
-VERSION 0.7
+VERSION --global-cache 0.7
 
 builder:
     FROM ./../../earthly/python+python-base

docs/src/guides/languages/rust.md

@@ -49,7 +49,7 @@ Also we will take a look how we are setup Rust projects and what configuration i
 ### Prepare base builder
 
 ```Earthfile
-VERSION 0.7
+VERSION --global-cache 0.7
 
 # Set up our target toolchains, and copy our files.
 builder:

docs/src/style/index.md

@@ -19,7 +19,7 @@ Any `Earthfile` which does not adhere to this style guide will be rejected if no
 The following structure should be used to provide a consistent structure to `Earthfile`s:
 
 ```Earthfile
-VERSION 0.7  # Should be the same across the repository
+VERSION --global-cache 0.7  # Should be the same across the repository
 
 deps:
     FROM <base image>
@@ -87,7 +87,7 @@ This target is made up of the commands that appear outside of an existing target
 For example:
 
 ```Earthfile
-VERSION 0.7
+VERSION --global-cache 0.7
 FROM ubuntu:latest  # Apart of the base target
 WORKDIR /work  # Apart of the base target
 ```
@@ -104,7 +104,7 @@ As such, the base target should be avoided, and individual targets should be
 clear about their intentions:
 
 ```Earthfile
-VERSION 0.7
+VERSION --global-cache 0.7
 
 deps:
     FROM ubuntu:latest