add sops support (#107)

README.md

@@ -53,7 +53,7 @@ Install it with the [ArgoCD community chart](https://github.com/argoproj/argo-he
       extraContainers:
       - name: cmp-subst
         args: [/var/run/argocd/argocd-cmp-server]
-        image: ghcr.io/buttahtoast/subst-cmp:v0.3.0-alpha1
+        image: ghcr.io/buttahtoast/subst-cmp:v0.3.0
         imagePullPolicy: Always
         securityContext:
           allowPrivilegeEscalation: false
@@ -86,16 +86,6 @@ Install it with the [ArgoCD community chart](https://github.com/argoproj/argo-he
 
 Change version accordingly.
 
-
-
-
-
-
-**Applications in Projects**
-
-If an application is in a project, the value of `$ARGOCD_APP_NAME` looks like this: `<project-name>_<application-name>`. For example, if the application `my-app` is in the project `my-project`, the value of `$ARGOCD_APP_NAME` is `my-project_my-app`. 
-All special characters within are converted to `-` (dash). For example, if the application `my-app` is in the project `my-project`, the value of `$ARGOCD_APP_NAME` is `my-project-my-app`. So the secret reference is then `my-project-my-app` in the secret namespace.
-
 ## Available Substitutions
 
 You can display which substitutions are available for a kustomize build by running:
@@ -145,30 +135,38 @@ Note that directories do not resolve by recursion (eg. `/test/build/` only colle
 
 for environment variables which come from an argo application (`^ARGOCD_ENV_`) we remove the `ARGOCD_ENV_` and they are then available in your substitutions without the `ARGOCD_ENV_` prefix. This way they have the same name you have given them on the application ([Read More](https://argo-cd.readthedocs.io/en/stable/operator-manual/config-management-plugins/#using-environment-variables-in-your-plugin)). All the substions are available as flat key, so where needed you can use environment substitution.
 
-### Substitution (Envsubst)
-
-Subst does not support Environment Substitution. However we have a transformator, which allows to perform environment substitutions (outside of the subst context). [Read More](https://github.com/buttahtoast/transformers/tree/main/transformers/substitution)
-
 ## Spruce
 
 [Spruce](https://github.com/geofffranks/spruce) is used to access the substition variables, it has more flexability than [envsubst](#environment-substitution). You can grab values from the available substitutions using [Spruce Operators](https://github.com/geofffranks/spruce/blob/main/doc/operators.md). Spurce is greate, because it's operators are valid YAML which allows to build the kustomize without any further hacking.
 
 ## Secrets
 
-You can both encrypt files which are part of the kustomize build or which are used for substitution. Currently for secret decryption we support both [ejson](https://github.com/Shopify/ejson) and [sops](https://github.com/mozilla/sops). You can use any combination of these decryption providers together. The principal for all decryption provider is, that they should load the private keys while a substiution build is made instead of having a permanent keystore. This allows for secret tenancy (eg. one secret per argo application). 
+You can both encrypt files which are part of the kustomize build or which are used for substitution. Currently for secret decryption we support both [ejson](https://github.com/Shopify/ejson) and [sops](https://github.com/mozilla/sops). You can use any combination of these decryption providers together. The principal for all decryption provider is, that they should load the private keys while a substiution build is made instead of having a permanent keystore. This allows for secret tenancy (eg. one secret per argo application). The private keys are loaded from kubernetes secrets, therefor the plugin also creates it's own kubeconfig. 
 
-Decryption can be disabled, in that case the files are just loaded, without their encryption properties (might be useful if you dont have access to the private keys to decrypt the secrets):
+The secrets are loaded based on the environment variables `$ARGOCD_APP_NAME` and `$ARGOCD_APP_NAMESPACE` are used. If an application is in a project, the value of `$ARGOCD_APP_NAME` looks like this: `<project-name>_<application-name>`. For example, if the application `my-app` is in the project `my-project`, the value of `$ARGOCD_APP_NAME` is `my-project_my-app`. All special characters within are converted to `-` (dash). For example, if the application `my-app` is in the project `my-project`, the value of `$ARGOCD_APP_NAME` is `my-project-my-app`. So the secret reference is then `my-project-my-app` in the secret namespace (Assuming `--convert-secret-name=false`). 
+
+By default the `--convert-secret-name` is enabled. This removes the project prefix from the secret. If you create an application `test` in the namespace `test-reserved` the plugin is looking for private keys in the  secret `test` in the namespace `test-reserved`. The is not considered in this approach which helps endusers to keep it simple.
+
+The values for the secret name and namespace can also be set constant, however this way you lose the multi-tenancy aspect of the secrets management:
 
 ```
-subst render . --skip-decrypt
+subst render --secret-name static-name --secret-namespace static-namespace .
+```
+
+You can disable the lookup of the private keys in Kubernetes secrets. This is useful if you want to use the substition without access to the kubernetes clusters. The decryption providers allow to enter the private keys directly. This is useful for CI/CD pipelines or local testing (See decryption provider documentation).
+
+```
+subst render . --skip-secret-lookup
 ```
 
-Decryption can be enforced, if a secret can not be decrypted it's treated as an error (recommended):
+Decryption can be disabled, in that case the files are just loaded, without their encryption properties (might be useful if you dont have access to the private keys to decrypt the secrets):
 
 ```
-subst render . --must-decrypt
+subst render . --skip-decrypt
 ```
 
+See below how to work with the different decryption providers.
+
 ### EJSON
 
 [EJSON](https://github.com/Shopify/ejson) allows simple secrets management. I like it, because you can rencrypt secrets without having the private key, which is sometimes useful. 

go.mod

@@ -2,6 +2,8 @@ module github.com/buttahtoast/subst
 
 go 1.19
 
+replace github.com/buttahtoast/pkg/decryptors => /Users/pariah/Projects/pkg/decryptors
+
 require (
 	github.com/BurntSushi/toml v1.2.1
 	github.com/MakeNowJust/heredoc v1.0.0

pkg/config/config.go

@@ -18,6 +18,7 @@ type Configuration struct {
 	EnvRegex          string        `mapstructure:"env-regex"`
 	RootDirectory     string        `mapstructure:"root-dir"`
 	FileRegex         string        `mapstructure:"file-regex"`
+	SecretSkip        bool          `mapstructure:"secret-skip"`
 	SecretName        string        `mapstructure:"secret-name"`
 	SecretNamespace   string        `mapstructure:"secret-namespace"`
 	EjsonKey          []string      `mapstructure:"ejson-key"`
@@ -27,6 +28,8 @@ type Configuration struct {
 	KubeAPI           string        `mapstructure:"kube-api"`
 	Output            string        `mapstructure:"output"`
 	ConvertSecretname bool          `mapstructure:"convert-secret-name"`
+	SopSKeyring       string        `mapstructure:"sops-keyring"`
+	SopsTempKeyring   bool          `mapstructure:"sops-temp-keyring"`
 }
 
 func LoadConfiguration(cfgFile string, cmd *cobra.Command, directory string) (*Configuration, error) {

pkg/subst/build.go

@@ -20,7 +20,6 @@ type Build struct {
 	Kustomization *kustomize.Kustomize
 	Substitutions *Substitutions
 	cfg           config.Configuration
-	decryptors    []decrypt.Decryptor
 	kubeClient    *kubernetes.Clientset
 }
 
@@ -36,37 +35,64 @@ func New(config config.Configuration) (build *Build, err error) {
 		Kustomization: k,
 	}
 
-	err = init.initialize()
+	return init, err
+}
+
+func (b *Build) BuildSubstitutions() (err error) {
+	decryptors, cleanups, err := b.decryptors()
 	if err != nil {
-		return nil, err
+		return err
 	}
+
+	defer func() {
+		for _, cleanup := range cleanups {
+			cleanup()
+		}
+	}()
+
 	SubstitutionsConfig := SubstitutionsConfig{
-		EnvironmentRegex: init.cfg.EnvRegex,
-		SubstFileRegex:   init.cfg.FileRegex,
+		EnvironmentRegex: b.cfg.EnvRegex,
+		SubstFileRegex:   b.cfg.FileRegex,
 	}
 
-	s, err := NewSubstitutions(SubstitutionsConfig, init.decryptors, k.Build)
+	b.Substitutions, err = NewSubstitutions(SubstitutionsConfig, decryptors, b.Kustomization.Build)
 	if err != nil {
-		return nil, err
+		return err
 	}
-	init.Substitutions = s
 
-	err = init.loadSubstitutions()
+	err = b.loadSubstitutions()
 	if err != nil {
-		return nil, err
+		return err
 	}
-	return init, err
+	return nil
+
 }
 
 func (b *Build) Build() (err error) {
 
+	if b.Substitutions == nil {
+		logrus.Debug("no resources to build")
+		return nil
+	}
+
+	decryptors, cleanups, err := b.decryptors()
+	if err != nil {
+		return err
+	}
+
+	defer func() {
+		for _, cleanup := range cleanups {
+			cleanup()
+		}
+	}()
+
 	// Run Build
 	logrus.Debug("substitute manifests")
 	for _, manifest := range b.Substitutions.Resources.Resources() {
 		var c map[interface{}]interface{}
 
 		mBytes, _ := manifest.MarshalJSON()
-		for _, d := range b.decryptors {
+		for _, d := range decryptors {
 			isEncrypted, err := d.IsEncrypted(mBytes)
 			if err != nil {
 				logrus.Errorf("Error checking encryption for %s: %s", mBytes, err)
@@ -97,6 +123,7 @@ func (b *Build) Build() (err error) {
 		}
 		b.Manifests = append(b.Manifests, f)
 	}
+
 	return nil
 }
 
@@ -108,9 +135,6 @@ func (b *Build) loadSubstitutions() (err error) {
 	if err != nil {
 		return err
 	}
-	logrus.Debug("subtitution files loaded")
-
-	logrus.Debug("loaded substitutions: ", b.Substitutions.Subst)
 
 	// Final attempt to evaluate
 	eval, err := b.Substitutions.Eval(b.Substitutions.Subst, nil, false)
@@ -129,22 +153,32 @@ func (b *Build) loadSubstitutions() (err error) {
 }
 
 // initialize decryption
-func (b *Build) initialize() (err error) {
+func (b *Build) decryptors() (decryptors []decrypt.Decryptor, cleanups []func(), err error) {
 
 	c := decrypt.DecryptorConfig{
 		SkipDecrypt: b.cfg.SkipDecrypt,
 	}
 
 	ed, err := ejson.NewEJSONDecryptor(c, "", b.cfg.EjsonKey...)
 	if err != nil {
-		return err
+		return nil, nil, err
 	}
-	b.decryptors = append(b.decryptors, ed)
-	sd := sops.NewSOPSDecryptor(c, "")
-	if err != nil {
-		return err
+	decryptors = append(decryptors, ed)
+
+	if b.cfg.SopsTempKeyring {
+		sd, sopsCleanup, err := sops.NewSOPSTempDecryptor(c)
+		if err != nil {
+			return nil, nil, err
+		}
+		cleanups = append(cleanups, sopsCleanup)
+		decryptors = append(decryptors, sd)
+	} else {
+		decryptors = append(decryptors, sops.NewSOPSDecryptor(c, b.cfg.SopSKeyring))
+	}
+
+	if b.cfg.SecretSkip {
+		return
 	}
-	b.decryptors = append(b.decryptors, sd)
 
 	if !b.cfg.SkipDecrypt && (b.cfg.SecretName != "" && b.cfg.SecretNamespace != "") {
 
@@ -159,7 +193,7 @@ func (b *Build) initialize() (err error) {
 				logrus.Warnf("could not load kubernetes client: %s", err)
 			} else {
 				ctx := context.Background()
-				for _, decr := range b.decryptors {
+				for _, decr := range decryptors {
 					err = decr.KeysFromSecret(b.cfg.SecretName, b.cfg.SecretNamespace, b.kubeClient, ctx)
 					if err != nil {
 						logrus.Warnf("failed to load secrets from Kubernetes: %s", err)
@@ -170,5 +204,5 @@ func (b *Build) initialize() (err error) {
 		}
 	}
 
-	return nil
+	return
 }

subst/cmd/render.go

@@ -41,13 +41,19 @@ func addRenderFlags(flags *flag.FlagSet) {
 	}
 	flags.Bool("convert-secret-name", true, heredoc.Doc(`
 			Assuming the secret name is derived from ARGOCD_APP_NAME, this option will only use the application name (without project-name_)`))
+	flags.Bool("skip-secret-lookup", false, heredoc.Doc(`
+		Skip reading from decryption keys from Secret`))
 	flags.String("secret-name", "", heredoc.Doc(`
 	        Specify Secret name (each key within the secret will be used as a decryption key)`))
 	flags.String("secret-namespace", "", heredoc.Doc(`
 	        Specify Secret namespace`))
 	flags.StringSlice("ejson-key", []string{}, heredoc.Doc(`
 			Specify EJSON Private key used for decryption.
 			May be specified multiple times or separate values with commas`))
+	flags.String("sops-keyring", "", heredoc.Doc(`
+	        Path to local GPG keyring`))
+	flags.Bool("sops-temp-keyring", true, heredoc.Doc(`
+			Creates for each execution a dedicated keyring which is automatically deleted after execution. If false, uses the default keyring`))
 	flags.Bool("skip-decrypt", false, heredoc.Doc(`
 			Skip decryption`))
 	flags.String("env-regex", "^ARGOCD_ENV_.*$", heredoc.Doc(`
@@ -72,6 +78,11 @@ func render(cmd *cobra.Command, args []string) error {
 		return err
 	}
 
+	err = m.BuildSubstitutions()
+	if err != nil {
+		return err
+	}
+
 	start := time.Now() // Start time measurement
 	if m != nil {
 		err = m.Build()

subst/cmd/root.go

@@ -72,7 +72,7 @@ func setUpLogs(out io.Writer, level string) error {
 
 func addCommonFlags(flags *flag.FlagSet) {
 	flags.StringVar(&cfgFile, "config", "", "Config file")
-	flags.String("file-regex", "(.*subst\\.yaml|.*(ejson|sops))", heredoc.Doc(`
+	flags.String("file-regex", "(.*subst\\.yaml|.*(ejson))", heredoc.Doc(`
 			Regex Pattern to discover substitution files`))
 	flags.Bool("debug", false, heredoc.Doc(`
 			Print CLI calls of external tools to stdout (caution: setting this may