Merge pull request #17 from SmartColumbusOS/external_module_support

Support pre-defined module compose stacks

README.md

@@ -9,9 +9,9 @@ If [available in Hex](https://hex.pm/docs/publish), the package can be installed
 by adding `divo` to your list of dependencies in `mix.exs`:
 
 ```elixir
-def deps do
+def deps() do
   [
-    {:divo, "~> 1.0.2", only: [:dev, :integration], organization: "smartcolumbus_os"}
+    {:divo, "~> 1.1.0", only: [:dev, :integration], organization: "smartcolumbus_os"}
   ]
 end
 ```
@@ -23,9 +23,34 @@ be found at [https://hexdocs.pm/divo](https://hexdocs.pm/divo).
 ## Usage
 
 Define services in your mix configuration file (typically under a `config/integration.exs` file)
-to define the dockerized service(s) you want to run as a dependency of your Elixir app.
+to define the dockerized service(s) you want to run as a dependency of your Elixir app. Define divo config in one of three ways"
 
+### Method 1 - Pre-existing definition from a behaviour-derived module stack
+In your mix file, include the additional dependency
+```elixir
+def deps() do
+  [
+    {:divo, "~> 1.1.0", only: [:dev, :integration], organization: "smartcolumbus_os"},
+    {:divo_redis, "~> 0.1.0", only: [:dev, :integration], organization: "smartcolumbus_os"}
+  ]
 ```
+And in your environment config, include the imported dependency module(s) as a list of tuples along with any environment variables the stack takes as a keyword list
+```elixir
+config :myapp,
+  divo: [
+    {DivoRedis, [initial_key: "myapp:secret"]}
+  ]
+```
+
+### Method 2 - Pre-existing definition from a supplied compose file on the file system
+In your environment config, include the path to the yaml- or json-formatted compose file
+```elixir
+config :myapp,
+  divo: "test/support/docker-compose.yaml
+```
+
+### Method 3 - Define the custom compose file as an elixir map directly in your config
+```elixir
 config :myapp,
   divo: [
     kafka: %{
@@ -51,47 +76,19 @@ config :myapp,
   ]
 ```
 
-## Options
-
-These options can be added to a service definition's config to customize it.
-
-* `image` - the name of the docker image to be started. This is the only required key.
-
-* `env` - a keyword list of environment variables to be set in the started container. Each element is translated to a `--env=VARIABLE=VALUE` option in the run command.
-
-* `ports` - a list of ports to be exposed to the system. Each element is translated to a `--publish=LOCAL:REMOTE` option in the run command.
-
-* `volumes` - a list of tuples of the format `{local_volume, remote_volume}`. Each element is translated to a `--volume=LOCAL:REMOTE` option in the run command
+## Compose File Definition
 
-* `command` - a command to be run in the created container. Does not support piping `ls | grep logs` or additional commands `ls && cd ..`
+Docker Compose instructions are passed to the `docker-compose` binary on your Docker engine host as either a yaml- or json-formatted document with maps defining the container services, networks, and volumes needed to run the services as an interconnected stack of components (as well as a compose file version). The keys in the underlying map structure generally have a one-to-one relationship to the various arguments available to the `docker run` command.
 
-* `net` - a different service defined by Divo that this container needs to be linked to. Translated to `--network=container:APP_NAME-SERVICE_NAME`
+For more details, see the full [docker compose documentation](https://docs.docker.com/compose/compose-file/)
 
-* `additional_opts` - a list of strings representing extra options to be passed to `docker run`. This allows for options not explicitly supported by Divo to be used if needed. Any option defined in the (Docker Run)[https://docs.docker.com/engine/reference/commandline/run/] docs can be used.
-
-* `wait_for` - see [Wait For]()
-
-
-## Wait For
+## Divo Wait
 
 Sometimes services take a moment to start up and Elixir apps tend to start (and attempt to run their tests)
-too quickly for their dependencies to be ready. For those situations, add the key `:wait_for` to the map
-that defines each services that will need to be fully initialized before accepting interactions from your
-service-under-test. The value of that key should be a map containing a log message to expect from the service
-indicating it is ready to accept requests, a interval in seconds to wait between log parsing attempts, and a
-number of retries to make the attempt.
+too quickly for their dependencies to be ready. For those situations, add the key `:divo_wait` to the app config that defines a wait period in milliseconds and a maximum number of tries to check for the containerized services to be healthy before aborting. In order for the wait to hold execution for the containers to register as healthy with the Docker engine, a [healthcheck](https://docs.docker.com/compose/compose-file/#healthcheck) must be built into the Dockerfile for the image or defined in the compose file.
 
-```
-...
-  kafka: %{
-    image: ...,
-    env: [
-      ...
-    ],
-    wait_for: %{
-        log: "home",
-        dwell: 400,
-        max_retries: 10
-      }
-  }
+```elixir
+config :myapp,
+  divo: "test/support/docker-compose.yaml",
+  divo_wait: [dwell: 700, max_tries: 50]
 ```

lib/divo/compose.ex

@@ -12,6 +12,12 @@ defmodule Divo.Compose do
   require Logger
   alias Divo.{File, Helper, Validate}
 
+  @doc """
+  Builds and/or validates the compose file and executes the `docker-compose up`
+  call to start the entirety of the defined stack or a subset of the services
+  defined in the stack based on supplying an optional list of service keys.
+  """
+  @spec run(keyword()) :: none()
   def run(opts \\ []) do
     services = get_services(opts)
 
@@ -21,10 +27,22 @@ defmodule Divo.Compose do
     await()
   end
 
+  @doc """
+  Builds and/or validates the compose file and executes the `docker-compose stop`
+  call to stop the containerized services without removing the resources created
+  by the compose file.
+  """
+  @spec stop() :: none()
   def stop() do
     execute("stop")
   end
 
+  @doc """
+  Builds and/or validates the compose file and executes the `docker-compose down`
+  call to stop the containerized services and removes all resources created by
+  the compose file such as containers, networks, and volumes.
+  """
+  @spec kill() :: none()
   def kill() do
     execute("down")
   end

lib/divo/file.ex

@@ -7,6 +7,12 @@ defmodule Divo.File do
   require Logger
   alias Divo.Helper
 
+  @doc """
+  Returns the name of the compose file to run, either as a
+  pass-through from an existing compose file or the path of
+  the file dynamically created by divo.
+  """
+  @spec file_name() :: String.t()
   def file_name() do
     app = Helper.fetch_name()
 
@@ -16,16 +22,36 @@ defmodule Divo.File do
     end
   end
 
+  @doc """
+  Passes through the file name when the compose file is
+  pre-existing and supplied via file system path or builds
+  and writes a dynamic compose file to a temp directory before
+  returning the path to that temp file.
+  """
+  @spec ensure_file(String.t() | [tuple()] | map()) :: String.t()
   def ensure_file(app_config) when is_binary(app_config) do
     Logger.info("Using : #{app_config}")
 
     app_config
   end
 
+  def ensure_file(app_config) when is_list(app_config) do
+    file = file_name()
+
+    Logger.info("Generating : #{file} from stack module")
+
+    app_config
+    |> Divo.Stack.concat_compose()
+    |> Jason.encode!()
+    |> write(file)
+
+    file
+  end
+
   def ensure_file(app_config) when is_map(app_config) do
     file = file_name()
 
-    Logger.info("Generating : #{file}")
+    Logger.info("Generating : #{file} from map")
 
     app_config
     |> Jason.encode!()

lib/divo/helper.ex

@@ -5,10 +5,20 @@ defmodule Divo.Helper do
   commands.
   """
 
+  @doc """
+  Returns the name of the calling app from the mix config.
+  """
+  @spec fetch_name() :: atom()
   def fetch_name() do
     Mix.Project.config()[:app]
   end
 
+  @doc """
+  Returns the configuration for divo from the environment config
+  exs file that defines the container services to run or the path
+  to the config given an existing compose file.
+  """
+  @spec fetch_config() :: map() | String.t() | [tuple()]
   def fetch_config() do
     with {:ok, config} <- Application.fetch_env(fetch_name(), :divo) do
       config

lib/divo/stack.ex

@@ -0,0 +1,39 @@
+defmodule Divo.Stack do
+  @moduledoc """
+  Implements a behaviour for importing
+  predefined compose stacks from external
+  library complementary to divo for quickly
+  standing up well-defined services with
+  little variation in configuration.
+  """
+
+  @doc """
+  Defines the behaviour that must be implemented to
+  supply configs from external modules to divo. The
+  configuration values are expected as a keyword list
+  of attributes specific to each module adopting the
+  behaviour.
+  """
+  @callback gen_stack(keyword()) :: {atom(), map()}
+
+  @doc """
+  Iterates over modules supplied in the app :divo
+  config, calling the `gen_stack` function on each
+  module's implementation of the behavior, collecting
+  the resulting maps into a single map and passing
+  the accumulated result out for writing out the file.
+  """
+  @spec concat_compose([tuple()]) :: map()
+  def concat_compose(config) do
+    compose_file = %{version: "3.4"}
+
+    services =
+      config
+      |> Enum.map(fn {module, envars} ->
+        apply(module, :gen_stack, [envars])
+      end)
+      |> Enum.reduce(%{}, fn service, acc -> Map.merge(service, acc) end)
+
+    Map.put(compose_file, :services, services)
+  end
+end

lib/divo/validate.ex

@@ -6,6 +6,11 @@ defmodule Divo.Validate do
   """
   require Logger
 
+  @doc """
+  Wraps the function of `docker-compose` to validate the structure of
+  the compose file for correct format/syntax and required keys are supplied.
+  """
+  @spec validate(binary()) :: none()
   def validate(file) do
     System.cmd("docker-compose", ["--file", file, "config"], stderr_to_stdout: true)
     |> case do

mix.exs

@@ -4,7 +4,7 @@ defmodule Divo.MixProject do
   def project do
     [
       app: :divo,
-      version: "1.0.2",
+      version: "1.1.0",
       elixir: "~> 1.8",
       start_permanent: Mix.env() == :prod,
       deps: deps(),
@@ -26,10 +26,10 @@ defmodule Divo.MixProject do
     [
       {:credo, "~> 1.0", only: [:dev, :test], runtime: false},
       {:jason, "~> 1.1"},
-      {:placebo, "~> 1.2.1", only: [:dev, :test]},
+      {:placebo, "~> 1.2", only: [:dev, :test]},
       {:ex_doc, "~> 0.19", only: :dev},
-      {:patiently, "~> 0.2.0"},
-      {:temporary_env, "~> 2.0.1", only: [:dev, :test]}
+      {:patiently, "~> 0.2"},
+      {:temporary_env, "~> 2.0", only: [:dev, :test]}
     ]
   end
 

mix.lock

@@ -1,7 +1,7 @@
 %{
   "bunt": {:hex, :bunt, "0.2.0", "951c6e801e8b1d2cbe58ebbd3e616a869061ddadcc4863d0a2182541acae9a38", [:mix], [], "hexpm"},
-  "credo": {:hex, :credo, "1.0.2", "88bc918f215168bf6ce7070610a6173c45c82f32baa08bdfc80bf58df2d103b6", [:mix], [{:bunt, "~> 0.2.0", [hex: :bunt, repo: "hexpm", optional: false]}, {:jason, "~> 1.0", [hex: :jason, repo: "hexpm", optional: false]}], "hexpm"},
-  "earmark": {:hex, :earmark, "1.3.1", "73812f447f7a42358d3ba79283cfa3075a7580a3a2ed457616d6517ac3738cb9", [:mix], [], "hexpm"},
+  "credo": {:hex, :credo, "1.0.4", "d2214d4cc88c07f54004ffd5a2a27408208841be5eca9f5a72ce9e8e835f7ede", [:mix], [{:bunt, "~> 0.2.0", [hex: :bunt, repo: "hexpm", optional: false]}, {:jason, "~> 1.0", [hex: :jason, repo: "hexpm", optional: false]}], "hexpm"},
+  "earmark": {:hex, :earmark, "1.3.2", "b840562ea3d67795ffbb5bd88940b1bed0ed9fa32834915125ea7d02e35888a5", [:mix], [], "hexpm"},
   "ex_doc": {:hex, :ex_doc, "0.19.3", "3c7b0f02851f5fc13b040e8e925051452e41248f685e40250d7e40b07b9f8c10", [:mix], [{:earmark, "~> 1.2", [hex: :earmark, repo: "hexpm", optional: false]}, {:makeup_elixir, "~> 0.10", [hex: :makeup_elixir, repo: "hexpm", optional: false]}], "hexpm"},
   "jason": {:hex, :jason, "1.1.2", "b03dedea67a99223a2eaf9f1264ce37154564de899fd3d8b9a21b1a6fd64afe7", [:mix], [{:decimal, "~> 1.0", [hex: :decimal, repo: "hexpm", optional: true]}], "hexpm"},
   "makeup": {:hex, :makeup, "0.8.0", "9cf32aea71c7fe0a4b2e9246c2c4978f9070257e5c9ce6d4a28ec450a839b55f", [:mix], [{:nimble_parsec, "~> 0.5.0", [hex: :nimble_parsec, repo: "hexpm", optional: false]}], "hexpm"},

test/file_test.exs

@@ -26,8 +26,38 @@ defmodule Divo.FileTest do
   test "generates compose file from app config" do
     allow(File.write!(any(), any()), return: :ok)
     allow(System.get_env("TMPDIR"), return: "/var/tmp/foo")
-    file = Divo.Helper.fetch_config()
+    config = Divo.Helper.fetch_config()
 
-    assert Divo.File.ensure_file(file) == "/var/tmp/foo/divo.compose"
+    assert Divo.File.ensure_file(config) == "/var/tmp/foo/divo.compose"
+  end
+
+  test "generates compose file from a behaviour implementation" do
+    allow(File.write!(any(), any()), return: :ok)
+    allow(System.get_env("TMPDIR"), return: "/var/tmp/bar")
+
+    services = [{DivoBarbaz, []}]
+
+    TemporaryEnv.put :divo, :divo, services do
+      config = Divo.Helper.fetch_config()
+
+      assert Divo.File.ensure_file(config) == "/var/tmp/bar/divo.compose"
+    end
+  end
+end
+
+defmodule DivoBarbaz do
+  @behaviour Divo.Stack
+
+  @impl Divo.Stack
+  def gen_stack(_envars) do
+    %{
+      barbaz: %{
+        image: "library/barbaz",
+        healthcheck: %{
+          test: ["CMD-SHELL", "/bin/true || exit 1"]
+        },
+        ports: ["2345:2345", "7777:7777"]
+      }
+    }
   end
 end

test/stack_test.exs

@@ -0,0 +1,57 @@
+defmodule Divo.StackTest do
+  use ExUnit.Case
+  require TemporaryEnv
+
+  test "behaviour returns the service definition" do
+    services = [
+      {DivoFoobar, [db_password: "we-are-divo", db_name: "foobar-db", something: "else"]}
+    ]
+
+    expected = %{
+      version: "3.4",
+      services: %{
+        foobar: %{
+          image: "foobar:latest",
+          ports: ["8080:8080"],
+          command: ["/bin/server", "foreground"]
+        },
+        db: %{
+          image: "cooldb:5.0.9",
+          environment: [
+            "PASSWORD=we-are-divo",
+            "DB=foobar-db"
+          ]
+        }
+      }
+    }
+
+    actual = Divo.Stack.concat_compose(services)
+
+    assert expected == actual
+  end
+end
+
+defmodule DivoFoobar do
+  @behaviour Divo.Stack
+
+  @impl Divo.Stack
+  def gen_stack(envars) do
+    password = envars[:db_password]
+    db_name = envars[:db_name]
+
+    %{
+      foobar: %{
+        image: "foobar:latest",
+        ports: ["8080:8080"],
+        command: ["/bin/server", "foreground"]
+      },
+      db: %{
+        image: "cooldb:5.0.9",
+        environment: [
+          "PASSWORD=#{password}",
+          "DB=#{db_name}"
+        ]
+      }
+    }
+  end
+end