Dynamic blocks

[traut]

Background

The content structure in FCL templates is static at the moment. Fabric supports content filtering during rendering (#99), but users can't mutate the document structure based on data.

Requirements

• be able to create multiple documents from the context data
• be able to create multiple content and section blocks from the context data
• support jq query for producing a list to iterate over for multiple items

Design

We introduce dynamic block type. It is similar to Terraform's dynamic blocks but has a different syntax.

dynamic block works with specific block types, similar to config block: dynamic block can be applied to document, section and content blocks.

For example, dynamic document would look like this:

dynamic document "test_doc" {

  vars {
    doc_texts = [
      { title = "A", text = "a" },
      { title = "B", text = "b" }
    ]
  }

  dynamic_items_var = query_jq(".vars.doc_texts")

  # Dynamic block populates `.vars.dynamic_item` for every child block
  title = "Document {{ .vars.dynamic_item.title }}"

  # Dynamic block populates `.vars.dynamic_item_index` for every child block
  content text {
    value = "Doc text is {{ .vars.dynamic_item.text }} with index {{ .vars.dynamic_item_index }}"
  }
}

Note

Dynamic blocks clone the blocks they wrap. For example dynamic document block is document block with additional attributes, so data blocks can be defined. No nested blocks (apart from meta) can be defined in the 'dynamic content' block.

dynamic block attributes:

• dynamic_items_var -- a syntactic sugar that extends the vars block with dynamic_items variable. The variable's value will be used as a collection to iterate over while cloning.
• dynamic_condition_var -- a syntactic sugar that extends the vars block with dynamic_condition variable. The variable's value will be used to evaluate if the dynamic block should produce 0 or more than 0 results.

For example, this snippet shows how we can choose which block to render based on available data:

document test1 {

  vars {
    baz = query_jq(".data.foobar.some.x")
  }

  dynamic content text {
    dynamic_condition_var = query_jq("(.vars.baz == {} or .vars.baz == [] or .vars.baz == null) | not")

    value = "The elements: {{ .vars.baz }}"
  }
}

document test2 {

  vars {
    baz = {}
  }

  dynamic content text {
    vars {
      baz = {}
    }
    dynamic_condition_var = query_jq("(.vars.baz == {}) | not")

    value = "There are no elements"
  }
}

Behaviour

Execution flow

• if it is a dynamic document block, all data blocks and vars block are executed and set in the context
• if it is a dynamic non-document block, vars block is NOT executed
• if dynamic_condition_var is defined, it is executed.
  • if the query returns a boolean false or null, the dynamic block is skipped completely
  • if the query returns a boolean true or a non-null object, the execution continues
• if dynamic_items_var is defined, it is executed, and the blocks is created for every element of the collection in .vars.dynamic_items
• in every child non-document block, vars block is evaluated

Context for produced blocks

Dynamic block extends the context available for produced blocks with these variables:

• .vars.dynamic_item -- the current element of the collection
• .vars.dynamic_item_index -- the index of the current element of the collection

If the dynamic block has another dynamic block nested inside (if section is wrapped in dynamic block), .vars.dynamic_* variables are overwritten by the values for the inner block.

Naming

If the dynamic block is named, the names of produced blocks are suffixed with the .vars.dynamic_item_index value, for example, section.foo[<.vars.dynamic_item_index value>] as in section.foo[1]

If the block is unnamed, it's given a random name, and the names of the copies are suffixed too: for example, section.24a086f[<.vars.dynamic_item_index value>] as in section.24a086f[0]

Example

Dynamic section and dynamic content block:

document "test3" {

  vars {
    doc_texts = [
      { title = "A", text = "a" },
      { title = "B", text = "b" }
    ]
  }

  title = "Parent doc with {{ len .vars.doc_texts }} docs"

  dynamic section {
    dynamic_items_var = query_jq(".vars.doc_texts")

    title = "Section about document {{ .vars.dynamic_item.title }}"

    content text {
      value = "Doc text is {{ .vars.dynamic_item.text }} with index {{ .vars.dynamic_item_index }}"
    }

    dynamic content text {
      dynamic_items_var = ["foo", "bar"]

      vars {
        title = query_jq(".vars.doc_texts[.dynamic_item_index].title")
      }
      # Using local `.vars.dynamic_item_index` to select a title from `.vars.doc_texts` defined in a parent
      value = "Dynamic text: item={{ .vars.dynamic_item }}, index={{ .vars.dynamic_item_index }}, title={{ .vars.title }}"
    }
  }
}

renders into

# Parent doc with 2 docs

## Section about document A

Doc text is a with index 0

Dynamic text: item=foo, index=0, title=A

Dynamic text: item=bar, index=1, title=B

## Section about document B

Doc text is b with index 1

Dynamic text: item=foo, index=0, title=A

Dynamic text: item=bar, index=1, title=B

The ref blocks should still be supported:

dynamic section ref {
  dynamic_items_var = query_jq(".vars.doc_texts")

  base = section.foo
}

and #29 is supported as well.

Dynamic documents

The behavior of the dynamic document blocks differs slightly from the behavior of other dynamic blocks.

dynamic document "test_doc" {

  # Normal data definition. This data might be used for `dynamic_items_var`, so
  # the data blocks must be executed _before_ `dynamic_items_var` is evaluated
  data elasticsearch "alerts" {
    index = "test"
  }

  # Vars defined on the root level are at the highest level inline data can be defined in the template.
  # The dynamic document block can not inherit `vars` from somewhere else.
  # This means we need to treat `vars` block the same way we treat `data` blocks in the template:
  # both `data` blocks and `vars` block must be executed before dynamic clones are produced.

  vars {
     foo = ".vars.dynamic_item"  # null since there are no vars or `.vars.dynamic_item`
  }

  # Dynamic block attributes
  dynamic_items_var = query_jq(".data.elasticsearch.alerts.hits.hits")

  title = "Document {{ .vars.dynamic_item.title }}"

  content text {
    value = "Doc text is {{ .vars.dynamic_iteam.text }} with index {{ .vars.dynamic_item_index }}, foo={{ .vars.foo }}"
  }
}

renders into 2 separate documents:

# Document A

Doc text is a with index 0, foo=null

and

# Document B

Doc text is b with index 1, foo=null

CLI issues

• render CLI command should support dynamic.document.<document-name> as a target
• data CLI command should support dynamic.document.<document-name>.data.<...> as a target

The behavior of publish / output flags in CLI should be adjusted in dynamic blocks as the target introduces ambiguity.

References

• Support for unanchored dynamic blocks hashicorp/terraform#21219
• Streamline Fabric CLI interface  #70
• Introduce publish blocks and data sinks #126
• https://github.com/hashicorp/hcl2/blob/master/ext/dynblock/
• https://developer.hashicorp.com/terraform/language/expressions/dynamic-blocks

[dobarx]

I think dedicated dynamic be a bit easier to implement and understand than embedding these parameters and functionality in selected few section & document blocks.

Not using dedicated block for this would move all content blocks to the lower level on content tree. There is no way to use this dynamic feature to create blocks on the top level of the document or on the same level where it is used.

[traut]

@dobarx I almost wrote the whole issue about dynamic blocks! I decided against it because I kept inventing the behavior that reminded me very much of the combo of section / document blocks!

The problem with a dedicated dynamic block is that we must define its behavior in all situations: Where can it be placed? How does it behave when defined inside and outside the document? If it is outside, where does the data come from? Should we allow data blocks inside dynamic block? Should we reference blocks outside dynamic block on the root level? If we allow data blocks inside dynamic blocks, it opens the door to non-document-level data blocks, which will create another set of problems. Can dynamic blocks be referenced? etc

If we take a step back, dynamic block is just a container around some content with a couple of instructions. And we already have 2 containers with defined behaviour -- document and section blocks! Extending these containers allows us to build up on all behaviours the blocks support, which is so much easier.

Not using dedicated block for this would move all content blocks to the lower level on content tree. There is no way to use this dynamic feature to create blocks on the top level of the document or on the same level where it is used.

It might need a dedicated unpacking step, indeed, where all these "dynamic" blocks are unpacked into "normal" blocks. We don't need to retain the knowledge that some nodes were originally dynamic—we can adjust the template tree and continue.

[dobarx]

Should we allow data blocks inside dynamic block? Should we reference blocks outside dynamic block on the root level? If we allow data blocks inside dynamic blocks, it opens the door to non-document-level data blocks, which will create another set of problems.

But doesn't allowing document being dynamic open the door for data blocks being also dynamic already? That inline data in given example is already dynamic. My understanding of dynamic is that we should take for_each property of it and take all children blocks of it and recreate them dynamically without executing them. Allowing any block to be dynamic.

[traut]

But doesn't allowing document being dynamic open the door for data blocks being also dynamic already?

they are dynamic in a sense that they will be copied as part of the document, but they still follow the rules and limitations we established for them

in the description above, for_each controls the behavior of the block it is defined in (not the children), so if it is inside document, multiple document blocks will be created (the same for section). This allows us to use the existing definitions without defining new ones: for example, if for_each would affect only children, we would need to invent a block that can include document to have dynamic documents.

Potentially, we can allow for_each for every block type -- since it affects the block it is defined in, it can be applied to any block. It might be interesting to do in the future but for the first iteration, enabling for_each only for document and section is easier to explain and to use.

[traut]

I've updated the issue description with the new design

[traut]

the issue is updated to align better with #166 and #169

[traut]

bouncing to 0.5

[traut]

me and @Andrew-Morozko agreed to postpone dynamic feature for the document blocks