Macros

Macros highly facilitate the writing of complex CFoods. Consider the following common example:

ExperimentalData:
  type: Directory
  match: ExperimentalData
  subtree:
    README:
      type: SimpleFile
      match: ^README.md$
      records:
        ReadmeFile:
          parents:
          - MarkdownFile
          role: File
          path: $README
          file: $README

This example just inserts a file called README.md contained in Folder ExpreimentalData/ into CaosDB, assigns the parent (RecordType) MarkdownFile and allows for later referencing this entity within the cfood. As file objects are created in the cfood specification using the records section with the special role File, defining and using many files can become very cumbersome and make the cfood file difficult to read.

The same version using cfood macros could be defined as follows:

---
metadata:
  macros:
  - !defmacro
    name: MarkdownFile
    params:
      name: null
      filename: null
    definition:
      ${name}_filename:
        type: SimpleFile
        match: $filename
        records:
          $name:
            parents:
            - MarkdownFile
            role: File
            path: ${name}_filename
            file: ${name}_filename
---
ExperimentalData:
  type: Directory
  match: ExperimentalData
  subtree: !macro
    MarkdownFile:
    - name: README
      filename: ^README.md$

The “MarkdownFile” key and its value will be replaced by everything that is given below “definition” in the Macro.

The expanded version of ExperimentalData will look like:

ExperimentalData:
  match: ExperimentalData
  subtree:
    README_filename:
      match: ^README.md$
      records:
        README:
          file: README_filename
          parents:
          - MarkdownFile
          path: README_filename
          role: File
      type: SimpleFile
  type: Directory

This example can also be found in the macro unit tests (see unittests.test_macros.test_documentation_example_2()).

Complex Example

The following, more complex example, demonstrates the use of macro variable substitutions that generate crawler variable substitutions:

• $$$nodename will lead to a macro variable substitution of variable $nodename during macro expansion.

• $$ will be turned into $

• So in the crawler cfood, the string will appear as $value if variable nodename would be set to value when using the macro.

macros:
- !defmacro
  name: SimulationDatasetFile
  params:
    match: null
    recordtype: null
    nodename: null
  definition:
    $nodename:
      match: $match
      type: SimpleFile
      records:
        File:
          parents:
          - $recordtype
          role: File
          path: $$$nodename
          file: $$$nodename
        Simulation:
          $recordtype: +$File

The expanded version of example can be seen in example.

SimulationData:
  match: SimulationData
  subtree:
    Dataset:
      match: .*
      records:
        File:
          file: $Dataset
          parents:
          - DatasetFile
          path: $Dataset
          role: File
        Simulation:
          DatasetFile: +$File
      type: SimpleFile
  type: Directory

This example can also be found in the macro unit tests (see unittests.test_macros.test_documentation_example_1()).

Using Macros Multiple Times

To use the same macro multiple times in the same yaml node, lists can be used:

---
metadata:
  macros:
    - !defmacro
      name: test_twice
      params:
        macro_name: default_name
        a: 4
      definition:
        $macro_name:
          something:
            a: $a
---
extroot: !macro
  test_twice:
  - macro_name: once  # <- This is the first replacement of the macro
  - macro_name: twice # <- This is the second one, with different arguments
    a: 5
  - {}                # <- This is the third one, just using default arguments

This example is taken from the macro unit tests (see unittests.test_macros.test_use_macro_twice()).

The example will be expanded to:

extroot:
  default_name:
    something:
      a: '4'
  once:
    something:
      a: '4'
  twice:
    something:
      a: '5'

Note, that you need to make sure that subsequent macro calls do not use the same top level key. Because later versions would overwrite previous ones. Here we used $macro_name to prevent that.

Limitation

Currently it is not possible to use the same macro twice in the same yaml node, but in different positions. Consider:

---
metadata:
  macros:
    - !defmacro
      name: test_twice
      params:
        macro_name: default_name
        a: 4
      definition:
        $macro_name:
          something:
            a: $a
---
extroot: !macro
  test_twice:
  - macro_name: once  # <- This is the first replacement of the macro

  Other_node:
    type: test

  test_twice:  # This is NOT possible as each
               #  dictionary element can only appear once in a yaml node.
  - macro_name: twice # <- This is the second one, with different arguments
    a: 5
  - {}                # <- This is the third one, just using default arguments

However, this should not be a real limitation, as the crawler is designed in a way, that the order of the nodes in the same level should not matter.

Using macros within macro definitions

It is possible to use other macros in macro definitions. Again, examples can be found in the macro unit tests (see e.g. unittests.test_macros.test_macros_in_macros()):

---
metadata:
  crawler-version: 0.3.1
  macros:
    - !defmacro
      name: one_macro
      params:
        a: 25
      definition:
        macro_sub_$a:
          b: $a
          another_param: 3
    - !defmacro
      name: test_macrodef
      params: {}
      definition:
        macro_top: !macro
          one_macro:
          - a: 17
          - {}
          - a: 98
          not_macro:
            a: 26
---
extroot: !macro
    test_macrodef:

TODO: to be continued