Macros

Introduction

Macros highly facilitate the writing of complex CFoods. Consider the following common example:

ExperimentalData:
  type: Directory
  match: ExperimentalData
  subtree:
    README:
      type: SimpleFile
      match: ^README.md$
      records:
        ReadmeFile:
          parents:
          - MarkdownFile
          role: File
          path: $README
          file: $README

This example just inserts a file called README.md contained in Folder ExpreimentalData/ into CaosDB, assigns the parent (RecordType) MarkdownFile and allows for later referencing this entity within the cfood. As file objects are created in the cfood specification using the records section with the special role File, defining and using many files can become very cumbersome and make the cfood file difficult to read.

The same version using cfood macros could be defined as follows:

---
metadata:
  macros:
  - !defmacro
    name: MarkdownFile
    params:
      name: null
      filename: null
    definition:
      ${name}_filename:
        type: SimpleFile
        match: $filename
        records:
          $name:
            parents:
            - MarkdownFile
            role: File
            path: ${name}_filename
            file: ${name}_filename
---
ExperimentalData:
  type: Directory
  match: ExperimentalData
  subtree: !macro
    MarkdownFile:
    - name: README
      filename: ^README.md$

The “MarkdownFile” key and its value will be replaced by everything that is given below “definition” in the Macro.

The expanded version of ExperimentalData will look like:

ExperimentalData:
  match: ExperimentalData
  subtree:
    README_filename:
      match: ^README.md$
      records:
        README:
          file: README_filename
          parents:
          - MarkdownFile
          path: README_filename
          role: File
      type: SimpleFile
  type: Directory

This example can also be found in the macro unit tests (see unittests.test_macros.test_documentation_example_2()).

Mixing macros and plain definitions

You can also mix macros and plain definitions. Whenever a name cannot be resolved to a macro, a plain yaml node definition is used as a fallback:

---
metadata:
  macros:
  - !defmacro
    name: MarkdownFile
# ... Definition here ...
---
ExperimentalData:
  type: Directory
  match: ExperimentalData
  subtree: !macro
    MarkdownFile:
    - name: README
      filename: ^README.md$
    OtherContent:  # There is no macro named "OtherContent", so this is parsed as normal content.
      type: SimpleFile
      match: .*txt
      records:
        # ... Normal content ...

Complex example

Let’s try something more complex: what happens to multiple $? This example demonstrates the use of macro variable substitutions to generate crawler variable substitutions:

• $$ will be converted into $.

• $$$nodename will retain a single $ and substitute $nodename during macro expansion.

• So in the cfood, if nodename: value, the string $$$nodename will be converted to $value.

macros:
- !defmacro
  name: SimulationDatasetFile
  params:
    match: null
    recordtype: null
    nodename: null
  definition:
    $nodename:
      match: $match
      type: SimpleFile
      records:
        File:
          parents:
          - $recordtype
          role: File
          path: $$$nodename
          file: $$$nodename
        Simulation:
          $recordtype: +$File

The expanded version of the example above (with nodename: Dataset) can be seen here:

SimulationData:
  match: SimulationData
  subtree:
    Dataset:
      match: .*
      records:
        File:
          file: $Dataset
          parents:
          - DatasetFile
          path: $Dataset
          role: File
        Simulation:
          DatasetFile: +$File
      type: SimpleFile
  type: Directory

This example can also be found in the macro unit tests (see unittests.test_macros.test_documentation_example_1()).

Using macros multiple times

To use the same macro multiple times in the same yaml node, lists can be used:

---
metadata:
  macros:
    - !defmacro
      name: test_twice
      params:
        macro_name: default_name
        a: 4
      definition:
        $macro_name:
          something:
            a: $a
---
extroot: !macro
  test_twice:
  - macro_name: once  # <- This is the first replacement of the macro
  - macro_name: twice # <- This is the second one, with different arguments
    a: 5
  - {}                # <- This is the third one, just using default arguments

This example is taken from the macro unit tests (see unittests.test_macros.test_use_macro_twice()).

The example will be expanded to:

extroot:
  default_name:
    something:
      a: '4'
  once:
    something:
      a: '4'
  twice:
    something:
      a: '5'

Note, that you need to make sure that subsequent macro calls do not use the same top level key. Because later versions would overwrite previous ones. Here we used $macro_name to prevent that.

Limitations

Currently it is not possible to use the same macro twice in the same yaml node, if it occurs in different positions. Consider:

---
metadata:
  macros:
    - !defmacro
      name: test_twice
      params:
        macro_name: default_name
        a: 4
      definition:
        $macro_name:
          something:
            a: $a
---
extroot: !macro
  test_twice:
  - macro_name: once  # <- This is the first replacement of the macro

  Other_node:
    type: test

  test_twice:  # This is NOT possible as each key
               # can only appear once in a yaml node.
  - macro_name: twice # <- This is the second one, with different arguments
    a: 5
  - {}                # <- This is the third one, just using default arguments

This should not be a real limitation however, as the order of nodes does not matter for the crawler.

Using macros within macro definitions

It is possible to use other macros in macro definitions. Again, examples can be found in the macro unit tests (see e.g. unittests.test_macros.test_macros_in_macros()):

---
metadata:
  crawler-version: 0.3.1
  macros:
    - !defmacro
      name: one_macro
      params:
        a: 25
      definition:
        macro_sub_$a:
          b: $a
          another_param: 3
    - !defmacro
      name: test_macrodef
      params: {}
      definition:
        macro_top: !macro
          one_macro:
          - a: 17
          - {}
          - a: 98
          not_macro:
            a: 26
---
extroot: !macro
    test_macrodef:

TODO: to be continued