Macros
------

Macros highly facilitate the writing of complex :doc:`CFoods<cfood>`. Consider the following common
example:

.. _example_files:
.. code-block:: yaml

  ExperimentalData:
    type: Directory
    match: ExperimentalData
    subtree:
      README:
        type: SimpleFile
        match: ^README.md$
        records:
          ReadmeFile:
            parents:
            - MarkdownFile
            role: File
            path: $README
            file: $README

This example just inserts a file called ``README.md`` contained in Folder ``ExpreimentalData/`` into CaosDB, assigns the parent (RecordType) ``MarkdownFile`` and allows for later referencing this entity within the cfood. As file objects are created in the cfood specification using the ``records`` section with the special role ``File``, defining and using many files can become very cumbersome and make the cfood file difficult to read.

The same version using cfood macros could be defined as follows:

.. _example_files_2:
.. code-block:: yaml

  ---
  metadata:
    macros:
    - !defmacro
      name: MarkdownFile
      params:
        name: null
        filename: null
      definition:
        ${name}_filename:
          type: SimpleFile
          match: $filename
          records:
            $name:
              parents:
              - MarkdownFile
              role: File
              path: ${name}_filename
              file: ${name}_filename
  ---
  ExperimentalData:
    type: Directory
    match: ExperimentalData
    subtree: !macro
      MarkdownFile:
      - name: README
        filename: ^README.md$

The "MarkdownFile" key and its value will be replaced by everything that is 
given below "definition" in the Macro.
        
The expanded version of `ExperimentalData` will look like:

.. _example_files_2_expanded:
.. code-block:: yaml

  ExperimentalData:
    match: ExperimentalData
    subtree:
      README_filename:
        match: ^README.md$
        records:
          README:
            file: README_filename
            parents:
            - MarkdownFile
            path: README_filename
            role: File
        type: SimpleFile
    type: Directory

This :ref:`example<example_files_2>` can also be found in the macro unit tests (see :func:`unittests.test_macros.test_documentation_example_2`).


Complex Example
===============

The following, more complex example, demonstrates the use
of macro variable substitutions that generate crawler variable substitutions:

- `$$$nodename` will lead to a macro variable substitution of variable `$nodename` during macro expansion.
- `$$` will be turned into `$`
- So in the crawler cfood, the string will appear as `$value` if variable `nodename` would be set to `value` when using the macro.


.. _example_1:
.. code-block:: yaml
   
     macros:
     - !defmacro
       name: SimulationDatasetFile
       params:
         match: null
         recordtype: null
         nodename: null
       definition:
         $nodename:
           match: $match
           type: SimpleFile
           records:
             File:
               parents:
               - $recordtype
               role: File
               path: $$$nodename
               file: $$$nodename
             Simulation:
               $recordtype: +$File

The expanded version of :ref:`example<example_1>` can be seen in :ref:`example<example_1_expanded>`.


.. _example_1_expanded:
.. code-block:: yaml
                
  SimulationData:
    match: SimulationData
    subtree:
      Dataset:
        match: .*
        records:
          File:
            file: $Dataset
            parents:
            - DatasetFile
            path: $Dataset
            role: File
          Simulation:
            DatasetFile: +$File
        type: SimpleFile
    type: Directory

This :ref:`example<example_1>` can also be found in the macro unit tests (see :func:`unittests.test_macros.test_documentation_example_1`).



Using Macros Multiple Times
===========================

To use the same macro multiple times in the same yaml node, lists can be used:

.. _example_multiple:
.. code-block:: yaml

    ---
    metadata:
      macros:
        - !defmacro
          name: test_twice
          params:
            macro_name: default_name
            a: 4
          definition:
            $macro_name:
              something:
                a: $a
    ---
    extroot: !macro
      test_twice:
      - macro_name: once  # <- This is the first replacement of the macro
      - macro_name: twice # <- This is the second one, with different arguments
        a: 5
      - {}                # <- This is the third one, just using default arguments

        
This :ref:`example<example_multiple>` is taken from the macro unit tests (see :func:`unittests.test_macros.test_use_macro_twice`).

The example will be expanded to:

.. _example_multiple_expanded:
.. code-block:: yaml

    extroot:
      default_name:
        something:
          a: '4'
      once:
        something:
          a: '4'
      twice:
        something:
          a: '5'


Note, that you need to make sure that subsequent macro calls do not
use the same top level key. Because later versions would overwrite previous
ones. Here we used ``$macro_name`` to prevent that.


Limitation
==========

Currently it is not possible to use the same macro twice in the same yaml node, but in different
positions. Consider:

.. _example_multiple_limitation:
.. code-block:: yaml

    ---
    metadata:
      macros:
        - !defmacro
          name: test_twice
          params:
            macro_name: default_name
            a: 4
          definition:
            $macro_name:
              something:
                a: $a
    ---
    extroot: !macro
      test_twice:
      - macro_name: once  # <- This is the first replacement of the macro
        
      Other_node:
        type: test
        
      test_twice:  # This is NOT possible as each
                   #  dictionary element can only appear once in a yaml node.
      - macro_name: twice # <- This is the second one, with different arguments
        a: 5
      - {}                # <- This is the third one, just using default arguments

However, this should not be a real limitation, as the crawler is designed in a way,
that the order of the nodes in the same level should not matter.


Using macros within macro definitions
=====================================

It is possible to use other macros in macro definitions. Again, examples can be found in
the macro unit tests (see e.g. :func:`unittests.test_macros.test_macros_in_macros`):

.. _example_macros_in_macros:
.. code-block:: yaml

  ---
  metadata:
    crawler-version: 0.3.1
    macros:
      - !defmacro
        name: one_macro
        params:
          a: 25
        definition:
          macro_sub_$a:
            b: $a
            another_param: 3
      - !defmacro
        name: test_macrodef
        params: {}
        definition:
          macro_top: !macro
            one_macro:
            - a: 17
            - {}
            - a: 98
            not_macro:
              a: 26
  ---
  extroot: !macro
      test_macrodef:

TODO:
to be continued