Server-Side Scripting API (v0.1)

The CaosDB Server can execute scripts (bash/python/perl) and compiled executables. The scripts can be invoked by a remote-procedure-call (RPC) protocol. Both, the requirements for the scripts and the RPC are described in this document.

Configuration of the Server

The CaosDB Server has two relevant properties:

SERVER_SIDE_SCRIPTING_BIN_DIR

is the directory where executable scripts are to be placed. The server will not execute scripts which are out side of this directory. This directory must be readable and executable for the server. But it should not be readable or executable for anyone else. Executable files in this directory or its subdirectories are called SSS.

All other files in that directory MUST be ignored by the server, i.e. the server will never call them directly. However, they MAY contain additional data, different implementations, libraries etc.

A symlink pointing to an executable MUST be treated as SSS, too.

Example SERVER_SIDE_SCRIPTING_BIN_DIR

/
+- script1.py (EXECUTABLE)
+- script2.sh (EXECUTABLE)
+- subdir1/
   +-script3.pl (EXECUTABLE)
+- subdir2/
   +- data_for_script2
   +- another_script.py (EXECUTABLE)
+- script4 -> ./subdir2/another_script.py (SIMLINK to EXECUTABLE)
+- script5 -> /usr/local/bin/external (SIMLINK to EXECUTABLE)

The files scripts1.py, scripts2.sh, scripts3.pl and another_script.py are SSS. Also the script4 can be called which would result in calling another_script.py The script5 points to an executable which is not stored in the SSD_DIR.

SERVER_SIDE_SCRIPTING_WORKING_DIR

is the directory under which the server creates temporary working directories. The server needs writing, reading and executing permissions. The temporary working directories are deleted after the scripts have finished and the server has collected the results of the scripts.

Calling a Server-Side Script Via Remote Procedure Call

  • Users can invoke scripts via HTTP under the uri https://$HOST:$HTTPS_PORT/$CONTEXT_ROOT/scripting. The server accepts POST requests with content types application/x-www-form-urlencoded and multipart/form-data.

  • There are 6 types of form fields which are processed by the server:

    1. A single parameter form field with name=”call” (the path to script, relative to the SERVER_SIDE_SCRIPTING_BIN_DIR). If there are more than one fields with that name, the behavior is not defined.

    2. Zero or more parameter form fields with a unique name which starts with the string -O (command line options).

    3. Zero or more parameter form fields with a unique name which starts with the string -p (positional command line arguments).

    4. Zero or more file form fields which have a unique field name and a unique file name (upload files). If the field names or file names are not unique, the behavior is not defined.

    5. A parameter form field with name=”timeout” (the request timeout)

    6. A parameter form field with name=”auth-token” and a valid CaosDB AuthToken string or “generate” as value.

How Does the Server Call the Script?

  • The server executes the script in a temporary working directory, called PWD (which will be deleted afterwards).

  • The form parameter call, the options, the arguments and file fields construct the string which is executed in a shell. The value of call begins the command line.

  • For any option paramter with the name -Ooption_name and a value option_value a resulting --option_name=option_value is appended to the commandLine in no particular order.

  • The values of the positional arguments are appended sorted alphabetically by their field name. E.g. a parameter -p0 with value v0 and a parameter -p1 with value v1 result in appending v0 v1 to the command line.

  • All files will be loaded into the directory $PWD/.upload_files with their file name (i.e. the form field property). If the file names contain slashes ‘/’ they will build sub-directories in the ./upload_files.

  • If a file form field has a field name which begins with either -p or -O the file name with prefix .upload_files/ is passed as value of an option or as a positional argument to the script. Thus it is possible to distinguish several uploaded files from one another.

  • If there is a “auth-token” field present, another command line options --auth-token=... is appended. The value is either the string which was submitted with the POST request, or, if the value was “generate”, a refreshed, valid AuthToken which authenticates as the user of the request (why? see below).

Example HTML Form

<form action="/scripting" method="post" enctype="multipart/form-data">
<input type="hidden" name="call" value="my/script.py"/>
<input type="file" name="-Oconfig-file"/>
<input type="file" name="-p1"/>
<input type="text" name="-p0" value="analyze"/>
<input type="text" name="user"/>
<input type="text" name="-Oalgorithm" value="fast"/>
<input type="submit" value="Submit">
</form>

where the user uploads my.conf as -Oconfig-file and my.input.tsv as -p1, would result in this command line:

$SERVER_SIDE_SCRIPTING_BIN_DIR/my/script.py --config-file=.upload_files/my.conf --algorithm=fast analyze .upload_files/my.input.tsv

CaosDB Server Response

The CaosDB Server responds with an xml document. The root element is the usual /Response. If no errors occurred (which would be represented with /Response/Error elements) the result of the script execution is represented as a /Response/script/ element.

  • It has a code attribute which contains the exit code value of the execution.

  • It has stdout and stderr children which contain the dump of the stdout and stderr file of the execution environment.

  • It has a call child which contains the command line which was executed (but without a possible --auth-token option and with a relative path to the executable).

Example XML Response

<Response>
<script code="0">
<call>my/script.py --config-file=.upload_files/my.conf --algorithm=fast analyze .upload_files/my.input.tsv</call>
<stdout>Result: 0.5</stdout>
<stderr>Warning: 8 Lines did not contain enough columns</stderr>
</Response>

CaosDB Clients and Authentication Token

A special use case for server side scripting is the automated execution of CaosDB clients. These clients need to connect to the CaosDB Server and thus need a way to authenticate themselves. For this special case the server can pass an Authentication Token which can be used by the script to authenticate itself. If the invocation request send a particulare AuthToken in the form, this AuthToken will be passed to the script with the --auth-token option. Otherwise, if the auth-token field has “generate” as value, a fresh AuthToken is generated which belongs to the user who requested the script execution. Thus the script is executed (and connects back to the server) as the user who called the script in the first place.

A CaosDB client might use the python client library to connect to the server with that AuthToken.

Example Script

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
#
# ** header v3.0
# This file is a part of the CaosDB Project.
#
# Copyright (C) 2018 Research Group Biomedical Physics,
# Max-Planck-Institute for Dynamics and Self-Organization Göttingen
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU Affero General Public License as
# published by the Free Software Foundation, either version 3 of the
# License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU Affero General Public License for more details.
#
# You should have received a copy of the GNU Affero General Public License
# along with this program. If not, see <https://www.gnu.org/licenses/>.
#
# ** end header
#
"""server_side_script.py.

An example which implements a minimal server-side script.

1) This script expects to find a *.txt file in the .upload_files dir which is
printed to stdout.

2) It executes a "Count stars" query and prints the result to stdout.

3) It will return with code 0 if everything is ok, or with any code that is
specified with the commandline option --exit
"""

import sys
from os import listdir
from caosdb import configure_connection, execute_query


# parse --auth-token option and configure connection
CODE = 0
QUERY = "COUNT stars"
for arg in sys.argv:
    if arg.startswith("--auth-token="):
        auth_token = arg[13:]
        configure_connection(auth_token=auth_token)
    if arg.startswith("--exit="):
        CODE = int(arg[7:])
    if arg.startswith("--query="):
        QUERY = arg[8:]


############################################################
# 1 # find and print *.txt file ############################
############################################################

try:
    for fname in listdir(".upload_files"):
        if fname.endswith(".txt"):
            with open(".upload_files/{}".format(fname)) as f:
                print(f.read())
except FileNotFoundError:
    pass


############################################################
# 2 # query "COUNT stars" ##################################
############################################################

RESULT = execute_query(QUERY)
print(RESULT)

############################################################
# 3 ########################################################
############################################################

sys.exit(CODE)