Server-Side Scripting

Introduction

Server-Side Scripting is a LinkAhead feature which allows to install executables in the LinkAhead Server and trigger the execution via the Server-Side Scripting API.

Small computation task, like some visualization, might be easily implemented in Python or some other language, but cumbersome to integrate into the server. Furthermore, the LinkAhead server should stay a general tool without burden from specific projects. Also, one might want to trigger some standardized processing task from the web interface for convenience. For these situations the “server side scripting” is intended.

The basic idea is that a script or program (script in the following) can be called to run on the server (or elsewhere in future) to do some calculations. This triggering of the script is done over the API so it can be done with any client. Input arguments can be passed to the script and the STDOUT and STDERR are returned.

Each script is executed in a temporary home directory, which is automatically clean up. Scripts can store files in a special folder and provide a link that allows to download those afterwards.

This section contains hands-on, step-by-step instructions on:

Please also consider to look at the more technical specification of the API or take the more step-by-step Deep Dive into the Scripting API.

Recipes

Install a Minimal Script

This is our minimal example script, minimal.bash:

#!/bin/bash
printf "$0"
printf " %s" "$@"
printf "\n"

  1. Move it to your scripting directory. By default, this in the custom/caosdb-server/scripting/bin/ directory under your profile directory.

  2. Add executable permissions at least for the user account of the server (user in the docker container, see user_group in the profile.yml): chmod +x custom/caosdb-server/scripting/bin/minimal.bash

Note: The scripting directories can be configure via the SERVER_SIDE_SCRIPTING_BIN_DIRS option of the server. See Server Configuration for further information.

In principal, the script can be invoked via the scripting API now after restarting the server. You can trigger the script via a browser based HTML form or by using the linkahead python library. See below for more on that.

If you face any issues try the Deep Dive section which explains how to test and debug the installation and how exactly scripts are being called and parameter and files are being passed to the script by the server.

How to Trigger Scripts

Server side scripts are triggered by sending a POST to the /scripting resource using a multipart/form-data or application/x-www-form-urlencoded content type. There are the following arguments that can be provided via the requests fields.

  • call: the name of the script to be called

  • -pN: positional arguments (e.g. -p0, -p1 etc.)

  • -ONAME: named arguments (e.g. -Otest, -Onumber etc.)

The arguments will be passed on to the script. See the Writing Server-Side Scripts section or take the Deep Dive for more information.

Simple, Static Form

The scripting API was designed to interact with HTML forms.

A simple HTML form like this

<h1>Trigger Analysis</h1>
<form action="/scripting" method="post">
<input type="hidden" name="call" value="minimal.bash"/>
<input type="hidden" name="-p0" value="analyze"/>
<label>Some parameter<input type="text" name="-p1"/></label>
<label>Algorithm<input type="text" name="-Oalgorithm" value="fast"/></label>
<input type="submit" value="Submit">
</form>

will trigger a call to the scripting API when submitted. The server will execute minimal.bash --auth-token=[...] --algorithm=fast analyze [...] with the respective parameters the user specified in the form.

To install a static web form like this one, just store this html file to custom/caosdb-server/caosdb-webui/src/ext/html/minimal.html.

Then (re-)start the server. You can browse to https://localhost:10443/webinterface/html/minimal.html where the webform will be served.

Note: Usually you need to sign in first, otherwise you might not have the permissions to execute the script. In this case, browse to https://localhost:10443 first.

Note: While static forms are a straightforward way to trigger scripts via the browser they produce rather ugly responses. See the section on the form_elements module for other means to create web forms and present the script’s output in a way that is more pleasing to the eye.

Using the form_elements module.

The linkahead-webui contains the form_elements module. It provides functionality to dynamically generate forms which interact with the scripting API and present the script’s results to the user.

A configuration for a simple form triggering the minimal.bash script might look like this:

const config = {
  script: "minimal.bash",
  fields: [
    { type: "integer", name: "number", label: "A Number", required: true },
    { type: "date", name: "date", label: "A Date", required: false },
    { type: "text", name: "comment", label: "A Comment", required: false },
  ],
};

The data from this kind of forms will be converted into a json file form.json and uploaded to the scripting API. The script will be called as minimal.bash --auth-token=[...] ./upload_files/form.json

This sure is an opinionated implementation as it only calls scripts in a very particular way and uses a json file instead of passing the parameters directly to the script. However, forms generated by the form_elements module may contain deeply nested fields which is not conveniently mapping to the flat structure of optional and positional parameters.

Note: This section is obviouly just a teaser. Find more information about forms for triggering server-side scripts in the webui documentation.

Note: The form_elements module of the webui works particularly good for python scripts using the convenience methods of the caosadvancedtools package. See next section.

Using a Python Script

The linkahead-pylib library comes with convenient functionality for triggering server-side scripts. This can be as easy as this:

from linkahead.utils.server_side_scripting import run_server_side_script

with open("test_file.txt", "w") as f:
    f.write("this is a test")

response = run_server_side_script("my_script.py",
                                  "pos0",
                                  "pos1",
                                  option1="val1",
                                  option2="val2",
                                  files={"-Ofile": "test_file.txt"})
assert response.stderr is None
assert response.code == 0
assert response.call == ('my_script.py '
                         '--option1=val1 --option2=val2 --file=.upload_files/test_file.txt '
                         'pos0 pos1')

See linkahead.utils.server_side_scripting module for more information.

Writing Server-Side Scripts

Up until now this document has been about triggering server-side scripts, mainly. This section is about writing convenient scripts which are to be deployed and executed as server-side scripts.

Minimal Requirements

  • Be executable. The server will spawn a process and execute the script as the server user. So the script needs to be executable for the server user.

  • Be located in the correct directory. The server needs to find the script in one of it’s configure SERVER_SIDE_SCRIPTING_BIN_DIRS.

How to Find Uploaded Files

The uploaded files are being stored in the ./upload_files directory, relative to the working directory of the server-side script.

The script may search in that directory for a particular file name pattern. However, if the HTTP request to the scripting API (using multipart/form-data type) speficied the name of the file field the way described above (as optional or positional parameters) the files location is passed to the script via the respective parameter. This is the recommended approach.

How to Create a Downloadable File

The runtime environment of the script has a special SHARED_DIR variable pointing to a directory where the script may dump files. If the script wants to output links to the files for downloading it can construct a relative link like this: /${CONTEXT_ROOT_OF_THE_SERVER}/Shared/${SHARED_DIR}/${FILE_NAME}

Note: The Caosadvancedtools package provides a convenience function for this under caosadvancedtools.serverside.helper.get_shared_filename. See below for more information.

How to Invoke Transactions

To insert, retrieve, update or delete entities from within a script, use a suitable client, e.g. the python client.

Note: The linkahead python client library is pre-installed in every linkahead instance and can be used by server-side scripts right away.

How to Send an Email

If linkahead is configured appropriately, a sendmail client can be accessed to send emails by a script.

Note: The caosadvancedtools package provides a convenience function for this under caosadvancedtools.serverside.helper.send_mail. See below for more information.

Using the LinkAhead Python Library

The LinkAhead python client library is pre-installed such that you can use it for writing scripts right away.

In order to configure the connection to the server you need to place a .pylinkahead.ini file at custom/caosdb-server/scripting/home/.pylinkahead.ini with the following content (or similar):

# -- custom/caosdb-server/scripting/home/.pylinkahead.ini

[Connection]
url = https://localhost:10443
cacert = /opt/caosdb/cert/caosdb.cert.pem
#ssl_insecure = True

[Misc]
#optional:
#sendmail=/usr/sbin/sendmail

Use the hostname in the URL for which the certificate is created and the correct port. If the certificate is signed by a well-known CA you may leave cacert blank since the certificate can then be verified without the file.

Note: The script is executed inside Docker. Thus, the URL should be publicly valid (i.e. the external hostname and port) or at least valid inside the container (which is typically the one given above with localhost). If you cannot make it work because of ssl errors try to add ssl_insecure = True to the

Now you can use the Auth Token which is being passed to the script to configure the authentication and use the library:

import linkahead as la
from caosadvancedtools.serverside import helper

def main():
    parser = helper.get_argument_parser()
    args = parser.parse_args()
    auth_token = args.auth_token
    la.configure_connection(auth_token=auth_token)
    la.execute_query("FIND Experiment")
...

Note: we use Caosadvancedtools for parsing the parameters passed to the script. See the next section for more information.

Using the Convenience Methods of Caosadvancedtools

The python package caosadvancedtools comes with the module caosadvancedtools.serverside which provides functionality for sending mails, making files available for download, rendering the output as HTML, and much more.

Note: see the documentation of the caosadvancedtools for more information.

Auth Token

Caosadvancedtools come with a special argument parser which parses the Auth Token right away:

import linkahead as la
from caosadvancedtools.serverside import helper

def main():
    parser = helper.get_argument_parser()
    args = parser.parse_args()
    auth_token = args.auth_token
    la.configure_connection(auth_token=auth_token)
    la.execute_query("FIND Experiment")
...

If for some reason you need information about the user who called the script (such as name, roles, or realm), you will find that the linkahead.get_connection().get_username() method will not work because the connection is configured with an auth token. However, you can still access the user information using the Info resource and the UserInfo therein:

import linkahead as la

inf = la.Info()
username = inf.user_info.name
Using the Logger for Output

Caosadvancedtools comes with a convenient function for configuring the native python logging for html output. This helps to produce pretty output for users calling the script via the webinterace.

...
import logging

def main():
    # setup logging and reporting
    configure_server_side_logging("my-logger")

    ...

    logger = logging.getLogger("my-logger")
    logger.info("The script terminated successfully.")

This renders as

<div class="alert alert-info alert-dismissible" role="alert">The script terminated successfully.</div>
Send an Email

You can use Caosadvancedtools to send an email via the sendmail daemon. You need to configure LinkAhead appropriatly (see administration/Email Notification) and add sendmail=/usr/sbin/sendmail to the [Misc] section of the pylinkahead.ini of the server-side scripting home directory.

Now you can do

def notify_user(user_email, subject, message):
    helper.send_mail(from_addr="admin@example.com", to=user_email, subject=subject, body=message)

Note: See caosadvancedtools for more information.

Uploaded File

Caosadvancedtools integrates particularly well with the form_elements module of the webui.

To read the the json file provided by the form_elements trigger request you can use the argument parser as well:

...
import json

def main():
    parser = helper.get_argument_parser()
    args = parser.parse_args()

    # Read the input from the form (form.json)
    with open(args.filename) as form_json:
        form_data = json.load(form_json)
...
Create Files for Download

Caosadvancedtools has a helper method named get_shared_filename. It generates a pair of paths. The first path is intented for generating the download links for the user. The second is intended for opening and write the file:

...
def write_output_file(result):
    display_path, internal_path = helper.get_shared_filename("output.json")
    with open(internal_path, 'w') as f:
        json.dump(result, f)

    # use the logger to inform the user where the can download the file
    logger.info(f"Download the <a href=/Shared/{display_path}>output.json</a>.")
...

Changing Permissions

Up until now, our examples only worked for users with the administration role. If anyone else tried to trigger a script they most likely got a permission error. This is the case when a user lacks the appropriate permissions to call a script.

Permissions to Call a Script

You can allow users to run a particular script by granting the role permission SCRIPTING:EXECUTE:?PATH? where the path is the path of the script in the server’s scripting directory (SERVER_SIDE_SCRIPTING_BIN_DIRS).

Note: the path can also specify a glob pattern using *. E.g. SCRIPTING:EXECUTE:* is the permission to execute any script. SCRIPTING:EXECUTE:my_scripts/* is the permission to execute any script under ./my_scripts/ in one of the servers scripting directories.

Note: Find more information about permissions in general and how to grant them in the server docs on permissions.

Permissions of the Auth Token passed to the Script

By default, the Auth Token passed to the script as the --auth-token optional parameter authenticates the script against the server as the user who called the script in the first place.

However, there are use cases, where we want the script to have 1) more permissions than the caller or use a different user. One example are scripts which are to be called by unprivileged users or the anonymous user which need to update entities. This is a pattern to allow unprivileged users to exactly one operation (like adding a new entry to a list) but nothing else.

In this case, you can configure the authtoken.yaml (server option: AUTHTOKEN_CONFIG) to issue a more powerfull Auth Token for the script.

Note: See the server’s documentation for the AUTHTOKEN_CONFIG for more information.

Example: Putting it All Together

The sections show how to create a web form using form_element which triggers a server-side script. The script will be executable for any user having the team role and it will act with administration permissions.

Create the HTML Form

It has one field which is a file upload with the name “csvfile” and one text field with the name “myparameter”.

The javascript snippet to create such a form using the webui’s form_elements module looks like this:

// -- trigger_sss_form.js
const form_config = {
    script: "handle_csv.py",
    fields: [{
        type: "text",
        name: "myparameter",
        label: "My Parameter",
        required: true,
        help: "Define a parameter",
    }, {
        type: "file",
        name: "csvfile",
        label: "CSV Table",
        required: true,
        accept: "*.csv",
        help: "Upload a CSV Table",
    }, ],
};

// create the form
const form = form_elements.make_form(form_config);

// now inject it into the document
document.body.append(form);

This snippet needs to be installed under custom/caosdb-server/caosdb-webui/src/ext/js/trigger_sss_form.js

Create a Server-Side Script

This script accepts a form.json file containing myparameter and csvfile fields as generated by the above form. It uses the Auth Token to communicate with the server, it uses the logger configured by Caosadvancedtools to print the output nicely renderen as HTML and it returns a download link to the user for downloading a generated file.

# -- handle_csv.py
import os
import logging
import linkahead as la
from caosadvancedtools.serverside import helper
from caosadvancedtools.serverside.logging import configure_server_side_logging

LOGGER_NAME="my-logger"
logger = logging.getLogger(LOGGER_NAME)

def do_something(csv_file_path, my_parameter):
    """Do something with the uploaded csv file and the parameter, then return a dict."""
    la.execute_query("FIND Something")

    return {"data": "example"}

def main():
    # setup logging and reporting
    configure_server_side_logging(LOGGER_NAME)

    parser = helper.get_argument_parser()
    args = parser.parse_args()

    # configure the connection using the auth_token.
    la.configure_connection(auth_token=args.auth_token)

    # Read the input from the form (form.json)
    with open(args.filename) as form_json:
        form_data = json.load(form_json)

    # files are uploaded to this directory
    upload_dir = os.path.dirname((args.filename))
    # Read content of the uploaded form data
    csv_file_path = os.path.join(upload_dir, form_data["csvfile"])
    my_parameter = form_data["myparameter"]

    result = do_something(csv_file_path, my_parameter)

    # create a downloadable file and write the results into that file
    display_path, internal_path = helper.get_shared_filename("output.json")
    with open(internal_path, 'w') as f:
        json.dump(result, f)

    # use the logger to inform the user where the can download the file
    logger.info(f"Download the <a href=/Shared/{display_path}>output.json</a>.")

if __name__ == "__main__":
   main()

This file needs to be installed under custom/caosdb-server/scripting/bin/handle_csv.py as an executable file.

Configure the Server

You can add a configuration file at custom/caosdb-server/conf/ext/server.conf.d/sss.conf

Be sure to have the server configured with SERVER_SIDE_SCRIPTING_BIN_DIRS=./scripting/bin`. This is the default, so probably you are just fine.

Add add a configuration file at custom/caosdb-server/conf/ext/authtoken.yaml for the Auth Tokens passed to the script:

# -- custom/caosdb-server/conf/ext/authtoken.yaml
- purpose: SCRIPTING:EXECUTE:handle_csv.py
  roles:
    - administration

Note: Find more examples for the configuration in the authtoken.example.yaml

Put the file to custom/caosdb-server/conf/authtoken.yaml and configure the server with

# -- custom/caosdb-server/conf/ext/server.conf.d/sss.conf
AUTHTOKEN_CONFIG=`./conf/ext/authtoken.yaml`

Deep Dive into the Scripting API

This section shows how parameters or files are being passed to the script and how you can test an installation.

Testing an Installation

Given that you installed the minimal bash script from the previous section, you can start the server and test it with curl:

curl -F call=minimal.bash 'https://localhost:10443/scripting' -H 'Cookie: SessionToken=[...]'

Note: The scripting API expects a HTTP POST request with content types application/x-www-form-urlencoded or multipart/form-data. It has one mandatory parameter, which is call and the value must be the script’s path below the scripting directory.

Note: You need a valid session token. Easiest way to get one: Log in with the browser, open the developer tools and copy paste it from “Storage” -> “Cookies” -> “SessionToken” (firefox).

Note: If curl fails with “curl: (60) SSL certificate problem: self-signed certificate” or a similar error message you must give curl the path to the public certificate or a signing certificate authority using the --cacert or --capath options. If you want to ignore the issue because you only do this for testing purposes you may use the --insecure|-k option.

The server’s response should look like this:

<Response>
  ...
  <script code="0">
    <call>minimal.bash</call>
    <stdout>/opt/caosdb/git/caosdb-server/scripting/bin/minimal.bash --auth-token=["S","PAM","admin",...]</stdout>
    <stderr />
  </script>
</Response>

You can see the exit code <script code="0"> was 0 (no errors). The standard output and error output are there and you can see that the server called the script with one additional --auth-token=["S", "PAM", "admin",...] option. This AuthToken can be used by the script for further communication with the server.

Note: The session token should never be leaked by the script. This script does it only for didactic reasons.

Note: You need to restart linkahead every time you install a new script or make changes to an existing one. If you need to add or change a script in a running system you must to copy your script into the docker container to the correct directory, manually.

Call a Script With Parameters

You can add parameter to the scripting call by adding key-value pairs to your POST request. There are two types of parameters that will be processed:

  1. Keys matching -p[1-9][0-9]*, e.g. -p0, -p1,… These are positional parameters which are goint to be passed to the script.

  2. Keys starting with -O. These are options which are going to be parsed to the script.

Using our minimal script and calling it via curl:

curl -F call=minimal.bash -F -p0=Positional0 -F -p1=Positional1 -F -Ooption1= -F -Ooption2=some-value 'https://localhost:10443/scripting' -H 'Cookie: SessionToken=[...]'

This server’s response should look like this:

<Response>
  ...
  <script code="0">
    <call>minimal.bash --option1= --option2=some-value Positional0 Positional1</call>
    <stdout>/opt/caosdb/git/caosdb-server/scripting/bin/minimal.bash --auth-token=["S","PAM","admin",...] --option1= --option2=some-value Positional0 Positional1</stdout>
    <stderr />
  </script>
</Response>

You can see that the parameters have been passed to the script and the effective call was minimal.bash --option1= --option2=some-value Positional0 Positional1.

Note: The session token should never be leaked by the script. This script does it only for didactic reasons. The <call>... element prints the complete call but omits the session token.

Pass Files to the Script

You can upload files and pass the files to the script via the scripting API using the POST request with the multipart/form-data content type (this is the standard way of uploading files via HTTP).

Let’s assume you have a file data.json like this:

{ "my_key": "my value" }

You can upload the file and call the script via

curl -F call=minimal.bash -F -Omy-file=@data.json 'https://localhost:10443/scripting' -H 'Cookie: SessionToken=[...]'

The server will store the file under .upload_files/data.json relative to the scripts working directory.

...
<script code="0">
  <call>minimal.bash --my-file=.upload_files/data.json </call>
  ...
</script>
...

You can see that the server passes the file’s location to the script.

Note: We used a key starting with -O here, resulting in an optional parameter. However, you can also use a positional paramter by using the -pN keys for the file.

We can now add another minimal script print_file.bash which prints the content of the uploaded file and install it as shown above.

#!/bin/bash
cat $2

Note: Since the first parameter is always the server-generated --auth-token the $2 refers to the first parameter specified by the user.

This script expects to be called with one positional parameter. So we call it via

curl -F call=print_file.bash -F -p0=@data.json 'https://localhost:10443/scripting' -H 'Cookie: SessionToken=[...]'

The server’s response will look like this:

...
<script code="0">
  <call>print_file.bash .upload_files/data.json</call>
  <stdout>{"my-key":"my-value"}</stdout>
  <stderr />
</script>
...

You can see that the script can indeed access the file and print the content.

Using the Auth Token From Within the Script

Up until now, we ignored the --auth-token parameter. It can be used by the script to make calls to the server. By default, the authentication token passed to script is just a session token for the current session of the user account who is calling the script: If you call the script as “user1” the script can execute any operation on the server as the same “user1” user.

Using curl inside a minimal script user_info.bash we can see this happening:

#!/bin/bash

# extract the session token from the --auth-token option.
SESSION_TOKEN="${1#--auth_token=}"

curl -o /dev/null -s -w "\nNo SessionToken: %{http_code}" --insecure 'https://localhost:10443/'
curl -o /dev/null -s -w "\nWith SessionToken: %{http_code}" --insecure -H "Cookie: SessionToken=${SESSION_TOKEN}" 'https://localhost:10443'

If we call this script via the scripting API without any further parameters

curl -k -F call=user_info.bash 'https://localhost:10443/scripting' -H 'Cookie: SessionToken=[...]'

the server’s response shows a “401” (Unauthorized) response for the first request and a “200” (Ok) response for the second request:

...
<script code="0">
  <call>user_info.bash</call>
  <stdout>No SessionToken: 401
th SessionToken: 200</stdout>
  <stderr />
</script>
...

A Pattern: The Server as a Client of Itself.

So what did we do in the last step?

We installed a script user_info.bash. If we call that script via the scripting API, the server executes that script on our behalf. The scripts sends two request back to the server and the scripting API response reaches us after both curl requests of the script terminated.

| USER: calls `user_info.bash`
|       | SERVER: receives the scripting API request
|       | SERVER: executes `user_info.bash`
|       |         | SCRIPT: curl sends request 1
|       |         |         | SERVER receives the request
|       |         |         * SERVER responds with 401
|       |         | SCRIPT: curl terminates 1st time
|       |         | SCRIPT: curl sends request 2
|       |         |         | SERVER receives the request
|       |         |         * SERVER responds with 200
|       |         | SCRIPT: curl terminates 2nd time
|       |         * SCRIPT: terminates
|       * SERVER: sends response `<script code="0"> ...`
* USER: receives the servers response.

So in the lines in the middle, the server is talking to itself. Actually, this is a very common pattern. This pattern can be used to encapsulate a complex series of transactions into one single command with parameters.