# Server-Side Scripting ## Introduction Server-Side Scripting is a LinkAhead feature which allows to install executables in the LinkAhead Server and trigger the execution via the Server-Side Scripting API. Small computation task, like some visualization, might be easily implemented in Python or some other language, but cumbersome to integrate into the server. Furthermore, the LinkAhead server should stay a general tool without burden from specific projects. Also, one might want to trigger some standardized processing task from the web interface for convenience. For these situations the "server side scripting" is intended. The basic idea is that a script or program (script in the following) can be called to run on the server (or elsewhere in future) to do some calculations. This triggering of the script is done over the API so it can be done with any client. Input arguments can be passed to the script and the STDOUT and STDERR are returned. Each script is executed in a temporary home directory, which is automatically clean up. Scripts can store files in a special folder and provide a link that allows to download those afterwards. This section contains hands-on, step-by-step instructions on: * [The installation of server-side scripts](<#install-a-minimal-script>) * How to trigger scripts * [With a plain html form](<#simple-static-form>) * [With the webui's `form_elements` module](<#using-the-form-elements-module>) * [With the linkahead python library](<#using-a-python-script>) * How to implement a server-side script: * [As a minimal bash script](<#install-a-minimal-script>) * [With the linkahead python library](<#using-the-linkahead-python-library>) * [Using the Caosadvancedtools package package](<#using-the-convenience-methods-of-caosadvancedtools>) * [For sending email](<#how-to-send-an-email>) * [For creating a downloadable file](<#how-to-create-a-downloadable-file>) * How to configure the permissions: * [Who can trigger a particular script](<#permissions-to-call-a-script>) * [What are the permissions of the script when the script interacts with the LinkAhead server.](<#permissions-of-the-auth-token-passed-to-the-script>) Please also consider to look at the more technical [specification](https://docs.indiscale.com/caosdb-server/administration/server_side_scripting.html) of the API or take the more step-by-step [Deep Dive into the Scripting API](<#deep-dive-into-the-scripting-api>). ## Recipes ### Install a Minimal Script This is our minimal example script, `minimal.bash`: ```bash #!/bin/bash printf "$0" printf " %s" "$@" printf "\n" ``` 1. Move it to your scripting directory. By default, this in the `custom/caosdb-server/scripting/bin/` directory under your profile directory. 2. Add executable permissions at least for the user account of the server (in the docker container): `chmod +x custom/caosdb-server/scripting/bin/minimal.bash` > Note: The scripting directories can be configure via the > `SERVER_SIDE_SCRIPTING_BIN_DIRS` option of the server. See > [Server Configuration]() > for further information. In principal, the script can be invoked via the scripting API now after restarting the server. You can trigger the script via a browser based HTML form or by using the linkahead python library. See below for more on that. If you face any issues try the Deep Dive section which explains how to test and debug the installation and how exactly scripts are being called and parameter and files are being passed to the script by the server. ### How to Trigger Scripts Server side scripts are triggered by sending a POST to the `/scripting` resource using a `multipart/form-data` or `application/x-www-form-urlencoded` content type. There are the following arguments that can be provided via the requests fields. - `call`: the name of the script to be called - `-pN`: positional arguments (e.g. `-p0`, `-p1` etc.) - `-ONAME`: named arguments (e.g. `-Otest`, `-Onumber` etc.) The arguments will be passed on to the script. See the [Writing Server-Side Scripts](<#writing-server-side-scripts>) section or take the [Deep Dive](<#deep-dive-into-the-scripting-api>) for more information. #### Simple, Static Form The scripting API was designed to interact with HTML forms. A simple HTML form like this ```html

Trigger Analysis

``` will trigger a call to the scripting API when submitted. The server will execute `minimal.bash --auth-token=[...] --algorithm=fast analyze [...]` with the respective parameters the user specified in the form. To install a static web form like this one, just store this html file to `custom/caosdb-server/caosdb-webui/src/ext/html/minimal.html`. Then (re-)start the server. You can browse to `https://localhost:10443/webinterface/html/minimal.html` where the webform will be served. > Note: Usually you need to sign in first, otherwise you might not have the > permissions to execute the script. In this case, browse to > `https://localhost:10443` first. > Note: While static forms are a straightforward way to trigger scripts via the > browser they produce rather ugly responses. See the section on the > `form_elements` module for other means to create web forms and present the > script's output in a way that is more pleasing to the eye. #### Using the `form_elements` module. The [linkahead-webui]() contains the `form_elements` module. It provides functionality to dynamically generate forms which interact with the scripting API and present the script's results to the user. A configuration for a simple form triggering the `minimal.bash` script might look like this: ```javascript const config = { script: "minimal.bash", fields: [ { type: "integer", name: "number", label: "A Number", required: true }, { type: "date", name: "date", label: "A Date", required: false }, { type: "text", name: "comment", label: "A Comment", required: false }, ], }; ``` The data from this kind of forms will be converted into a json file `form.json` and uploaded to the scripting API. The script will be called as `minimal.bash --auth-token=[...] ./upload_files/form.json` This sure is an opinionated implementation as it only calls scripts in a very particular way and uses a json file instead of passing the parameters directly to the script. However, forms generated by the `form_elements` module may contain deeply nested fields which is not conveniently mapping to the flat structure of optional and positional parameters. > Note: This section is obviouly just a teaser. Find more information about > forms for triggering server-side scripts in the webui documentation. > Note: The `form_elements` module of the webui works particularly good for > python scripts using the convenience methods of the caosadvancedtools > package. See next section. #### Using a Python Script The [linkahead-pylib](https://docs.indiscale.com/caosdb-pylib) library comes with convenient functionality for triggering server-side scripts. This can be as easy as this: ```python from linkahead.utils.server_side_scripting import run_server_side_script with open("test_file.txt", "w") as f: f.write("this is a test") response = run_server_side_script("my_script.py", "pos0", "pos1", option1="val1", option2="val2", files={"-Ofile": "test_file.txt"}) assert response.stderr is None assert response.code == 0 assert response.call == ('my_script.py ' '--option1=val1 --option2=val2 --file=.upload_files/test_file.txt ' 'pos0 pos1') ``` See [`linkahead.utils.server_side_scripting`](https://docs.indiscale.com/caosdb-pylib/_apidoc/linkahead.utils.server_side_scripting.html) module for more information. ### Writing Server-Side Scripts Up until now this document has been about *triggering* server-side scripts, mainly. This section is about writing convenient scripts which are to be deployed and executed as server-side scripts. #### Minimal Requirements * Be executable. The server will spawn a process and execute the script as the server user. So the script needs to be executable for the server user. * Be located in the correct directory. The server needs to find the script in one of it's configure `SERVER_SIDE_SCRIPTING_BIN_DIRS`. #### How to Find Uploaded Files The uploaded files are being stored in the `./upload_files` directory, relative to the working directory of the server-side script. The script may search in that directory for a particular file name pattern. However, if the HTTP request to the scripting API (using `multipart/form-data` type) speficied the name of the file field the way described above (as optional or positional parameters) the files location is passed to the script via the respective parameter. This is the recommended approach. #### How to Create a Downloadable File The runtime environment of the script has a special `SHARED_DIR` variable pointing to a directory where the script may dump files. If the script wants to output links to the files for downloading it can construct a relative link like this: `/${CONTEXT_ROOT_OF_THE_SERVER}/Shared/${SHARED_DIR}/${FILE_NAME}` > Note: The Caosadvancedtools package provides a convenience function for this > under `caosadvancedtools.serverside.helper.get_shared_filename`. See below > for more information. #### How to Invoke Transactions To insert, retrieve, update or delete entities from within a script, use a suitable client, e.g. the python client. > Note: The linkahead python client library is pre-installed in every linkahead > instance and can be used by server-side scripts right away. #### How to Send an Email If linkahead is configured appropriately, a `sendmail` client can be accessed to send emails by a script. > Note: The caosadvancedtools package provides a convenience function for this > under `caosadvancedtools.serverside.helper.send_mail`. See [below](<#send-an-email>) for more > information. #### Using the LinkAhead Python Library The LinkAhead python client library is pre-installed such that you can use it for writing scripts right away. In order to configure the connection to the server you need to place a .pylinkahead.ini file at `custom/caosdb-server/scripting/home/.pylinkahead.ini` with the following content (or similar): ```ini # -- custom/caosdb-server/scripting/home/.pylinkahead.ini [Connection] url = https://localhost:10443 cacert = /opt/caosdb/cert/caosdb.cert.pem #ssl_insecure = True [Misc] #optional: #sendmail=/usr/sbin/sendmail ``` Use the hostname in the URL for which the certificate is created and the correct port. If the certificate is signed by a well-known CA you may leave `cacert` blank since the certificate can then be verified without the file. > Note: The script is executed inside Docker. Thus, the URL should be publicly > valid (i.e. the external hostname and port) or at least valid inside the > container (which is typically the one given above with `localhost`). If you > cannot make it work because of ssl errors try to add `ssl_insecure = True` to the Now you can use the Auth Token which is being passed to the script to configure the authentication and use the library: ```python import linkahead as la from caosadvancedtools.serverside import helper def main(): parser = helper.get_argument_parser() args = parser.parse_args() auth_token = args.auth_token la.configure_connection(auth_token=auth_token) la.execute_query("FIND Experiment") ... ``` > Note: we use Caosadvancedtools for parsing the parameters passed to the > script. See the next section for more information. #### Using the Convenience Methods of Caosadvancedtools The python package `caosadvancedtools` comes with the module `caosadvancedtools.serverside` which provides functionality for sending mails, making files available for download, rendering the output as HTML, and much more. > Note: see the [documentation of the caosadvancedtools](https://docs.indiscale.com/caosdb-advanced-user-tools) for more information. ##### Auth Token Caosadvancedtools come with a special argument parser which parses the Auth Token right away: ```python import linkahead as la from caosadvancedtools.serverside import helper def main(): parser = helper.get_argument_parser() args = parser.parse_args() auth_token = args.auth_token la.configure_connection(auth_token=auth_token) la.execute_query("FIND Experiment") ... ``` ##### Using the Logger for Output Caosadvancedtools comes with a convenient function for configuring the native python logging for html output. This helps to produce pretty output for users calling the script via the webinterace. ```python ... import logging def main(): # setup logging and reporting configure_server_side_logging("my-logger") ... logger = logging.getLogger("my-logger") logger.info("The script terminated successfully.") ``` This renders as ```html ``` ##### Send an Email You can use Caosadvancedtools to send an email via the `sendmail` daemon. You need to configure LinkAhead appropriatly (see [administration/Email Notification](https://docs.indiscale.com/caosdb-deploy/administration/email.html)) and add `sendmail=/usr/sbin/sendmail` to the `[Misc]` section of the `pylinkahead.ini` of the server-side scripting home directory. Now you can do ```python def notify_user(user_email, subject, message): helper.send_mail(from_addr="admin@example.com", to=user_email, subject=subject, body=message) ``` > Note: See > [caosadvancedtools](https://docs.indiscale.com/caosdb-advanced-user-tools/_apidoc/caosadvancedtools.serverside.html#caosadvancedtools.serverside.helper.send_mail) > for more information. ##### Uploaded File Caosadvancedtools integrates particularly well with the `form_elements` module of the webui. To read the the json file provided by the `form_elements` trigger request you can use the argument parser as well: ```python ... import json def main(): parser = helper.get_argument_parser() args = parser.parse_args() # Read the input from the form (form.json) with open(args.filename) as form_json: form_data = json.load(form_json) ... ``` ##### Create Files for Download Caosadvancedtools has a helper method named `get_shared_filename`. It generates a pair of paths. The first path is intented for generating the download links for the user. The second is intended for opening and write the file: ```python ... def write_output_file(result): display_path, internal_path = helper.get_shared_filename("output.json") with open(internal_path, 'w') as f: json.dump(result, f) # use the logger to inform the user where the can download the file logger.info(f"Download the output.json.") ... ``` ### Changing Permissions Up until now, our examples only worked for users with the administration role. If anyone else tried to trigger a script they most likely got a permission error. This is the case when a user lacks the appropriate permissions to *call* a script. #### Permissions to Call a Script You can allow users to run a particular script by granting the role permission `SCRIPTING:EXECUTE:?PATH?` where the path is the path of the script in the server's scripting directory (`SERVER_SIDE_SCRIPTING_BIN_DIRS`). > Note: the path can also specify a glob pattern using `*`. E.g. > `SCRIPTING:EXECUTE:*` is the permission to execute any script. > `SCRIPTING:EXECUTE:my_scripts/*` is the permission to execute any script > under `./my_scripts/` in one of the servers scripting directories. > Note: Find more information about permissions in general and how to grant > them in the server docs on permissions. #### Permissions of the Auth Token passed to the Script By default, the Auth Token passed to the script as the `--auth-token` optional parameter authenticates the script against the server as the user who called the script in the first place. However, there are use cases, where we want the script to have 1) more permissions than the caller or use a different user. One example are scripts which are to be called by unprivileged users or the anonymous user which need to update entities. This is a pattern to allow unprivileged users to exactly one operation (like adding a new entry to a list) but nothing else. In this case, you can configure the `authtoken.yaml` (server option: `AUTHTOKEN_CONFIG`) to issue a more powerfull Auth Token for the script. > Note: See the server's documentation for the `AUTHTOKEN_CONFIG` for more > information. ## Example: Putting it All Together The sections show how to create a web form using `form_element` which triggers a server-side script. The script will be executable for any user having the `team` role and it will act with `administration` permissions. ### Create the HTML Form It has one field which is a file upload with the name "csvfile" and one text field with the name "myparameter". The javascript snippet to create such a form using the webui's `form_elements` module looks like this: ```javascript // -- trigger_sss_form.js const form_config = { script: "handle_csv.py", fields: [{ type: "text", name: "myparameter", label: "My Parameter", required: true, help: "Define a parameter", }, { type: "file", name: "csvfile", label: "CSV Table", required: true, accept: "*.csv", help: "Upload a CSV Table", }, ], }; // create the form const form = form_elements.make_form(form_config); // now inject it into the document document.body.append(form); ``` This snippet needs to be installed under `custom/caosdb-server/caosdb-webui/src/ext/js/trigger_sss_form.js` ### Create a Server-Side Script This script accepts a form.json file containing `myparameter` and `csvfile` fields as generated by the above form. It uses the Auth Token to communicate with the server, it uses the logger configured by Caosadvancedtools to print the output nicely renderen as HTML and it returns a download link to the user for downloading a generated file. ```python # -- handle_csv.py import os import logging import linkahead as la from caosadvancedtools.serverside import helper from caosadvancedtools.serverside.logging import configure_server_side_logging LOGGER_NAME="my-logger" logger = logging.getLogger(LOGGER_NAME) def do_something(csv_file_path, my_parameter): """Do something with the uploaded csv file and the parameter, then return a dict.""" la.execute_query("FIND Something") return {"data": "example"} def main(): # setup logging and reporting configure_server_side_logging(LOGGER_NAME) parser = helper.get_argument_parser() args = parser.parse_args() # configure the connection using the auth_token. la.configure_connection(auth_token=args.auth_token) # Read the input from the form (form.json) with open(args.filename) as form_json: form_data = json.load(form_json) # files are uploaded to this directory upload_dir = os.path.dirname((args.filename)) # Read content of the uploaded form data csv_file_path = os.path.join(upload_dir, form_data["csvfile"]) my_parameter = form_data["myparameter"] result = do_something(csv_file_path, my_parameter) # create a downloadable file and write the results into that file display_path, internal_path = helper.get_shared_filename("output.json") with open(internal_path, 'w') as f: json.dump(result, f) # use the logger to inform the user where the can download the file logger.info(f"Download the output.json.") if __name__ == "__main__": main() ``` This file needs to be installed under `custom/caosdb-server/scripting/bin/handle_csv.py` as an executable file. ### Configure the Server You can add a configuration file at `custom/caosdb-server/conf/ext/server.conf.d/sss.conf` Be sure to have the server configured with `SERVER_SIDE_SCRIPTING_BIN_DIRS=`./scripting/bin`. This is the default, so probably you are just fine. Add add a configuration file at `custom/caosdb-server/conf/ext/authtoken.yaml` for the Auth Tokens passed to the script: ```yaml # -- custom/caosdb-server/conf/ext/authtoken.yaml - purpose: SCRIPTING:EXECUTE:handle_csv.py roles: - administration ``` > Note: Find more examples for the configuration in the [authtoken.example.yaml]() Put the file to `custom/caosdb-server/conf/authtoken.yaml` and configure the server with ``` # -- custom/caosdb-server/conf/ext/server.conf.d/sss.conf AUTHTOKEN_CONFIG=`./conf/ext/authtoken.yaml` ``` ## Deep Dive into the Scripting API This section shows how parameters or files are being passed to the script and how you can test an installation. ### Testing an Installation Given that you installed the minimal bash script from the previous section, you can start the server and test it with curl: ```shell curl -F call=minimal.bash 'https://localhost:10443/scripting' -H 'Cookie: SessionToken=[...]' ``` > Note: The scripting API expects a HTTP POST request with content types > `application/x-www-form-urlencoded` or `multipart/form-data`. It has one > mandatory parameter, which is `call` and the value must be the script's path > below the scripting directory. > Note: You need a valid session token. Easiest way to get one: Log in with the > browser, open the developer tools and copy paste it from "Storage" -> > "Cookies" -> "SessionToken" (firefox). > Note: If curl fails with "curl: (60) SSL certificate problem: self-signed > certificate" or a similar error message you must give curl the path to the > public certificate or a signing certificate authority using the `--cacert` or > `--capath` options. If you want to ignore the issue because you only do this > for testing purposes you may use the `--insecure|-k` option. The server's response should look like this: ```xml ... ``` You can see the exit code ` ``` You can see that the parameters have been passed to the script and the effective call was `minimal.bash --option1= --option2=some-value Positional0 Positional1`. > Note: The session token should never be leaked by the script. This script > does it only for didactic reasons. The `...` element prints the > complete call but omits the session token. ### Pass Files to the Script You can upload files and pass the files to the script via the scripting API using the POST request with the `multipart/form-data` content type (this is the standard way of uploading files via HTTP). Let's assume you have a file `data.json` like this: ```json { "my_key": "my value" } ``` You can upload the file and call the script via ```shell curl -F call=minimal.bash -F -Omy-file=@data.json 'https://localhost:10443/scripting' -H 'Cookie: SessionToken=[...]' ``` The server will store the file under `.upload_files/data.json` relative to the scripts working directory. ```xml ... ... ``` You can see that the server passes the file's location to the script. > Note: We used a key starting with `-O` here, resulting in an optional > parameter. However, you can also use a positional paramter by using the `-pN` > keys for the file. We can now add another minimal script `print_file.bash` which prints the content of the uploaded file and install it as shown above. ```bash #!/bin/bash cat $2 ``` > Note: Since the first parameter is *always* the server-generated > `--auth-token` the `$2` refers to the first parameter specified by the user. This script expects to be called with one positional parameter. So we call it via ```shell curl -F call=print_file.bash -F -p0=@data.json 'https://localhost:10443/scripting' -H 'Cookie: SessionToken=[...]' ``` The server's response will look like this: ```xml ... ... ``` You can see that the script can indeed access the file and print the content. ### Using the Auth Token From Within the Script Up until now, we ignored the `--auth-token` parameter. It can be used by the script to make calls to the server. By default, the authentication token passed to script is just a session token for the current session of the user account who is calling the script: If you call the script as "user1" the script can execute any operation on the server as the same "user1" user. Using curl inside a minimal script `user_info.bash` we can see this happening: ```bash #!/bin/bash # extract the session token from the --auth-token option. SESSION_TOKEN="${1#--auth_token=}" curl -o /dev/null -s -w "\nNo SessionToken: %{http_code}" --insecure 'https://localhost:10443/' curl -o /dev/null -s -w "\nWith SessionToken: %{http_code}" --insecure -H "Cookie: SessionToken=${SESSION_TOKEN}" 'https://localhost:10443' ``` If we call this script via the scripting API without any further parameters ```shell curl -k -F call=user_info.bash 'https://localhost:10443/scripting' -H 'Cookie: SessionToken=[...]' ``` the server's response shows a "401" (Unauthorized) response for the first request and a "200" (Ok) response for the second request: ```xml ... ... ``` ### A Pattern: The Server as a Client of Itself. So what did we do in the last step? We installed a script `user_info.bash`. If we call that script via the scripting API, the server executes that script on our behalf. The scripts sends two request back to the server and the scripting API response reaches us after both curl requests of the script terminated. ``` | USER: calls `user_info.bash` | | SERVER: receives the scripting API request | | SERVER: executes `user_info.bash` | | | SCRIPT: curl sends request 1 | | | | SERVER receives the request | | | * SERVER responds with 401 | | | SCRIPT: curl terminates 1st time | | | SCRIPT: curl sends request 2 | | | | SERVER receives the request | | | * SERVER responds with 200 | | | SCRIPT: curl terminates 2nd time | | * SCRIPT: terminates | * SERVER: sends response `