Getting Started with the CaosDB Server

Here, you find information on requirements, the installation, configuration and more. Note, that building the CaosDB Server from the sources and maintaining the server requires the knowledge/skill of an experienced Linux administrator.

Note: If you are looking for a quick and easy way to get startet, check out the docker deployment.

Requirements

See DEPENDENCIES.md.

Install the requirements on Debian

On Debian, the required packages can be installed with:

apt-get install make mariadb-server maven openjdk-11-jdk-headless \
  python3-pip libpam0g-dev unzip

Note that installing MariaDB will uninstall existing MySQL packages and vice versa.

Install the requirements on Fedora

On Fedora, the required packages can be installed with:

sudo dnf install make pam-devel mariadb-server mariadb python3 java-17-openjdk-headless unzip gcc

System

  • >=Linux 4.0.0, x86_64, e.g. Ubuntu 18.04

  • Mounted filesytem(s) with enough space

  • Working internet connection (for up-to-date python and java libraries)

Extensions

Web UI

  • If the WebUI shall run, check out the respective submodule: git submodule update --init caosdb-webui

  • Then configure and compile it according to its documentation.

PAM

Authentication via PAM is possible, for this the PAM development library must be installed and the pam user tool must be compiled:

  • cd misc/pam_authentication/

  • make

  • If you want, you can run a test now: ./pam_authentication.sh asdf asks for a password for user asdf. If no such user exists or the wrong passowrd is entered, it print [FAILED] and return with a non-zero exit code.

  • If you want to run the CaosDB server without root privilege, you need to use the setuid bit for the binary. For example, if the user caosdb runs the server process the permissions of bin/pam_authentication should be the following: -rwsr-x--- 1 root caosdb  pam_authentication

Troubleshooting

If make fails with pam_authentication.c:4:31: fatal error: security/pam_appl.h: No such file or directory the header files are probably not installed. You can do so under Debian and Ubuntu with apt-get install libpam0g-dev. Then try again.

First Setup

After a fresh clone of the repository, this is what you need to setup the server:

  1. Install the proto submodule (and submodules for those extensions you want, see above): git submodule update --init caosdb-proto

  2. Compile the server with make compile. This may take a while and there needs to be an internet connection as packages are downloaded to be integrated in the java file.

    1. It is recommended to run the unit tests with make test. It may take a while.

  3. Create an SSL certificate somewhere with a Java Key Store file. For self-signed certificates (not recommended for production use) you can do:

    • mkdir certificates; cd certificates

    • keytool -genkey -keyalg RSA -alias selfsigned -keystore caosdb.jks -validity 375 -keysize 2048 -ext san=dns:localhost Replace localhost by your host name, if you want.

    • keytool -importkeystore -srckeystore caosdb.jks -destkeystore caosdb.p12 -deststoretype PKCS12 -srcalias selfsigned

    • Export the public part only: openssl pkcs12 -in caosdb.p12 -nokeys -out cert.pem. The resulting cert.pem can safely be given to users to allow ssl verification.

    • You can check the content of the certificate with openssl x509 -in cert.pem -text

    Alternatively, you can create a keystore from certificate files that you already have:

    • openssl pkcs12 -export -inkey privkey.pem -in fullchain.pem -out all-certs.pkcs12

    • keytool -importkeystore -srckeystore all-certs.pkcs12 -srcstoretype PKCS12  -deststoretype pkcs12 -destkeystore caosdb.jks

  4. Install/configure the MySQL back-end: see the README_SETUP.md of the caosdb-mysqlbackend repository

  5. Create an authtoken config (e.g. copy conf/core/authtoken.example.yaml to conf/ext/authtoken.yml and change it)

  6. If any of the settings in conf/core/server.conf do not fit your needs, create conf/ext/server.conf and add settings as required:

    • Setup for MySQL back-end: specify the fields MYSQL_USER_NAME, MYSQL_USER_PASSWORD, MYSQL_DATABASE_NAME, and MYSQL_HOST.

    • Choose the host and ports under which CaosDB will be accessible. The host is the IP address the server should listen on. Blank (the default) means all interfaces, and 127.0.0.1 means only localhost.

    • Setup the SSL certificate: Assuming that there is an appropriate Java Key Store file (see above), change the fields CERTIFICATES_KEY_PASSWORD, CERTIFICATES_KEY_STORE_PATH, and CERTIFICATES_KEY_STORE_PASSWORD. Make sure that the conf file is not readable by other users because the certificate passwords are stored in plaintext.

    • Set the path to the authtoken config (see step 4)

    • Set the file system paths:

      • FILE_SYSTEM_ROOT: The root for all the files managed by CaosDB.

      • DROP_OFF_BOX: Files can be put here for insertion into CaosDB.

      • TMP_FILES: Temporary files go here, for example during script execution or when uploading or moving files.

      • SHARED_FOLDER: Folder for sharing files via cryptographic tokens, also those created by scripts.

      • SERVER_SIDE_SCRIPTING_BIN_DIRS: A comma or white space separated list of directories (relative or absolute) where the server will be looking for executables which are then callable as server-side scripts. By default this list only contains ./scripting/bin. If you want to include e.g. scripts which are maintained as part of the caosdb-webui repository (because they are intended for usage by the webui), you should add ./caosdb-webui/sss_bin/ as well.

      • INSERT_FILES_IN_DIR_ALLOWED_DIRS: add mounted filesystems here that shall be accessible by CaosDB

    • Maybe set another SESSION_TIMEOUT_MS.

    • See also CONFIGURATION.rst

  7. Copy conf/core/usersources.ini.template to conf/ext/usersources.ini.

    • You can skip this if you do not want to use an external authentication. Local users (CaosDB realm) are always available.

    • Define the users/groups who you want to include/exclude.

    • Assign at least one user the administration role.

      • For example, if the admin user is called caosdb, there should be the following lines:

        include.user = caosdb
user.caosdb.roles = administration

    • It is important that the file complies with the ini file specification. Especially that there are no properties (aka keys) without a value. An emtpy value can be represented by "". Comments are everything from # or ; to the end of the line.

  8. Possibly install the PAM caller in misc/pam_authentication/ if you have not do so already. See above.

Done!

Start Server

$ make run

This can take a while. Once you see Starting org.caosdb.server.CaosDBServer application the server is ready and you can try it out by connecting with a client, e.g. the web client, if you installed it. Typically, you just need to type https://localhost:10443 in your Browser, assuming you used 10443 as port. Note, that you will get a security warning if you are using a self-signed certificate.

You can use make run-single to directly start the server without rebuilding sources (if you called make run previously).

Run Unit Tests

$ make test

You can run single unit test with mvn test -X -Dtest=TestCQL#testDecimalNumber

Setup Eclipse

  1. Open Eclipse (tested with 2022-R12)

  2. File > Import > Maven > Existing Maven Projects: Specify location.

  3. You will most likely encounter “Plugin execution not covered by lifecycle configuration: …” errors. Adapt the file <eclipse-workspace>/.metadata/.plugins/org.eclipse.m2e.core/lifecycle-mapping-metadata.xml.

    Example:

    <?xml version="1.0" encoding="UTF-8"?>
<lifecycleMappingMetadata>
  <pluginExecutions>
    <pluginExecution>
      <pluginExecutionFilter>
        <groupId>com.coveo</groupId>
        <artifactId>fmt-maven-plugin</artifactId>
        <versionRange>2.5.1</versionRange>
        <goals>
          <goal>format</goal>
        </goals>
      </pluginExecutionFilter>
      <action>
        <ignore />
      </action>
    </pluginExecution>
    <pluginExecution>
      <pluginExecutionFilter>
        <groupId>org.codehaus.mojo</groupId>
        <artifactId>buildnumber-maven-plugin</artifactId>
        <versionRange>1.4</versionRange>
        <goals>
          <goal>create-metadata</goal>
        </goals>
      </pluginExecutionFilter>
      <action>
        <ignore />
      </action>
    </pluginExecution>
  </pluginExecutions>
</lifecycleMappingMetadata>

Done!

Migration

From 0.1 to 0.2

A major change in the code is the renaming of the java packages (from caosdb.[...] to org.caosdb.[...]).

This makes its necessary to change some of your config files as well. Whenever you configured a class (e.g. the EMAIL_HANDLER, or the realms in your usersources.ini) you would need to reconfigure it there.

The following sed command could be useful. However, use it with care and backup before you execute it.

sed -i.bak -e "s/\(\s*\)\([^.]\)caosdb\.server/\1\2org.caosdb.server/g" FILE_TO_BE_CHANGED

Build the documentation

Stand-alone documentation is built using Sphinx: make doc

Requirements

  • plantuml

  • recommonmark

  • sphinx

  • sphinx-rtd-theme

  • sphinx-a4doc

  • sphinxcontrib-plantuml

  • javasphinx :: pip3 install --user javasphinx

    • Alternative, if javasphinx fails because python3-sphinx is too recent: (l_ not found):

git clone https://github.com/simgrid/javasphinx.git
cd javasphinx
git checkout 659209069603a
pip3 install .

Installation under special conditions

Installation without or with restricted internet connection on the host system

It is of course possible to install CaosDB on servers which do not have an internet connection or only have limited access to the internet. However, maven, which is used to build the CaosDB server, typically accesses the internet very often, and some reconfiguration might be necessary to prevent timeout issues and similar headache.

A recommendation to proceed is:

  • Clone the repositories (caosdb-server, caosdb-mysqlbackend and subrepository caosdb-webui) on a second machine that has internet connection.

  • Run make install and make run in the caosdb-server repository. This will download all required dependencies and setup all maven-related artifacts in a folder called .m2 typically located in the home directory.

  • In addition to copying over the repositories for the server, mysqlbackend and webui to the target host machine, also copy the complete .m2 folder to the home directory of the user that is running the CaosDB server.

  • On the host machine, open the file pom.xml in the server repository.

  • Look for the section <repositories>. There should be a <respository> with <id>local-maven-repo</id>. It should be moved on top of the list of repositories, so that the xml block appears directly after <repositories> and before the repository maven-central.

Troubleshooting / FAQ

I set up and run the server, but I get an error that XSLT parsing failed in the WebUI

You probably forgot to make the webui.

Make sure that you:

  • Installed the WebUI according to the section [Web UI]

  • Run make install in the webui sub folder

How can I run the CaosDB server using ports 443 and 80?

On standard linux setups, ports 443 and 80 cannot be opened by non-root processes.

There are multiple solutions for this problem:

  • Use iptables to redirect a low port to a high port.

  • Use CAP_NET_BIND_SERVICE.

  • Use authbind.

Here are some resources to read about these options:

Using authbind is a simple solution that is briefly described here:

  • Install the package authbind

  • Create (empty) files /etc/authbind/byport/80 (and .../443), e.g. using touch

  • Grant execution permissions for the user who runs the server to the new files.

  • Run the CaosDB server using authbind with the --deep option: authbind --deep make run. The --deep option is necessary because the server starts a subprocess which actually opens the ports.