Getting Started with the CaosDB Server

Here, you find information on requirements, the installation, configuration and more.

Requirements

CaosDB Packages

  • caosdb-webui=0.2.1
  • caosdb-mysqlbackend=3.0

Third-party Software

  • >=Java 8
  • >=Apache Maven 3.0.4
  • >=Python 3.4
  • >=pip 9.0.1
  • >=git 1.9.1
  • >=Make 3.81
  • >=Screen 4.01
  • >=MySQL 5.5 (better >=5.6) or >=MariaDB 10.1
  • libpam (if PAM authentication is required)
  • unzip
  • openpyxl (for XLS/ODS export)
  • openssl (if a custom TLS certificate is required)
  • easy-units >= 0.0.1 https://gitlab.com/timm.fitschen/easy-units

Install the requirements on Debian

On Debian, the required packages can be installed with:

apt-get install git make mariadb-server maven openjdk-11-jdk-headless \
  python3-pip screen libpam0g-dev unzip

Note that installing MariaDB will uninstall existing MySQL packages and vice versa.

System

  • >=Linux 4.0.0, x86_64, e.g. Ubuntu 18.04
  • Mounted filesytem(s) with enough space
  • Working internet connection (for up-to-date python and java libraries)

Extensions

Web UI

  • If the WebUI shall run, check out the respective submodule: git submodule update --init caosdb-webui
  • Then configure and compile it according to its documentation.

PAM

Authentication via PAM is possible, for this the PAM development library must be installed and the pam user tool must be compiled:

  • cd misc/pam_authentication/
  • make
  • If you want, you can run a test now: ./pam_authentication.sh asdf ghjk should print [FAILED] and return with a non-zero exit code. Unless there is a user asdf with password ghjk on your system, of course.
Troubleshooting

If make fails with pam_authentication.c:4:31: fatal error: security/pam_appl.h: No such file or directory the header files are probably not installed. You can do so under Debian and Ubuntu with apt-get install libpam0g-dev. Then try again.

First Setup

After a fresh clone of the repository, this is what you need to setup the server:

  1. Compile the server with make compile. This may take a while and there needs to be an internet connection as packages are downloaded to be integrated in the java file.

    1. It is recommended to run the unit tests with make test. It may take a while.
  2. Create an SSL certificate somewhere with a Java Key Store file. For self-signed certificates (not recommended for production use) you can do:

    • mkdir certificates; cd certificates
    • keytool -genkey -keyalg RSA -alias selfsigned -keystore caosdb.jks -validity 375 -keysize 2048 -ext san=dns:localhost Replace localhost by your host name, if you want.
    • keytool -importkeystore -srckeystore caosdb.jks -destkeystore caosdb.p12 -deststoretype PKCS12 -srcalias selfsigned
    • Export the public part only: openssl pkcs12 -in caosdb.p12 -nokeys -out cert.pem. The resulting cert.pem can safely be given to users to allow ssl verification.
    • You can check the content of the certificate with openssl x509 -in cert.pem -text

    Alternatively, you can create a keystore from certificate files that you already have:

    • openssl pkcs12 -export -inkey privkey.pem -in fullchain.pem -out all-certs.pkcs12
    • keytool -importkeystore -srckeystore all-certs.pkcs12 -srcstoretype PKCS12  -deststoretype pkcs12 -destkeystore caosdb.jks
  3. Install/configure the MySQL back-end: see the README_SETUP.md of the caosdb-mysqlbackend repository

  4. Create an authtoken config (e.g. copy conf/core/authtoken.example.yaml to conf/ext/authtoken.yml and change it)

  5. Copy conf/core/server.conf to conf/ext/server.conf and change it appropriately:

    • Setup for MySQL back-end: specify the fields MYSQL_USER_NAME, MYSQL_USER_PASSWORD, MYSQL_DATABASE_NAME, and MYSQL_HOST.
    • Choose the ports under which CaosDB will be accessible.
    • Setup the SSL certificate: Assuming that there is an appropriate Java Key Store file (see above), change the fields CERTIFICATES_KEY_PASSWORD, CERTIFICATES_KEY_STORE_PATH, and CERTIFICATES_KEY_STORE_PASSWORD. Make sure that the conf file is not readable by other users because the certificate passwords are stored in plaintext.
    • Set the path to the authtoken config (see step 4)
    • Set the file system paths:
      • FILE_SYSTEM_ROOT: The root for all the files managed by CaosDB.
      • DROP_OFF_BOX: Files can be put here for insertion into CaosDB.
      • TMP_FILES: Temporary files go here, for example during script execution or when uploading or moving files.
      • SHARED_FOLDER: Folder for sharing files via cryptographic tokens, also those created by scripts.
      • SERVER_SIDE_SCRIPTING_BIN_DIRS: A comma or white space separated list of directories (relative or absolute) where the server will be looking for executables which are then callable as server-side scripts. By default this list only contains ./scripting/bin. If you want to include e.g. scripts which are maintained as part of the caosdb-webui repository (because they are intended for usage by the webui), you should add ./caosdb-webui/sss_bin/ as well.
      • INSERT_FILES_IN_DIR_ALLOWED_DIRS: add mounted filesystems here that shall be accessible by CaosDB
    • Maybe set another SESSION_TIMEOUT_MS.
    • See also README_CONFIGURATION.md
  6. Copy conf/core/usersources.ini.template to conf/ext/usersources.ini.

    • You can skip this if you do not want to use an external authentication. Local users (CaosDB realm) are always available.

    • Define the users/groups who you want to include/exclude.

    • Assign at least one user the administration role.

      • For example, if the admin user is called caosdb, there should be the following lines:

        include.user = caosdb
        user.caosdb.roles = administration
        
    • It is important that the file complies with the ini file specification. Especially that there are no properties (aka keys) without a value. An emtpy value can be represented by "". Comments are everything from # or ; to the end of the line.

  7. Possibly install the PAM caller in misc/pam_authentication/ if you have not do so already. See above.

Done!

Start Server

$ make run

This can take a while. Once you see Starting org.caosdb.server.CaosDBServer application the server is ready and you can try it out by connecting with a client, e.g. the web client, if you installed it. Typically, you just need to type https://localhost:10443 in your Browser, assuming you used 10443 as port. Note, that you will get a security warning if you are using a self-signed certificate.

Run Unit Tests

$ make test

Setup Eclipse

  1. Open Eclipse (recommended version: Oxygen.1a Release (4.7.1a))
  2. File > New > Java Project: Choose a project name and specify the location of this repo. The JRE and Project layout should be configured automatically. Now, the project should initially have two source-folders: ./src/main/java and ./src/test/java. After a build, another one, ./target/generated-sources/antlr4 should be generated. If there are more than these three source-folders, reconfigure the projects source folders appropriately with Project > Properties > Java Build Path > Source.
  3. In the Package Explorer view, right-click on the project and Configure > Convert to Maven Project.
  4. In the Package Explorer view, right-click on the project and Maven > Update Project.
  5. Usually a build of the project is started automatically. Otherwise Project > Build Project.

Done!

Migration

From 0.1 to 0.2

A major change in the code is the renaming of the java packages (from caosdb.[...] to org.caosdb.[...]).

This makes its necessary to change some of your config files as well. Whenever you configured a class (e.g. the EMAIL_HANDLER, or the realms in your usersources.ini) you would need to reconfigure it there.

The following sed command could be useful. However, use it with care and backup before you execute it.

sed -i.bak -e "s/\(\s*\)\([^.]\)caosdb\.server/\1\2org.caosdb.server/g" FILE_TO_BE_CHANGED

Build the documentation

Stand-alone documentation is built using Sphinx: make doc

Requirements

  • plantuml
  • recommonmark
  • sphinx
  • sphinx-rtd-theme
  • sphinxcontrib-plantuml
  • javasphinx :: pip3 install --user javasphinx
    • Alternative, if javasphinx fails because python3-sphinx is too recent: (l_ not found):
git clone git@github.com:simgrid/javasphinx.git
cd javasphinx
git checkout 659209069603a
pip3 install .