# Getting Started with the CaosDB Server

Here, you find information on requirements, the installation, configuration and
more. Note, that building the CaosDB Server from the sources and maintaining
the server requires the knowledge/skill of an experienced Linux administrator.

**Note:** If you are looking for a quick and easy way to get startet, check
out [the docker deployment](https://gitlab.indiscale.com/caosdb/src/caosdb-docker).

## Requirements

See [DEPENDENCIES.md](DEPENDENCIES.md).

#### Install the requirements on Debian

On Debian, the required packages can be installed with:

    apt-get install make mariadb-server maven openjdk-11-jdk-headless \
      python3-pip libpam0g-dev unzip

Note that installing MariaDB will uninstall existing MySQL packages and vice
versa.

#### Install the requirements on Fedora

On Fedora, the required packages can be installed with:

    sudo dnf install make pam-devel mariadb-server mariadb python3 java-17-openjdk-headless unzip gcc

### System

* `>=Linux 4.0.0`, `x86_64`, e.g. Ubuntu 18.04
* Mounted filesytem(s) with enough space
* Working internet connection (for up-to-date python and java libraries)

### Extensions ##

#### Web UI ###
- If the WebUI shall run, check out the respective submodule: `git submodule
  update --init caosdb-webui`
- Then configure and compile it according to its
  [documentation](https://docs.indiscale.com/caosdb-webui/getting_started.html).

#### PAM ###
Authentication via PAM is possible, for this the PAM development library must be
installed and the pam user tool must be compiled:

- `cd misc/pam_authentication/`
- `make`
- If you want, you can run a test now: `./pam_authentication.sh asdf` asks for a password for user
  `asdf`.  If no such user exists or the wrong passowrd is entered, it print `[FAILED]` and return
  with a non-zero exit code.
- If you want to run the CaosDB server without root privilege, you need to use
  the setuid bit for the binary. For example, if the user `caosdb` runs the
  server process the permissions of `bin/pam_authentication` should be the
  following:
  `-rwsr-x--- 1 root caosdb  pam_authentication`

##### Troubleshooting ####
If `make` fails with `pam_authentication.c:4:31: fatal error:
security/pam_appl.h: No such file or directory` the header files are probably
not installed. You can do so under Debian and Ubuntu with `apt-get install
libpam0g-dev`. Then try again.


## First Setup

After a fresh clone of the repository, this is what you need to setup the
server:

1. Install the `proto` submodule (and submodules for those extensions you want, see above):
   `git submodule update --init caosdb-proto`
2. Compile the server with `make compile`. This may take a while and there
   needs to be an internet connection as packages are downloaded to be
   integrated in the java file.
   1. It is recommended to run the unit tests with `make test`. It may take a
      while.
3. Create an SSL certificate somewhere with a `Java Key Store` file.  For
   self-signed certificates (not recommended for production use) you can do:
   - `mkdir certificates; cd certificates`
   - `keytool -genkey -keyalg RSA -alias selfsigned -keystore caosdb.jks -validity 375 -keysize 2048 -ext san=dns:localhost`
     Replace `localhost` by your host name, if you want.
   - `keytool -importkeystore -srckeystore caosdb.jks -destkeystore caosdb.p12 -deststoretype PKCS12 -srcalias selfsigned`
   - Export the public part only: `openssl pkcs12 -in caosdb.p12 -nokeys -out cert.pem`.
     The resulting `cert.pem` can safely be given to users to allow ssl
     verification.
   - You can check the content of the certificate with `openssl x509 -in cert.pem -text`

   Alternatively, you can create a keystore from certificate files that you already have:
   - `openssl pkcs12 -export -inkey privkey.pem -in fullchain.pem -out all-certs.pkcs12`
   - `keytool -importkeystore -srckeystore all-certs.pkcs12 -srcstoretype PKCS12  -deststoretype pkcs12 -destkeystore caosdb.jks`
4. Install/configure the MySQL back-end: see the `README_SETUP.md` of the
   `caosdb-mysqlbackend` repository
5. Create an authtoken config (e.g. copy `conf/core/authtoken.example.yaml` to
   `conf/ext/authtoken.yml` and change it)
6. If any of the settings in `conf/core/server.conf` do not fit your needs, create
   `conf/ext/server.conf` and add settings as required:
    * Setup for MySQL back-end:
      specify the fields `MYSQL_USER_NAME`, `MYSQL_USER_PASSWORD`,
      `MYSQL_DATABASE_NAME`, and `MYSQL_HOST`.
    * Choose the host and ports under which CaosDB will be accessible. The host
      is the IP address the server should listen on. Blank (the default) means
      all interfaces, and `127.0.0.1` means only localhost.
    * Setup the SSL certificate: Assuming that there is an appropriate `Java Key
      Store` file (see above), change the fields `CERTIFICATES_KEY_PASSWORD`,
      `CERTIFICATES_KEY_STORE_PATH`, and `CERTIFICATES_KEY_STORE_PASSWORD`.
      Make sure that the conf file is not readable by other users because the
      certificate passwords are stored in plaintext.
    * Set the path to the authtoken config (see step 4)
    * Set the file system paths:
      - `FILE_SYSTEM_ROOT`: The root for all the files managed by CaosDB.
      - `DROP_OFF_BOX`: Files can be put here for insertion into CaosDB.
      - `TMP_FILES`: Temporary files go here, for example during script
        execution or when uploading or moving files.
      - `SHARED_FOLDER`: Folder for sharing files via cryptographic tokens,
        also those created by scripts.
      - `SERVER_SIDE_SCRIPTING_BIN_DIRS`: A comma or white space separated list
        of directories (relative or absolute) where the server will be looking
        for executables which are then callable as server-side scripts. By
        default this list only contains `./scripting/bin`. If you want to
        include e.g. scripts which are maintained as part of the caosdb-webui
        repository (because they are intended for usage by the webui), you
        should add `./caosdb-webui/sss_bin/` as well.
      - `INSERT_FILES_IN_DIR_ALLOWED_DIRS`: add mounted filesystems here that
        shall be accessible by CaosDB
    * Maybe set another `SESSION_TIMEOUT_MS`.
     * See also [CONFIGURATION.rst](src/doc/administration/configuration.rst)
7. Copy `conf/core/usersources.ini.template` to `conf/ext/usersources.ini`.
    * You can skip this if you do not want to use an external authentication. 
      Local users (CaosDB realm) are always available.
    * Define the users/groups who you want to include/exclude.
    * Assign at least one user the `administration` role.
      * For example, if the admin user is called `caosdb`, there should be the
        following lines:
        ```
        include.user = caosdb
        user.caosdb.roles = administration
        ```
    * It is important that the file complies with the ini file specification.
      Especially that there are no `properties` (aka `keys`) without a
      `value`. An emtpy value can be represented by `""`. Comments are
      everything from `#` or `;` to the end of the line.
8. Possibly install the PAM caller in `misc/pam_authentication/` if you have
   not do so already. See above.
   
Done!

## Start Server

`$ make run`

This can take a while. Once you see `Starting org.caosdb.server.CaosDBServer
application` the server is ready and you can try it out by connecting with a
client, e.g. the web client, if you installed it. Typically, you just need to
type `https://localhost:10443` in your Browser, assuming you used 10443 as port.
Note, that you will get a security warning if you are using a self-signed
certificate.

You can use `make run-single` to directly start the server without rebuilding
sources (if you called `make run` previously).

## Run Unit Tests

`$ make test`

You can run single unit test with
`mvn test -X -Dtest=TestCQL#testDecimalNumber`


## Setup Eclipse

1. Open Eclipse (tested with 2022-R12)
2. File > Import > Maven > Existing Maven Projects: Specify location.
3. You will most likely encounter "Plugin execution not covered by lifecycle
   configuration: ..." errors. Adapt the file
   `<eclipse-workspace>/.metadata/.plugins/org.eclipse.m2e.core/lifecycle-mapping-metadata.xml`.

   Example:

   ```xml 
   <?xml version="1.0" encoding="UTF-8"?>
   <lifecycleMappingMetadata>
     <pluginExecutions>
       <pluginExecution>
         <pluginExecutionFilter>
           <groupId>com.coveo</groupId>
           <artifactId>fmt-maven-plugin</artifactId>
           <versionRange>2.5.1</versionRange>
           <goals>
             <goal>format</goal>
           </goals>
         </pluginExecutionFilter>
         <action>
           <ignore />
         </action>
       </pluginExecution>
       <pluginExecution>
         <pluginExecutionFilter>
           <groupId>org.codehaus.mojo</groupId>
           <artifactId>buildnumber-maven-plugin</artifactId>
           <versionRange>1.4</versionRange>
           <goals>
             <goal>create-metadata</goal>
           </goals>
         </pluginExecutionFilter>
         <action>
           <ignore />
         </action>
       </pluginExecution>
     </pluginExecutions>
   </lifecycleMappingMetadata>
   ```

Done!

## Migration

### From 0.1 to 0.2

A major change in the code is the renaming of the java packages (from
`caosdb.[...]` to `org.caosdb.[...]`).

This makes its necessary to change some of your config files as well. Whenever
you configured a class (e.g. the `EMAIL_HANDLER`, or the realms in your
`usersources.ini`) you would need to reconfigure it there.

The following `sed` command could be useful. However, use it with care and backup
before you execute it.

```sh
sed -i.bak -e "s/\(\s*\)\([^.]\)caosdb\.server/\1\2org.caosdb.server/g" FILE_TO_BE_CHANGED
```

## Build the documentation #

Stand-alone documentation is built using Sphinx: `make doc`

### Requirements ##

- plantuml
- recommonmark
- sphinx
- sphinx-rtd-theme
- sphinx-a4doc
- sphinxcontrib-plantuml
- javasphinx :: `pip3 install --user javasphinx`
  - Alternative, if javasphinx fails because python3-sphinx is too recent:
    (`l_` not found):

```sh
git clone https://github.com/simgrid/javasphinx.git
cd javasphinx
git checkout 659209069603a
pip3 install .
```

## Installation under special conditions

### Installation without or with restricted internet connection on the host system

It is of course possible to install CaosDB on servers which do not have an
internet connection or only have limited access to the internet.
However, maven, which is used to build the CaosDB server, typically accesses
the internet very often, and some reconfiguration might be necessary to prevent
timeout issues and similar headache.

A recommendation to proceed is:
- Clone the repositories (caosdb-server, caosdb-mysqlbackend and subrepository
  caosdb-webui) on a second machine that has internet connection.
- Run `make install` and `make run` in the caosdb-server repository. This will
  download all required dependencies and setup all maven-related artifacts in
  a folder called `.m2` typically located in the home directory.
- In addition to copying over the repositories for the server, mysqlbackend and
  webui to the target host machine, also copy the complete `.m2` folder to the
  **home directory of the user that is running the CaosDB server**.
- On the host machine, open the file `pom.xml` in the server repository.
- Look for the section `<repositories>`. There should be a `<respository>` with
  `<id>local-maven-repo</id>`. It should be moved on top of the list of
  repositories, so that the xml block appears directly after `<repositories>`
  and before the repository maven-central.

## Troubleshooting / FAQ

### I set up and run the server, but I get an error that XSLT parsing failed in the WebUI

You probably forgot to `make` the webui.

Make sure that you:
- Installed the WebUI according to the section [Web UI]
- Run `make install` in the webui sub folder

### How can I run the CaosDB server using ports 443 and 80?

On standard linux setups, ports 443 and 80 cannot be opened by non-root processes.

There are multiple solutions for this problem:
- Use iptables to redirect a low port to a high port.
- Use `CAP_NET_BIND_SERVICE`.
- Use authbind.

Here are some resources to read about these options:
- [Question on Stackoverflow](https://stackoverflow.com/questions/413807/is-there-a-way-for-non-root-processes-to-bind-to-privileged-ports-on-linux)
- [Questoin on Superuser](https://superuser.com/questions/710253/allow-non-root-process-to-bind-to-port-80-and-443)
- [iptables tutorial](https://www.frozentux.net/iptables-tutorial/iptables-tutorial.html#REDIRECTTARGET)

Using authbind is a simple solution that is briefly described here:
- Install the package authbind
- Create (empty) files `/etc/authbind/byport/80` (and `.../443`), e.g. using `touch`
- Grant execution permissions for the user who runs the server to the new files.
- Run the CaosDB server using authbind with the `--deep` option:
  `authbind --deep make run`. The `--deep` option is necessary because the
  server starts a subprocess which actually opens the ports.