Backup LinkAhead Data

Create a backup of LinkAhead via

linkahead -p path/to/profile.yml backup -d path/to/backup/directory

in order to save the SQL dump of your data (the *.dump.sql file) and a backup of LinkAhead’s internal file system (the *.tar.gz).

Regular Backups

Of course, it is recommended to do regular backups. One way to do this is to use cronjob. This can be nicely combined with rsnapshot to prevent the backups from growing too much. With rsnapshot, rotating backups can be created that are kept for predefined intervals. To combine this with LinkAhead backups, do the following.

  1. Create a specific LinkAhead backup script that will be executed by rsnapshot and that stores the backups in its current working directory. For example, create a /usr/local/bin/backup_linkahead_rsnapshot.sh which contains the following lines (may have to be adapted to your LinkAhead installation, see above).

    #!/bin/sh

linkahead backup -d ./

  2. Adjust the rsnapshot configuration in /etc/rsnapshot.conf (see its documentation):

    • Create reasonable intervals that correspond to the frequency with which backups are rotated, e.g., daily, weekly, and monthly of which 14, 8, and 12 copies are kept, respectively. I.e., you’ll be keeping daily backups for two weeks, weekly for the following two months, and monthly for a year.

    • Choose a meaningful snapshot_root in which the backups will be stored.

    • Add the script created above to the backup scripts in /etc/rsnapshot.conf. In the above example, the line looks like

    backup_script   /usr/local/bin/backup_linkahead_rsnapshot.sh    linkahead/

    • Possibly remove standard backup directories like /etc/ and /home/ from the /etc/rsnapshot.conf

  3. Test the config with rsnapshot configtest for correctness.

  4. Make a test run. For example rsnapshot -t daily or rsnapshot daily. The former performs a dry run, the latter actually creates a backup.

  5. Add the rsnapshot commands to execute the backup intervals defined above to your crontab. In the above example, daily could be executed each day at 2 am, weekly on every Sunday at 1:30, and monthly on each first day of the month at 1 am with the following lines.

    0 2 * * *  /usr/bin/rsnapshot daily
30 1 * * 1 /usr/bin/rsnapshot weekly
0 1 1 * *  /usr/bin/rsnapshot monthly

  6. Check (after some time) logs in /var/log/syslog (default log location)

Upgrading the SQL backend

Some LinkAhead releases which upgrade the SQL backend may introduce incompatibilities with stored SQL dump backup files. For these cases, the backend repository provides upgrade scripts which must be called on the SQL dump before restoring data from the dump.

For detailed information, please read the file dump_updates/README.md in the SQL backend repository.

Known upgrade problems and solutions

These are the known problems and their solutions for specific LinkAhead releases:

  • 0.15.0: This release updates the backend to MariaDB 10.11 which introduces a new keyword. To fix your SQL dumps run
    2024-10-02.dump_fix_mariadb_10_6.sh < yourdump.sql > yourdump.fixed.sql

Manage the SQL database manually

How to export the SQL database folder manually (e.g. for backup)

If it should ever be necessary to export the SQL database folder manually (for example for low-level backup, development or debugging), use these hints.

The idea is: Mount the Docker volume with the database into a temporary container, then export it from there.

docker run --rm --mount "src=$(pwd),dst=/sql_export,type=bind" \
  --mount "src=compose_caosdb-sqldata,dst=/sql,type=volume,ro" debian bash \
  -c "tar -czf /sql_export/sql_data.tar.gz /sql/"

The data will be now inside sql_data.tar.gz.

How to migrate the SQL database from a bind mount directory to a Docker volume

Attention: Be aware that you can delete Docker volumes with the Docker client (e.g. docker volume prune or docker system prune --volumes). Thus be careful not to lose data when you use these commands.

In order to migrate an SQL database from a directory that was mounted via bind mounts to a Docker volume, you simply need to populate the right volume with the existing data. In the following let us assume that the default project name is used (otherwise replace compose with the project name).

  • Note that if a volume named compose_caosdb-sqldata existed before, you should remove it first (docker volume rm compose_caosdb-sqldata).

  • Go to the parent folder of the sql data directory (typically somepath/profile/paths) and start a container which mounts the old sql data folder and a new volume with the correct name:

docker run --rm --mount "src=$(pwd)/sql_data,dst=/sql_data,type=bind,ro" \
  --mount "src=compose_caosdb-sqldata,dst=/new,type=volume" debian bash \
  -c "cp -a /sql_data/* /new"

You can now start LinkAhead using Docker volumes. Check that the migration from a plain directory to a Docker volume succeeded, then you can remove the old directory.

How to migrate from LinkAhead to a plain CaosDB installation

This section describes how to migrate from LinkAhead, the CaosDB distribution based on Docker, to a plain CaosDB installation, compiled from the sources.

Note: Even without a LinkAhead subscription, running LinkAhead installations will still work, but are not guaranteed to be compatible with future versions of other CaosDB components.

In order to use your custom extensions, your LinkAhead configuration, and, most importantly, the contents of your database with a plain CaosDB installation, the following steps should to be taken. Note that you should be familiar with the configuration in your profile.yml in your profile directory. Also, some basic knowledge of SQL, MySQL and MariaDB may be helpful.

Backup

Create a backup of the data as described above. Both, the tar ball and the SQL dump, will be restored in the new CaosDB installation later on. Shutdown LinkAhead (systemctl stop linkahead or linkahead -p path/to/profile.yml depending on your LinkAhead installation) afterwards for the setup of the new CaosDB installation.

Install CaosDB server and the WebUI

Get and install the CaosDB server and the web interface by following the installation instructions for the server, the MySQL backend, and the web interface. If you declared specific commits, branches or tags for CaosDB components in your profile.yml, you may want to use these for the manual installations as well to ensure the compatibility of your custom extensions to the new CaosDB installation.

You may also want to reuse the SSL certificates of your LinkAhead installation. You’ll usually find them in the custom/other/cert directory. See below on how to use the configuration files from your LinkAhead installation for the configuration of the CaosDB server, the MySQL backend, and the web interface.

Restore the backups

Restoring the backups from a LinkAhead installation is a two-step process:

  1. Restore the SQL dump of the database.

  2. Transfer the internal file system from the LinkAhead installation to the new CaosDB installation.

Restoring the SQL dump

Use the script provided in the MySQL backend repository as described in the documentation to restore the SQL dump that you created earlier.

Restoring the CaosDB file system

Untar the file-system backup created above into the directory specified as FILE_SYSTEM_ROOT in your server.conf according to CaosDB server’s maintenance documentation.

Adapt the configuration

Note: This manual covers only the case of managing user accounts directly in CaosDB. See the server documentation if you use an external authentication service like PAM.

Configuring the MySQL backend

To set up the MySQL backend, configure it according to the documentation and make sure that the database name and the user match those of your LinkAhead installation. Usually the defaults, i.e. database caosdb and user caosdb should do fine. For the tests to run through, you may have to create a 'caosdb'@'%' user in your MySQL service.

Configuring the server

Copy the server configuration files from your LinkAhead installation (usually in custom/caosdb-server/) to your CaosDB server directory instead of creating them during the corresponding steps of the server installation.

Transfer the contents of all customizations from your LinkAhead server configuration (usually the contents of the files in custom/other/caosdb-server/conf/ext/server.conf.d/) to the /conf/ext/server.conf file that is created during the installation of the server. Mind that some server configurations (e.g. whether there is an anonymous user, the network settings and much more) are contained in your profile.yml. Also, don’t forget to enter the passwords for your SSL certificates in the /conf/ext/server.conf file.

If you use server-side scripts in your LinkAhead installation, you should copy them (usually from custom/caosdb-server/scripting/bin) to a directory specified in the SERVER_SIDE_SCRIPTING_BIN_DIRS variable in your server.conf.

Configuring the web interface

Again, copy your configuration files from your LinkAhead installation (usually in custom/caosdb-server/caosdb-webui/conf/ext) to the corresponding locations in conf/ext of your new WebUI installation. Do the same for the custom build properties (usually in custom/caosdb-server/caosdb-webui/build.properties.d). Remember to also copy the settings from the theme-linkahead module (usually located in theme-linkahead) in the same way.

If you have custom extensions for your web interface, copy them (usually from the contents of custom/caosdb-server/caosdb-webui/src/ext) to the src/ext directory of your new WebUI installation. Remember to do the same for the theme-linkahead module if you want to.

(Re-)Compile the web interface after these steps.

Testing the new setup

After installation and configuration of the server, backend, and web interface, start the server by executing make run from the caosdb-server directory. Open the web interface with a browser of your choice at the address and port you specified (e.g. https://localhost:10443) and check whether all extensions work as expected. Also try and connect to the server via e.g. the Python client.