# Changelog

All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [0.12.2] - 2024-03-18

### Fixed

* Unknown error when trying to access the user list.
  [linkahead-server#250](https://gitlab.com/linkahead/linkahead-server/-/issues/250)
* Legacy integer ids are incremented by 1 again, not by 2.

## [0.12.1] - 2023-12-13
(Timm Fitschen)

### Fixed

* Insufficient permission checks during subproperty filters of SELECT queries
  when an entity with retrieve permissions references one without
  [linkahead-server#244](https://gitlab.com/linkahead/linkahead-server/-/issues/244)
* Insufficient permission checks in queries when a name of an invisible record
  is used in a filter where a visible record references the invisible one
  [linkahead-server#242](https://gitlab.com/linkahead/linkahead-server/-/issues/242)

### Security

This is an important security patch release. The bugs
[linkahead-server#244](https://gitlab.com/linkahead/linkahead-server/-/issues/244)
and
[linkahead-server#242](https://gitlab.com/linkahead/linkahead-server/-/issues/242)
possibly leak sensitive data when an attacker with read access to linkahead
(i.e. the attacker needs an active user account or anonymous needs to be
enabled) can guess the name of entities or properties of referenced entities
and construct a malicious FIND or SELECT statement and when the attacker has
read permissions for an entity which references the entities containing the
sensitive information. See the bug reports for more information.

## [0.12.0] - 2023-10-25
(Timm Fitschen)

### Fixed

* `FIND ENTITY <ID> is broken`.
  [linkahead-server#323](https://gitlab.indiscale.com/caosdb/src/caosdb-server/-/issues/323)
* Unknown Server Error when inserting an Entity.
  [linkahead-mariadbbackend](https://gitlab.indiscale.com/caosdb/src/caosdb-mysqlbackend/-/issues/48)

## [0.11.0] 2023-10-13 ##

### Added ###

* Configuration options `REST_RESPONSE_LOG_FORMAT` and
  `GRPC_RESPONSE_LOG_FORMAT` which control the format and information included
  in the log message of any response of the respective API. See
  `conf/core/server.conf` for more information.
* REST API: Permanent redirect from "FileSystem" to "FileSystem/".

### Fixed ###

* Inheritance of the unit is not working. (GRPC API)
  [linkahead-server#264](https://gitlab.indiscale.com/caosdb/src/caosdb-server/-/issues/264)
* Curly brackets in query lead to unexpected server error.
  [linkahead-server#138](https://gitlab.com/linkahead/linkahead-server/-/issues/138)
* Wrong url returned by FileSystem resource behind proxy.
* `NullPointerException` in GRPC API converters when executing SELECT query on
  NULL values.
* Fix parsing of decimal numbers. Fixes https://gitlab.com/linkahead/linkahead-server/-/issues/239

## [0.10.0] - 2023-06-02 ##
(Florian Spreckelsen)

### Changed ###

* The default behavior of the query `FIND SomeName [...]` (as well as COUNT and SELECT) is being
  made configurable and changes:
  * `FIND SomeName` will be interpreted as `FIND <FIND_QUERY_DEFAULT_ROLE>
    SomeName` from now on where `FIND_QUERY_DEFAULT_ROLE` is a newly introduced
    server property.
  * The new `FIND_QUERY_DEFAULT_ROLE` server property defaults to `RECORD`
    which is why the behavior of the server api has a **breaking change**.
  * The semantics of `FIND *` are affected as well. `FIND *` is equivalent to
    `FIND <FIND_QUERY_DEFAULT_ROLE>`.
  * Of course, administrators can choose to retain the old behavior by setting
    `FIND_QUERY_DEFAULT_ROLE=ENTITY`.
* CQL now treats `WITH` and `WITH A` equivalently. Issue: [#192](https://gitlab.com/caosdb/caosdb-server/-/issues/192)
* The InsertFilesInDir FlagJob now creates File entities without a name.  The previous behavior
  caused severe performance problems for very large numbers of files. Issue: [#197](https://gitlab.com/caosdb/caosdb-server/-/issues/197)

### Fixed ###

* Unexpected Server Error when inserting an Entity.
  [#216](https://gitlab.com/caosdb/caosdb-server/-/issues/216)
* Bad performance due to the execution of unnecessary jobs during retrieval.
  [#189](https://gitlab.com/caosdb/caosdb-server/-/issues/189)
* Query Language: Parentheses change filter to subproperty filter
  [#203](https://gitlab.com/caosdb/caosdb-server/-/issues/203)
* Searching for values in scientific notation
  [#143](https://gitlab.com/caosdb/caosdb-server/-/issues/143)
* Denying a role permission has no effect
  [#196](https://gitlab.com/caosdb/caosdb-server/-/issues/196). See security
  notes below.
* Missing RecordType leads to unexpected server error
  [#166](https://gitlab.com/caosdb/caosdb-server/-/issues/166)

### Security ###

* Fixed [#196](https://gitlab.com/caosdb/caosdb-server/-/issues/196). This was
  an error in the authorization procedure which allowed unprivileged users
  execute insert, update or delete transactions on entities. However, the
  unprivileged users would also need the correct entity permissions to do that.

  Without backup, this means possible data loss. Also there was the possibility
  to spam the database by creating unwanted entities.

### Documentation ###

- Nested queries.
- Global entity permissions.
- DOC: Data model tutorial.
- Removed old documentation directory `/doc/`, migrated non-duplicate content to `/src/doc/`.

## [0.9.0] - 2023-01-19

### Added

* SELECT query support for the GRPC API

## [0.8.1] - 2022-11-07
(Timm Fitschen)

### Added

* Make time zone configurable via server properties during runtime, but only in
  debug mode.

### Changed

### Deprecated

### Removed

### Fixed

### Security

### Documentation

* Extended documentation on role and entity permissions

## [0.8.0] - 2022-07-12
(Timm Fitschen)

### Added

* Configurable requirements for user names and passwords. The default is the old hard-coded configuration.

### Changed

* Minimal changes to the error messages for invalid user names and passwords.

### Deprecated

### Removed

### Fixed

* [caosdb-server#130](https://gitlab.com/caosdb/caosdb-server/-/issues/130)
  CQL: Spaces are not recognised in sub-property queries.
* [caosdb-server#148](https://gitlab.com/caosdb/caosdb-server/-/issues/148)
  Cannot delete file entities containing unescaped space characters
* [caosdb-server#145](https://gitlab.com/caosdb/caosdb-server/-/issues/145)
  Searching for large numbers results in wrong results if integer values are
  used.

### Security

## [0.7.3] - 2022-05-03
(Daniel Hornung)

### Changed

* `misc/pam_authentication/ldap.conf` is not used for configuring the
  `ldap_authentication.sh` script anymore.
  Use `misc/pam_authentication/ldap.env` instead and view the documentation
  inside the file itself for more information.

### Fixed

* [caosdb-server#142](https://gitlab.com/caosdb/caosdb-server/-/issues/142)
  Can't create users with dots in their user names
* `ldap_authentication.sh <username>` failed on every attempt when used in
  combination with OpenLDAP with default configuration.
* `ldap_authentication.sh` allowed empty and even wrong passwords when used in
  combination with MS Active Directory when AD is configured to allow binding
  with an empty password.

### Security

* `ldap_authentication.sh` allowed empty and even wrong passwords when used in
  combination with MS Active Directory when AD is configured to allow binding
  with an empty password. This is only relevant for non-default configurations
  of the `PAM.pam_script` option in the `usersources.ini`.

## [0.7.2] - 2022-03-25
(Timm Fitschen)

This is an important security update.

### Added

* Implementation for the ACM GRPC-API (caosdb-proto 0.2)
* Implementation for the EntityACL GRPC-API (caosdb-proto 0.2)

### Changed

### Deprecated

### Removed

### Fixed

* Wrong serialization of date time values in the GRPC-API (resulting in
  org.caosdb.server.datatime@12347abcd or similar).
* Missing serialization of file descriptors in the GRPC-API during retrievals.
* [caosdb-server#131](https://gitlab.com/caosdb/caosdb-server/-/issues/131)
  Query: AND does not work with sub-properties
* Add previously missing `Value.equals` functions
* [caosdb-server#132](https://gitlab.com/caosdb/caosdb-server/-/issues/132)
  Query: subproperties should not require parentheses
* [caosdb-server#174](https://gitlab.indiscale.com/caosdb/src/caosdb-server/-/issues/174)
  GRPC-API: Server should gracefully handle non-file entities with file-like
  attributes.
* [caosdb-server#217](https://gitlab.indiscale.com/caosdb/src/caosdb-server/-/issues/217)
  Server gets list property datatype wrong if description is updated.
* [caosdb-server#220](https://gitlab.indiscale.com/caosdb/src/caosdb-server/-/issues/220)
  Entities can be retrieved via GRPC despite insufficient permissions.
* [caosdb-server#221](https://gitlab.indiscale.com/caosdb/src/caosdb-server/-/issues/221)
  Unknown error during update of property leaving out datatype.
* [caosdb-server#223](https://gitlab.indiscale.com/caosdb/src/caosdb-server/-/issues/223)
  State is being leaked even though RETRIEVE:ENTITY permission is not granted.

### Security

* Update of logging backend log4j to 2.17.2
* Fix for [caosdb-server#220](https://gitlab.indiscale.com/caosdb/src/caosdb-server/-/issues/220)
  Entities can be retrieved via GRPC despite insufficient permissions.
* Fix for [caosdb-server#223](https://gitlab.indiscale.com/caosdb/src/caosdb-server/-/issues/223)
  State is being leaked even though RETRIEVE:ENTITY permission is not granted.

## [v0.7.1] - 2021-12-13
(Timm Fitschen)

This is an important security update.

### Added

### Changed

### Deprecated

### Removed

### Fixed

### Security

* Update of logging backend log4j after a critical security vulnerability
  [CVE-2021-44228](https://nvd.nist.gov/vuln/detail/CVE-2021-44228) to v2.15.0.
* [caosdb-deploy#225](https://gitlab.indiscale.com/caosdb/src/caosdb-deploy/-/issues/225)
  - Denied Edit permission leads to retrieve permission.

## [v0.6.1] - 2021-11-13 [YANKED]
(Timm Fitschen)

This version's release was pulled after some problems during the release
process. It is identical to v0.7.1


## [v0.6.0] - 2021-11-17
(Timm Fitschen)

### Added

* Endpoint for CaosDB GRPC API 0.1 (see https://gitlab.com/caosdb-proto.git for
  more).
  Authentication is supported via a Basic scheme, using the well-known
  "authentication" header.
  Notable limitations of the current implementation of the API:
  * It is currently not possible to mix retrievals
    (caosdb.entity.v1.RetrieveRequest) with any other transaction type - so
    transaction are either read-only or write-only. The server throws an error
    if it finds mixed read/write transactions.
  * It is currently not possible to have more that one query
    (caosdb.entity.v1.Query) in a single transaction. The server throws an
    error if it finds more than one query.


### Changed

### Deprecated

* Legacy XML/HTTP API (also known as the REST API). The API will not be removed
  until the web interface (caosdb-webui) and the python client libraries have
  been updated and freed from any dependencies. However, new clients should not
  implement this API anymore.

### Removed

### Fixed

### Security

## [v0.5.0] - 2021-10-19

### Added

* An openAPI specification of the XML api
* New server configuration option `SERVER_BIND_ADDRESS`, which is the address to listen to. See
  [server.conf](conf/core/server.conf).

### Changed

### Deprecated

### Removed

* `IdOnly` flag (see https://gitlab.indiscale.com/caosdb/src/caosdb-server/-/issues/187).
  The flags was not working anyways. However, `SELECT id FROM ...` queries are
  now optimized in the way the `IdOnly` flag was supposed to do.

### Fixed

* #181 CQL's `UPDATED` filter.
  (https://gitlab.indiscale.com/caosdb/src/caosdb-server/-/issues/181)
* #183 No reasonable error when using bad datetime format.
  (https://gitlab.indiscale.com/caosdb/src/caosdb-server/-/issues/183)
* #127 "nan" as value (list item) in properties with data type "LIST<DOUBLE>"
  return with "Cannot parse value to double" error.
* #170 Updating an abstract list-type property with a default value fails with
  "unkown error".
* #145 Documentation of importances and inheritance
* Missing sources of the easy-unit dependency.
* #178 Formatting of tables in documentation

### Security

## [v0.4.0] - 2021-06-21

### Added

* Related to #146, a new flag for entities and complete transactions:
  `force-missing-obligatory=[ignore|warn|error]`. The flag overrides the
  default behavior of the server (throwing an error when an obligatory property
  is missing). `ignore` just discards the consistency check, `warn` only issues
  a warning when obligatory properties are missing and `error` throws an error
  in that case. The flag can be set for the complete transaction and each
  single entity, while the entity flag takes precedence.
* New EntityState plug-in. The plug-in disabled by default and can be enabled
  by setting the server property `EXT_ENTITY_STATE=ENABLED`. See
  [!62](https://gitlab.com/caosdb/caosdb-server/-/merge_requests/62) for more
  information.
* `ETag` property for the query. The `ETag` is assigned to the query cache
  each time the cache is cleared (currently whenever the server state is being
  updated, i.e. the stored entities change).
  This can be used to debug the query cache and also allows a client
  to determine whether the server's state has changed between queries.
* Basic caching for queries. The caching is enabled by default and can be
  controlled by the usual "cache" flag.
* Documentation for the overall server structure.
* Add `BEFORE`, `AFTER`, `UNTIL`, `SINCE` keywords for query transaction

### Changed

* The default session timeout changed from 10 min to 60 min. Please set it to
  your needs via the server config option `SESSION_TIMEOUT_MS`.

### Deprecated

### Removed

### Fixed

* #146 - Default behavior for missing obligatory properties
* #131 - CQL Parsing error when white space characters before some units.
* #134 - CQL Parsing error when multiple white space characters after `FROM`.
* #130 - Error during `FIND ENTITY` when
  `QUERY_FILTER_ENTITIES_WITHOUT_RETRIEVE_PERMISSIONS=False`.
* #125 - `bend_symlinks` script did not allow whitespace in filename.
* #122 - Dead-lock due to error in the DatabaseAccessManager.
* #120 - Editing entities that were created with a no longer existing user
  leads to a server error.
* #31 - Queries with keywords in the path (e.g. `... STORED AT 0in.txt`)
* #116 - Queries `FIND [ANY VERSION OF] *` and `FIND [ANY VERSION OF] ENTITY`.

### Security

## [0.3.0] - 2021-02-10

### Added

* New version history feature. The "H" container flag retrieves the full
  version history during a transaction (e.g. during Retrievals) and constructs
  a tree of successors and predecessors of the requested entity version.
* New query functionality: `ANY VERSION OF` modifier. E.g. `FIND ANY VERSION OF
  RECORD WITH pname=val` returns all current and old versions of records where
  `pname=val`. For further information, examples and limitations see the wiki
  page on [CQL](https://gitlab.com/caosdb/caosdb/-/wikis/manuals/CQL/CaosDB%20Query%20Language)
* New server property `SERVER_SIDE_SCRIPTING_BIN_DIRS` which accepts a comma or
  space separated list as values. The server looks for scripts in all
  directories in the order or the list and uses the first matching file.
* Automated documentation builds: `make doc`

### Changed

* Server can be started without TLS even when not in debug mode.
* Select queries would originally only select the returned properties by their
  names and would not check if a property is a subtype of a selected property. This
  has changed now and select queries will also return subtypes of selected
  properties.

### Deprecated

* `SERVER_SIDE_SCRIPTING_BIN_DIR` property is deprecated.
  `SERVER_SIDE_SCRIPTING_BIN_DIRS` should be used instead (note the plural
  form!)

### Removed

* Text user interface (CaosDBTerminal).

### Fixed

* Bug: When the user password is updated the user is deactivated.
* Semi-fixed a bug which occurs when retrieving old versions of entities which
  reference entities which have been deleted in the mean time. The current fix
  adds a warning message to the reference property in question and sets the
  value to NULL. This might even be desired behavior, however this would have
  to finally specified during the Delete/Forget phase of the implementation of
  the versioning.
- Inheritance job cannot handle inheritance from same container (!54)
* Bug in the query parser (MR!56) - The parser would throw an error when the
  query contains a conjunction or disjunction filter with a first element which
  is another disjunction or conjunction and being wrapped into parenthesis.

### Security

## [0.2.0] - 2020-09-02

### Added

- Support for deeply nested selectors in SELECT queries.
- One-time Authentication Tokens for login without credentials and login with
  particular permissions and roles for the course of the session.
- `Entity/names` resource for retrieving all known entity names.
- Scripting is simplified by adding a `home` directory, of which a copy is
  created for each called script and set as the `HOME` environment variable.
- [bend_symlinks.sh](misc/bend_symlinks/bend_symlinks.sh) (version 0.1, experimental)
  fix broken symlinks in the internal file system. See
  [README.md](misc/bend_symlinks/README.md)
- [move_files.py](misc/move_files/move_files.py) (version 0.1, experimental)
  Script for moving files (change their path) in the internal file system based
  on a two-column tsv file (with columns "from" and "to"). See
  [README.md](misc/move_files/README.md).
- LDAP server may now be given and may be different from LDAP domain. See
  `misc/pam_authentication/ldap.conf`
- #47 - Sub-properties can now be queried, such as in
  `SELECT window.width FROM house`.
- Added support for versioning, if it is enabled on the backend.


### Changed

* All caosdb server java classes moved from `caosdb.[...]` to
  `org.caosdb.[...]` because the new root package is compliant with the java
  package naming conventions while the old was not. This has some implications
  for configuring the server. See [README_SETUP.md](./README_SETUP.md), section
  "Migration" for additional information.
- The server by default now only serves TLS 1.2 and 1.3, all previous versions
  have been disabled in the default settings.  Make sure that your clients
  (especially the Python client) are up to date.

### Deprecated

- CaosDBTerminal

### Removed

### Fixed

* Missing handling of list of reference properties in SELECT queries.
* #51 - name queries (e.g. `FIND ENTITY WITH name = ...`)
- #27 - star matches slashes (e.g. for `FIND ... STORED AT /*.dat`).
- #30 - file path cannot be in quotes
- #46 - Server-side scripting failed as an unprivileged user because there was
  no writable home directory.
- NaN Double Values (see #41)
- #14 - Handle files on file system without File entity: Those entries are
  returned without ID but with a notice now.
- #11 - pam_authentication leaks the password to unprivileged processes on the
  same machine.
- #39 - quotes around datetimes in queries
- #99 - Checksum updating resulted in infinite loop on server.

### Security (in case of vulnerabilities)

- TLS is by default restricted to v1.2 and v1.3 now.
- #11 - PAM and LDAP authentication no longer leak the password to unprivileged
  processes on the same machine.
- #68 - Shadow sensitive information when logging for debugging purposes.

## [0.1.0] - 2018-10-09

Tag `v0.1` - Commit 3b17b49

### Added

- everything