6. Operations

6.1. Search index lifecycle

The search index is derived data — it can always be rebuilt from the database, and it needs no backup. At every startup the application compares the current indexing configuration (the indexed fields of types.yaml, the content locale search.content-locale and the internal document layout version) with the values stored in the index’s own metadata:

  • unchanged → the index is reused as-is;

  • changed → the schema is dropped, recreated and all APUs are reindexed automatically.

No manual full-reindex procedure exists or is needed (selected records can be re-indexed through the dirty set, see below); to force a rebuild of a persisted embedded-Lucene index, delete its directory (search.lucene.path) and restart.

Index synchronization after an import

An import writes the database first and the search index afterwards, from the database, so a failed import leaves the index as it was. Inside its transaction the import fills the dirty set (table index_dirty) with the identifiers of every record whose document must be re-derived; right after the import — and, should the application stop in between, at the next start — the set is worked off: a record that exists is (re)indexed, a record that is gone has its document deleted. A log line Search index synchronized: … reports each run.

Because the start works off the whole set, inserting rows by hand is the supported way to re-index selected records without a full rebuild — for example after correcting data directly in the database:

INSERT INTO index_dirty(apu_uuid, requested_at)
SELECT uuid, now() FROM apu WHERE apu_source_id = 42;

The records are written to the index at the next start (or the next import). An orphaned search document — one no database record backs — is removed the same way: insert its uuid.

6.2. Data ingest

Transfagent (SOAP file transfer)

The regular ingest path: the separate Transfagent application collects data from source systems (Elza, PEvA, …) and pushes APUX packages and files over the SOAP file-transfer interface at /cxf/ft. This endpoint is internal — see the proxy rules in Installation.

Input-directory import

A file-based alternative for supported data inputs, enabled by import.input-dir. Each subdirectory of the input directory is one transfer (apusrc-*.xml or dao-*.xml, optionally a files/ directory). The directory is scanned at startup; transfers whose content has not changed are skipped using a content-hash journal stored in the database (import_journal table), so leaving processed transfers in place is harmless.

Withdrawing published data

Data is deleted the same way it arrives — on request of the publishing system. Transfagent calls the internal management interface at /cxf/management (SOAP, contract wsdl/aron_core.wsdl, operation DeleteApuSources) with the uuids of the APUX packages to withdraw. For each package the portal deletes its APUs, their attachments and relations, and removes them from the search index; digital objects are only detached, because they are transferred separately and are reattached if the package is imported again. Uuids that are not present are ignored, so a repeated request is harmless.

Two consequences worth knowing:

  • Binary files of deleted attachments stay in the file storage (files.storage) — as they do after a reimport.

  • Withdrawal does not touch the input-directory journal. A package deleted this way and still present as a transfer folder is not reimported at the next startup unless its content changes; delete the folder, or its import_journal row, if the data should come back.

6.3. Database

PostgreSQL is the primary store and the only component that requires backup. Schema changes are applied automatically at startup via Liquibase; the upgrade procedure — what to back up, what to compare, what the first start does — is in Upgrading.

6.4. Monitoring

Spring Boot Actuator is available under /actuator (outside the /api namespace):

  • /actuator/health — liveness/readiness, suitable for probes;

  • further endpoints (metrics, info) according to the actuator configuration in application.yml.

6.5. Why a request failed

When a request fails, the portal answers with a status and, in the log, the reason: which record was not found, which filter the section does not have, which piece of a description a citation could not be built from. The reason is not in the response body by default: Spring’s server.error.include-message is never, so a client sees only

{"timestamp":"…","status":422,"error":"Unprocessable Entity","path":"/api/v1/apu/…/citations"}

That is deliberate. Setting server.error.include-message: always adds the reason to every error response. For the errors the portal reports on purpose that is exactly what someone debugging needs (“No such APU.”, the missing part of a citation). For an unexpected error — status 500 — the same field carries the exception’s own message, which in this stack can name a database column, a file path on the server or an internal component. Stack traces are a separate setting and stay off either way.

Hence the recommendation:

  • public portal — leave it off and read the reason in the log. The reader is shown the portal’s own message anyway; the portal’s reasons are English diagnostics written for whoever debugs, not text a reader could act on.

  • test or evaluation server — turn it on. It is the same reasoning as system.expose-version: such a server exists to be inspected, and having the reason in the response saves a trip to the log. The commented block is in application.yml.template.

Failed requests are never silent in the UI either: they appear in a visible error bar, with the server’s message when the deployment sends one.

Todo

Describe the recommended logging configuration and log locations once the deployment packaging (service unit, log rotation) is settled.