.. _operate-troubleshooting:

===============
Troubleshooting
===============

This chapter is intentionally short. It will grow as real
incidents are observed in the field; we deliberately do not list
generic "have you tried turning it off and on again" advice.

Where to start
==============

For any unexpected behaviour the first source of truth is the server
log (see :ref:`operate-logs`):

* ``./logs/toa-server.log`` for the rolling on-disk file, or
* the captured stdout if the operator runs the service under
  ``systemd`` / ``NSSM`` / similar.

If the symptom is "the request never reaches the server" - no entry
in either log for the call the user just made - look at the **reverse
proxy access log** next (see :ref:`client-hosting` for the proxy
setup). A request rejected by the proxy (TLS handshake failure,
401/403 from an upstream auth layer, 502 because the backend is down)
never appears in the TOA log at all. That access log is also where
the originating client IP and ``User-Agent`` are recorded - the TOA
server does not log either by default.

The on-disk import folder under ``<dataRoot>`` (see
:ref:`operate-data-layout`) is the second source of truth - the
``import.json`` of a problematic import carries its current
``status``, the ``error`` message of any failed Damis submit, and the
``userEmail`` that owns it.

Known issues
============

Damis submit ends with ``status: FAILED``
-----------------------------------------

**Symptom.** The user clicks the final "Send" button on the
confirmation page; the UI eventually surfaces a submission error.
``import.json`` for the affected import shows::

   "submitStatus": "FAILED",
   "submitError":  "<one-line error string>"

**Where to look.** The same error string is logged by the server.
Both code paths produce a grep-able line:

* If Damis returned a non-OK status, ``SubmitService`` logs
  ``Import <id> submission failed: Damis rejected batch: status=...``
  at WARN level.
* If the call threw before reaching Damis (network error, malformed
  response, batch builder failure, ...), ``SubmitService`` logs
  ``Submission failed for import <id> in domain <code>: <message>``
  at ERROR level **with** the full stack trace.

Grep the log for the import id to land on the right line directly.

**Recovery.** A ``FAILED`` import is not automatically retried. The
import is left in ``FAILED`` state with the original page binaries
intact on disk, so the operator's options are:

1. Hand the error string from ``submitError`` (or the stack trace
   from the ERROR log entry) to the Damis administrator. Most
   ``Damis rejected batch`` messages reflect a content/template
   mismatch the user has to correct rather than a transport problem.
2. Once the underlying issue is addressed, the user can re-trigger
   the submit from the confirmation page; the controller accepts a
   resubmit when the status is ``FAILED`` (it refuses only
   ``PENDING`` and ``SUBMITTED``).
3. If the import must be discarded, delete the
   ``<dataRoot>/<domain>/<date>/<importId>/`` folder by hand - see
   :ref:`operate-data-layout`. Otherwise it will be removed by the
   normal retention sweep (see :ref:`operate-retention`).

There is no "force submitted" override, by design - flipping the
status without an actual Damis acknowledgement would lose the audit
trail of which batch id received the documents.

*(further entries to be added as incidents are diagnosed in
production)*