Configure Runtime logging

When using Semarchy xDI Runtime, you can log two kinds of information:

Session information: Data and messages that a runtime generates when executing deliveries.
Debug information: Supplementary technical data and messages that a runtime generates at any time while it is running.

A runtime stores these logs in two places.

First, a runtime always stores session information to a log database, which you can monitor from xDI Designer, xDI Analytics, or the command line. You can change which database to use. The default log database is an HSQL database built into xDI Runtime which provides a ready-to-use configuration.

xDI Runtime also uses the Log4j library to store both session and debug information to multiple locations. Configure Log4j settings to change storage locations, as well as how much information you want to log. You can monitor these logs with external monitoring tools

The built-in HSQL database is, by default, only accessible from the runtime. You should not use it for production use. Use a certified database server instead.

Configure session logging to a database

The main session logging settings are in the main xDI Runtime configuration file engineParameters.xml. This file is in the properties subdirectory of the Runtime installation directory by default.

The main configuration sections are in a <parameters> element, inside the engineParameters and logs nodes.

Here is an overview of how you configure runtime session logging to a database:

Open the engineParameters.xml configuration file.
Under <parameters>, find the engineParameters and the logs elements.
Configure the initial logging engine parameters.
Set the log database parameters.
Restart the runtime.

Configure initial logging engine parameters

In the engineParameters node of the engineParameters.xml file, set two parameters to configure logging:

The userLogDefaultName parameter defines a name for a database configuration. Create and edit that configuration further in the configuration file, under <logs>.
The defaultSessionLogLevel parameter defines how much information the runtime logs by default.

Refer to the runtime parameters reference for detailed information parameters.

Log levels

The defaultSessionLogLevel parameter defines how much information the runtime saves in a session. It applies only if the session does not set its log level elsewhere. Possible values are as follows:

Table 1. Logged information by level
logLevel	Running session	Successful session	Session with errors
-1	Not logged	Not logged	Not logged
0	Full details	Not logged	Full details
100	Full details	Session header and statistics	Full details
200	Full details	[logLevel 100] + Process information and statistics	Full details
300	Full details	[logLevel 200] + Actions information and statistics	Full details
400	Full details	Full details	Full details

Example 1. Sample logging configuration

  <engineParameters>
    <...>
    <parameter name="userLogDefaultName" value="logDatabase"/>
    <parameter name="defaultSessionLogLevel" value="400"/>
    <...>
  </engineParameters>

Set an external logging engine database

If you want to use the default HSQL database instead of an external database, see the section about HSQL.

A log database configuration contains the connection information to a database that hosts the log. It must be inside the <logs> element, in its own <log> node.

To set a log database used by the runtime, follow these steps:

Copy a <log> element and its contents, or find a <log> entry to modify.
Set the userLogName attribute of the <log> node to match the initial userLogDefaultName engine parameter.
Set the JDBC URL and driver in userLogRdbmsUrl and userLogRdbmsDriver parameters.
Set the database username and password in userLogRdbmsUser and userLogRdbmsPassword parameters.
Set the userLogSchemaName parameter to the database schema you want to use.
Set the userLogRdbmsModule parameter to the name of an xDI module containing the required JDBC database drivers.

You do not need to configure a database schema ahead of time to prepare it for logging. The runtime creates the necessary tables the first time it connects to the database.

Example configuration for an external database

The following configuration example uses a log database named oracleLogDatabase using an Oracle storage.

Adapt the following parameters to your database.

userLogRdbmsUrl
userLogRdbmsUser
userLogRdbmsPassword
userLogSchemaName
userLogRdbmsModule

Refer to the runtime parameters reference for detailed information about each parameter.

<parameters>
  <...>
  <engineParameters>
  <...>
    <parameter name="userLogDefaultName"
      value="oracleLogDatabase"/>
    <parameter name="defaultSessionLogLevel"
      value="400"/>
    <...>
  </engineParameters>
  <...>
  <logs>
    <...>
    <log userLogName="oracleLogDatabase"
      autoUpdate="true"
      userLogClass="com.semarchy.xdi.runtime.sessionlog.RdbmsLogger">
      <parameter name="userLogRdbmsDriver"
        value="oracle.jdbc.driver.OracleDriver"/>
      <parameter name="userLogRdbmsUrl"
        value="jdbc:oracle:thin:@[host]:[port]:[sid]"/>
      <parameter name="userLogRdbmsUser"
        value="[USER]"/>
      <parameter name="userLogRdbmsPassword"
        value="[PASSWORD]"/>
      <parameter name="userLogRdbmsModule"
        value="[MODULE_NAME]"/>
      <parameter name="userLogRdbmsSchemaName"
        value="[SCHEMA_NAME]"/>
      <parameter name="userLogRdbmsVarcharType"
        value="varchar2"/>
      <parameter name="userLogRdbmsVarcharMaxSize"
        value="4000"/>
      <parameter name="userLogRdbmsClobType"
        value="clob"/>
      <parameter name="userLogRdbmsBlobType"
        value="blob"/>
      <parameter name="userLogRdbmsNumericType"
        value="number"/>
      <parameter name="userLogRdbmsDeleteSyntaxe"
        value="Delete from"/>
      <parameter name="userLogRdbmsDeliveryFormat"
        value="text"/>
      <parameter name="userLogRdbmsPropertyMaxVarcharSize"
        value="1000"/>
      <parameter name="userLogRdbmsPropertyMaxClobSize"
        value="10000"/>
      <parameter name="userLogRdbmsPropertyBinaryFormat"
        value="compressed"/>
      <parameter name="userLogRdbmsPoolEnabled"
        value="true"/>
      <parameter name="userLogRdbmsPoolConnectionTimeout"
        value="30000"/>
      <parameter name="userLogRdbmsPoolIdleTimeout"
        value="600000"/>
      <parameter name="userLogRdbmsPoolKeepAliveTime"
        value="0"/>
      <parameter name="userLogRdbmsPoolMaxLifetime"
        value="1800000"/>
      <parameter name="userLogRdbmsPoolMinimumIdle"
        value="0"/>
      <parameter name="userLogRdbmsPoolMaximumSize"
        value="20"/>
      <parameter name="userLogRdbmsPoolValidationTimeout"
        value="5000"/>
    </log>
    <...>
  </logs>
  <...>
</parameters>

The default engineParameters.xml installed with the runtime contains commented log database configuration samples. Uncomment and adapt these samples to configure to your log storage. Make sure to set the userLogDefaultName engine parameter to the appropriate log database configuration.

Set or configure the built-in HSQL database

If you want to use an external database instead of the default HSQL database, see the section about external databases.

The runtime ships a built-in HSQL quick start database. The default configuration uses this database to store logs for delivery execution sessions and the scheduler.

The built-in HSQL database is intended for development environments. It is only accessible from the runtime itself by default. You should use a certified database server where possible.

On runtime versions prior to 2023.4.0, the built-in database was an H2 database. See Migrate the built-in database from H2 to HSQL to migrate your data to HSQL.

Set the HSQL directory

In its default configuration, the runtime creates an HSQL database inside a sessions directory to hold logging information. The runtime tries to create the directory if it does not exist already. If an HSQL database already exists, make sure that the user and password in the configuration file match the ones expected by the database. This is most important when upgrading from a previous runtime version.

The directory name and location are set by the userLogRdbmsUrl engine parameter. The following example is from a runtime that writes to the default sessions/internalDb/sessionLogs location:

<parameter name="userLogRdbmsUrl" value="jdbc:hsqldb:file:internalDb/sessionLogs/sessionLogs"/>

The parameter can take a different file path, including absolute filesystem paths. Make sure that the directories you specify have write permissions enabled.

Open external access

The built-in HSQL database is only accessible from the runtime itself by default. The runtime logs to the database in-memory, and does not expose the database as an external service.

To open access to the built-in HSQL database from the outside, you must manually start it as a server. This can be useful to access its logs from xDI Analytics, for example. To do this:

Stop the runtime.
Update the runtime configuration
1. Log Database:
  1. Change the userLogRdbmsUrl parameter value from jdbc:hsqldb:file:internalDb/sessionLogs/sessionLogs to jdbc:hsqldb:tcp://localhost:42100/sessions/internalDb/sessionLogs
2. Scheduler:
  1. Change the org.quartz.dataSource.internal.URL parameter value from jdbc:hsqldb:file:internalDb/scheduler/scheduler to jdbc:hsql:tcp://localhost:42100/internalDb/scheduler/scheduler
Start the built-in HSQL database as a server.
1. According to your operating system, run the start-database.bat or start-database.sh from the runtime’s installation root folder.
Start the runtime.

After this change, you must always start the HSQL database server before starting the runtime itself.

Disable session logging

You can disable delivery session logging in the runtime in two ways:

Explicitly define a special, non-logging database entry using the EmptyLogger class as shown in the further example. Use this method first.
Remove or comment out all log database configurations under the <logs> node. This stops the runtime from logging delivery sessions.

Example 2. Configure the runtime for no logging

<parameters>
  <...>
  <engineParameters>
    <...>
    <parameter name="userLogDefaultName" value="noLogs"/>
    <...>
  </engineParameters>
  <...>
  <logs>
    <log
    userLogName="noLogs"
    autoUpdate="false"
    userLogClass="com.semarchy.xdi.runtime.sessionlog.EmptyLogger">
    </log>
  </logs>
  <...>
</parameters>

Purge session logs

You can purge runtime session logs manually, or schedule automatic purging, from multiple views.

Manually from Designer

In Designer, purge logs manually from the Sessions view. Purging logs from this view takes effect on the connected runtime.

Choose Delete until to purge a range of logs, or choose Delete all to purge all session logs from the log database.

Scheduled from Designer or Analytics

In Designer or Analytics, set up scheduled purges from the Runtime Editor. Select the Purge tab in the Runtime editor, and choose Add to add a schedule.

With Runtime commands

Using runtime commands, you can purge logs manually, as well as schedule regular log purges. Access the runtime command line from:

The runtime startCommand.bat|sh script.
The runtime editor in Designer or Analytics.

The runtime two commands for log purging: the purge keep command is for manual use, while the schedule purge keep command creates a regular schedule.

For information about these commands, see the runtime session command reference.

Write session logs asynchronously

On a first execution, active runtime sessions typically wait for session logs to be written at each step before execution continues. While this behavior guarantees that logs are in step with process execution, this behavior may reduce performance.

If you want to change this behavior, set the attribute synchMode in a log configuration section. The attribute goes on the line with the userLogName attribute, and accepts these values:

synch: Session logs are written synchronously. Session execution is always in step with its logs.
asynch: Session logs are written asynchronously. Session execution does not wait on logs.
synchFirstLog: The first log of a session is written synchronously, while subsequent logs are written asynchronously.

synchFirstLog is the default when the parameter is not set.

Example 3. Configure the runtime for asynchronous logging

<parameters>
  <...>
  <logs>
    <log
    userLogName="logDatabase" autoUpdate="true"
    userLogClass="com.indy.engine.userLog.RdbmsUserLog"
    synchMode="asynch" >
      <parameter name="userLogRdbmsDriver" value="oracle.jdbc.driver.OracleDriver"/>
    </log>
  </logs>
  <...>
</parameters>

Configure session and debug logging with Log4j

When xDI Runtime saves logs using Log4j, it reads settings from the XML file log4j.xml. This file is in the properties subdirectory of the Runtime installation directory by default.

The configuration file starts with a Configuration element. Inside it, you can find Appenders and Loggers.

Appenders send logging information to its final destination, like a log file or a console. Loggers set which elements log4j records, and need at least one reference to an appender.

Example 4. Example configuration

<Configuration shutdownHook="disable">
  <Appenders>
    <RollingFile fileName="log/xdi-runtime.log" filePattern="log/xdi-runtime.log.%i" immediateFlush="true" name="runtimeAppender">
          <PatternLayout>
              <Pattern>%d{dd/MM/yyyy HH:mm:ss,SSS} [%t] %c{10} - %-5p - %m%n%throwable</Pattern>
          </PatternLayout>
          <SizeBasedTriggeringPolicy size="20 MB"/>
          <DefaultRolloverStrategy fileIndex="min" max="6"/>
    </RollingFile>
  </Appenders>
  <Logger name="com.indy.engine.user.console" level="all">
    <AppenderRef ref="consoleAppender"/>
  </Logger>
  <Logger name="com.semarchy.xdi.runtime.httpserver" level="warn">
    <AppenderRef ref="consoleAppender"/>
    <AppenderRef ref="runtimeAppender"/>
  </Logger>
</Configuration>

Aside from the loggers built into the Log4j library, xDI Runtime implements some of its own, among which:

Session start
Session end
Action start
Action end
Java compiler
Module service
HTTP server

The default configuration disables some of these loggers. You can re-enable them.

Change the log location

By default, the Log4j library writes to the xdi-runtime.log file in a log subdirectory of the runtime installation path. If this subdirectory does not exist, the runtime tries to create it. The parameters that set the directory are in the log4j.xml file, as attributes of the RollingFile appender:

<RollingFile fileName="log/xdi-runtime.log" filePattern="log/xdi-runtime.log.%i" immediateFlush="true" name="runtimeAppender">

The attributes can take a different folder and file path, including absolute filesystem paths. If the path points to external directories, make sure those directories have write permissions enabled.

Logs can also be sent to other places, such as the runtime console using the Console appender.

Set appender patterns

When writing to disk, appenders determine how to write the log to disk from a Layout. One such layout is the PatternLayout, which sets string patterns to enrich log messages with additional information. The pattern strings have their own internal syntax and pattern parameters.

xDI has pattern parameters of its own. To use them, add them with the syntax %X{parameterName}.

You can also use %X on its own to get all the parameters.

Example 5. Example pattern for file appender

<PatternLayout>
     <Pattern>%d{dd/MM/yyyy HH:mm:ss,SSS} [%t] %c{10} - %-5p - %m%n%throwable</Pattern>
</PatternLayout>

List of xDI patterns

Table 2. All events
Parameter Name	Description
version	Version of the logging system
type	Type of event
message	event message
sessionId	Id of the session
sessionIteration	Iteration of the session
sessionName	Name of the session

Table 3. Session start
Parameter Name	Description
sessionStatus	Status of the session (always `RUNNING` for this event)
sessionParentId	Id of the parent session
sessionBeginDate	Start date of the session
processId	Id of the process
processName	Name of the process
deliveryId	Id of the delivery
deliveryName	Name of the delivery
deliveryBuildDate	Build date of the delivery
deliveryBuildUser	Build user of the delivery
configuration	Configuration used for the execution
runtimeHost	Hostname of the runtime used for the execution
runtimePort	Port of the runtime used for the execution
executionMode	Mode of execution (MEMORY/AUTONOMOUS)
launchMode	Launch mode (RESTART/COMMAND_LINE/SCHEDULE/DESIGNER/ACTION/WEB_INTERACTIVE/WEB_SERVICE/ENGINE_COMMAND)
sessionLogLevel	Log level of the session
maxparallelExecutions	Max parallel executions setting
autoRestartInterval	Auto restart interval setting
autoRestartMaxTries	Auto restart max tries setting
launchUserName	Name of the user who launched the session

Table 4. Session end
Parameter Name	Description
sessionStatus	Status of the session
sessionErrorMessage	Error message of the session
sessionEndDate	End date of the session
sessionDuration	Total duration of the session
var.<varName>	Variables
stat.<aggregateName>	Statistics

Table 5. All action events
Parameter Name	Description
actionId	ID of the action
actionIteration	Iteration of the action
actionType	Type of the action (Code/Process)
actionPath	Path of the action
actionPathWithIteration	Unique identifier of the action (<path>_<iteration>)
actionStatus	Status of the action (`RUNNING` for the Action Start event)

Table 6. Action start
Parameter Name	Description
actionBeginDate	Start date of the action
actionParentID	ID of the parent session
actionParentIteration	Iteration of the parent session
actionIsBegin	Is the action begin`
actionNumber	Unique number of the action

Table 7. Action end
Parameter Name	Description
actionErrorMessage	Error message of the Action [NOTE] This is not for errors that occur during action executions. It is for all other action errors, such as if actions did not generate.
actionEndDate	End date of the action
actionDuration	Total duration of the action
actionBindIteration	Final bind iteration of the action
actionExecutionCount	Execution count of the action (when used in a loop)
var.<varName>	Variables
stat.<aggregateName>	Statistics

For more on patterns and layouts, see the official Log4j documentation.

Further information

Apache Log4j website