Platform logging

Platform logging is used to monitor and troubleshoot issues in the xDM platform once it is fully operational.

For daily monitoring of job executions, it is recommended to use the job logs and the job notifications.

View the logs

By default, important log events are recorded in the Semarchy error log. This log shows events sent to the PDE appender.

You can access the error log by following these steps:

  • In the Application Builder, select Error Log in the user menu.
    The Error Log view opens.

  • In the Configuration module, select View Logs in the navigation drawer.
    The Error Log editor opens.

Configure logging

The default logging configuration of Semarchy is sufficient for most situations. However, you may need to modify this configuration to troubleshoot complex issues.

Logging is managed through the Logging Configuration editor.

Starting from the 5.3.9 release, xDM uses Apache Log4j 2 to log its activities. This section offers key concepts and configuration samples for working with Log4j. For detailed reference information, see the official Log4j documentation.

Instructions for upgrading the logging configuration from Log4j 1 to Log4j 2 are available in the Upgrade guide.

To configure logging:

  1. In the Configuration module, select Logging Configuration in the navigation drawer.
    The Logging Configuration editor opens.

  2. Change the configuration in the editor. For explanations about the configuration, see Understand the logging below.

  3. Press Control+S (or Command+S on macOS) to save your changes. The new logging configuration is immediately applied.

To reset the logging configuration to its default values, click the Restore Logging Configuration to Default button in the Logging Configuration editor toolbar.

Understand the logging

Log4j operates based on three key concepts: appenders, loggers, and log levels.

  • Log levels determine the granularity and importance of log events.

  • Loggers capture log events generated by libraries and classes within the xDM platform.

  • Appenders deliver log events to a specified destination. For example, they can write events to a file, send them to a message queue, or display them in the Semarchy error log.

Log events generated by xDM have an associated log level. These events are captured by a logger and then delivered to their destination through an appender.

Log levels

The log level defines both the granularity and importance of a log event. The available log levels are: TRACE, DEBUG, INFO, WARN, ERROR, and FATAL.

These levels are ordered from highest to lowest granularity, and simultaneously, from least to most importance. For instance, the DEBUG level is highly granular, capturing many messages that may not be relevant during normal operations. Conversely, ERROR and FATAL messages occur less frequently but hold greater significance.

When a logger is set to capture events at a certain level, it also captures all events of higher importance. For example, if you configure a logger to capture INFO events, it will also capture WARN, ERROR, and FATAL events.

When configuring a logger, you can set the level to ALL to capture all log events or to OFF to disable log capture entirely.

INFO is typically a good level to start with, as it captures informational messages along with any warnings or errors.

Appenders

Appenders are responsible for delivering log events to a specific destination. Each appender is defined by a name and an appender class, which is a Java class that determines how and where the log events are written, such as to a file, a JMS message queue, an email, etc.

Appender classes can be configured with various properties to control their behavior. They also support layout properties, which allow you to define the format of the logged events.

xDM includes a built-in appender class, com.semarchy.commons.log4j.appender.PDEAppender, which is used to deliver logged events to the Semarchy error log.

Configure appenders

In the logging configuration, appenders are declared within the <Appenders> element.

An appender is defined using the following syntax:

<Appenders>
    <appender_class (1)
        name="appender_name" (2)
        ...
    </appender_class>

    ...
<Appenders>
1 The appender’s class (see Log4j 2 Appenders).
2 The name used by loggers to refer to this appender.

The parameters for each appender are defined using an XML structure consisting of elements and attributes specific to the appender’s class. Configuration examples for the most commonly used appender classes are provided in the following section.

Configuration samples

This section provides configuration samples for appenders.

Example 1. FILE appender writing log events to a daily rolling log file.
<!-- Rolling file appender -->
<RollingFile (1)
  name="appender_name"
  fileName="${sys:user.home}/.semarchy/logfile.log" (2)
  filePattern="${sys:user.home}/.semarchy/logfile.%d{yyyy-MM-dd}.log">
  <PatternLayout pattern="%d{yyyy-MM-dd HH:mm s S}-%X{authenticatedUser}-%t-%-5p-%-10c:%m%n"/> (3)
  <Policies>
    <TimeBasedTriggeringPolicy modulate="true"/> (4)
  </Policies>
  <DefaultRolloverStrategy max="2147483647"/> (5)
</RollingFile>
1 The DailyRollingFileAppender appender class creates a new log file each day. Alternatively, you can use the FileAppender to write logs to a single file.
2 The daily log file is stored in the .semarchy/ sub-directory of the user’s home directory. Its filename is logfile.log, with the date appended in the specified format.
3 An appender uses a layout to format log events. For RollingFileAppender, logged events are usually formatted using a PatternLayout pattern string, which can include details like the date and priority of the event to diagnose. This pattern can also incorporate a specific %X{authenticatedUser} context key representing the authenticated user in xDM, if applicable.
4 The triggering policy determines if a rollover should be performed.
5 The RolloverStrategy specifies how the rollover is executed. If no RolloverStrategy is configured, RollingFileAppender will use the DefaultRolloverStrategy.
Example 2. PDE appender sending log events to the Semarchy error log.
<!-- PDE Appender -->
<PDE name="appender_name" />
Example 3. SMTP appender sending log events by email
<!-- The SMTP class sends logs in emails via an SMTP host.         -->
<!-- This appender should be preferably used for important events. -->
<SMTP  (1)
        name="appender_name"
        smtpHost="smtp_host_name"  (2)
        smtpPort="smtp_host_port"  (2)
        from="email@my-domain.com" (2)
        to="admin1@my-domain.com,admin2@my-domain.com"  (3)
        subject="Log Event">                            (3)
    <layout class="org.apache.log4j.PatternLayout">
        <param name="ConversionPattern" value="%d %-5p [%t] %C{2} (%F:%L) - %m%n"/>  (4)
    </layout>
</SMTP>
1 This appender class sends log events via email.
2 Mail server configuration. Additional properties may be set for the SMTP user, password, protocol, and other settings.
3 Email recipients and subject.
4 Log events are formatted using a pattern layout pattern string. If no layout is provided, the HTML layout will be used y default.

Loggers

Loggers are organized in a hierarchical structure based on class names, with each logger capturing events for itself and all of its sub-classes. For example, the com.semarchy logger captures events for all classes that begin with com.semarchy, including com.semarchy.commons.sql and com.semarchy.mdm.

The highest level in the hierarchy is the root logger.

Configure loggers

Loggers are declared within the <Loggers> element.

To configure a logger, specify a log level and one or more appenders. The logger captures and sends to the appender any event emitted by the logger class that is equal to or more important than the specified log level.

The configuration for loggers includes:

  • The root logger.

  • A list of additional loggers.

Example 4. Loggers sample configuration
<Loggers>

    <!-- Root Logger -->
    <Root level="root_log_level">             (1)
        <AppenderRef ref="appender_name"/>    (2)
        ...
    </Root>

    <Logger level="log_level"  (3)
            name="logger_name">
        <AppenderRef ref="appender_name_1"/>  (4)
        <AppenderRef ref="appender_name_2"/>
        ...
    </Logger>
    ...
</Loggers>
1 Threshold level for the root logger. Accepted values are: TRACE, DEBUG, INFO, WARN, ERROR, and FATAL.
2 Appender to use for the root logger.
3 Threshold level for this logger.
4 Appender to use for this logger.
Example 5. Send DEBUG (and more important) events from the IPlatformManager class to the PDE appender
<Logger level="DEBUG" name="com.semarchy.platform.setup.IPlatformManager">
    <AppenderRef ref="PDE"/>
</Logger>

Logger inheritance

Loggers inherit appenders additively from their parent loggers in the hierarchy. They can also override the log level specified by their parent.

In the following configuration, the com.semarchy.platform.setup.IPlatformManager logger inherits the PDE appender from its parent com.semarchy logger and sends:

  • INFO events to the PDE appender

  • ERROR events to the PDE appender and the SMTP appender.

<Logger level="INFO"
        name="com.semarchy">
    <AppenderRef ref="PDE"/>
</Logger>
<Logger level="ERROR"
        name="com.semarchy.platform.setup.IPlatformManager">
    <AppenderRef ref="SMTP"/>
</Logger>

To prevent inheritance, use the following syntax:

<Logger additivity="false"
        name="logger_name">
    ...
</Logger>

For example, in the following configuration, com.semarchy.platform.setup.IPlatformManager will log events only to the SMTP appender, and will no longer use the PDE appender.

<Logger level="INFO"
        name="com.semarchy">
    <AppenderRef ref="PDE"/>
</Logger>
<Logger level="ERROR"
        additivity="false"
        name="com.semarchy.platform.setup.IPlatformManager">
    <AppenderRef ref="SMTP"/>
</Logger>

Configuration samples

The following configuration represents the default setup for xDM.

Example 6. Sample logger configuration for xDM.
<!-- External libs shutdown -->
<Logger level="ERROR" name="org.apache.directory.shared.asn1.ber.Asn1Decoder"/>
<Logger level="WARN" name="org.apache.commons.beanutils.converters"/>
<Logger level="WARN" name="org.quartz"/>
<Logger level="WARN" name="org.ops4j.pax.logging"/>
<Logger level="WARN" name="org.apache.cxf"/>
<Logger level="WARN" name="org.apache.cxf.services"/>
<Logger level="ERROR" name="org.apache.cxf.jaxrs.impl.WebApplicationExceptionMapper"/>
<Logger level="WARN" name="org.apache.cxf.services.WebClient"/>
<Logger level="WARN" name="org.apache.aries.blueprint"/>
<Logger level="OFF" name="org.apache.aries.blueprint.container.ServiceRecipe"/>
<Logger level="OFF" name="org.apache.aries.blueprint.container.BlueprintContainerImpl"/>
<Logger level="INFO" name="org.apache.jasper.servlet"/>
<Logger level="WARN" name="org.apache.aries.spifly"/>
<Logger level="INFO" name="com.sun.xml"/>
<Logger level="INFO" name="com.zaxxer.hikari"/>
<Logger level="INFO" name="javax.xml.bind"/>
<!-- Remove verbose WARN on Tika Parser see: https://issues.apache.org/jira/browse/TIKA-2518 -->
<Logger level="ERROR" name="org.apache.tika.parser.SQLite3Parser"/>
<Logger level="INFO" name="org"/>

<!-- Semarchy Loggers -->
<Logger level="INFO" name="com.semarchy"/>
<Logger level="INFO" name="com.semarchy.commons.sql"/>
<Logger level="WARN" name="org.springframework.jdbc.core.JdbcTemplate"/>
<Logger level="WARN" name="org.springframework.jdbc.core.StatementCreatorUtils"/>
<Logger level="WARN" name="org.apache.aries.blueprint.container">
    <AppenderRef ref="CONSOLE"/>
    <AppenderRef ref="PDE"/>
</Logger>
<Logger level="INFO" name="com.semarchy.platform.engine.core.impl.product.SL4JExecutionMonitor"/>
<Logger level="INFO" name="com.semarchy.platform.engine"/>
<Logger level="INFO" name="com.semarchy.platform.setup.IPlatformManager"/>
<Logger level="INFO" name="com.semarchy.mdm.datahub.services.query.datamgr.IDataManager"/>
<!-- Plugin Execution logging. Set to Debug to troubleshoot plugins. -->
<Logger level="INFO" name="com.semarchy.platform.engine.PluginExecution"/>
<Logger level="INFO" name="com.semarchy.platform.repository.domain.restclient.RestClientInstance"/>
<!-- Integration Batch Notification logger to be notified when notification errors occurs -->
<Logger level="INFO" name="com.semarchy.platform.product.notification.PlatformNotificationHandler"/>
<!-- Import / Export -->
<Logger level="INFO" name="com.semarchy.mdm.dataui.domain.dataimport"/>
<Logger level="INFO" name="com.semarchy.mdm.dataui.domain.dataexport"/>
<!-- Consistency Checker -->
<Logger level="INFO" name="com.semarchy.ui.platform.console.actions.CheckConsistencyAction">
    <AppenderRef ref="PDE"/>
</Logger>
<!-- Public REST API logger -->
<Logger level="INFO" name="com.semarchy.xdm.rest.accesslog"/>

Logger reference

The following table lists the various loggers available in Semarchy xDM:

Logger name Description

com.semarchy

Root logger for the Semarchy platform.

com.semarchy.commons.sql

TRACE logs SQL queries made by the Application Builder and Configuration user interfaces.

com.semarchy.mdm.datahub.services.query.datamgr.IDataManager

Logs database operations performed by the platform for the MDM applications while browsing data. DEBUG logs all queries and execution times.

com.semarchy.platform.engine.core.impl.DefaultStandaloneEngineImpl

Logs execution engine events. DEBUG logs all submitted jobs, and ERROR can be used to troubleshoot engine issues.

com.semarchy.platform.engine.core.impl.product.SL4JExecutionMonitor

Logs execution engine job processing. Use DEBUG to see execution engine details.

com.semarchy.platform.engine.PluginExecution

Logs enricher and validation plugins execution. DEBUG logs plugin feedback, TRACE logs the processing of every row.

com.semarchy.platform.integration.polling.IntegrationLoadDequeuer

Logs the batch poller activity. DEBUG logs every polled interval.

com.semarchy.platform.product.notification.JobNotificationHandler

Logs job notifications. ERROR traces notification failures.

com.semarchy.platform.setup.IPlatformManager

Logs platform status changes. INFO traces normal status changes. ERROR traces abnormal statuses.

org.apache.cxf.services

INFO traces REST requests, responses, and faults between the MDM applications and the server. This includes also REST requests performed by plugins on external services (e.g., Melissa Data).

org.springframework.jdbc.core.JdbcTemplate

DEBUG traces the SQL queries made by the MDM applications and REST API calls as well as dashboards-related SQL queries.

org.springframework.security.ldap

DEBUG traces the authentication attempts made with an LDAP or Active Directory identity provider.

org.springframework.security.oauth2
org.springframework.security.oauth2.jwt

Set oauth2 to DEBUG and oauth2.jwt to TRACE to log the authentication attempts made with an OpenID Connect identity provider.

waffle.spring

DEBUG traces the authentication attempts made with a Windows Authentication identity provider.

org.springframework.security.saml2
org.opensaml

DEBUG traces the authentication attempts made with a SAML identity provider.

com.semarchy.xdm.rest.accesslog

DEBUG traces REST API calls. This log is in JSON format and contains: username, API key name (or null for basic authentication), origin IP address, date and time, endpoint (relative to base server URL), response code and execution duration.

com.semarchy.mdm.discovery.core.impl.DefaultProfilingService

Logs the profiling activity for xDM Discovery. DEBUG traces the details of the profiling processes and queues.

com.semarchy.platform.repository.infrastructure.services.HttpClientRequestExecutor

DEBUG traces the REST requests and responses sent and received by data notifications.