Continuous loads

With continuous loads:

  • Integration developers do not need to initialize and submit individual external loads. They directly load data into the hub using the Load ID or Name of the continuous load.

  • At regular intervals, Semarchy xDM automatically creates then submits an external load with the data loaded in the continuous load. This external load is submitted with a program name, a job, and a submitter name.

  • The continuous load remains, with the same Load ID and Name. Subsequent data loads made with this continuous load are processed at the next interval.

Continuous loads are configured and managed by the data location manager. Unlike external loads, they cannot be created, submitted or canceled via integration points.

Configure a continuous load

To configure a continuous load:

  1. In the Management view, expand the Data Locations node, then expand the data location for which you want to configure a continuous load.

  2. Right-click the Continuous Loads node and select New Continuous Load. The Create New Continuous Load wizard opens.

  3. Enter the following values:

    • Active: Check this option to make the continuous load active. Only active loads integrate data at a regular interval.

    • Name: Name of the continuous load. This name is used to uniquely identify this load.

    • Program Name: This value is for information only. It describes the submitted external loads.

    • On Submit Job: Integration job submitted with the external loads. This job is selected among those available in the deployed model edition.

    • Submit Interval: Interval in second between submissions.

    • Submit as: name of the user submitting the external loads. This user may or may not be a user declared in Semarchy.

    • Wait for Previous Loads to Finish: Check this option to make the continuous load wait for any queued or running load to be completed before submitting a new one (default: selected).

  4. Click Finish to close the wizard. The Continuous Load editor opens.

  5. In the Description field, optionally enter a description for this load.

  6. Press Control+S (or Command+S on macOS) to save the editor.

Note the Load ID and Name values of the continuous load as you will need them for integrating data using this load.

Using the Name instead of the Load ID in your integration processes and flows gives the flexibility to use the same integration process or flow definition regardless of the data location. When deploying a model to another data location (for example, to move from development to production), you just need to create another continuous load with the same Name.

The data location manager can deactivate a continuous load to prevent it from processing its data.

Activate continuous loads

To activate or deactivate continuous loads:

  1. In the Management view, expand the Data Locations node, then expand the data location for which you want to configure a continuous load.

  2. Double-click the Continuous Loads node. The Data Location editor opens on the Continuous Loads tab.

  3. Select one or more continuous loads in the list, and then click the Activate or Deactivate button in the toolbar.

  4. Press Control+S (or Command+S on macOS) to save the editor.

You do not need to restart any other component after creating, activating or deactivating a continuous load. The changes are immediately taken into account.
When deploying a new model edition that deprecates a job, continuous loads using this job are automatically made inactive. They must be updated to use the updated integration job and then reactivated by the data location manager.

Publish data with continuous loads

The SQL Interface or the REST API can use continuous loads via their Load ID or Name to push data into the hub without having to perform the load ID management operation.

  • The REST API calls support both the Load ID or the Name of the continuous load in their URL.

  • The SQL interface exposes tables with a B_LOADID column to load the Load ID of the continuous load.
    It provides a utility function called GET_CONTINUOUS_LOAD, stored in the repository (and in the INTEGRATION_LOAD package for Oracle), to return the Load ID for a given continuous load name.

Example 1. Retrieving a load ID from a continuous load name (Oracle)
vLoad_id := <repository_schema>.INTEGRATION_LOAD.GET_CONTINUOUS_LOADID(
	'<continuous_load_name>' /* Name of the continuous load. */
	);
Example 2. Retrieving a load ID from a continuous load name (PostgreSQL)
vLoad_id := <repository_schema>.get_continuous_loadid(
	'<continuous_load_name>' /* Name of the continuous load. */
	);
Example 3. Retrieving a load ID from a continuous load name (SQL Server)
EXEC <repository_schema>.dbo.GET_CONTINUOUS_LOADID
    @LOAD_ID OUTPUT,
    @P_CONTINUOUS_LOAD_NAME  = '<continuous_load_name>' /* Name of the continuous load. */
Using the Name instead of the Load ID in your integration processes and flows gives the flexibility to use the same integration process or flow definition regardless of the data location. When deploying a model to another data location (for example, to move from development to production), you just need to create another continuous load with the same Name.
When using a continuous load, make sure that it is configured with a job that processes the entities that are loaded.