This is documentation for Semarchy xDI 5.3 or earlier, which is no longer supported.

For more information, see our Global Support and Maintenance Policy.

Spark Component

Overview

Semarchy xDI allows to work with Spark to produce fully customized Data Flows.

Install the Spark Component

The Spark Component is installed from Semarchy xDI Designer using the component installation feature.

Supported Features

Spark 2

Feature	Description
LOAD	Data can be loaded to Spark: HBase, HDFS, Hive, RBDMS, Vertica, Parquet, Elasticsearch Data can also be loaded from Spark: Hive, RDBMS, Vertica, Parquet, Elasticsearch
INTEGRATE	Data can in integrated from Spark: HDFS , Hive, RDBMS
STAGE	Spark Metadata can be used as a stage (between loading and integration) to boost Hadoop Mappings. Spark Stage can be: SQL, Java

Feature

Description

LOAD

Data can be loaded to Spark: HBase, HDFS, Hive, RBDMS, Vertica, Parquet, Elasticsearch

Data can also be loaded from Spark: Hive, RDBMS, Vertica, Parquet, Elasticsearch

INTEGRATE

Data can in integrated from Spark: HDFS , Hive, RDBMS

STAGE

Spark Metadata can be used as a stage (between loading and integration) to boost Hadoop Mappings.

Spark Stage can be: SQL, Java