acxiom / metalus   1.3.0

Apache License 2.0 GitHub

This project aims to make writing Spark applications easier by abstracting the effort to assemble the driver into reusable steps and pipelines.

Scala versions: 2.11
Branch Build Coverage
Develop Develop Build Develop Coverage
Master Release Master Coverage

Metalus Pipeline Library

The Metalus library was created as a way to create Spark applications at runtime without the need to write or compile code. The library is written in Scala and provides binaries for different version of Spark and Scala. Developers build applications by providing a JSON configuration file which gets loaded and executed by the metalus core library.

Documentation for this project may be found here.

Instructions for contributing to this project and instructions on building may be found here.

Projects

There are several sub-projects:

This project contains the core library and is the minimum requirement for any application.

This step library contains steps that are considered generic enough to be used in any project.

This step library contains AWS specific components. The Kinesis driver provides a basic implementation that gathers data and then initiates the Metalus Pipeline Core for processing of the incoming data.

This step library contains GCP specific components. The Pub/Sub driver provides a basic implementation that gathers data and then initiates the Metalus Pipeline Core for processing of the incoming data.

This step library contains GCP specific components. The Kafka driver provides a basic implementation that gathers data and then initiates the Metalus Pipeline Core for processing of the incoming data.

This step library adds support for working with Mongo.

This project provides several examples to help demonstrate how to use the library.

This project provides utilities that help work with the project.

This project provides a single jar that can be used to run the application. Additional components provide jars that can be added to the classpath.

Examples

Examples of building pipelines can be found in the metalus-examples project.