ebiznext / comet-data-pipeline   0.2.6

Apache License 2.0 Website GitHub

Comet Data Pipeline is a Spark Based On Premise and Cloud Ingestion & Transformation Framework for Batch & Stream Processing Systems

Scala versions: 2.12 2.11

Build Status Scala Steward badge codecov Codacy Badge Documentation Maven Central Comet Spark 3 Join the chat at https://gitter.im/comet-data-pipeline/community License

About Comet Data Pipeline

Complete documentation available here


The purpose of this project is to efficiently ingest various data sources in different formats and make them available for analytics. Usually, ingestion is done by writing hand made custom parsers that transform input files into datasets of records.

This project aims at automating this parsing task by making data ingestion purely declarative.

The workflow below is a typical use case :

  • Export your data as a set of DSV (Delimiter-separated values) or JSON files
  • Define each DSV/JSON file with a schema using YAML syntax
  • Configure the ingestion process
  • Start watching your data being available as Hive Tables in your datalake

The main advantages of the Comet Data Pipeline project are :

  • Eliminates manual coding for data ingestion
  • Assign metadata to each dataset
  • Expose data ingestion metrics and history
  • Transform text files to strongly typed records
  • Support semantic types
  • Force privacy on specific fields (RGPD)
  • very, very simple piece of software to administer

How it works

Comet Data Pipeline automates the loading and parsing of files and their ingestion into a Hadoop Datalake where datasets become available as Hive tables.

Complete Comet Data pipeline

  1. Landing Area : Files are first stored in the local file system
  2. Staging Area : Files associated with a schema are imported into the datalake
  3. Working Area : Staged Files are parsed against their schema and records are rejected or accepted and made available in parquet/orc/... files as Hive Tables.
  4. Business Area : Tables in the working area may be joined to provide a hoslictic view of the data through the definition of AutoJob.
  5. Data visualization : parquet/orc/... tables may be exposed in datawarehouses or elasticsearch indexes