Search *

7 results

swoop-inc/spark-alchemy

Collection of open-source Spark tools & frameworks that have made the data engineering and data science teams at Swoop highly productive

Scala versions: 2.12

data-engineering data-science scala spark

178 7
setl-framework/setl

A simple Spark-powered ETL framework that just works 🍺

Scala versions: 2.12 2.11

data-engineering etl-pipeline etl pipeline data-science spark scala machine-learning setl framework big-data modularization data-transformation dataset data-analysis

172 7
treeverse/lakefs

lakeFS - Data version control for your data lake | Git for data

Scala versions: 2.12 2.11

apache-sparksql data-version-control data-engineering datalake aws-s3 data-quality go lakefs data-lake git-for-data azure-storage object-storage hadoop-filesystem golang apache-spark data-versioning datalakes google-cloud-storage azure-blob-storage

3779 1162 77
galliaproject/gallia-core

A schema-aware Scala library for data transformation

Scala versions: 2.13 2.12

data-manipulation data-engineering nesting json etl data-science spark scala data-transformation feature-engineering

79 1
coxautomotivedatasolutions/waimak

Waimak is an open-source framework that makes it easier to create complex data flows in Apache Spark.

Scala versions: 2.13 2.12 2.11

data-engineering hadoop scala spark

75 9
coxautomotivedatasolutions/spark-distcp

A re-implementation of Hadoop DistCP in Apache Spark

Scala versions: 2.13 2.12 2.11

apache-spark data-engineering distcp hadoop spark

37 3
harrystech/hyppo-worker

The hyppo data ingestion system worker components

Scala versions: 2.11

data-engineering

0 1