hydrospheredata/spark-ml-serving   0.3.3

Spark ML Lib serving library

hablapps/sparkoptics   0.1.1

Optics for Spark DataFrames

coxautomotivedatasolutions/spark-distcp   0.2

A re-implementation of Hadoop DistCP in Apache Spark

absaoss/hyperdrive   4.7.0

Extensible streaming ingestion pipeline on top of Apache Spark

tharwaninitin/etlflow   1.7.3

EtlFlow is an ecosystem of functional libraries in Scala based on ZIO for running complex Auditable workflows which can interact with Google Cloud Platform, AWS, Kubernetes, Databases, SFTP servers, On-Prem Systems and more.

locationtech-labs/geopyspark   0.3.0

GeoTrellis for PySpark

xskipper-io/xskipper   1.6.0

An Extensible Data Skipping Framework

benfradet/struct-type-encoder   0.6.0

Deriving Spark DataFrame schemas from case classes

heartsavior/spark-sql-kafka-offset-committer   0.2.0

Kafka offset committer for structured streaming query

g-research/spark-dgraph-connector   0.2.0

A connector for Apache Spark and PySpark to Dgraph databases.

absaoss/spark-hats   0.3.0

Nested array transformation helper extensions for Apache Spark

tupol/spark-utils   0.6.2

Basic framework utilities to quickly start writing production ready Apache Spark applications

zuinnote/spark-hadoopoffice-ds   1.7.0

A Spark datasource for the HadoopOffice library

agile-lab-dev/darwin   1.2.2

Avro Schema Evolution made easy

music-of-the-ainur/almaren-framework   2.4.5-2.4.5

The Almaren Framework provides a simplified consistent minimalistic layer over Apache Spark. While still allowing you to take advantage of native Apache Spark features. You can still combine it with standard Spark code.

intenthq/pucket   1.7.3

Bucketing and partitioning system for Parquet

sansa-stack/archived-sansa-query   0.7.1

SANSA Query Layer

fsanaulla/chronicler-spark   0.6.0

InfluxDB connector to Apache Spark on top of Chronicler

agile-lab-dev/wasp   3.0.1

WASP is a framework to build complex real time big data applications. It relies on a kind of Kappa/Lambda architecture mainly leveraging Kafka and Spark. If you need to ingest huge amount of heterogeneous data and analyze them through complex pipelines, this is the framework for you.

sansa-stack/archived-sansa-inference   0.7.1

A general Inference API based on two of the most popular Big Data processing engines: Apache Spark and Apache Flink