apache/spark

Apache Spark - A unified analytics engine for large-scale data processing

apache/predictionio

PredictionIO, a machine learning server for developers and ML engineers.

delta-io/delta

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs

microsoft/synapseml

Simple and Distributed Machine Learning

apache/flink

Apache Flink

bigdatagenomics/adam

ADAM is a genomics analysis platform with specialized file formats built using Apache Avro, Apache Spark, and Apache Parquet. Apache 2 licensed.

yotpoltd/metorikku

A simplified, lightweight ETL Framework based on Apache Spark

nvidia/spark-rapids

Spark RAPIDS plugin - accelerate Apache Spark with GPUs

microsoft/hyperspace

An open source indexing subsystem that brings index-based query acceleration to Apache Spark™ and big data workloads.

delta-io/delta-sharing

An open protocol for secure data sharing

hydrospheredata/mist

Serverless proxy for Spark cluster

touk/nussknacker

A visual tool to define and run real-time decision algorithms. Brings agility to business teams, liberates developers to focus on technology.

setl-framework/setl

A simple Spark-powered ETL framework that just works 🍺

sparkling-graph/sparkling-graph

SparklingGraph provides easy to use set of features that will give you ability to proces large scala graphs using Spark and GraphX.

51zero/eel-sdk

Big Data Toolkit for the JVM

clustering4ever/clustering4ever

C4E, a JVM friendly library written in Scala for both local and distributed (Spark) Clustering.

apache/samza

Mirror of Apache Samza

swoop-inc/spark-records

Bulletproof Apache Spark jobs with fast root cause analysis of failures.

hablapps/doric

Type safety for spark columns

locationtech-labs/geopyspark

GeoTrellis for PySpark