Search *

32 results

microsoft/synapseml

Simple and Distributed Machine Learning

Scala versions: 2.11

deep-learning databricks apache-spark onnx lightgbm azure cognitive-services opencv big-data data-science spark ai microsoft machine-learning ml pyspark synapse scala http model-deployment

4562 97
lucacanali/sparkmeasure

This is the development repository for sparkMeasure, a tool for performance troubleshooting of Apache Spark workloads. It simplifies the collection and analysis of Spark task and stage metrics data.

Scala versions: 2.13 2.12 2.11

apache-spark performance-metrics performance-troubleshooting spark scala python

594 11
hydrospheredata/mist

Serverless proxy for Spark cluster

Scala versions: 2.12 2.11 2.10

apache-spark api big-data serverless

327 23
azure/azure-event-hubs-spark

Enabling Continuous Data Processing with Apache Spark and Azure Event Hubs

Scala versions: 2.12 2.11 2.10

apache-spark streaming azure eventhubs kafka stream scala real-time microsoft spark-streaming bigdata continuous databricks structured-streaming spark apache ingestion connector event-hubs

227 37
azure/azure-cosmosdb-spark

Apache Spark Connector for Azure Cosmos DB

Scala versions: 2.11 2.10

databricks apache-spark lambda-architecture cosmos-db databricks-notebooks azure-databricks connector azure-cosmos-db jupyter-notebook pyspark spark changefeed

194 24
treeverse/lakefs

lakeFS - Data version control for your data lake | Git for data

Scala versions: 2.12 2.11

apache-sparksql data-version-control data-engineering datalake aws-s3 data-quality go lakefs data-lake git-for-data azure-storage object-storage hadoop-filesystem golang apache-spark data-versioning datalakes google-cloud-storage azure-blob-storage

3642 1137 76
streamnative/pulsar-spark

Spark Connector to read and write with Pulsar

Scala versions: 2.12 2.11

structured-streaming apache-spark apache-pulsar flink data-science batch-processing stream-processing spark-sql spark data-processing

102 19
microsoft/mobius

C# and F# language binding and extensions to Apache Spark

Scala versions: 2.11 2.10

bigdata apache-spark streaming mapreduce dataframe eventhubs kafka-streaming rdd dstream spark-streaming near-real-time csharp mobius spark dataset fsharp

942 37
chermenin/spark-states

Custom state store providers for Apache Spark

Scala versions: 2.12 2.11

structured-streaming apache-spark state spark-structured-streaming stateful state-store spark-streaming spark apache

92 3
kakao/cuesheet

A framework for writing Spark 2.x applications in a pretty way

Scala versions: 2.11 2.10

apache-spark yarn scala magic spark mango

85 3
sansa-stack/sansa-stack

Big Data RDF Processing and Analytics Stack built on Apache Spark and Apache Jena http://sansa-stack.github.io/SANSA-Stack/

Scala versions: 2.12 2.11

rdf apache-spark flink spark semantic-web apache-jena distributed-computing

139 35
swoop-inc/spark-records

Bulletproof Apache Spark jobs with fast root cause analysis of failures.

Scala versions: 2.12

apache-spark big-data sparksql scala spark-records spark swoop

72 4
databrickslabs/automl-toolkit

Toolkit for Apache Spark ML for Feature clean-up, feature Importance calculation suite, Information Gain selection, Distributed SMOTE, Model selection and training, Hyper parameter optimization and selection, Model interprability.

Scala versions: 2.11

apache-spark feature-engineering spark scala ml pyspark machinelearning

187 0 7
uosdmlab/spark-nkp

Natural Korean Processor for Apache Spark

Scala versions: 2.11

nlp apache-spark text-mining korean-nlp spark natural-language-processing spark-mllib

53 2
absaoss/hyperdrive

Extensible streaming ingestion pipeline on top of Apache Spark

Scala versions: 2.12 2.11

apache-spark streaming spark-structured-streaming framework pipeline kafka streaming-etl spark ingestion

42 12
coxautomotivedatasolutions/spark-distcp

A re-implementation of Hadoop DistCP in Apache Spark

Scala versions: 2.13 2.12 2.11

apache-spark data-engineering distcp hadoop spark

36 3
heartsavior/spark-state-tools

Spark Structured Streaming State Tools

Scala versions: 2.12 2.11

apache-spark structured-streaming

34 2
tupol/spark-utils

Basic framework utilities to quickly start writing production ready Apache Spark applications

Scala versions: 2.12 2.11

apache-spark convenience data-source framework data-sink spark scala spark-applications spark-streaming

33 1
itspawanbhardwaj/spark-fuzzy-matching

Fuzzy matching function in spark (https://spark-packages.org/package/itspawanbhardwaj/spark-fuzzy-matching)

Scala versions: 2.11 2.10

levenshtein fuzzy-matching algorithm scala similarity-metric soundex apache-spark

23 1
isarn/isarn-sketches-spark

Routines and data structures for using isarn-sketches idiomatically in Apache Spark

Scala versions: 2.12 2.11 2.10

dataframes sketching-algorithm apache-spark aggregator data-sketches dataframe variable-importance datasets t-digest data-sketching spark scala spark-ml pyspark dataset udaf python feature-importance

30 2