An open source indexing subsystem that brings index-based query acceleration to Apache Spark™ and big data workloads.
This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.
When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.
Please review our contribution guide.
Development on Windows
This repository contains symbolic links which don't work properly on Windows. To build this project on Windows, you can use our provided Git aliases to replace symbolic links with junctions.
$ git config --local include.path ../dev/.gitconfig $ git replace-symlinks # replace symlinks with junctions $ #git restore-symlinks # use this to restore symlinks if you need
You can use the built-in sbt shell in IntelliJ without any problems. However, the built-in "Build Project" command may not work. To fix the issue, go to Project Structure -> Project Settings -> Modules and follow these steps:
src/main/scala-spark2as "Sources" and
src/test/scala-spark2as "Tests" for the
src/main/scala-spark3as "Sources" and
src/test/scala-spark3as "Tests" for the
- Remove the
- An example of Project Structure
Additionally, you might have to run
sbt buildInfo if you encounter an error like
object BuildInfo is not a member of package com.microsoft.hyperspace for the first build.
Inspiration and Special Thanks
This project would not have been possible without the outstanding work from the following communities:
- Apache Spark: Unified Analytics Engine for Big Data, the engine that Hyperspace builds on top of.
- Delta Lake: Delta Lake is an open-source storage layer that brings ACID transactions to Apache Spark™ and big data workloads. Hyperspace derives quite a bit of inspiration from the way the Delta Lake community operates and pioneering of some surrounding ideas in the context of data lakes (e.g., their novel use of optimistic concurrency).
- Databricks: Unified analytics platform. Many thanks to all the inspiration they have provided us.
- .NET for Apache Spark™: Hyperspace offers .NET bindings for developers, thanks to the efforts of this team in collaborating and releasing the bindings just-in-time.
- Minimal Mistakes: The awesome theme behind Hyperspace documentation.
Code of Conduct
Apache License 2.0, see LICENSE.