Flint is a Scala library, server and Web UI for managing on-demand Spark clusters.

Add

libraryDependencies += "com.videoamp" %% "flint" % "1.2.6"

to your project's build.sbt. See the REPL example below for a Flint code snippet. The Scaladoc for the latest build is also hosted online.

To run Flint in a REPL, you will need configured aws_service.conf and creds.conf files. The other configuration files are unnecessary and required only by the server. See TUTORIAL.md for guidance on creating these files for your environment.

Follow the Tutorial for a complete guide to deploying the Flint Web app.

There are several ways to use Flint in a REPL.

Run sbt core/console.

You can use the files in this gist to integrate Ammonite into sbt globally. Copy ammonite.sbt into your global sbt build directory. For example, for sbt 0.13 this is ~/.sbt/0.13. This will add an amm task to your sbt build. If you run core/amm or server/amm in sbt you will drop into an Ammonite console with that project's source and library dependencies on the classpath. If you wish to run some code from a file every time you run the amm task, integrate predef.sc from the gist into ~/.ammonite/predef.sc (or copy it over if you have none). This will load the contents of any Scala script named .amm_init.sc from the subproject's directory when the amm task is run. For example, running core/amm will execute the code in core/.amm_init.sc, if that file exists. (Note that due to com-lihaoyi/Ammonite#658, this functionality is broken as of Ammonite 1.0.0. As a workaround you can run repl.load.exec(localInitPath) within the Ammonite console until this bug is fixed.)

Once in a REPL (sbt or Ammonite), you can use a ClusterService to launch, manage and terminate clusters. For example:

import flint._, service._, aws._
import ContainerState._
import ClusterTerminationReason._
import java.io.File
import com.typesafe.config.ConfigFactory

import scala.concurrent.Await
import scala.concurrent.ExecutionContext.Implicits.global

val flintConfig = ConfigFactory.parseFile(new File("conf/aws_service.conf")).getConfig("flint")
validateFlintConfig(flintConfig)

val cs: ClusterService = new AwsClusterService(flintConfig)

def newClusterSpec(flintSparkImage: DockerImage, workerInstanceType: String, numWorkers: Int, subnetId: String) =
  ClusterSpec(
    ClusterId(),
    flintSparkImage,
    "Bob Data User",
    None,
    None,
    "t2.micro",
    workerInstanceType,
    numWorkers,
    subnetId)

// Pick one of your Flint Spark images. See https://github.com/VideoAmp/flint-spark for more info
val flintSparkImage: DockerImage = ???
// Pick one of your VPC subnet ids
val subnetId: String = ???
val spec = newClusterSpec(flintSparkImage, "c3.8xlarge", 4, subnetId)

// Launch it
cs.launchCluster(spec).foreach { managedCluster =>
  managedCluster.cluster.master.containerState.collectFirst { case ContainerRunning =>
    println("Compute, compute, compute...")
    Thread.sleep(3000)
    println("Terminating...")
    Await.ready(managedCluster.terminate(ClientRequested))
  }
}

If you'd like to contribute to the Flint codebase, please read about the contribution process in CONTRIBUTING.md. Then consult DEVELOPING.md for build instructions and guidance.