Generate code of case class from BigQuery schema. The generated case class also provides functionality to convert Scala objects to BigQuery row content.
Add to build.sbt
libraryDependencies += "com.github.satouso0401" %% "bigquery-class-gen" % "0.2.1"
This library needs to get the schema from BigQuery. Set the credential that has "bigquery.tables.list" and "bigquery.tables.get" permissions to the environment variable.
export GOOGLE_APPLICATION_CREDENTIALS="[CREDENTIAL_JSON]"
Generate a case class based on the table contained in the dataset.
implicit val bigQuery = BigQueryOptions.getDefaultInstance.getService
val datasetId = "test_dataset"
val outputDir = "src/main/scala"
val outputPkg = "output.bq.testdataset"
BigQueryCaseClassGenerator().run(datasetId, outputDir, outputPkg)
The generated case class can be converted to a BigQuery row using .toBqRow
val bigQuery = BigQueryOptions.getDefaultInstance.getService
val datasetId = "test_dataset"
val tableId = "play_list"
val rowContent = PlayList(1, "Lotus", 265, "Susumu Hirasawa").toBqRow // convert to BQ row
bigQuery.insertAll(
InsertAllRequest
.newBuilder(TableId.of(datasetId, tableId))
.addRow(rowContent)
.build)
Usually, BigQuery TIMESTAMP is mapped to ZonedDateTime. It can be overwritten by giving BigQueryCaseClassGenerator a Partial Function for mapping.
val importClass = "java.util.Date" :: Nil
val overwriteType: PartialFunction[StandardSQLTypeName, String] = {
case INT64 => "Int"
case TIMESTAMP => "Date"
}
val overwriteClassToRow: PartialFunction[(StandardSQLTypeName, String), String] = {
case (INT64, fieldName) => s"x.$fieldName"
case (TIMESTAMP, fieldName) => s"x.$fieldName.getTime / 1000"
}
BigQueryCaseClassGenerator(
importClass,
overwriteType,
overwriteClassToRow,
BigQueryCaseClassGenerator.NothingOverwriteFunctorToRow
).run(datasetId, outputDir, outputPkg)
