Wednesday, September 30, 2015

Publishing custom artifacts with SBT

In today's world the code is packaged and re-packaged in different ways. The code is typically packaged in a jar file. Along with that there is another jar file for the sources, and another for the javadoc. But then an application will use more than one jar along with other resources which results in the need for a "distribution" which is typically a zip file with a bunch of resources copied in the appropriate structure to be run. But then there is Docker which yet another packaging solution for delivering the application. With docker you would re-package the distribution in a different format. All these files need to be published to some repository as part of the build or release process.
I guess typically, the UI is last piece that with the most dependencies in a large application. In my Company for instance we develop the back-end code using plain Java and Maven and UIs using Play Framework and SBT. There is always the possibility of Mavenizing the Play project but somehow I never liked to complicate things when there is no absolute need for it.
When it comes to dependency management, the Maven builds will publish their artifacts in a repository (Artifactory for instance) and the UI will use the artifacts via managed dependencies.
SBT then compiles and packages the UI code and the resulting artifacts can be published to Artifactory via the simple "publish" command.
There are however some gaps here, and they all can be solved with a few lines of code in your build.sbt. :

  1. Play will generate multiple jar files but not all are published by default. For instance, try creating a Play module that is not just code, but is a full Play app complete with images, CSS and other assets. Play will package the assets separately from the project jar. It all works fine if you use a distribution created via "dist" because Play knows to add all the jars. But if you try to use the application as a dependency in another, you will need the assets jar which is not typically published. Fortunately it does not take too much code to convince SBT to do so. So once you encounter the problem you will ask Google for a solution and you will find that adding this to your build.sbt will result in publishing the assets jar too:
    packagedArtifacts in publish := {
      val artifacts: Map[sbt.Artifact, java.io.File] = (packagedArtifacts in publish).value
      val assets: java.io.File = (playPackageAssets in Compile).value
      artifacts + (Artifact(moduleName.value, "jar", "jar", "assets") -> assets)
    }
    Then you can add both the regular artifact and the asset jar as a dependency (note the "assets" classifier):
    libraryDependencies ++= Seq(
      "com.acme" %% "foo" % "1.0",  "com.acme" %% "foo" % "1.0" classifier "assets")
  2. Play/SBT do a nice job of generating a distribution. The distribution step, however is not included in the release process (see sbt-release plugin). This can be fixed easily by customizing the release process:
    releaseProcess := Seq[ReleaseStep](
      checkSnapshotDependencies,  inquireVersions,  // runTest,           // do not run tests - assume everything is perfect  setReleaseVersion,  commitReleaseVersion,  tagRelease,  releaseStepTask(dist in myapp), // create distribution  publishArtifacts,  setNextVersion,  commitNextVersion,  pushChanges)
    In the above snippet I am removing the test phase because we are automatically running tests as part of each commit so when it gets to release, everything is in order. This is just to speed up things.
    Also, I am adding a step for creating the distribution. This step need to be run before the version is changed or the zip file name will be stamped with the next version. You'll see later why I do it before publishing artifacts as well.
  3. Now the  zip file is created but it is not included in the list of artifacts being published. Oh well, the trick at point 1) can be applied here too.  
  4. I don't know about others but I do not like to mix jars with distributions so I really wanted to publish the distribution to a different repository. This also is fixable and you can find the solution on stackoverflow. The idea is that you can create a custom task that extends the "publish" task and overrides the "publishTo" setting.   
  5. Finally, since I am working on a relatively large project, there are in fact multiple submodule/applications that are being built and they do not always need to be deployed together. As such, I am creating multiple distributions that package different components. So now I need to put it all together - publish multiple distributions to a different repository during the release process.
    lazy val publishDist = taskKey[Unit]("Publish distributions - Custom Task")
    publishDist := {
      println("Publishing distributions ...")
      val extracted = Project.extract(state.value)
      Project.runTask(publish, extracted.append(List(
        publishTo := Some("distributions" at distRepo),    publishMavenStyle := true,    publishArtifact in Compile := false,    publishArtifact in Test := false,    publishArtifact in Universal := false,    packagedArtifacts in publish := {
        val artifacts: Map[sbt.Artifact, java.io.File] = 
    
    (packagedArtifacts in publish).value
    artifacts + (Artifact("root", "zip", "zip") -> baseDirectory.value / "target" / "universal" / ("root-" + version.value + ".zip")) + (Artifact("submodule1", "zip", "zip") -> baseDirectory.value / "modules/submodule1/target" / "universal" / ("submodule1-" + version.value + ".zip")) + (Artifact("submodule1", "zip", "zip") -> baseDirectory.value / "modules/submodule2/target" / "universal" / ("submodule2-" + version.value + ".zip")) + } ), state.value), true) }
    Here I am adding to SBT's default list of artifacts my files - the zips along with customizations on where and how to publish.
    This task can be invoked manually in the activator console, but I created it so that I can invoke it as part of the release process. It is important that the task is invoked after the distributions are created:
    releaseProcess := Seq[ReleaseStep](
      checkSnapshotDependencies,  inquireVersions,  // runTest,           // do not run tests - assume everything is perfect  setReleaseVersion,  commitReleaseVersion,  tagRelease,  releaseStepTask(dist in root),
      releaseStepTask(dist in submodule1),
      releaseStepTask(dist in submodule2),
      publishArtifacts,  releaseStepTask(publishDist), // publish the distributions to Artifactory  setNextVersion,  commitNextVersion,  pushChanges)

As it's always the case with SBT it looks easy and obvious when done, but it's not as obvious when you are not very clear on its concepts.