OOPSLA ~ External Benchmarks Streams

This page describes an external benchmark: a full description is outside of the scope of this wiki, but we recommend seeing the scala-streams repository for more information. The reason this benchmark was included in the tutorial is because miniboxing uses a mechanism similar to ildl to transform the function representation.

The benchmark files described in the reminder of this presentation are available in the scala-streams repository in the oopsla15 tag.

Description

A stream type has a single constructor. The Stream constructor takes as a parameter a function which takes as a parameter another function with type (T => Boolean) => Unit. A stream holds a function that represents the entire iteration over elements of an array, with another function applied to each one. The bool return value serves to indicate whether the next consumer down the operator chain should be called on the element or not. The iteration itself is encoded as a loop . The ofArray operator creates such a stream with the initial looping function passed as the iterf function. Intermediate operators (e.g., map) alter these iteration functions and a terminal operator (e.g., fold) will evaluate the stream function, transforming again the iteration function if needed. fold holds the state in a separate accumulator, initializes it according to the passed state, and returns it.

The project uses heavily function types (e.g., T => Stream[R] for flatMap, f: T => R for map) where the type parameters can be generic or primitive types. The transformation of the layout of functions employed by the miniboxing plugin increases performance significantly.

The project uses JMH through a necessary wrapper for the build tool that the project uses. The wrapper(sbt plugin) is called sbt-jmh.

To explore the scala-streams benchmarks you can either:

• clone the project via git clone:

$ git clone git@github.com:biboudis/scala-streams.git
$ cd scala-streams
$ git checkout oopsla15
$ sbt

• download the archive, disconnected from git:

$ wget https://github.com/biboudis/scala-streams/archive/oopsla15.zip 
$ unzip oopsla15.zip 
$ rm oopsla15.zip
$ cd scala-streams-oopsla15
$ sbt

Running the Benchmarks

To enable or disable the miniboxing plugin edit the build.sbt configuration file. To run the benchmarks enter the command below in the sbt command line prompt. This runs JMH five times, with five warmup iterations, for 1 fork (more are recommended to measure run-to-run variance) and selects only the streams benchmarks included in the paper. More are included in the benchmark file (baseline and scala-views that are omitted for our purposes here):

run -i 5 -wi 5 -f 1 ".*streams.*"

Generic/Erased

The last lines in the build.sbt file should all be commented:

// Enable miniboxing
// addCompilerPlugin("org.scala-miniboxing.plugins" %% "miniboxing-plugin" % "0.4-M4")

// Miniboxed
// scalacOptions ++= Seq("-P:minibox:mark-all", "-P:minibox:Ykeep-functionX-values")

// Miniboxed + functions
// scalacOptions ++= Seq("-P:minibox:mark-all")

And the results are:

[info] Benchmark                           Mode  Cnt    Score    Error  Units
[info] Pipelines.streams_sum               avgt    5   98.212 ± 4.558  ms/op
[info] Pipelines.streams_sumOfSquares      avgt    5  131.557 ± 7.803  ms/op
[info] Pipelines.streams_sumOfSquaresEven  avgt    5   92.299 ± 4.658  ms/op
[info] Pipelines.streams_cart              avgt    5  217.351 ± 6.302  ms/op

Miniboxing without function support

Enable Miniboxing by un-commenting the following declarations:

// Enable miniboxing
addCompilerPlugin("org.scala-miniboxing.plugins" %% "miniboxing-plugin" % "0.4-M4")

// Miniboxed
scalacOptions ++= Seq("-P:minibox:mark-all", "-P:minibox:Ykeep-functionX-values")

// Miniboxed + functions
// scalacOptions ++= Seq("-P:minibox:mark-all")

and re-run sbt:

[info] Benchmark                           Mode  Cnt    Score    Error  Units
[info] Pipelines.streams_sum               avgt    5  158.565 ±  8.642  ms/op
[info] Pipelines.streams_sumOfSquares      avgt    5  193.141 ± 11.282  ms/op
[info] Pipelines.streams_sumOfSquaresEven  avgt    5  189.583 ±  7.803  ms/op
[info] Pipelines.streams_cart              avgt    5  214.849 ±  3.725  ms/op

Miniboxed, with function support

Enable miniboxing function support:

// Enable miniboxing
addCompilerPlugin("org.scala-miniboxing.plugins" %% "miniboxing-plugin" % "0.4-M4")

// Miniboxed
// scalacOptions ++= Seq("-P:minibox:mark-all", "-P:minibox:Ykeep-functionX-values")

// Miniboxed + functions
scalacOptions ++= Seq("-P:minibox:mark-all")

and re-run sbt:

[info] Benchmark                           Mode  Cnt   Score   Error  Units
[info] Pipelines.streams_sum               avgt    5  17.953 ± 3.994  ms/op
[info] Pipelines.streams_sumOfSquares      avgt    5  12.010 ± 0.414  ms/op
[info] Pipelines.streams_sumOfSquaresEven  avgt    5  48.699 ± 2.148  ms/op
[info] Pipelines.streams_cart              avgt    5  57.464 ± 2.215  ms/op

Great! Now you've seen how the ildl-like function support in miniboxing can improve the performance of the Streams library!

From Here:

• see the second set of external benchmarks
• get back to the home page