Documentation site overhaul (#576)

Author: sritchie

.travis.yml

@@ -32,3 +32,11 @@ before_cache:
   - du -h -d 2 $HOME/.sbt/
   - find $HOME/.sbt -name "*.lock" -type f -delete
   - find $HOME/.ivy2/cache -name "ivydata-*.properties" -type f -delete
+
+notifications:
+  webhooks:
+    urls:
+      - https://webhooks.gitter.im/e/83f5f34730d7a004992f
+    on_success: change
+    on_failure: always
+    on_start: never

CONTRIBUTING.md

@@ -7,19 +7,85 @@ position: 5
 
 # Contributing to Algebird
 
-This page lists recommendations and requirements for how to best contribute to Algebird.
+This page lists recommendations and requirements for how to best contribute to Algebird. We strive to obey these as best as possible. As always, thanks for contributing--we hope these guidelines make it easier and shed some light on our approach and processes.
 
-We strive to obey these as best as possible. As always, thanks for contributing--we hope these guidelines make it easier and shed some light on our approach and processes.
+## Key branches
 
-### Key branches
 - `master` is the latest, deployed version.
 - `develop` is where development happens and all pull requests should be submitted.
 
-### Pull requests
+## Pull requests
 
-- Submit pull requests against the `develop` branch.
-- Try not to pollute your pull request with unintended changes--keep them simple and small.
+Submit pull requests against the `develop` branch. Try not to pollute your pull request with unintended changes. Keep it simple and small.
 
-### License
+## Contributing Documentation
+
+The documentation for Algebird's website is stored in the `docs/src/main/tut` directory of the [docs subproject](https://github.com/twitter/algebird/tree/develop/docs).
+
+Algebird's documentation is powered by [sbt-microsites](https://47deg.github.io/sbt-microsites/) and [tut](https://github.com/tpolecat/tut). `tut` compiles any code that appears in the documentation, ensuring that snippets and examples won't go out of date.
+
+We would love your help making our documentation better. If you see a page that's empty or needs work, please send us a pull request making it better. If you contribute a new data structure to Algebird, please add a corresponding documentation page. To do this, you'll need to:
+
+- Add a new Markdown file to `docs/src/main/tut/datatypes` with the following format:
+
+```markdown
+---
+layout: docs
+title:  "<Your Page Title>"
+section: "data"
+source: "algebird-core/src/main/scala/com/twitter/algebird/<YourDataType>.scala"
+scaladoc: "#com.twitter.algebird.<YourDataType>"
+---
+
+# Your Data Type
+
+.....
+```
+
+- Make sure to add some code examples! Any code block of this form will get compiled using `tut`:
+
+
+    ```toot:book
+    <your code>
+    ```
+
+(Please replace `toot` with `tut`!) `tut` will evaluate your code as if you'd pasted it into a REPL and insert each line's results in the output. State persists across `tut` code blocks, so feel free to alternate code blocks with text discussion. See the [tut README](https://github.com/tpolecat/tut) for more information on the various options you can use to customize your code blocks.
+- Add your page to the appropriate section in [the menu](https://github.com/twitter/algebird/tree/develop/docs/src/main/resources/microsite/data/menu.yml)
+
+### Generating the Site
+
+run `sbt docs/makeMicrosite` to generate a local copy of the microsite.
+
+### Previewing the site
+
+1. Install jekyll locally, depending on your platform, you might do this with any of the following commands:
+
+```
+yum install jekyll
+apt-get install jekyll
+gem install jekyll
+```
+
+2. In a shell, navigate to the generated site directory in `docs/target/site`
+3. Start jekyll with `jekyll serve --incremental`
+4. Navigate to http://127.0.0.1:4000/algebird/ in your browser
+5. Make changes to your site, and run `sbt docs/makeMicrosite` to regenerate the site. The changes should be reflected as soon as `sbt docs/makeMicrosite` completes.
+
+## Post-release
+
+After the release occurs, you will need to update the documentation. Here is a list of the places that will definitely need to be updated:
+
+ * `README.md`: update version numbers
+ * `CHANGES.md`: summarize changes since last release
+
+(Other changes may be necessary, especially for large releases.)
+
+You can get a list of changes between release tags `v0.1.2` and `v0.2.0` via `git log v0.1.2..v0.2.0`. Scanning this list of commit messages is a good way to get a summary of what happened, although it does not account for conversations that occured on Github.
+
+Once the relevant documentation changes have been committed, new [release notes](https://github.com/twitter/algebird/releases) should be added. You can add a release by clicking the "Draft a new release" button on that page, or if the relevant release already exists, you can click "Edit release".
+
+The website should then be updated via `sbt docs/publishMicrosite`.
+
+## License
 
 By contributing your code, you agree to license your contribution under the terms of the [APLv2](LICENSE).

README.md

@@ -9,7 +9,7 @@
 
 Abstract algebra for Scala. This code is targeted at building aggregation systems (via [Scalding](https://github.com/twitter/scalding) or [Apache Storm](http://storm.apache.org/)). It was originally developed as part of Scalding's Matrix API, where Matrices had values which are elements of Monoids, Groups, or Rings. Subsequently, it was clear that the code had broader application within Scalding and on other projects within Twitter.
 
-See the [current API documentation](http://twitter.github.com/algebird) for more information.
+See the [Algebird website](https://twitter.github.io/algebird) for more information.
 
 ### What can you do with this code?
 
@@ -61,7 +61,7 @@ A list of contributors to the project can be found here: [Contributors](https://
 
 ## Maven
 
-Algebird modules are available on maven central. The current groupid and version for all modules is, respectively, `"com.twitter"` and  `0.11.0`.
+Algebird modules are available on maven central. The current groupid and version for all modules is, respectively, `"com.twitter"` and  `0.12.2`.
 
 Current published artifacts are
 

algebird-core/src/main/scala/com/twitter/algebird/ExpHist.scala

@@ -194,7 +194,7 @@ object ExpHist {
 
     // Drops all buckets with an expired timestamp, based on the
     // configured window and the supplied current time.
-    def dropExpired(buckets: Vector[Bucket], currTime: Timestamp): (Long, Vector[Bucket]) =
+    private[algebird] def dropExpired(buckets: Vector[Bucket], currTime: Timestamp): (Long, Vector[Bucket]) =
       ExpHist.dropExpired(buckets, expiration(currTime))
 
     /**
@@ -225,7 +225,7 @@ object ExpHist {
    * @param cutoff buckets with ts <= cutoff are expired
    * @return the sum of evicted bucket sizes and the unexpired buckets
    */
-  def dropExpired(buckets: Vector[Bucket], cutoff: Timestamp): (Long, Vector[Bucket]) = {
+  private[algebird] def dropExpired(buckets: Vector[Bucket], cutoff: Timestamp): (Long, Vector[Bucket]) = {
     val (dropped, remaining) = buckets.reverse.span(_.timestamp <= cutoff)
     (dropped.map(_.size).sum, remaining.reverse)
   }

algebird-core/src/main/scala/com/twitter/algebird/SummingIterator.scala

@@ -29,7 +29,7 @@ import scala.annotation.tailrec
 
 /**
  * Creates an Iterator that emits partial sums of an input Iterator[V].
- * Generally this is useful to change from processing individiual V's to
+ * Generally this is useful to change from processing individual Vs to
  * possibly blocks of V @see SummingQueue or a cache of recent Keys in
  * a V=Map[K,W] case: @see SummingCache
  */

algebird-core/src/main/scala/com/twitter/algebird/macros/Cuber.scala

@@ -31,38 +31,6 @@ trait Cuber[I] {
   def apply(in: I): TraversableOnce[K]
 }
 
-/**
- * Given a TupleN, produces a sequence of (N + 1) tuples each of arity N
- * such that, for all k from 0 to N, there is a tuple with k Somes
- * followed by (N - k) Nones.
- *
- * This is useful for comparing some metric across multiple layers of
- * some hierarchy.
- * For example, suppose we have some climate data represented as
- * case class Data(continent: String, country: String, city: String, temperature: Double)
- * and we want to know the average temperatures of
- *   - each continent
- *   - each (continent, country) pair
- *   - each (continent, country, city) triple
- *
- * Here we desire the (continent, country) and (continent, country, city)
- * pair because, for example, if we grouped by city instead of by
- * (continent, country, city), we would accidentally combine the results for
- * Paris, Texas and Paris, France.
- *
- * Then we could do
- * > import com.twitter.algebird.macros.Roller.roller
- * > val data: List[Data]
- * > val averageTemps: Map[(Option[String], Option[String], Option[String]), Double] =
- * > data.flatMap { d => roller((d.continent, d.country, d.city)).map((_, d)) }
- * >   .groupBy(_._1)
- * >   .mapValues { xs => val temps = xs.map(_.temperature); temps.sum / temps.length }
- */
-trait Roller[I] {
-  type K
-  def apply(in: I): TraversableOnce[K]
-}
-
 object Cuber {
   implicit def cuber[T]: Cuber[T] = macro cuberImpl[T]
 
@@ -110,52 +78,3 @@ object Cuber {
     c.Expr[Cuber[T]](cuber)
   }
 }
-
-object Roller {
-  implicit def roller[T]: Roller[T] = macro rollerImpl[T]
-
-  def rollerImpl[T](c: Context)(implicit T: c.WeakTypeTag[T]): c.Expr[Roller[T]] = {
-    import c.universe._
-
-    ensureCaseClass(c)
-
-    val params = getParams(c)
-    val arity = params.length
-    if (arity > 22)
-      c.abort(c.enclosingPosition, s"Cannot create Roller for $T because it has more than 22 parameters.")
-    if (arity == 0)
-      c.abort(c.enclosingPosition, s"Cannot create Roller for $T because it has no parameters.")
-
-    val tupleName = {
-      val types = getParamTypes(c)
-      val optionTypes = types.map { t => tq"_root_.scala.Option[$t]" }
-      val tupleType = newTypeName(s"Tuple${arity}")
-      tq"_root_.scala.$tupleType[..$optionTypes]"
-    }
-
-    val somes = params.zip(Stream.from(1)).map {
-      case (param, index) =>
-        val name = newTermName(s"some$index")
-        q"val $name = _root_.scala.Some(in.$param)"
-    }
-
-    val items = (0 to arity).map { i =>
-      val args = (1 to arity).map { index =>
-        val some = newTermName(s"some$index")
-        if (index <= i) q"$some" else q"_root_.scala.None"
-      }
-      q"new K(..$args)"
-    }
-
-    val roller = q"""
-    new _root_.com.twitter.algebird.macros.Roller[${T}] {
-      type K = $tupleName
-      def apply(in: ${T}): _root_.scala.Seq[K] = {
-        ..$somes
-        Seq(..$items)
-      }
-    }
-    """
-    c.Expr[Roller[T]](roller)
-  }
-}

algebird-core/src/main/scala/com/twitter/algebird/macros/Roller.scala

@@ -0,0 +1,86 @@
+package com.twitter.algebird.macros
+
+import scala.language.experimental.{ macros => sMacros }
+import scala.reflect.macros.Context
+import scala.reflect.runtime.universe._
+
+/**
+ * Given a TupleN, produces a sequence of (N + 1) tuples each of arity N
+ * such that, for all k from 0 to N, there is a tuple with k Somes
+ * followed by (N - k) Nones.
+ *
+ * This is useful for comparing some metric across multiple layers of
+ * some hierarchy.
+ * For example, suppose we have some climate data represented as
+ * case class Data(continent: String, country: String, city: String, temperature: Double)
+ * and we want to know the average temperatures of
+ *   - each continent
+ *   - each (continent, country) pair
+ *   - each (continent, country, city) triple
+ *
+ * Here we desire the (continent, country) and (continent, country, city)
+ * pair because, for example, if we grouped by city instead of by
+ * (continent, country, city), we would accidentally combine the results for
+ * Paris, Texas and Paris, France.
+ *
+ * Then we could do
+ * > import com.twitter.algebird.macros.Roller.roller
+ * > val data: List[Data]
+ * > val averageTemps: Map[(Option[String], Option[String], Option[String]), Double] =
+ * > data.flatMap { d => roller((d.continent, d.country, d.city)).map((_, d)) }
+ * >   .groupBy(_._1)
+ * >   .mapValues { xs => val temps = xs.map(_.temperature); temps.sum / temps.length }
+ */
+trait Roller[I] {
+  type K
+  def apply(in: I): TraversableOnce[K]
+}
+
+object Roller {
+  implicit def roller[T]: Roller[T] = macro rollerImpl[T]
+
+  def rollerImpl[T](c: Context)(implicit T: c.WeakTypeTag[T]): c.Expr[Roller[T]] = {
+    import c.universe._
+
+    ensureCaseClass(c)
+
+    val params = getParams(c)
+    val arity = params.length
+    if (arity > 22)
+      c.abort(c.enclosingPosition, s"Cannot create Roller for $T because it has more than 22 parameters.")
+    if (arity == 0)
+      c.abort(c.enclosingPosition, s"Cannot create Roller for $T because it has no parameters.")
+
+    val tupleName = {
+      val types = getParamTypes(c)
+      val optionTypes = types.map { t => tq"_root_.scala.Option[$t]" }
+      val tupleType = newTypeName(s"Tuple${arity}")
+      tq"_root_.scala.$tupleType[..$optionTypes]"
+    }
+
+    val somes = params.zip(Stream.from(1)).map {
+      case (param, index) =>
+        val name = newTermName(s"some$index")
+        q"val $name = _root_.scala.Some(in.$param)"
+    }
+
+    val items = (0 to arity).map { i =>
+      val args = (1 to arity).map { index =>
+        val some = newTermName(s"some$index")
+        if (index <= i) q"$some" else q"_root_.scala.None"
+      }
+      q"new K(..$args)"
+    }
+
+    val roller = q"""
+    new _root_.com.twitter.algebird.macros.Roller[${T}] {
+      type K = $tupleName
+      def apply(in: ${T}): _root_.scala.Seq[K] = {
+        ..$somes
+        Seq(..$items)
+      }
+    }
+    """
+    c.Expr[Roller[T]](roller)
+  }
+}

algebird-test/src/test/scala/com/twitter/algebird/CombinatorTest.scala

@@ -28,23 +28,17 @@ class CombinatorTest extends CheckProperties {
     Arbitrary.arbitrary[T].map { t => Max(t) }
   }
 
-  implicit val sg: Semigroup[(Max[Int], List[Int])] =
-    new SemigroupCombinator({ (m: Max[Int], l: List[Int]) =>
-      val sortfn = { (i: Int) => i % (scala.math.sqrt(m.get.toLong - Int.MinValue).toInt + 1) }
-      l.sortWith { (l, r) =>
-        val (sl, sr) = (sortfn(l), sortfn(r))
-        if (sl == sr) l < r else sl < sr
-      }
-    })
+  private def fold(m: Max[Int], l: List[Int]): List[Int] = {
+    val sortfn = { (i: Int) => i % (scala.math.sqrt(m.get.toLong - Int.MinValue).toInt + 1) }
+    l.sortWith { (l, r) =>
+      val (sl, sr) = (sortfn(l), sortfn(r))
+      if (sl == sr) l < r else sl < sr
+    }
+  }
+
+  implicit val sg: Semigroup[(Max[Int], List[Int])] = new SemigroupCombinator(fold)
+  implicit val mond: Monoid[(Max[Int], List[Int])] = new MonoidCombinator(fold)
 
-  implicit val mond: Monoid[(Max[Int], List[Int])] =
-    new MonoidCombinator({ (m: Max[Int], l: List[Int]) =>
-      val sortfn = { (i: Int) => i % (scala.math.sqrt(m.get.toLong - Int.MinValue).toInt + 1) }
-      l.sortWith { (l, r) =>
-        val (sl, sr) = (sortfn(l), sortfn(r))
-        if (sl == sr) l < r else sl < sr
-      }
-    })
   // Make sure the lists start sorted:
   implicit def pairArb(implicit lista: Arbitrary[List[Int]]): Arbitrary[(Max[Int], List[Int])] =
     Arbitrary {
@@ -84,5 +78,4 @@ class CombinatorTest extends CheckProperties {
   property("MonoidCombinator with top-K forms a Monoid") {
     monoidLaws[(Map[Int, Int], Set[Int])]
   }
-
 }