Skip to content

Improve the build and test times and add Codecov.io #541

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Sep 6, 2016
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 19 additions & 3 deletions .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,24 @@ language: scala
sudo: false
matrix:
include:
- scala: 2.10.5
- scala: 2.10.6
script: ./sbt ++$TRAVIS_SCALA_VERSION clean test

- scala: 2.11.7
script: ./sbt ++$TRAVIS_SCALA_VERSION clean test mimaReportBinaryIssues
- scala: 2.11.8
script: ./sbt ++$TRAVIS_SCALA_VERSION clean coverage test coverageReport mimaReportBinaryIssues
after_success:
- bash <(curl -s https://codecov.io/bash)

cache:
directories:
- $HOME/.sbt/0.13/dependency
- $HOME/.sbt/boot/scala*
- $HOME/.sbt/launchers
- $HOME/.ivy2/cache
- $HOME/.nvm

before_cache:
- du -h -d 1 $HOME/.ivy2/cache
- du -h -d 2 $HOME/.sbt/
- find $HOME/.sbt -name "*.lock" -type f -delete
- find $HOME/.ivy2/cache -name "ivydata-*.properties" -type f -delete
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
## Algebird [![Build status](https://img.shields.io/travis/twitter/algebird/develop.svg)](http://travis-ci.org/twitter/algebird) [![Coverage status](https://img.shields.io/coveralls/twitter/algebird/develop.svg)](https://coveralls.io/r/twitter/algebird?branch=develop)
## Algebird [![Build status](https://img.shields.io/travis/twitter/algebird/develop.svg)](http://travis-ci.org/twitter/algebird)
[![Codecov branch](https://img.shields.io/codecov/c/github/twitter/algebird/develop.svg?maxAge=2592000)](https://codecov.io/github/twitter/algebird)


Abstract algebra for Scala. This code is targeted at building aggregation systems (via [Scalding](https://github.com/twitter/scalding) or [Apache Storm](http://storm.apache.org/)). It was originally developed as part of Scalding's Matrix API, where Matrices had values which are elements of Monoids, Groups, or Rings. Subsequently, it was clear that the code had broader application within Scalding and on other projects within Twitter.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -250,8 +250,8 @@ class CMSContraMapSpec extends WordSpec with Matchers with GeneratorDrivenProper
val minWidth = data1.distinct.size

forAll(
(Gen.choose(1, 709), "depth"),
(Gen.choose(minWidth, 10000), "width"),
(Gen.choose(1, 70), "depth"),
(Gen.choose(minWidth, 1000), "width"),
(Gen.choose(Int.MinValue, Int.MaxValue), "seed")) { (depth: Int, width: Int, seed: Int) =>
val cms1: TopCMS[Seq[Byte]] = TopPctCMS.monoid[Seq[Byte]](depth, width, seed, 0.01).create(data1)
cms1.heavyHitters should be(Set(oneKey, twoKey, threeKey, fourKey, fiveKey))
Expand Down Expand Up @@ -373,6 +373,9 @@ abstract class CMSTest[K: CMSHasher: FromIntLike] extends WordSpec with Matchers
val EPS = 0.001
val SEED = 1

private[this] val maxDepth = 70

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use these in the changes to the Gens above?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a different class and I didn't want to yak shave too much. Just want the minimal change to enable code coverage to work.

private[this] val maxWidth = 1000

// We use TopPctCMS for testing CMSCounting functionality. We argue that because TopPctCMS[K] encapsulates CMS[K]
// and uses it for all its counting/querying functionality (like an adapter) we can test CMS[K] indirectly through
// testing TopPctCMS[K].
Expand Down Expand Up @@ -579,8 +582,8 @@ abstract class CMSTest[K: CMSHasher: FromIntLike] extends WordSpec with Matchers
val minWidth = data1.distinct.size

forAll(
(Gen.choose(minDepth, 709), "depth"),
(Gen.choose(minWidth, 10000), "width"),
(Gen.choose(minDepth, maxDepth), "depth"),
(Gen.choose(minWidth, maxWidth), "width"),
(Gen.choose(Int.MinValue, Int.MaxValue), "seed")) { (depth: Int, width: Int, seed: Int) =>
val cms1 = TopPctCMS.monoid[K](depth, width, seed, 0.01).create(data1)
cms1.heavyHitters should be(Set(1, 2, 3, 4, 5).toK[K])
Expand All @@ -600,11 +603,11 @@ abstract class CMSTest[K: CMSHasher: FromIntLike] extends WordSpec with Matchers
val data1 = Seq(1, 2, 2, 3, 3, 3, 4, 4, 4, 4, 5, 5, 5, 5, 5).toK[K]

val minDepth = 2 // Use 2 to be on the safe side in case we happen to run into hash collisions
val minWidth = data1.distinct.size
val minWidth = data1.distinct.size * 4

forAll(
(Gen.choose(minDepth, 709), "depth"),
(Gen.choose(minWidth, 10000), "width"),
(Gen.choose(minDepth, maxDepth), "depth"),
(Gen.choose(minWidth, maxWidth), "width"),
(Gen.choose(Int.MinValue, Int.MaxValue), "seed")) { (depth: Int, width: Int, seed: Int) =>
val cms1 = TopPctCMS.aggregator[K](depth, width, seed, 0.01).apply(data1)
cms1.heavyHitters should be(Set(1, 2, 3, 4, 5).toK[K])
Expand Down Expand Up @@ -784,50 +787,30 @@ abstract class CMSTest[K: CMSHasher: FromIntLike] extends WordSpec with Matchers
val minWidth = data1.distinct.size

forAll(
(Gen.choose(minDepth, 709), "depth"),
(Gen.choose(minWidth, 10000), "width"),
(Gen.choose(minDepth, maxDepth), "depth"),
(Gen.choose(minWidth, maxWidth), "width"),
(Gen.choose(Int.MinValue, Int.MaxValue), "seed")) { (depth: Int, width: Int, seed: Int) =>
val cms1 = TopNCMS.monoid[K](depth, width, seed, 5).create(data1)
cms1.heavyHitters should be(Set(1, 2, 3, 4, 5).toK[K])

val cms2 = TopNCMS.monoid[K](depth, width, seed, 4).create(data1)
cms2.heavyHitters should be(Set(2, 3, 4, 5).toK[K])

val cms3 = TopNCMS.monoid[K](depth, width, seed, 3).create(data1)
cms3.heavyHitters should be(Set(3, 4, 5).toK[K])

val cms4 = TopNCMS.monoid[K](depth, width, seed, 2).create(data1)
cms4.heavyHitters should be(Set(4, 5).toK[K])

val cms5 = TopNCMS.monoid[K](depth, width, seed, 1).create(data1)
cms5.heavyHitters should be(Set(5).toK[K])
val size = math.abs(seed % 5) + 1 // a number 1 to 5
val hh = ((6 - size) to 5).toSet
val cms = TopNCMS.monoid[K](depth, width, seed, size).create(data1)
cms.heavyHitters should be(hh.toK[K])
}
}

"work as an Aggregator when created from a single, small stream" in {
val data1 = Seq(1, 2, 2, 3, 3, 3, 4, 4, 4, 4, 5, 5, 5, 5, 5).toK[K]

val minDepth = 2 // Use 2 to be on the safe side in case we happen to run into hash collisions

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are these being removed because of redundancy?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. They are tested by the replaced line (we only check one for each of
the checks, but that is fine).
On Mon, Aug 29, 2016 at 14:18 Ben Pence [email protected] wrote:

In
algebird-test/src/test/scala/com/twitter/algebird/CountMinSketchTest.scala
#541 (comment):

     (Gen.choose(Int.MinValue, Int.MaxValue), "seed")) { (depth: Int, width: Int, seed: Int) =>
  •      val cms1 = TopNCMS.monoid[K](depth, width, seed, 5).create(data1)
    

- cms1.heavyHitters should be(Set(1, 2, 3, 4, 5).toK[K])

  •      val cms2 = TopNCMS.monoid[K](depth, width, seed, 4).create(data1)
    

- cms2.heavyHitters should be(Set(2, 3, 4, 5).toK[K])

  •      val cms3 = TopNCMS.monoid[K](depth, width, seed, 3).create(data1)
    

- cms3.heavyHitters should be(Set(3, 4, 5).toK[K])

  •      val cms4 = TopNCMS.monoid[K](depth, width, seed, 2).create(data1)
    

- cms4.heavyHitters should be(Set(4, 5).toK[K])

  •      val cms5 = TopNCMS.monoid[K](depth, width, seed, 1).create(data1)
    
  •      cms5.heavyHitters should be(Set(5).toK[K])
    

Are these being removed because of redundancy?


You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/twitter/algebird/pull/541/files/973cb4f94e8734ff7b8a201b40c645500b95dd5e#r76711645,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAEJdmpCnzN5bUr4g4wzIeZEi3wGaVyKks5qk3bOgaJpZM4JbPXi
.

val minWidth = data1.distinct.size
val minWidth = data1.distinct.size * 2

forAll(
(Gen.choose(minDepth, 709), "depth"),
(Gen.choose(minWidth, 10000), "width"),
(Gen.choose(minDepth, maxDepth), "depth"),
(Gen.choose(minWidth, maxWidth), "width"),
(Gen.choose(Int.MinValue, Int.MaxValue), "seed")) { (depth: Int, width: Int, seed: Int) =>
val cms1 = TopNCMS.aggregator[K](depth, width, seed, 5).apply(data1)
cms1.heavyHitters should be(Set(1, 2, 3, 4, 5).toK[K])

val cms2 = TopNCMS.aggregator[K](depth, width, seed, 4).apply(data1)
cms2.heavyHitters should be(Set(2, 3, 4, 5).toK[K])

val cms3 = TopNCMS.aggregator[K](depth, width, seed, 3).apply(data1)
cms3.heavyHitters should be(Set(3, 4, 5).toK[K])

val cms4 = TopNCMS.aggregator[K](depth, width, seed, 2).apply(data1)
cms4.heavyHitters should be(Set(4, 5).toK[K])

val cms5 = TopNCMS.aggregator[K](depth, width, seed, 2).apply(data1)
cms5.heavyHitters should be(Set(4, 5).toK[K])
val size = math.abs(seed % 5) + 1 // a number 1 to 5
val hh = ((6 - size) to 5).toSet
val cms = TopNCMS.aggregator[K](depth, width, seed, size).apply(data1)
cms.heavyHitters should be(hh.toK[K])
}
}

Expand Down
19 changes: 6 additions & 13 deletions algebird-test/src/test/scala/com/twitter/algebird/QTreeTest.scala
Original file line number Diff line number Diff line change
Expand Up @@ -18,25 +18,19 @@ package com.twitter.algebird

import org.scalatest._

import org.scalatest.{ PropSpec, Matchers }
import org.scalatest.prop.PropertyChecks
import org.scalacheck.Arbitrary
import org.scalatest.{ PropSpec, Matchers }
import org.scalatest.prop.PropertyChecks
import org.scalacheck.Arbitrary.arbitrary
import org.scalatest.{ PropSpec, Matchers }
import org.scalatest.prop.PropertyChecks
import org.scalacheck.Gen.choose
import org.scalacheck.Properties
import org.scalatest.{ PropSpec, Matchers }
import org.scalatest.prop.PropertyChecks
import org.scalacheck.Gen.choose
import org.scalatest.{ PropSpec, Matchers }

import java.util.Arrays

class QTreeLaws extends CheckProperties {
import BaseProperties._

implicit val qtSemigroup = new QTreeSemigroup[Long](6)
implicit val qtSemigroup = new QTreeSemigroup[Long](4)
implicit val qtGen = Arbitrary {
for (
v <- choose(0L, 10000L)
Expand All @@ -46,7 +40,6 @@ class QTreeLaws extends CheckProperties {
property("QTree is associative") {
isAssociative[QTree[Long]]
}

}

class QTreeTest extends WordSpec with Matchers {
Expand All @@ -56,7 +49,7 @@ class QTreeTest extends WordSpec with Matchers {

def buildQTree(k: Int, list: Seq[Double]) = {
val qtSemigroup = new QTreeSemigroup[Double](k)
list.map{ QTree(_) }.reduce{ qtSemigroup.plus(_, _) }
qtSemigroup.sumOption(list.map{ QTree(_) }).get
}

def trueQuantile[T: Ordering](list: Seq[T], q: Double): T = {
Expand All @@ -72,7 +65,7 @@ class QTreeTest extends WordSpec with Matchers {
s"QTree with elements (1 to $k)" should {
val trueMedian = (1 + k) / 2
s"have median $trueMedian" in {
implicit val sg = new QTreeSemigroup[Unit](k)
implicit val sg = new QTreeSemigroup[Unit](6)

val list = (1 to k).map(_.toDouble)
val qtree = sg.sumOption(list.map(QTree.value(_))).get
Expand Down Expand Up @@ -113,7 +106,7 @@ class QTreeTest extends WordSpec with Matchers {
assert(truth <= upper)
}
"have size bounded by 2^(k+2)" in {
val list = randomList(100000)
val list = randomList(10000)
val qt = buildQTree(k, list)
assert(qt.size <= (1 << (k + 2)))
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ import org.scalatest.prop.PropertyChecks
import org.scalacheck.{ Gen, Arbitrary }

object SketchMapTestImplicits {
val DELTA = 1E-8
val DELTA = 1E-6
val EPS = 0.001
val SEED = 1
val HEAVY_HITTERS_COUNT = 10
Expand All @@ -16,7 +16,7 @@ class SketchMapLaws extends CheckProperties {
import SketchMapTestImplicits._
import HyperLogLog.int2Bytes

val params = SketchMapParams[Int](SEED, EPS, DELTA, HEAVY_HITTERS_COUNT)
val params = SketchMapParams[Int](SEED, EPS, 1e-3, HEAVY_HITTERS_COUNT)
implicit val smMonoid = SketchMap.monoid[Int, Long](params)
implicit val smGen = Arbitrary {
for (key: Int <- Gen.choose(0, 10000)) yield (smMonoid.create((key, 1L)))
Expand Down
8 changes: 4 additions & 4 deletions build.sbt
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@ import com.typesafe.tools.mima.plugin.MimaPlugin.mimaDefaultSettings
import pl.project13.scala.sbt.JmhPlugin
import scalariform.formatter.preferences._

val paradiseVersion = "2.0.1"
val quasiquotesVersion = "2.0.1"
val paradiseVersion = "2.1.0"
val quasiquotesVersion = "2.1.0"
val bijectionVersion = "0.9.0"
val utilVersion = "6.20.0"

Expand All @@ -22,8 +22,8 @@ def isScala210x(scalaVersion: String) = scalaBinaryVersion(scalaVersion) == "2.1

val sharedSettings = Project.defaultSettings ++ scalariformSettings ++ Seq(
organization := "com.twitter",
scalaVersion := "2.11.7",
crossScalaVersions := Seq("2.10.5", "2.11.7"),
scalaVersion := "2.11.8",
crossScalaVersions := Seq("2.10.6", "2.11.8"),
ScalariformKeys.preferences := formattingPreferences,

resolvers ++= Seq(
Expand Down
17 changes: 8 additions & 9 deletions project/plugins.sbt
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,11 @@ resolvers ++= Seq(
url("http://dl.bintray.com/content/sbt/sbt-plugin-releases"))(Resolver.ivyStylePatterns)
)

addSbtPlugin("com.github.gseitz" % "sbt-release" % "1.0.0")
addSbtPlugin("com.jsuereth" % "sbt-pgp" % "1.0.0")
addSbtPlugin("com.typesafe" % "sbt-mima-plugin" % "0.1.9")
addSbtPlugin("com.typesafe.sbt" % "sbt-ghpages" % "0.5.3")
addSbtPlugin("com.typesafe.sbt" % "sbt-scalariform" % "1.3.0")
addSbtPlugin("org.scoverage" % "sbt-coveralls" % "1.0.0.BETA1")
addSbtPlugin("org.scoverage" % "sbt-scoverage" % "1.0.4")
addSbtPlugin("org.xerial.sbt" % "sbt-sonatype" % "1.0")
addSbtPlugin("pl.project13.scala" % "sbt-jmh" % "0.2.2")
addSbtPlugin("com.github.gseitz" % "sbt-release" % "1.0.0")
addSbtPlugin("com.jsuereth" % "sbt-pgp" % "1.0.0")
addSbtPlugin("com.typesafe" % "sbt-mima-plugin" % "0.1.9")
addSbtPlugin("com.typesafe.sbt" % "sbt-ghpages" % "0.5.3")
addSbtPlugin("com.typesafe.sbt" % "sbt-scalariform" % "1.3.0")
addSbtPlugin("org.scoverage" % "sbt-scoverage" % "1.3.5")
addSbtPlugin("org.xerial.sbt" % "sbt-sonatype" % "1.0")
addSbtPlugin("pl.project13.scala" % "sbt-jmh" % "0.2.2")