atom.xml

<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">

  <title><![CDATA[adcloud dev]]></title>
  <link href="http://dev.adcloud.com/atom.xml" rel="self"/>
  <link href="http://dev.adcloud.com/"/>
  <updated>2013-10-25T11:33:24+02:00</updated>
  <id>http://dev.adcloud.com/</id>
  <author>
    <name><![CDATA[Dev Team]]></name>
    
  </author>
  <generator uri="http://octopress.org/">Octopress</generator>

  
  <entry>
    <title type="html"><![CDATA[DevHouse Friday Chillout]]></title>
    <link href="http://dev.adcloud.com/blog/2013/10/08/devhouse-friday-chillout/"/>
    <updated>2013-10-08T14:40:00+02:00</updated>
    <id>http://dev.adcloud.com/blog/2013/10/08/devhouse-friday-chillout</id>
    <content type="html"><![CDATA[<p><a href="http://devhousefriday.org/networks/events/show_event.82979"><img class="right" src="http://dev.adcloud.com/images/devhouse.gif"></a></p>

<h2><a href="http://en.wikipedia.org/wiki/Run_of_network">&#8220;Out Of Network&#8221;</a></h2>

<p>Adcloud GmbH, acquired in 2011 by Deutsche Post DHL as a startup
company, will cease its operations by the end of 2013.</p>

<p>We&#8217;re proud to host the DevHouse Friday Chillout for the last time.</p>

<p>Feel welcome and enjoy the DevHouse Friday with talks, food and
Beer.js in our office and on the rooftop.</p>

<h2>* UPDATE * Talks</h2>

<ol>
<li>Christian of <a href="http://www.rockethome.de">ROCKETHOME</a> will show  their
sensoring hard- and software in action.</li>
<li><a href="https://twitter.com/bluse1995">@bluse1995</a> will give an intro to
<a href="http://www.docker.io/">docker.io</a></li>
<li>Open Data ^ Wahlen - Was wäre wenn app</li>
</ol>


<h3>When?</h3>

<p>Friday, 25th Ocober 2013<br/>
Doors open at 19:00h</p>

<h3>Where?</h3>

<p>Adcloud GmbH<br/>
6th floor<br/>
<a href="https://maps.google.de/maps?f=q&amp;source=s_q&amp;hl=de&amp;geocode=&amp;q=Adcloud+GmbH,+K%C3%B6ln&amp;aq=t&amp;sll=51.151786,10.415039&amp;sspn=18.676297,42.407227&amp;ie=UTF8&amp;hq=Adcloud+GmbH,&amp;hnear=K%C3%B6ln,+Nordrhein-Westfalen&amp;t=h&amp;ll=50.941826,6.936321&amp;spn=0.03656,0.082827&amp;z=14&amp;layer=c&amp;cbll=50.941682,6.936591&amp;panoid=V7fZsdjW52tul8e5aBKh6g&amp;cbp=12,288.94,,0,-8.03">Venloer Straße 25-27</a><br/>
50672 Köln</p>

<h3>Tickets / Registration</h3>

<div>
  <script type="text/javascript" src="https://de.amiando.com/resources/js/amiandoExport.js"></script><iframe src="https://de.amiando.com/dhf-adcloud.html?viewType=iframe&distributionChannel=CHANNEL_IFRAME&panelId=2117572&useDefaults=false&resizeIFrame=true" frameborder="0" width="650px" height="450px" name="_amiandoIFrame2117572CNE1kL7w" id="_amiandoIFrame2117572CNE1kL7w"><p>This page requires frame support. Please use a frame compatible browser to see the amiando ticket sales widget.</p><p> Try out the <a href="http://www.amiando.com/">online event registration system</a> from amiando.</p></iframe><p style="text-align: left; font-size:10px;"><a href="http://www.amiando.com" target="_blank" alt="conference - Online Event Management" title="conference - Online Event Management" >conference - Online Event Management</a> with the ticketing solution from amiando</p>
</div>


<h3>See also</h3>

<ul>
<li><a href="http://devhousefriday.org/networks/events/show_event.82979">Event @ devhousefriday.org</a></li>
</ul>

]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[clojure introduction workshop]]></title>
    <link href="http://dev.adcloud.com/blog/2013/04/19/clojure-introduction-workshop/"/>
    <updated>2013-04-19T12:04:00+02:00</updated>
    <id>http://dev.adcloud.com/blog/2013/04/19/clojure-introduction-workshop</id>
    <content type="html"><![CDATA[<p><img class="right" src="http://dev.adcloud.com/images/sold_out.png"></p>

<p><strong>When?</strong> Wednesday, 22.05.2013</p>

<p><strong>Where?</strong><br/>
<em>Adcloud GmbH</em><br/>
<em><a href="https://maps.google.de/maps?f=q&amp;source=s_q&amp;hl=de&amp;geocode=&amp;q=Adcloud+GmbH,+K%C3%B6ln&amp;aq=t&amp;sll=51.151786,10.415039&amp;sspn=18.676297,42.407227&amp;ie=UTF8&amp;hq=Adcloud+GmbH,&amp;hnear=K%C3%B6ln,+Nordrhein-Westfalen&amp;t=h&amp;ll=50.941826,6.936321&amp;spn=0.03656,0.082827&amp;z=14&amp;layer=c&amp;cbll=50.941682,6.936591&amp;panoid=V7fZsdjW52tul8e5aBKh6g&amp;cbp=12,288.94,,0,-8.03">Venloer Straße 25-27</a></em><br/>
<em>50672 Köln</em></p>

<p>E-Mail: waldemar.schwan@adcloud.com, tjg@pentaside.org</p>

<p><strong>Doors open</strong> 16:30<br/>
<strong>Workshop</strong> 17:00 - 19:00</p>

<p>Free drinks and BBQ afterwards.</p>

<h2>What you will learn</h2>

<p>This will be a very pragmatic — hands on — workshop.
We will start with a short introduction (~30 minutes) to syntax and basic usage patterns, giving you the chance to ask questions.</p>

<p>After that, we will encourage people to pair up and start working on our <a href="https://github.com/adcloud/clojure-intro-workshop">example project</a>.</p>

<p>In the last 30 minutes, you will be encouraged to present your solution.</p>

<p>We will finish with a BBQ, where you can talk with other attendees or continue on your project.</p>

<p>Although the event is free we ask you to register since space is limitted:</p>

<!-- more -->


<p><a class="guestlist-event-161211" href="#"><img width="250" height="60" style="border: 0" src="https://guestlistapp.com/images/embed/register.png" alt="Register for Event" /></a></p>

<h2>What you will NOT learn</h2>

<p>Given the limited amount of time and zero-prior-knowledge assumption, we can&#8217;t explain philosophies, concepts and inner workings.
After the workshop, you will be able to write code in Clojure, but still lack a deeper understanding.</p>

<p>If you&#8217;d like to dive deeper into Clojure&#8217;s world, here&#8217;s some resources:</p>

<ul>
<li><a href="http://tryclj.com/">Try Clojure</a> - try Clojure in the browser</li>
<li><a href="http://www.youtube.com/watch?v=P76Vbsk_3J0">Clojure for Java Programmers</a> - by Rich Hickey (the inventor of Clojure)</li>
<li><a href="http://www.youtube.com/user/ClojureTV">ClojureTV</a> - videos of talks</li>
<li><a href="http://clojure-doc.org/">Clojure Documentation</a> - up-to-date community driven documentation</li>
<li><a href="http://www.clojure-toolbox.com/">The Clojure Toolbox</a> - a list of projects by use-case</li>
<li><a href="http://clojuredocs.org/">ClojureDocs</a> - API references and examples</li>
<li><a href="http://clojure.org/">Clojure Homepage</a></li>
<li><a href="http://www.clojurebook.com/">Clojure Programming</a> - book</li>
<li><a href="http://www.manning.com/fogus2/">The Joy of Clojure (2nd edition)</a> - book</li>
<li><a href="http://pragprog.com/book/shcloj2/programming-clojure">Programming Clojure (2nd edition)</a> - book</li>
</ul>


<h2>Please come prepared</h2>

<p>Please make sure that you have the <a href="https://github.com/adcloud/clojure-intro-workshop">basic project</a> forked and checked out on your machine.
(<a href="https://github.com/adcloud/clojure-intro-workshop#readme">The README</a> has more details.)</p>

<h2>About Clojure</h2>

<blockquote>
<p>
Clojure is a <strong>dynamic programming language</strong> that <strong>targets the Java Virtual Machine</strong> (and the CLR, and JavaScript). It is designed to be a general-purpose language, combining the approachability and interactive development of a scripting language with an efficient and robust infrastructure for <strong>multithreaded programming</strong>. Clojure is a compiled language - it compiles directly to JVM bytecode, yet remains completely dynamic. Every feature supported by Clojure is supported at runtime. Clojure provides <strong>easy access to the Java frameworks</strong>, with optional type hints and type inference, to ensure that calls to Java can avoid reflection.
</p>

<p>
Clojure is a dialect of <strong>Lisp</strong>, and shares with Lisp the code-as-data philosophy and a <strong>powerful macro system</strong>. Clojure is predominantly a <strong>functional</strong> programming language, and features a rich set of <strong>immutable, persistent data structures</strong>. When mutable state is needed, Clojure offers a <strong>software transactional memory system</strong> and <strong>reactive Agent system</strong> that ensure clean, correct, multithreaded designs.
</p>
</blockquote>


<p>— <a href="http://clojure.org/">clojure.org</a></p>

<ul>
<li>Example projects

<ul>
<li><a href="http://leiningen.org/">Leiningen (Build system)</a></li>
<li><a href="http://riemann.io/">Riemann.io (Monitoring)</a></li>
<li><a href="http://overtone.github.io/">Overtone (Collaborative Programmable Music)</a></li>
<li><a href="http://storm-project.net/">Storm (Distributed and fault-tolerant realtime computation)</a></li>
<li><a href="http://clojurerabbitmq.info/">Langohr (Messaging)</a></li>
<li><a href="https://github.com/nathanmarz/cascalog#readme">Cascalog (Query Big Data)</a></li>
<li><a href="https://github.com/ring-clojure/ring#readme">Ring</a> / <a href="https://github.com/weavejester/compojure#readme">Compojure (Webserver)</a></li>
<li><a href="https://github.com/quil/quil#readme">Quil (Generative art)</a></li>
</ul>
</li>
</ul>

]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[TechTalk #5 - Asgard and the AWS cloud]]></title>
    <link href="http://dev.adcloud.com/blog/2013/02/05/asgard-and-the-aws-cloud/"/>
    <updated>2013-02-05T15:22:00+01:00</updated>
    <id>http://dev.adcloud.com/blog/2013/02/05/asgard-and-the-aws-cloud</id>
    <content type="html"><![CDATA[<p>TechTalk #5 - Asgard and the AWS cloud</p>

<p>We are happy to announce our first Adcloud TechTalk in 2013:</p>

<p>Asgard and the AWS cloud</p>

<p>Manually managing infrastructure with a lot of components and servers is very error-prone.
Especially in dynamic environments like in the &#8220;cloud&#8221; it often leads to maintenance problems that require lots of time.
A solution could be the introduction of change management applications like puppet, chef, etc. but this of course needs sophisticated knowledge and experience and also adds another level of complexity to the exisiting infrastructure.
<a href="https://signup.netflix.com">NetFlix</a>, one of the biggest players in the big-data business, hosts its complete infrastructure in the AWS cloud and needs to maintain it automatically.
Therefore they developed a tool called <a href="https://github.com/Netflix/asgard">&#8216;Asgard&#8217;</a> which provides the lacking interface to already existing functionality of the AWS cloud thus enabling user-friendly management of cloud infrastructure</p>

<p>In this talk, Johannes Hörle and Roland Gude from <a href="http://www.yoochoose.com">Yoochoose</a> will give a short overview of Asgard&#8217;s functionality and how they use it to administer their cloud environment.</p>

<p>&#8220;Don&#8217;t ask what you can do for the cloud, ask what the cloud can do for you!&#8221;</p>

<p>Johannes holds a Master&#8217;s Degree in Computer Science from the University of Applied Sciences Bonn-Rhein-Sieg. He now works as Lead Engineer at Yoochoose and coordinates the development of the SaaS Recommender System in the AWS Cloud. In his free time he likes playing football and riding his road bike.</p>

<p>Roland holds a Master’s Degree in Computer Science from the University of Applied Science Bonn-Rhein Sieg. After working as a research fellow at the German Aerospace Center Institute for Simulation and Software technology he joined Yoochoose as an Engineer, where he is mainly concerned with scalability of the Yoochoose software stack. In the meantime he takes care for his 2 year old daughter. Apart from that he enjoys playing football, reading and going to the theater.</p>

<p>The event will be held on Wednesday, February 27th in our office: <a href="http://maps.google.de/maps?f=q&amp;source=s_q&amp;hl=de&amp;geocode=&amp;q=Adcloud+GmbH,+Köln&amp;aq=t&amp;sll=51.151786,10.415039&amp;sspn=18.676297,42.407227&amp;ie=UTF8&amp;hq=Adcloud+GmbH,&amp;hnear=Köln,+Nordrhein-Westfalen&amp;t=h&amp;ll=50.941826,6.936321&amp;spn=0.03656,0.082827&amp;z=14&amp;layer=c&amp;cbll=50.941682,6.936591&amp;panoid=V7fZsdjW52tul8e5aBKh6g&amp;cbp=12,288.94,,0,-8.03">Venloer Str. 25-27, Cologne</a>. Doors open at 17:30. Talk starts 18:00. Event ends around 19:30.</p>

<p>Although the event is free we ask you to register since space is limitted:</p>

<p><a class="guestlist-event-147090" href="#"><img width="250" height="60" style="border: 0" src="https://guestlistapp.com/images/embed/register.png" alt="Register for Event" /></a></p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[TechTalk #4 - The Sound of Adcloud]]></title>
    <link href="http://dev.adcloud.com/blog/2012/11/29/techtalk-number-4-the-sound-of-adcloud/"/>
    <updated>2012-11-29T16:00:00+01:00</updated>
    <id>http://dev.adcloud.com/blog/2012/11/29/techtalk-number-4-the-sound-of-adcloud</id>
    <content type="html"><![CDATA[<p><em>Winter is coming</em> and so is our last Tech Talk of the year. This time it is all about music and processing real time events.</p>

<h2>The Sound of Adcloud</h2>

<p>We have created an automated Adcloud music generator. The events (eg
clicks, conversions, retargeting) of our adservers are used to create
sound in real-time. Timo is going to introduce the different
components and literally going to rock the house. <a href="http://soundcloud.com/teemow/adcloud-music">Here is an
apetitzer.</a></p>

<p>The event will be on Wednesday, 12th of December in our office: <a href="http://maps.google.de/maps?f=q&amp;source=s_q&amp;hl=de&amp;geocode=&amp;q=Adcloud+GmbH,+K%C3%B6ln&amp;aq=t&amp;sll=51.151786,10.415039&amp;sspn=18.676297,42.407227&amp;ie=UTF8&amp;hq=Adcloud+GmbH,&amp;hnear=K%C3%B6ln,+Nordrhein-Westfalen&amp;t=h&amp;ll=50.941826,6.936321&amp;spn=0.03656,0.082827&amp;z=14&amp;layer=c&amp;cbll=50.941682,6.936591&amp;panoid=V7fZsdjW52tul8e5aBKh6g&amp;cbp=12,288.94,,0,-8.03">Venloer
Str. 25-27, Cologne</a>.
Doors open at 17:00. Talk starts 17:30. Event ends around 19:00.</p>

<p>Although the event is free we ask you to register since space is limitted:</p>

<p><a class="guestlist-event-138504" href="#"><img width="250"
height="60" style="border: 0"
src="https://guestlistapp.com/images/embed/register.png" alt="Register
for Event" /></a></p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[TechTalk #3 - Redis Internals]]></title>
    <link href="http://dev.adcloud.com/blog/2012/10/08/redis-techtalk/"/>
    <updated>2012-10-08T00:00:00+02:00</updated>
    <id>http://dev.adcloud.com/blog/2012/10/08/redis-techtalk</id>
    <content type="html"><![CDATA[<p>The summer is over. Our little break too. Lets continoue with our
Adcloud Techtalks. This time with some with a slighlty deeper topic:</p>

<p>As you might have seen in <a href="http://dev.adcloud.com/blog/2012/07/13/nosql-not-only-a-fairy-tale/">NoSQL: Not Only a Fairy
Tale</a>
<a href="http://redis.io/">Redis</a> is one our most important NoSQL databases.
We use it extensively throughout our different systems. Time to look
behind the curtain:</p>

<h2>Redis Internals</h2>

<p>This time <a href="https://twitter.com/a2800276">Tim Becker</a> will show us how
to pay more attention to the man behind the curtain! Redis is not
magic, in this talk we&#8217;ll rummage through source code, start reading
C, examine the redis protocol, event loop and some datastructures. It
should provide enough information that you won&#8217;t fear diving in the
code yourself and have a good overview of what&#8217;s going on behind the
curtain.</p>

<p>The event will be on Wednesday, 24th October in our office: <a href="http://maps.google.de/maps?f=q&amp;source=s_q&amp;hl=de&amp;geocode=&amp;q=Adcloud+GmbH,+K%C3%B6ln&amp;aq=t&amp;sll=51.151786,10.415039&amp;sspn=18.676297,42.407227&amp;ie=UTF8&amp;hq=Adcloud+GmbH,&amp;hnear=K%C3%B6ln,+Nordrhein-Westfalen&amp;t=h&amp;ll=50.941826,6.936321&amp;spn=0.03656,0.082827&amp;z=14&amp;layer=c&amp;cbll=50.941682,6.936591&amp;panoid=V7fZsdjW52tul8e5aBKh6g&amp;cbp=12,288.94,,0,-8.03">Venloer
Str. 25-27, Cologne</a>.
Doors open at 17:00. Talk starts 17:30. Event ends around 20:00.</p>

<p>Although the event is free we ask you to register since space is limitted:</p>

<p><a class="guestlist-event-128865" href="#"><img width="250"
height="60" style="border: 0"
src="https://guestlistapp.com/images/embed/register.png" alt="Register
for Event" /></a></p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[NoSQL: Not only a fairy tale]]></title>
    <link href="http://dev.adcloud.com/blog/2012/07/13/nosql-not-only-a-fairy-tale/"/>
    <updated>2012-07-13T00:00:00+02:00</updated>
    <id>http://dev.adcloud.com/blog/2012/07/13/nosql-not-only-a-fairy-tale</id>
    <content type="html"><![CDATA[<p><em>This is a cross-post from Sebastian Cohnens <a href="http://tisba.de/2012/06/28/nosql-not-only-a-fairy-tale/">personal blog</a>.</em></p>

<p>At this year&#8217;s <a href="http://www.nosql-matters.org/">NoSQL Matters</a> conference in Cologne, which was also the first one of its kind, I held a talk together with <a href="https://twitter.com/teemow">Timo Derstappen</a> about the evolution of <a href="http://adcloud.com">adcloud</a>&#8217;s adserver. We took the audience on a journey spanning multiple years of changing requirements and the resulting architectural decisions made during that time. A focus was on NoSQL databases involved in the adserver, of course :)</p>

<!--more-->


<p>Before I start, I&#8217;d like to quickly thank the organizers of the NoSQL Matters conference. It was a very interesting meetup with awesome speakers, great catering and nice location.</p>

<p>The reminder of this post is a transcript of the talk. Be warned, that this post is lengthy and quite detailed :) You&#8217;ll also find the slides at the end or directly at <a href="https://speakerdeck.com/u/tisba/p/nosql-not-only-a-fairy-tale">speakerdeck</a>. The talk was also recorded but the video is not available yet.</p>

<p><strong>Disclaimer</strong>: I&#8217;m not an employee of adcloud, I just work for them. If I say <em>we</em> in this article, I am referring to Timo and me or the adcloud dev team (depending on the context).</p>

<h2>Preface</h2>

<p>Before we can dive right into our story we are about to tell you, we first have to explain some terms related to the stuff we are doing at adcloud. At adcloud we are doing <a href="http://en.wikipedia.org/wiki/Performance-based_advertising">performance advertising</a>. To simplify things for this talk: We are serving ads.</p>

<p>We have two important data entities:</p>

<p><img class="right" src="http://dev.adcloud.com/images/posts/nosql-fairy-tale/website-placement.png"></p>

<ul>
<li><strong>placements</strong> are container on websites that can display ads to the user. A placement knows its size, style and the amount of ads that it holds.</li>
<li>one, two or more <strong>ads</strong> can be displayed within a placement. An ad consists of some text, an image and a link.</li>
<li>there is an n:m relationship between placements and ads, describing which ads can run on what placements. An attribute of this relationship is what we call <strong>ad priority</strong> or just priority. The priority specifies how likely it is for an ad to be displayed on a given placement.</li>
</ul>


<p><strong>Placements</strong> and <strong>ads</strong> are rather static data, whereas the <strong>priority</strong> changes all the time in order to optimize the performance.</p>

<p>To follow this talk you need to know that there are two distinct systems at adcloud (there are more actually, but they can be ignored for now): <strong>Platform</strong> and <strong>adserver</strong>.</p>

<ul>
<li>The <strong>Platform</strong> is a typical web application working primarily as the administrative back office. It is the place where all the business related data is managed, reports are generated, etc. Beside the traditional web stack, there are also tons of background workers to process and analyse data. There is only very little NoSQL here.</li>
<li>The <strong>adserver</strong> on the other hand is a dedicated and decoupled system, which serves ads and collects tracking data from the users (like clicks on ads). NoSQL databases are heavily used within the adserver.</li>
</ul>


<p><img class="center" src="http://dev.adcloud.com/images/posts/nosql-fairy-tale/system-context.png"></p>

<p>These systems are highly decoupled. The Platform <em>publishes</em> placement, ad and priority data to the adservers. The adserver reports statistical and tracking data back to the Platform for analysis.</p>

<p>Since only the adserver heavily uses NoSQL systems, we will focus on it in this talk.</p>

<h2>AWS S3: Simple Storage Service</h2>

<p>The first version of the &#8220;adserver&#8221; wasn&#8217;t actually a server at all. We simply used S3. We combined all placement and ad data, including priorities, together in one document, added some JavaScript and pushed everything to S3. This was the initial version of the publishing process.</p>

<p>When a user visited a website, we delivered the JavaScript with the embedded data via a <a href="http://www.akamai.com/cotendo">CDN</a> (fed by S3) and executed it on the client side. The JavaScript contained some logic to choose from the available ads, based on the ads&#8217; priorities; and in the end, it rendered the ads.</p>

<p>The problem with this simple approach was that, due to the heavy denormalization, the publishing process was rather expensive. As we already mentioned, placements and ads don&#8217;t change, but priorities change all the time. With our denormalized approach, we couldn&#8217;t update our data incrementally.</p>

<p>Eventually, the time required to fully publish everything was taking too long, and we need to publish the calculated priorities as quickly as possible to maximize the yield.</p>

<h2>The relaxed Knight</h2>

<p><img class="right" src="http://dev.adcloud.com/images/posts/nosql-fairy-tale/couchdb-logo.png"></p>

<p>In 2009, we took a closer look at CouchDB. We were pretty excited about the idea of using HTTP/REST with JavaScript map/reduce. The multi-master replication setup looked exactly like what we were looking for, in terms of having a scalable system.</p>

<p>The first design idea was to just replace S3 with CouchDB. We thought about normalizing the data (split by update frequency) to enable incremental updates, to speed up the publishing process. But in the end we realized that modeling n:m relationships with CouchDB&#8217;s MapReduce is a hard problem to solve. Another issue was that CouchDB&#8217;s views are incremental and persisted &mdash; which is quite useless if you are constantly changing the data underneath.</p>

<p>In the end, we added a application server written in node.js to the stack. We used node.js to assemble the normalized data served by CouchDB. We also added nginx with response caching in front of node.js. Additionally, we cached some data directly within the node.js process to speed up requests which were not already cached by nginx.</p>

<p>The request flow was like this:</p>

<ul>
<li>incoming request</li>
<li>given a nginx cache miss&#8230;</li>
<li>fetch placement data and priorities for this placement</li>
<li>process the placement data and preselect ads based on the priority</li>
<li>fetch the rest of the ad data from CouchDB</li>
<li>build and send the response back to the client</li>
</ul>


<p>With this in place, we were able to effectively cut the publishing time in half. We also laid out the foundation for upcoming features which required more business logic to serve ads.</p>

<h3>How to measure consistency with CouchDB in a multi-master setup?</h3>

<p>We don&#8217;t have strong requirements regarding consistency for most of our data. But we still need to make sure that an adserver with its CouchDB replication is not too far behind the others.</p>

<p>To solve this, we wrote a tracer document (containing a timestamp and the hostname) to the local CouchDB in each adserver instance. On the other hand, each adserver asked the ELB (Elastic Load Balancer) from time to time what other adservers were out there. Based on those hostnames, every adserver read the tracer documents of all the other servers and calculated the replication delay.</p>

<p>In case an adserver was too far behind, it removed itself from the ELB to allow the replication to catch up again. If the database became back in sync, the adserver automatically re-added itself to the ELB and started serving requests again.</p>

<h2>New Feature Requests</h2>

<p>We are now in early 2011. This was also about the time when I started to work for adcloud on a regular base as a freelancer.</p>

<p>The problem we were facing is that we know that eventually all requests to the adservers for serving ads are going to be unique and therefor full-response caching is no longer an option. The main reason for the uniqueness of the requests are targeting features, like IP-based location-awareness e.g.</p>

<p>A quick test immediately showed that CouchDB was not able to provide the necessary throughput and latency we need to disable the caching in nginx. We also realized that caching things in node.js is pretty a bad idea &mdash; stop world garbage collection is a no-go for good and stable service quality in terms of latency.</p>

<p><img class="right" src="http://dev.adcloud.com/images/posts/nosql-fairy-tale/redis-logo.png"></p>

<p>The solution was rather simple from an architectural perspective. We simply replaced our caching backend implemented from objects in node.js to Redis. Some cache objects and structures for lookups were changed to fully utilize native data types Redis provides. The migration was rather simple.</p>

<p>When we start an adserver, we have a cache warmup phase where we pre-fill all placement and ad data into Redis. Once we&#8217;ve finished that we start to accept requests and serve everything out of Redis. But since data is constantly being updated (the Platform publishes data), we needed to keep the data in Redis up to date. We did this with background workers that would handle this while the web worker can continue to serve (stale) data without creating a big impact on the performance.</p>

<p>The results of this step was pretty amazing. We were able to disable the full-response caching entirely while adding new features and maintaining a good and solid service quality and performance. We really love Redis for this!</p>

<h2>Scalability</h2>

<p>We are now in mid- to late-2011, and scalability was the next big topic on our roadmap. This step focussed on our usage of CouchDB, since our scalability was limited by it.</p>

<p>If you take a look at how we used CouchDB and what we did not require at all, you quickly see that there is a mismatch.</p>

<p>We updated <strong>>10k documents/h</strong> (with only about 20k documents total in our database). We had just a single source of updates, the Platform. And we had multi-master replication in a 4-node cluster in place.</p>

<p>On the other hand, we did not require CouchDB&#8217;s bullet-proof append-only durability, since all the data was essentially transient to the adserver since we constantly got new data. Another thing we did not need was the multi-version concurrency control (MVCC) that CouchDB offers: We only had one source of updates; there was no way that conflicts could happen.</p>

<p>The resulting issue was that we had huge problems with load caused by replication processes. CouchDB&#8217;s multi-master replication was the one thing hindering us from scaling out. When we added more instances, we got more replication going on, and in the end even more load on each machine &mdash; this is not something you have in mind when scaling out a system.</p>

<p>Compaction was painful too: We accumulated a two-digit gigabyte number of data per server per day due to append-only. For our purpose, we didn&#8217;t need our database to be rock-solid in terms of durability. And it was getting harder and harder to find a time window in the night when we could run the compaction.</p>

<p>We all realized that this was not CouchDB&#8217;s fault. We were simply trying to apply the wrong use case to CouchDB. In our case we could not utilize CouchDB&#8217;s strengths and instead triggered its weak points. Maybe we could have known better in the beginning, but nonetheless opting for CouchDB over S3 back then wasn&#8217;t a bad idea at all. It helped a lot in terms of evolving the system and the company.</p>

<p>The solution to this situation has a touch of irony. With Redis in place, we replaced CouchDB for placement- and ad-data with S3. Since we weren&#8217;t using any CouchDB-specific features, we simply published all the documents to S3 buckets instead. We still did the Redis cache warming upfront and data updates in the background. So by decoupling the application from the persistence layer using Redis, we also removed the need for a super fast database backend. We didn&#8217;t care that S3 is slower than a local CouchDB, since we updated everything asynchronously.</p>

<p>In the end, S3 simply fits our needs. We could remove lots of code like the replication delay check. One additional benefit is that we have fewer moving parts now and less state on our application servers, which is also always a good thing too.</p>

<h2>Once again, more features</h2>

<p>Of course there are always new feature requests on the horizon. We are now in early 2012. Before we can dive into our next challenge, we have to take a quick look at the status quo.</p>

<p>The first S3-based adserver did the entire ad selection on the client side (the user&#8217;s browser). To a certain degree this was still the case with the CouchDB-node.js solution. One difference is that we did a pre-selection of ads there.</p>

<p>The challenge now is the goal to prepare the system for <a href="http://en.wikipedia.org/wiki/Real-time_bidding">Real-time bidding</a> (RTB). RTB is quite a hot topic in the ad industry. Without going too much into the details what RTB is, let me try to describe it briefly. RTB establishes a auction, where you can bid on a single impression for a given user. For example: When we see a user, we are asking a number of third party services what they are willing to pay for this user/impression. We then need to compare the bids with the revenue we would generate from our own ads and decide whether to serve our ads or the 3rd party ads.<br />
<em>(Please excuse me if this is not 100% accurate &mdash; I hope you get the idea. I&#8217;m a tech person and not that super deep into the business mechanics behind the ad industry.)</em></p>

<p>To enable RTB in our adservers, we need to bring the entire decision logic onto the server-side in order to tell which ads we want to deliver. And we have to do it rather fast. Since we need to query external systems and we have a network roundtrip to our final response to the request, we only have little time processing the bids. We are speaking of ~25ms here to choose from hundreds of possible ads.</p>

<p>The solution to this still excites me quite a bit: Redis. We know and trust Redis&#8217; performance. We know it has <a href="http://redis.io/commands#sorted_set">sorted sets</a>. We also have kind of sets of ads to display, and they are sorted by a priority. Why not utilize Redis for this selection logic?</p>

<p>We now heavily use Redis&#8217; sorted sets. The score is our ad priority which helps us selecting ads in the end. We build lot&#8217;s of sets of ads that we can choose from for a given placement. But we also create sets of ads that mutually exclude each other &mdash; you don&#8217;t want to see a McDonalds ad next to Burger King. For the selection we use mainly <code>ZUNIONSTORE</code> and <code>ZRANGEBYSCORE</code>. We first create a temporary set of ads that we can choose from, where we removed ads that cannot be shown at all. Then we apply our selection logic to determine which ads to pick. Ads with a higher priority (higher member score) are more likely to be shown.</p>

<p>What can I say? Our use case was very easy to model with Redis and it became a deeply integrated part of our business logic. Besides laying the foundation for Real-time bidding, we also improved the performance and quality for our users, by removing code in the client-side JavaScript and by reducing the payload that we need to send to the client. Overall we reduced the size of data that needs to be transfered by over 75%!</p>

<h2>Conclusion</h2>

<p>We now have reached the end of our journey. We have shown you five generations of adcloud&#8217;s adserver and you might have noticed that we did some radical changes in the architecture. We started out with just S3, moved to CouchDB, then we added Redis, replaced CouchDB with S3 and lately we are using Redis even more&#8230;</p>

<p>There are three importent drivers for our design decisions: features, quality &amp; performance and scalability.</p>

<p>If we can deliver a new feature, that give us an advantage over competitors, than this is great. This step ahead of others financed the company very early on and enabled us to continuously grow and improve our systems. Service quality and performance is of course also a very important matter &mdash; only if the quality is good enough you can keep the competitive advantage. And last but not least, scalability. We need to enable our system and architecture to scale as the business grows.</p>

<p>When it comes to NoSQL we had to learn things the hard way sometimes. And one thing is for sure: There is no &#8220;one fits all&#8221; solution to the problems we are facing. We pick a tool that fits our needs and try to get the best out of it. If we are stuck, cannot scale any further, or we are hindered from implementing new features, we then consider re-evaluating the status quo. Don&#8217;t be afraid of questioning existing systems.</p>

<p>Adcloud started with a very small team of four developers. Beside the adserver was the Platform which was to be build. Having a very simple solution utilizing S3 was a good decision back then. Going with very small, incremental steps was the only way to go back then. And even now the developer team in Cologne has grown significantly we still try to go by the same principles.</p>

<h2>Slides</h2>

<script async class="speakerdeck-embed" data-id="4fc63943c41f0f002101150c" data-ratio="1.3333333333333333" src="http://dev.adcloud.com//speakerdeck.com/assets/embed.js"></script>

]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[TechTalk #2 - Why the SOLID principles matter - even for JavaScript]]></title>
    <link href="http://dev.adcloud.com/blog/2012/05/04/solid/"/>
    <updated>2012-05-04T00:00:00+02:00</updated>
    <id>http://dev.adcloud.com/blog/2012/05/04/solid</id>
    <content type="html"><![CDATA[<p>After our successful start with our <a href="http://img.ly/hoLs">first Adcloud TechTalk</a> we are proud to announce our second talk: Why the SOLID principles matter - even for JavaScript.</p>

<p>Even though the famous <a href="http://butunclebob.com/ArticleS.UncleBob.PrinciplesOfOod">SOLID</a> design principles where invented in the context of more classical object-oriented languages like C++ and Java, the fundamental principles are enormously useful way beyond the concrete programming environment and by far not restricted to a concrete programming language. And having a well-structured code base is one of the most important factors to keep software flexible and changeable over time.</p>

<p>Martin Lippert (co-founder of it-agile GmbH and now working at VMware on developer tooling for JavaScript, Spring, and Cloud Foundry, tweets <a href="https://twitter.com/#!/martinlippert">@martinlippert</a>) will give an overview of the SOLID principles, explain why they are so important, talk about what happens if you ignore them, and dive into some examples for JavaScript.</p>

<p>The event will be on Tuesday, 22nd May in our office: <a href="http://maps.google.de/maps?f=q&amp;source=s_q&amp;hl=de&amp;geocode=&amp;q=Adcloud+GmbH,+K%C3%B6ln&amp;aq=t&amp;sll=51.151786,10.415039&amp;sspn=18.676297,42.407227&amp;ie=UTF8&amp;hq=Adcloud+GmbH,&amp;hnear=K%C3%B6ln,+Nordrhein-Westfalen&amp;t=h&amp;ll=50.941826,6.936321&amp;spn=0.03656,0.082827&amp;z=14&amp;layer=c&amp;cbll=50.941682,6.936591&amp;panoid=V7fZsdjW52tul8e5aBKh6g&amp;cbp=12,288.94,,0,-8.03">Venloer Str. 25-27, Cologne</a>. Doors open at 17:00. Talk starts 17:30. Event ends around 19:00.</p>

<p>Although the event is free we ask you to register since space is limitted:</p>

<p><a class="guestlist-event-103512" href="#"><img width="250" height="60" style="border: 0" src="http://guestlistapp.com/images/embed/register.png" alt="Register for Event" /></a></p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[TechTalk #1 - Chaplin – an Application Architecture based on Backbone.js]]></title>
    <link href="http://dev.adcloud.com/blog/2012/04/16/chaplin/"/>
    <updated>2012-04-16T00:00:00+02:00</updated>
    <id>http://dev.adcloud.com/blog/2012/04/16/chaplin</id>
    <content type="html"><![CDATA[<p><strong>Update</strong>: We are out of space. Stay tuned for the next talk.</p>

<p>We are happy to announce our first public Adcloud TechTalk:</p>

<h2>Chaplin – JavaScript Application Architecture with Backbone.js</h2>

<p>Backbone.js is a small JavaScript library which helps building JavaScript-driven web applications. It’s widely used by companies like LinkedIn, Groupon, Foursquare and 37Signals. While Backbone.js is a good starting point, it is quite limited and does not provide a top-level architecture for scalable Web applications.</p>

<p><a href="http://molily.de/">Mathias Schäfer</a> of <a href="http://9elements.com">9elements</a> will introduce <a href="https://github.com/moviepilot/chaplin">Chaplin</a>, a field-tested application architecture on top of Backbone.js. Chaplin establishes conventions on how to structure an application and addresses the requirements of growing JavaScript applications. Like Backbone.js itself, Chaplin is free and open-source.</p>

<p>The event will be on Wednesday, 25th April in our office: <a href="http://maps.google.de/maps?f=q&amp;source=s_q&amp;hl=de&amp;geocode=&amp;q=Adcloud+GmbH,+K%C3%B6ln&amp;aq=t&amp;sll=51.151786,10.415039&amp;sspn=18.676297,42.407227&amp;ie=UTF8&amp;hq=Adcloud+GmbH,&amp;hnear=K%C3%B6ln,+Nordrhein-Westfalen&amp;t=h&amp;ll=50.941826,6.936321&amp;spn=0.03656,0.082827&amp;z=14&amp;layer=c&amp;cbll=50.941682,6.936591&amp;panoid=V7fZsdjW52tul8e5aBKh6g&amp;cbp=12,288.94,,0,-8.03">Venloer Str. 25-27, Cologne</a>. Doors open at 17:00. Talk starts 17:30. Event ends around 19:00.</p>

<p>Although the event is free we ask you to register since space is limitted:</p>

<p><a class="guestlist-event-98193" href="#"><img width="250" height="60" style="border: 0" src="http://guestlistapp.com/images/embed/register.png" alt="Register for Event" /></a></p>

<p>Looking forward to this and hopefully more events like this.</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Announcing "Adcloud TechTalks"]]></title>
    <link href="http://dev.adcloud.com/blog/2012/03/29/adcloud-techtalks/"/>
    <updated>2012-03-29T00:00:00+02:00</updated>
    <id>http://dev.adcloud.com/blog/2012/03/29/adcloud-techtalks</id>
    <content type="html"><![CDATA[<p>We are happy announce a new (hopefully) regular event: Adcloud TechTalks. These are informal meetings around technical challenges which we face, topics which we would like to know more about or things which we just find interesting. Speakers will range from internal developers and local friends to invited industry experts.</p>

<p>Our goal is to learn new things and foster the software development culture in Cologne.</p>

<p>These events will take place weekdays in the late afternoon. Doors open at 17:00. The talk starts at 17:30 and we calculate around 45 minutes for the talk and approx 45 minutes of discussion. Concrete dates will be announced a couple of weeks prior to the talk in this <a href="http://dev.adcloud.com/atom.xml">blog</a> and via <a href="http://twitter.com/adcloud">@adcloud</a>.</p>

<p>All events will be open to the public, although we will require some form of registration. Stay tuned for the first event.</p>
]]></content>
  </entry>
  
</feed>