Our team has been designing and building high performance data products for more than 10 years. We have dealt with millions of chat messages per second, served maps to hundreds of millions of people through the front page of the Wall Street Journal during an election night, helped Google deliver an application directly linked from their home page, and designed and run an analytics system to deal with 200 QPS over 7B rows.

During all these years we learnt a lot about what works and what doesn’t, and what’s the best approach to tackle different use cases, and it forced us to dig deeper into understanding how software and hardware interacts and the principles behind working with large amounts of data. Until now we had never devoted any time to collating all this knowledge that over the years we had spread in notes, presentations, references to books, etc. We started putting it together as an internal on-boarding guide to new employees but we thought it made sense to open it.

That is why we decided to create “Principles of real-time analytics on large datasets”, a course on how to design and build analytics systems at scale, an in-depth look at the core concepts behind our design methodology and principles.

This is a technology agnostic course, we will use different technologies to illustrate those concepts with a lot of easy to understand examples.

After a couple of live editions, we are opening this first revision everyone as a set of 9 short video lessons.