Bigger Than Twitter

How to build a notification stream application.

I was tasked with building a News Streaming service for a big company with millions of existing users. It was a video games company and the volume of “tweets” was expected to be higher than Twitter has on the New Years Eve. The volume comes from the fact that the news items are generated automatically, while players simply play their games.

The key aspect of the design rested on the fact that we were not obliged to keep all the news items. We did not expect users to ever scroll past a few screens of them. There would never be anyone interested in what happened last year. Later on, looking at the read/write ratio in production we realized that most of the data written to our system is never read even once by anyone.

But /dev/null was not a solution. Whenever the data was read, we were judged by how relevant and recent the information presented is. It did not matter what we chose to discard along the way. As long as what we kept was relevant and engaging, we were fulfilling our purpose.

The design was simple, yet powerful. We had a circular buffers for each user and feed type on top of Cassandra. Each user cost limited and predetermined amount of storage space to us. Cassandra is still one of the best write-intensive DBs I ever worked with. And even though it was challenging at times and rebalancing its ring under load was not a trivial operation, it did deliver the performance we were looking for.

The application sustained the highest throughput among all the company’s “web services” (meaning anything that is not a game server). And the most surprising part was the very modest hardware footprint it ran on all that time. It was literally a handful of servers.

The tech stack was JVM, Scala, Cassandra, Spring, Apache HTTP Server and the regular set of tools for metrics, log collection and monitoring.

Nuclear Option

The ultimate power of atomics

Designing and implementing a distributed system may be a challenge. More so when the system is highly concurrent. And even more so when it is destined to deal with the finance. I once designed and built just such an application. Let it be known as Auction House in this article.

On the surface of it Auction House is a simple service. User create lots, other users bid on them. Whoever has the higher bid at the closing time wins. Simple. And you have millions of open lots and hundreds to thousands of bids per item. And a powerful search engine to have a fuzzy match by description, current bid price, closing date, etc. This is where it may get a bit intimidating.

I once built such a system. It was not just one system with a monolytic DB, of course. You can say it had two microservices - the items/bids one and the search engine.

The items/bids service is the most interesting. All the operations in the system were designed to be atomic and non-blocking. An item creation, a bid on it and closing of a sale were all done atomically, via appending to the list. At the read time the system would restore the logical order as an in-memory operation.

The search engine was based on SOLR. ElasticSearch did not exist yet, so we had to design and implement some of its features manually. It was a very robust search engine. At any moment in time we had at least two clusters of indexes ready. We were streaming to both all the items and bids in real time. The index was always just a tiny bit behind, but the system was robust enough to buffer virtually unbounded amount of “changes” in memory for the search code if it goes offline.

Of course we had a relational DB as well. We were only storing the final data to it - the closed auction with the full history of bids and the winner information. It was never a bottleneck, since it was a relatively rare operation that would touch it.

After an extensive round of load tests we have put the system in production. It was running on a handful of servers, serving the higher-than-anticipated load without breaking sweat.

The tech stack was JVM, Java, Redis, Couchbase, Mongo, SOLR and the regular set of tools for metrics, log collection and monitoring.

The Good Agent

Calling home through the enemy lines

IoT made the need for secure communication channels more visible. I was tasked to design an “Agent”, which would run somewhere far, on the hardware we do not control and reach back home via network we can not trust. It would then get some code from “home” and execute it. And, of course, we would never want anybody random from the Internet or from within the company to be able to hijack the communication channel and send wrong code for execution.

The connection part is simple. WebSockets have been around long enough that any decent network, no matter how many proxies it has, can expect them to work. Meaning that using STOMP over websocket is a fairly straightforward design decision. But how do you authenticate?

To authenticate we bundle some shared secrets with the agent. Each user would get a unique combination. And we would never re-transmit them over the network. Instead, we are going to use JWT tokens with HMAC signature. The Agent generates them when it is about to connect, and the server can verify its validity. This would allow us to trust with a reasonable level of certainty that the Agent is a real one. But we still need to allow Agent to trust it is connected to the right server. And contrary to what your bank was telling you, SSL is not enough.

Every piece of code we are sending to the Agent is signed (GPG) with a private key. That key never left our trusted network, but the Agent has its public key and can verify the signature. It would not execute any unsigned code.

This project provided a set of interesting challenges beyond authentication. There are so many states in which an application may end up and making sure we can recover from any state was an interesting problem to solve.