While the development of new features is important, we had to stop for a while and rethink our infrastructure. We have completely recovered all data, but that was close. Two years ago we had a major incident that almost cost us an entire server with 3TB of data, holding several months worth of article data. We had redundancy on the application servers, but for databases with this size it’s very hard to keep everything redundant and fully backed up. It just didn’t feel right to have a rack full of expensive servers that we couldn’t fully utilize because their storage was full.įurthermore redundancy was an issue too. For a long time we knew we have to do something about it, otherwise we would soon need to rent a second rack, which would have doubled our bill.

CPUs and memory on those servers remain heavily underutilized while the storage is full. Once a database server fills up (usually at around 3TB) we buy another one and this one becomes read-only. All this was required to keep up with the needed throughput. Usually 4 to 8 spinning disks in RAID-10 mode with expensive RAID controllers equipped with cache modules and BBUs. More than half of those servers were databases, each with its own storage.

Last year we have reached a point where we had a 48U (rack-units) rack full of servers. We were not using virtualization except for some really small micro-services and it was practically one physical server with local storage broken down into several VMs. Until recently we were using bare-metal servers to operate our infrastructure, meaning we deployed services like database and application servers directly on the operating system of the server. The very basic feature that every RSS reader should have – being able to filter only unread articles, requires so much resource power that it contributes to around 30% of the storage pressure on our first-tier databases. Even calculating the unread counts for millions of users is enough of a challenge that it deserves a special module for caching and maintaining. Lots of them! Behind the deceivingly simple user interface there is a complex backend with huge datastore that should be able to fetch millions of feeds in time, store billions of articles indefinitely and make any of them available in just milliseconds – either by searching or simply by scrolling through lists. Building and maintaining a cloud RSS reader requires resources.

Inoreader