Bangladesh Advanced Computing Society (BACS) recently hosted their regional programming camps selection contest on Toph. There were 1792 registrants, of which 1043 participated in the contest. During the contest day, a whopping 2500 unique users were on the site.
These numbers may not sound very large, but they were large enough to help us identify a lot of bottlenecks and areas where Toph could be improved.
Based on what we learned during the contest and the feedback that we received from the participants, we have enhanced a few things in Toph.
Better Caching, Better Page Load Times
The first thing we did is reduce the number of database calls we need to make for certain things.
Take the submission list page for example. The page lists 64 submissions based on the filters you have selected on the right side.
Previously the back-end code would perform 64 queries on the database to determine the handle of the author of each of the submissions, even if all of the submissions belonged to the same user.
During the contest, we noticed that a lot of the database calls were merely trying to load data that would rarely change.
Since then we have added simple in-memory caching mechanisms for a lot of the database calls. This has significantly reduced the load times of a majority of the user-facing pages and APIs on Toph.
Appropriate Database Indexes
Although quite embarrassing, to be honest, this specific change affected only a very few of the areas in Toph. However, with thousands of users participating and following the BACS contest, the lack of a few very specific database indexes quickly became a nightmare for the entire infrastructure.
Both the standings page and the notifications drop down were affected by the lack of missing database indexes. Although the relevant database collections were indexed, some specific queries were not quite covered by the existing indexes.
One of the features (practice contests) that we are currently working on resulted in some discrepancies between the database calls the back-end code makes and the indexes that were present.
Now, with caching and proper database indexes, pages like the contest standings will load faster than ever!
Cursed Contest Context
One of the worst performance bottlenecks that happened during the contest was due to a missing $ in the code. Yes, just a dollar sign.
You may already know that Toph is built with Go – the programming language with one of the most feature-rich standard library.
Go has a standard templating library that we adore. The $ character in Go’s templating language helps chose the context from which variables are to be accessed.
A missing $ in front of a .Contest variable resulted in the database being called 256 × 6 every time someone tried to load that page. That page was requested tens of thousands of times during the contest.
Pages that were timing out for most users during the contest now load in a matter of seconds.
With a performance bottleneck in one of the most important and most requested pages, the performance of the entire site was affected. If you had a bad experience with any part of Toph during the BACS contest, it is very likely this specific issue was at the root of it.
Changes Worth Every Dime
Before these changes, this is what the database activity looked like when loading the contest standings page:
With all of the changes in place, the standings page, like most other pages load a lot faster, even when the caching layer is not warmed up:
Notice how, before the improvements, the database remained active for a duration that is 3 times longer.
With caching, everything becomes even faster:
Not only the pages are served fast the first time, they are served even faster on every subsequent requests.
A Nearby Datacenter
Beyond the contest day, Toph generally gets more than 90% of the visitors from Bangladesh and the neighboring countries. To make sure we can serve pages to these users faster, we moved a large part of Toph’s infrastructure to a nearby data center.
This will further improve the experience for a majority of the users that regularly participate in contests on Toph or solve problems from the archive.
We Are Always Listening
This contest, one of the largest in Toph’s history, and what we learned from it has helped us tremendously improve Toph over the last few days, and more improvements are already in the works.
We thank everyone who shared their feedback over the last few days. Our team owes you one!
If there is something you would like us to know, or if you have an idea on how you think Toph could be better, we are always listening.
You can send us a message through Toph’s contact form or start a discussion in Toph Community.
