Data Storage Intended For High Load Content Websites

One of the most frequent practice today for companies that have more than one website is to gather data from all the websites to collect and analyse just by using the only one source. The same approach was used by HIMSS — the company had more than 20 conference websites that had different data about users’ behaviour and needed to store the data in one space to transfer it next to their analytics service. The task was to develop the unified storage so the company could store data from all their websites just in one place.

We came up with a solution to develop the unified storage as a separate website based on Drupal which would use its API to collect data from other websites. In other words, we developed a micro service to collect and store data from all HIMSS websites.

The issue occurred later not with our solution but with the service the company used to analyse the data. Our task was basically to develop a unified data storage that would eventually transfer the data to that analytics service, however, the issue happened with the service itself. In 2018 the conference was attended by a record number of users which was about 45 000 just in 7 days. Once we started to transfer their data to the analytics service, it simply broke down as it could not keep up with that amount of load from the storage side. The service was used by the company as it had an advisory feature based on users’ behaviour which was a crucial point for HIMSS. Realising the fact that the server would not be able to work as expected, we managed to update the unified store so it would:

• store the data for much longer than before as it might have been needed in the future after the conference.

• serve as the previous analytics service so the company could continue operate at a conference as planned.

As a result, not only did we manage to develop a unified data storage so the company could keep their data from different sources just in one place, but also replace analytics services that did not work as expected. Moreover, it took us just one night to develop the latter feature and users did not even experience any issues with the main conference website while working on it that eventually allowed to sustain behaviour and revenue metrics on the website as planned.

If you have any tasks on mind that require working with data and would like to get a consultation for free — just let us know by filling up the form below. We will be happy to provide you with some insights on your project.