# Analyzing submissions in real time for social media updates

DevFeed: [Analyzing submissions in real time for social media updates](<https://devfeed.tech/articles/analyzing-submissions-in-real-time-for-social-media-updates-20003.md>)

Original publisher: [Read original article](<http://engineering.hackerearth.com/2016/02/02/analyzing-submissions-realtime-for-social-media-updates/>)

Published: 2016-02-02T00:00:00Z

Content type: article

Language: en

Sources: [HackerEarth](<https://devfeed.tech/sources/hackerearth.md>)

Topics: [real-time](<https://devfeed.tech/topics/real-time.md>), [Redis](<https://devfeed.tech/topics/redis.md>), [Kafka](<https://devfeed.tech/topics/kafka.md>), [Database](<https://devfeed.tech/topics/database.md>), [App](<https://devfeed.tech/topics/app.md>), [Persistence](<https://devfeed.tech/topics/persistence.md>), [payload](<https://devfeed.tech/topics/payload.md>), [data](<https://devfeed.tech/topics/data.md>), [coding](<https://devfeed.tech/topics/coding.md>)

Tags: [app](<https://devfeed.tech/tags/app.md>), [coding](<https://devfeed.tech/tags/coding.md>), [contests](<https://devfeed.tech/tags/contests.md>), [data](<https://devfeed.tech/tags/data.md>), [database](<https://devfeed.tech/tags/database.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [leaderboard](<https://devfeed.tech/tags/leaderboard.md>), [marketing](<https://devfeed.tech/tags/marketing.md>), [payload](<https://devfeed.tech/tags/payload.md>), [persistence](<https://devfeed.tech/tags/persistence.md>), [posts](<https://devfeed.tech/tags/posts.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [redis](<https://devfeed.tech/tags/redis.md>), [server](<https://devfeed.tech/tags/server.md>), [social-media](<https://devfeed.tech/tags/social-media.md>)

## AI overview

This article explains how HackerEarth built a real-time social media update system for coding and hiring challenges. It uses Redis to maintain submission counters and state, Kafka to deliver submission payloads to an asynchronous worker, database queries at selected thresholds, and scheduled or delayed posting to avoid duplicate tweets and excessive update frequency.

## Source excerpt

Objective In Jan 2015, HackerEarth conducted nearly 10-12 hiring challenges, 5-6 coding challenges and numerous college challenges. HackerEarth has a decent social media presence and we wanted to inform our followers about the events at HackerEarth. One of the main objectives of this project was to provide flexibility to the marketing team to automate simple jobs and to focus on sophisticated campaigns. Design Goals As a first step, we decided to post about our events and their highlights on twitter. We covered event reminders, start/end of contests, who scored first AC and leaderboard updates at the end of a contest. We chose to do it by reading from the biggest and the meanest table of our database of Submissions. Challenges The submissions table is a very large table. An additional query on the submissions table during peak hours was not favourable. Hence, we did not count the submissions in-place and instead queued them to be processed later. Preventing duplicate tweets while maintaining state is also a challenge. Solution The application made a high volume of reads, few writes/updates. So any key/value stored would do the job. We chose Redis in lieu of Memcached. Redis offers data persistence in the event of node failure. This is very useful to avoid duplicate tweets. For instance, Two different users being credited for first AC submission in an event. By setting key expiry time and less number of keys for a single event, we prevented our Redis server from being overloaded. We maintained a key in Redis to keep count of submissions for an event. Reading from the database was not recommended because it would make a read call per submission during peak time and Redis performed faster reads. The application is an asynchronous worker and payload containing submission_id and event_id are passed to it using Kafka. So when the Redis key counter hit the magic numbers (1, 100, 500, multiples of 1000), The worker makes a DB query and posts a tweet. Worker subscribes to a