Horizontal Scaling with Cron Jobs
Solving duplicate cron execution in auto-scaling environments
A client needed a cron job in their backend server with auto horizontal scaling — and this made things complicated.
The Problem
Horizontal scaling means having multiple copies of the server running together so that the load is divided between each server. If I were to run the cron job inside the server, then with horizontal scaling, the cron will run in every server instance.
Let's say we have to call Service B when a cron event occurs. Now if we have 5 server instances, then 5 crons would run and Service B would be called 5 times — creating duplicates.
The Solution
I solved this problem with a simple approach: having an API to trigger the cron job.
A Lambda cron would call the API, and because of the load balancer, the API would hit only one single server instance (not all) at any time. Thus, Service B will only be called once.
This elegantly solves the duplication issue!
