Background tasks

Originally, while running our services on DigitalOcean, we relied on Celery and Celery Beat (with Redis) for background tasks and scheduling. However, once we moved to Google Cloud Run, the integration became more complex and costly:

• Celery requires a always-running worker process, which isn’t straightforward in a serverless model.
• Running Celery in the cloud typically requires a Redis instance for task queuing.
• Maintaining a Redis cluster and the necessary network connectors in Google Cloud can exceed the cost of running Cloud Run itself.

The Switch to Cloud Tasks and Cloud Scheduler

• Cloud Tasks in place of Celery
  • Handles async task queuing and execution.
  • Integrates seamlessly with Cloud Run via authenticated HTTP requests.
• Cloud Scheduler in place of Celery Beat
  • Triggers recurring tasks (hourly in our case).
  • Works similarly to a traditional cron job service, but fully managed by Google Cloud.

Cloud tasks example:

Cloud scheduler example: