How not to overload a server when running thousands of Python applications with Celery and RabbitMQ

I have about 1000 small Python applications which run more or less at the same time. It’s running on a Ubuntu server.

These applications fetch data from a large database on a nightly basis and generate PDF reports. That’s quite time-consuming.

I was looking into some background jobs in Python and I found that Celery and RabbitMQ is the combination I should probably look after.

The idea was that I would have installed RabbitMQ on the server that would be managed through systemctl.
And then every single application Python application would start (and it would trigger starting Celery) and push data to RabbitMQ, from where it would be gradually taken and executed (generated a PDF document).

However, my concern is that if those 1000 Python scripts would run at the same time AND launched their Celery instance, it would probably overload the server.

How do I handle such a situation?

You need to sign in to view this answers