How to move user-specific log files to AWS S3 and update the user table in CodeIgniter 3 for 1.3 million users?

I am working on a project using CodeIgniter 3 where I store user-specific logs in separate files for each user. I have around 1.3 million users, and I need to move these log files to an AWS S3 bucket. After successfully uploading the files, I also need to update the corresponding entries in the users table in my database.

Given the large number of users, what is the best approach to:

Efficiently upload these user-specific log files to AWS S3.
Update the user data in the database once the files are successfully uploaded.
Any advice on handling this large-scale operation efficiently, along with code examples, would be greatly appreciated. Thank you!

What I’ve tried:

I’ve created two methods:

1. The first method generates text files in batches of 10,000 users. Each file contains an array of objects where each object has the local path of the file, the S3 path where the file should be stored, and the user ID.
2. The second method reads the generated text file, uploads the data to S3, updates the users table with the S3 path, and deletes the text file after the batch is complete.
   The batch generation and deletion process work fine, but I’m still looking for ways to optimize this workflow to handle all 1.3 million users efficiently.

The issue I’m facing is that after uploading 300 to 400 files, I’m getting a "maximum execution time exceeded" error, which halts the entire process.
What I was expecting:
I was hoping to find an efficient way to upload files in batches to AWS S3 without running into the execution time limit and ensuring that the database is updated correctly.

Questions:

How can I optimize the bulk file uploads to AWS S3 to avoid hitting the execution time limit?
Is there a way to batch or queue these uploads in CodeIgniter to prevent the process from timing out?
What strategies can I use to ensure that the entire upload and update process completes efficiently for 1.3 million users?
Any advice, code examples, or best practices for handling this large-scale operation without hitting the execution time limit would be greatly appreciated!

You need to sign in to view this answers