OiO.lk Community platform!

Oio.lk is an excellent forum for developers, providing a wide range of resources, discussions, and support for those in the developer community. Join oio.lk today to connect with like-minded professionals, share insights, and stay updated on the latest trends and technologies in the development field.
  You need to log in or register to access the solved answers to this problem.
  • You have reached the maximum number of guest views allowed
  • Please register below to remove this limitation

Running a .py algorithm on multiple csv files in python inside a for loop

  • Thread starter Thread starter Sagnik Kar
  • Start date Start date
S

Sagnik Kar

Guest
I am encountering a problem where I am trying to run an algorithm(.py file) on multiple csv files together (datasets). The algorithm computes values from the dataset and returns the output. The for loop is crashing after maximum of 250 files. Each file has approximately one year data. Even on VM with 128 GB RAM the code is crashing after few files. How to fix this problem?

  1. Tried optimizing the algorithm and enclose everything in functions.
  2. Used for loop where a file is read in pandas dataframe and then run the algorithm.
<p>I am encountering a problem where I am trying to run an algorithm(.py file) on multiple csv files together (datasets). The algorithm computes values from the dataset and returns the output. The for loop is crashing after maximum of 250 files. Each file has approximately one year data. Even on VM with 128 GB RAM the code is crashing after few files. How to fix this problem?</p>
<ol>
<li>Tried optimizing the algorithm and enclose everything in functions.</li>
<li>Used for loop where a file is read in pandas dataframe and then run the algorithm.</li>
</ol>
 

Latest posts

B
Replies
0
Views
1
Blundering Ecologist
B
Top