OiO.lk Community platform!

Oio.lk is an excellent forum for developers, providing a wide range of resources, discussions, and support for those in the developer community. Join oio.lk today to connect with like-minded professionals, share insights, and stay updated on the latest trends and technologies in the development field.
  You need to log in or register to access the solved answers to this problem.
  • You have reached the maximum number of guest views allowed
  • Please register below to remove this limitation

Parallel - Threading sqlite3 python issue

  • Thread starter Thread starter user25295381
  • Start date Start date
U

user25295381

Guest
Hello I am trying to add my data in parallel in sqlite3 but i seem to be getting the following errors:

Error inserting chunk: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 57940 and this is thread id 18120. Error inserting chunk: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 57940 and this is thread id 36052.

I have a file which I import the class for example:

Code:
class SQLiteDatabase:
    def __init__(self, db_path):
        self.db_path = db_path
    
    def connect(self):
        self.connection = sqlite3.connect(self.db_path)
        self.cursor = self.connection.cursor()

    def insert_data_from_list(self, columns, table_names, data) -> None:
       
        placeholders = ', '.join('?' for _ in range(2))
        insert_query = f"INSERT INTO {table_name} ({columns}) VALUES ({placeholders})"
        try:
            for row in data:
                self.cursor.execute(insert_query, row)
            self.connection.commit()
            print(f"Chunk inserted successfully")
        except sqlite3.Error as e:
            print(f"Error inserting chunk: {e}")
            self.connection.rollback()

and then when I try to do something like this :

Code:
from concurrent.futures import ThreadPoolExecutor
   
with ThreadPoolExecutor(max_workers=10) as executor:
        try:
            db = SQLiteDatabase(db_path)
            db.connect()
            executor.map(db.insert_data_from_list, data_chunks)
        finally:
            db.disconnect()

I get the errors I stated.

I tried using a for and a while loop I tried creating a function same result.
<p>Hello I am trying to add my data in parallel in sqlite3 but i seem to be getting the following errors:</p>
<p>Error inserting chunk: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 57940 and this is thread id 18120.
Error inserting chunk: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 57940 and this is thread id 36052.</p>
<p>I have a file which I import the class for example:</p>
<pre><code>class SQLiteDatabase:
def __init__(self, db_path):
self.db_path = db_path

def connect(self):
self.connection = sqlite3.connect(self.db_path)
self.cursor = self.connection.cursor()

def insert_data_from_list(self, columns, table_names, data) -> None:

placeholders = ', '.join('?' for _ in range(2))
insert_query = f"INSERT INTO {table_name} ({columns}) VALUES ({placeholders})"
try:
for row in data:
self.cursor.execute(insert_query, row)
self.connection.commit()
print(f"Chunk inserted successfully")
except sqlite3.Error as e:
print(f"Error inserting chunk: {e}")
self.connection.rollback()


</code></pre>
<p>and then when I try to do something like this :</p>
<pre><code>from concurrent.futures import ThreadPoolExecutor

with ThreadPoolExecutor(max_workers=10) as executor:
try:
db = SQLiteDatabase(db_path)
db.connect()
executor.map(db.insert_data_from_list, data_chunks)
finally:
db.disconnect()

</code></pre>
<p>I get the errors I stated.</p>
<p>I tried using a for and a while loop I tried creating a function same result.</p>
 

Latest posts

Top