OiO.lk Community platform!

Oio.lk is an excellent forum for developers, providing a wide range of resources, discussions, and support for those in the developer community. Join oio.lk today to connect with like-minded professionals, share insights, and stay updated on the latest trends and technologies in the development field.
  You need to log in or register to access the solved answers to this problem.
  • You have reached the maximum number of guest views allowed
  • Please register below to remove this limitation

How to avoid error ('cannot mix list and non-list, non-null values', 'Conversion failed for column position with type object') in Airflow?

  • Thread starter Thread starter John Doe
  • Start date Start date
J

John Doe

Guest
I got this task in my dag:

Code:
@task
def get_result_from_records_api(api, tries: int, result_from_salons_api: list):
    salon_ids_list = result_from_salons_api[1]
    # result is 4 pd.DataFrames
    result_from_records_api = get_data_or_raise_error_with_retry(api.get_records_staff_sales_of_services_goods, tries, salon_ids_list=salon_ids_list)
    # make lists for dfs
    records_lst = []
    staff_from_records_lst = []
    services_sales_lst = []
    good_sales_lst = []
    # put dfs in lists
    records_lst.append(result_from_records_api[0])
    staff_from_records_lst.append(result_from_records_api[1])
    services_sales_lst.append(result_from_records_api[2])
    good_sales_lst.append(result_from_records_api[3])
    # make dict with lists
    result_dict = {
        'records': records_lst
        , 'staff_from_records': staff_from_records_lst
        , 'services_sales': services_sales_lst
        , 'good_sales': good_sales_lst
    }
    return result_dict

This task returns 4 pandas.DataFrames without error (returned values is correct).

I've tryed put result in list, tuple, dictionary, but every time I get error:

Code:
{xcom.py:664} ERROR - ('cannot mix list and non-list, non-null values', 'Conversion failed for column position with type object'). If you are using pickle instead of JSON for XCom, then you need to enable pickle support for XCom in your airflow config or make sure to decorate your object with attr.

How to avoid this error?

P. S. I already enable_xcom_picking in docker-compose.yml:

Code:
AIRFLOW__CORE__ENABLE_XCOM_PICKLING=true

Airflow 2.8.2 Python 3.11
<p>I got this task in my dag:</p>
<pre><code>@task
def get_result_from_records_api(api, tries: int, result_from_salons_api: list):
salon_ids_list = result_from_salons_api[1]
# result is 4 pd.DataFrames
result_from_records_api = get_data_or_raise_error_with_retry(api.get_records_staff_sales_of_services_goods, tries, salon_ids_list=salon_ids_list)
# make lists for dfs
records_lst = []
staff_from_records_lst = []
services_sales_lst = []
good_sales_lst = []
# put dfs in lists
records_lst.append(result_from_records_api[0])
staff_from_records_lst.append(result_from_records_api[1])
services_sales_lst.append(result_from_records_api[2])
good_sales_lst.append(result_from_records_api[3])
# make dict with lists
result_dict = {
'records': records_lst
, 'staff_from_records': staff_from_records_lst
, 'services_sales': services_sales_lst
, 'good_sales': good_sales_lst
}
return result_dict
</code></pre>
<p>This task returns 4 pandas.DataFrames without error (returned values is correct).</p>
<p>I've tryed put result in list, tuple, dictionary, but every time I get error:</p>
<pre><code>{xcom.py:664} ERROR - ('cannot mix list and non-list, non-null values', 'Conversion failed for column position with type object'). If you are using pickle instead of JSON for XCom, then you need to enable pickle support for XCom in your airflow config or make sure to decorate your object with attr.
</code></pre>
<p>How to avoid this error?</p>
<p>P. S. I already enable_xcom_picking in docker-compose.yml:</p>
<pre><code>AIRFLOW__CORE__ENABLE_XCOM_PICKLING=true
</code></pre>
<p>Airflow 2.8.2
Python 3.11</p>
 

Latest posts

I
Replies
0
Views
1
impact christian
I
Top