OiO.lk Community platform!

Oio.lk is an excellent forum for developers, providing a wide range of resources, discussions, and support for those in the developer community. Join oio.lk today to connect with like-minded professionals, share insights, and stay updated on the latest trends and technologies in the development field.
  You need to log in or register to access the solved answers to this problem.
  • You have reached the maximum number of guest views allowed
  • Please register below to remove this limitation

Pyspark convert the structure of the data

  • Thread starter Thread starter Shravan K
  • Start date Start date
S

Shravan K

Guest
I have a bigquery table. Sample data in json format is

Code:
[{"cust_id": 12345,
"Seq_column": [1,2,3,4,5]}]

When i fetch the data using pyspark and save it back to bigquery table, the structure of column "seq_column" is getting changed

Code:
[{"cust_id": 12345,
"Seq_column": [{"element":1},{"element":2},{"element":3},{"element":4},{"element":5}
}]}]

How can i maintain the structure of that column as original? I tried array type function of pyspark. Closer solution i got is converting it to stringtype before saving. But that is not correct. Is there any other way?
<p>I have a bigquery table. Sample data in json format is</p>
<pre><code>[{"cust_id": 12345,
"Seq_column": [1,2,3,4,5]}]
</code></pre>
<p>When i fetch the data using pyspark and save it back to bigquery table, the structure of column "seq_column" is getting changed</p>
<pre><code>[{"cust_id": 12345,
"Seq_column": [{"element":1},{"element":2},{"element":3},{"element":4},{"element":5}
}]}]
</code></pre>
<p>How can i maintain the structure of that column as original?
I tried array type function of pyspark. Closer solution i got is converting it to stringtype before saving. But that is not correct. Is there any other way?</p>
 

Latest posts

M
Replies
0
Views
1
MusicLovingIndianGirl
M
Q
Replies
0
Views
1
quora question
Q
Top