OiO.lk Community platform!

Oio.lk is an excellent forum for developers, providing a wide range of resources, discussions, and support for those in the developer community. Join oio.lk today to connect with like-minded professionals, share insights, and stay updated on the latest trends and technologies in the development field.
  You need to log in or register to access the solved answers to this problem.
  • You have reached the maximum number of guest views allowed
  • Please register below to remove this limitation

How to randomly split a DataFrame into several smaller DataFrames?

  • Thread starter Thread starter Jerry Chen
  • Start date Start date
J

Jerry Chen

Guest
I'm having trouble randomly splitting DataFrame df into groups of smaller DataFrames.

Code:
df
  movie_id  1   2   4   5   6   7   8   9   10  11  12  borda
0   1       5   4   0   4   4   0   0   0   4   0   0   21
1   2       3   0   0   3   0   0   0   0   0   0   0   6   
2   3       4   0   0   0   0   0   0   0   0   0   0   4   
3   4       3   0   0   0   0   5   0   0   4   0   5   17  
4   5       3   0   0   0   0   0   0   0   0   0   0   3   
5   6       5   0   0   0   0   0   0   5   0   0   0   10  
6   7       4   0   0   0   2   5   3   4   4   0   0   22  
7   8       1   0   0   0   4   5   0   0   0   4   0   14  
8   9       5   0   0   0   4   5   0   0   4   5   0   23  
9   10      3   2   0   0   0   4   0   0   0   0   0   9   
10  11      2   0   4   0   0   3   3   0   4   2   0   18  
11  12      5   0   0   0   4   5   0   0   5   2   0   21  
12  13      5   4   0   0   2   0   0   0   3   0   0   14  
13  14      5   4   0   0   5   0   0   0   0   0   0   14  
14  15      5   0   0   0   3   0   0   0   0   5   5   18  
15  16      5   0   0   0   0   0   0   0   4   0   0   9   
16  17      3   0   0   4   0   0   0   0   0   0   0   7   
17  18      4   0   0   0   0   0   0   0   0   0   0   4   
18  19      5   3   0   0   4   0   0   0   0   0   0   12  
19  20      4   0   0   0   0   0   0   0   0   0   0   4   
20  21      1   0   0   3   3   0   0   0   0   0   0   7   
21  22      4   0   0   0   3   5   5   0   5   4   0   26  
22  23      4   0   0   0   4   3   0   0   5   0   0   16  
23  24      3   0   0   4   0   0   0   0   0   3   0   10

I've tried sample and arange, but with bad results.

Code:
ran1 = df.sample(frac=0.2, replace=False, random_state=1)
ran2 = df.sample(frac=0.2, replace=False, random_state=1)
ran3 = df.sample(frac=0.2, replace=False, random_state=1)
ran4 = df.sample(frac=0.2, replace=False, random_state=1)
ran5 = df.sample(frac=0.2, replace=False, random_state=1)

print(ran1, '\n')
print(ran2, '\n')
print(ran3, '\n')
print(ran4, '\n')
print(ran5, '\n')

This turned out to be 5 exact same DataFrames.

Code:
   movie_id  1  2  4  5  6  7  8  9  10  11  12  borda  
13    14     5  4  0  0  5  0  0  0   0   0   0     14  
18    19     5  3  0  0  4  0  0  0   0   0   0     12  
3     4      3  0  0  0  0  5  0  0   4   0   5     17  
14    15     5  0  0  0  3  0  0  0   0   5   5     18  
20    21     1  0  0  3  3  0  0  0   0   0   0      7

Also I've tried :

Code:
g = df.groupby(['movie_id'])
h = np.arange(g.ngroups)
np.random.shuffle(h)

df[g.ngroup().isin(h[:6])]

The output :

Code:
    movie_id    1   2   4   5   6   7   8   9   10  11  12  borda   
4      5        3   0   0   0   0   0   0   0   0   0   0   3   
6      7        4   0   0   0   2   5   3   4   4   0   0   22  
7      8        1   0   0   0   4   5   0   0   0   4   0   14  
16     17       3   0   0   4   0   0   0   0   0   0   0   7   
17     18       4   0   0   0   0   0   0   0   0   0   0   4   
18     19       5   3   0   0   4   0   0   0   0   0   0   12

But there's still only one smaller group, other datas from df aren't grouped.

I'm expecting the smaller groups to be split evenly by using percentage. And the whole df should be split into groups.
<p>I'm having trouble randomly splitting DataFrame <code>df</code> into groups of smaller <code>DataFrames</code>.</p>
<pre><code>df
movie_id 1 2 4 5 6 7 8 9 10 11 12 borda
0 1 5 4 0 4 4 0 0 0 4 0 0 21
1 2 3 0 0 3 0 0 0 0 0 0 0 6
2 3 4 0 0 0 0 0 0 0 0 0 0 4
3 4 3 0 0 0 0 5 0 0 4 0 5 17
4 5 3 0 0 0 0 0 0 0 0 0 0 3
5 6 5 0 0 0 0 0 0 5 0 0 0 10
6 7 4 0 0 0 2 5 3 4 4 0 0 22
7 8 1 0 0 0 4 5 0 0 0 4 0 14
8 9 5 0 0 0 4 5 0 0 4 5 0 23
9 10 3 2 0 0 0 4 0 0 0 0 0 9
10 11 2 0 4 0 0 3 3 0 4 2 0 18
11 12 5 0 0 0 4 5 0 0 5 2 0 21
12 13 5 4 0 0 2 0 0 0 3 0 0 14
13 14 5 4 0 0 5 0 0 0 0 0 0 14
14 15 5 0 0 0 3 0 0 0 0 5 5 18
15 16 5 0 0 0 0 0 0 0 4 0 0 9
16 17 3 0 0 4 0 0 0 0 0 0 0 7
17 18 4 0 0 0 0 0 0 0 0 0 0 4
18 19 5 3 0 0 4 0 0 0 0 0 0 12
19 20 4 0 0 0 0 0 0 0 0 0 0 4
20 21 1 0 0 3 3 0 0 0 0 0 0 7
21 22 4 0 0 0 3 5 5 0 5 4 0 26
22 23 4 0 0 0 4 3 0 0 5 0 0 16
23 24 3 0 0 4 0 0 0 0 0 3 0 10
</code></pre>
<p>I've tried <code>sample</code> and <code>arange</code>, but with bad results.</p>
<pre><code>ran1 = df.sample(frac=0.2, replace=False, random_state=1)
ran2 = df.sample(frac=0.2, replace=False, random_state=1)
ran3 = df.sample(frac=0.2, replace=False, random_state=1)
ran4 = df.sample(frac=0.2, replace=False, random_state=1)
ran5 = df.sample(frac=0.2, replace=False, random_state=1)

print(ran1, '\n')
print(ran2, '\n')
print(ran3, '\n')
print(ran4, '\n')
print(ran5, '\n')
</code></pre>
<p>This turned out to be 5 exact same <code>DataFrames</code>.</p>
<pre><code> movie_id 1 2 4 5 6 7 8 9 10 11 12 borda
13 14 5 4 0 0 5 0 0 0 0 0 0 14
18 19 5 3 0 0 4 0 0 0 0 0 0 12
3 4 3 0 0 0 0 5 0 0 4 0 5 17
14 15 5 0 0 0 3 0 0 0 0 5 5 18
20 21 1 0 0 3 3 0 0 0 0 0 0 7
</code></pre>
<p>Also I've tried :</p>
<pre><code>g = df.groupby(['movie_id'])
h = np.arange(g.ngroups)
np.random.shuffle(h)

df[g.ngroup().isin(h[:6])]
</code></pre>
<p>The output :</p>
<pre><code> movie_id 1 2 4 5 6 7 8 9 10 11 12 borda
4 5 3 0 0 0 0 0 0 0 0 0 0 3
6 7 4 0 0 0 2 5 3 4 4 0 0 22
7 8 1 0 0 0 4 5 0 0 0 4 0 14
16 17 3 0 0 4 0 0 0 0 0 0 0 7
17 18 4 0 0 0 0 0 0 0 0 0 0 4
18 19 5 3 0 0 4 0 0 0 0 0 0 12
</code></pre>
<p>But there's still only one smaller group, other datas from <code>df</code> aren't grouped.</p>
<p>I'm expecting the smaller groups to be split evenly by using percentage. And the whole <code>df</code> should be split into groups.</p>
 
Top