OiO.lk Community platform!

Oio.lk is an excellent forum for developers, providing a wide range of resources, discussions, and support for those in the developer community. Join oio.lk today to connect with like-minded professionals, share insights, and stay updated on the latest trends and technologies in the development field.
  You need to log in or register to access the solved answers to this problem.
  • You have reached the maximum number of guest views allowed
  • Please register below to remove this limitation

RandomizedSearchCV with NeuralNetClassifier gives different tuning parameters each run

  • Thread starter Thread starter Christopher Marlowe
  • Start date Start date
C

Christopher Marlowe

Guest
I am trying to get the best parameters for the DNN Model using RandomizedSearchCV, but each time I run the code I get different sets of best parameters.

I tried to add random_state=42 to GridSearchCV, but it seems not acceptable.

Step of my experiment:


  1. I set the random seed for my experiment:

    Code:
    seed = 42
    os.environ['PYTHONHASHSEED'] = str(seed)
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    # Enforce deterministic behavior in PyTorch
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False

  2. I have divided the dataset into training and test datasets:

Code:
data_indices = list(range(2400)) 
train_indices, test_indices = train_test_split(
    data_indices, random_state=42, test_size=0.20, shuffle=True, stratify=y)  

X_train, y_train = X[train_indices], Y[train_indices]
X_test, y_test =  X[test_indices], Y[test_indices]
  1. I am using NeuralNetClassifier from the Skorch package to tune the hyperparameter with RandomizedSearchCV:

Code:
import torch
import torch.nn as nn
import torch.optim as optim
from skorch import NeuralNetClassifier 
from sklearn.model_selection import RandomizedSearchCV 
 
# Define the model
class MlpNet(nn.Module):
    def __init__(self, layer_sizes=[], activation=nn.ReLU):
        super(MlpNet, self).__init__()
        layers = []
        self.act = activation
        for l_id in range(len(layer_sizes) - 1):
            layers.append(nn.Linear(layer_sizes[l_id], layer_sizes[l_id + 1]))
            if l_id < len(layer_sizes) - 2:  # No activation or batchnorm for the last layer
                layers.append(nn.BatchNorm1d(num_features=layer_sizes[l_id + 1], affine=False))
                layers.append(self.act())
        self.layers = nn.Sequential(*layers)
        self.prob = nn.Sigmoid()
        self.float()

    def forward(self, x):
        x = self.layers(x) 
        return self.prob(x) 

# Create the model
model = NeuralNetClassifier(
    module=MlpNet,
    criterion=nn.BCELoss, 
    optimizer=optim.Adam, 
    device=device,   
    verbose=False
) 

param_grid = {
    'optimizer__lr': [0.001, 0.01], 
    'module__layer_sizes': [    [ 108,  256, 256, 128, 1],
                                [ 108,  256, 256, 1],
                                [ 108,  128, 64, 1],
                                [ 108,  128, 128, 1], 
                            ],
    'module__activation': [nn.ReLU,  nn.Tanh, nn.Sigmoid], 
    'batch_size': [10, 30, 50, 80, 100],
    'max_epochs': [10, 50, 80, 100]  
}  
   
skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
random_search = RandomizedSearchCV(estimator=model, param_distributions=param_grid, n_iter=5, 
                                   scoring='accuracy', n_jobs=4, cv=skf.split(X_train,y_train), random_state=42 )

grid_result = random_search.fit(X_train.float(), torch.unsqueeze(y_train.float(), 1))
 

print("Best: %f using %s" % (grid_result.best_score_, grid_result.best_params_))

I have seen this question from StackOverflow, but none have solved my queries.

  1. Why i am getting different tuning parameters each run when using GridSearchCV with random forest regressor?
  2. Reproducing Model results from RandomizedSearchCV
  3. RandomizedSearchCV independently on models in an ensemble
  4. hyperparameter tuning in sklearn using RandomizedSearchCV taking lot of time
  5. Conditional tuning of hyperparameters with RandomizedSearchCV in scikit-learn

Each time I run the code, I get different sets of best parameters. These different best hyperparameters give different training and test accuracy.

I want to reproduce the result of hyperparameter tuning. How can I get the same set of hyperparameters in every run?
<p>I am trying to get the best parameters for the DNN Model using RandomizedSearchCV, but each time I run the code I get different sets of best parameters.</p>
<p>I tried to add random_state=42 to GridSearchCV, but it seems not acceptable.</p>
<p><strong>Step of my experiment:</strong></p>
<ol>
<li><p>I set the random seed for my experiment:</p>
<pre><code>seed = 42
os.environ['PYTHONHASHSEED'] = str(seed)
random.seed(seed)
np.random.seed(seed)
torch.manual_seed(seed)
torch.cuda.manual_seed(seed)
torch.cuda.manual_seed_all(seed)
# Enforce deterministic behavior in PyTorch
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
</code></pre>
</li>
<li><p>I have divided the dataset into training and test datasets:</p>
</li>
</ol>
<pre><code>data_indices = list(range(2400))
train_indices, test_indices = train_test_split(
data_indices, random_state=42, test_size=0.20, shuffle=True, stratify=y)

X_train, y_train = X[train_indices], Y[train_indices]
X_test, y_test = X[test_indices], Y[test_indices]
</code></pre>
<ol start="3">
<li>I am using NeuralNetClassifier from the Skorch package to tune the hyperparameter with RandomizedSearchCV:</li>
</ol>
<pre><code>import torch
import torch.nn as nn
import torch.optim as optim
from skorch import NeuralNetClassifier
from sklearn.model_selection import RandomizedSearchCV

# Define the model
class MlpNet(nn.Module):
def __init__(self, layer_sizes=[], activation=nn.ReLU):
super(MlpNet, self).__init__()
layers = []
self.act = activation
for l_id in range(len(layer_sizes) - 1):
layers.append(nn.Linear(layer_sizes[l_id], layer_sizes[l_id + 1]))
if l_id < len(layer_sizes) - 2: # No activation or batchnorm for the last layer
layers.append(nn.BatchNorm1d(num_features=layer_sizes[l_id + 1], affine=False))
layers.append(self.act())
self.layers = nn.Sequential(*layers)
self.prob = nn.Sigmoid()
self.float()

def forward(self, x):
x = self.layers(x)
return self.prob(x)

# Create the model
model = NeuralNetClassifier(
module=MlpNet,
criterion=nn.BCELoss,
optimizer=optim.Adam,
device=device,
verbose=False
)

param_grid = {
'optimizer__lr': [0.001, 0.01],
'module__layer_sizes': [ [ 108, 256, 256, 128, 1],
[ 108, 256, 256, 1],
[ 108, 128, 64, 1],
[ 108, 128, 128, 1],
],
'module__activation': [nn.ReLU, nn.Tanh, nn.Sigmoid],
'batch_size': [10, 30, 50, 80, 100],
'max_epochs': [10, 50, 80, 100]
}

skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
random_search = RandomizedSearchCV(estimator=model, param_distributions=param_grid, n_iter=5,
scoring='accuracy', n_jobs=4, cv=skf.split(X_train,y_train), random_state=42 )

grid_result = random_search.fit(X_train.float(), torch.unsqueeze(y_train.float(), 1))


print("Best: %f using %s" % (grid_result.best_score_, grid_result.best_params_))
</code></pre>
<p>I have seen this question from StackOverflow, but none have solved my queries.</p>
<ol>
<li><a href="https://stackoverflow.com/q/57498725">Why i am getting different tuning parameters each run when using GridSearchCV with random forest regressor?</a></li>
<li><a href="https://stackoverflow.com/q/68580962">Reproducing Model results from RandomizedSearchCV</a></li>
<li><a href="https://stackoverflow.com/q/77769033">RandomizedSearchCV independently on models in an ensemble</a></li>
<li><a href="https://stackoverflow.com/q/50664480">hyperparameter tuning in sklearn using RandomizedSearchCV taking lot of time</a></li>
<li><a href="https://stackoverflow.com/q/58168297">Conditional tuning of hyperparameters with RandomizedSearchCV in scikit-learn</a></li>
</ol>
<p>Each time I run the code, I get different sets of best parameters. These different best hyperparameters give different training and test accuracy.</p>
<p>I want to reproduce the result of hyperparameter tuning. How can I get the same set of hyperparameters in every run?</p>
 

Latest posts

M
Replies
0
Views
1
Mohit Pant
M
Top