OiO.lk Community platform!

Oio.lk is an excellent forum for developers, providing a wide range of resources, discussions, and support for those in the developer community. Join oio.lk today to connect with like-minded professionals, share insights, and stay updated on the latest trends and technologies in the development field.
  You need to log in or register to access the solved answers to this problem.
  • You have reached the maximum number of guest views allowed
  • Please register below to remove this limitation

Anomaly in PyTorch Vision Transformer Model, Validation Loss, Testing Accuracy and Normal Accuracy Calculation

  • Thread starter Thread starter dgezgin
  • Start date Start date
D

dgezgin

Guest
I'm trying to create a vision transformer model for my personal project using PyTorch.

The problem is when I'm running my testing codei, I'm not sure if I'm correctly calculating the training loss, validation loss, training accuracy and testing (+ top 2 testing) accuracy.

This is my code:

Code:
criterion = nn.CrossEntropyLoss()
optimizer = AdamW(vit.parameters(), lr=LEARNING_RATE, weight_decay=WEIGHT_DECAY)

# Training Loop
train_losses, val_losses, accuracies, top2_accuracies= [], [], [], []
training_start = time.time()

for epoch in range(NUM_EPOCHS):
    log_str = write_and_print_str(log_str, f"EPOCH [{epoch+1}/{NUM_EPOCHS}]")
    start = time.time()
    vit.train()
    running_loss = []
    for images, labels in train_dataloader:
        images, labels = images.to(device), labels.to(device)
        optimizer.zero_grad()
        outputs = vit(images)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()
        running_loss.append(loss.item())

    avg_train_loss = sum(running_loss) / len(running_loss)
    train_losses.append(avg_train_loss)
    log_str = write_and_print_str(log_str, f'Loss: {avg_train_loss}')

    vit.eval()
    val_loss = []
    correct = 0
    top2_correct = 0
    total = 0
    with torch.no_grad():
        for images, labels in test_dataloader:
            images, labels = images.to(device), labels.to(device)
            outputs = vit(images)
            loss = criterion(outputs, labels)
            val_loss.append(loss.item())
            _, predicted = torch.max(outputs.data, 1)
            total += labels.size(0)
            correct += (predicted == labels).sum().item()

            # Calculate top-2 accuracy
            top2_pred = torch.topk(outputs, 2, dim=1).indices
            top2_correct += (top2_pred == labels.unsqueeze(1)).sum().item()
        end = time.time()
    avg_val_loss = sum(val_loss) / len(val_loss)
    accuracy = 100 * correct / total
    top2_accuracy = 100 * top2_correct / total
   
    val_losses.append(avg_val_loss)
    accuracies.append(accuracy)
    top2_accuracies.append(top2_accuracy)

    log_str = write_and_print_str(log_str, f'Validation Loss: {avg_val_loss}, \nAccuracy: {accuracy}%,\nTop-2 Accuracy: {top2_accuracy}%\nTime:{round(end-start, 2)}\n\n')

training_end = time.time()

log_str = write_and_print_str(log_str, f'Training Duration: {round(training_end - training_start, 2)}\n')

print("EPOCHS were saved to the file successfully")

I ran it for more than 20 times and logged all of the runs in an extensive way.

In all of the results the validation loss started over 1 and reduced down to 0.20. But the problem is in these cases my accuracy was around 90% so I assume that I'm making something wrong in my code related to calculations.

To give more detail about the numerical valus of accuracy and loss, here are some of my EPOCH results

Code:
EPOCH [1/50]
Loss: 1.4692728799123032
Validation Loss: 1.1625839814995274, 
Accuracy: 49.58932238193019%,
Top-2 Accuracy: 75.77002053388091%
Time:29.71

EPOCH [10/50]
Loss: 0.1079550055715327
Validation Loss: 0.5106942771059094, 
Accuracy: 83.26488706365502%,
Top-2 Accuracy: 97.53593429158111%
Time:25.86

EPOCH [20/50]
Loss: 0.037730065656293076
Validation Loss: 0.4059527646185774, 
Accuracy: 89.52772073921972%,
Top-2 Accuracy: 97.94661190965093%
Time:26.12

EPOCH [30/50]
Loss: 0.00011380775267753052
Validation Loss: 0.22308006276955095, 
Accuracy: 94.6611909650924%,
Top-2 Accuracy: 99.48665297741273%
Time:24.41

EPOCH [40/50]
Loss: 3.5449059315886606e-05
Validation Loss: 0.23672400451808548, 
Accuracy: 94.76386036960986%,
Top-2 Accuracy: 99.48665297741273%
Time:25.46

EPOCH [50/50]
Loss: 1.367992779425829e-05
Validation Loss: 0.24671741761443572, 
Accuracy: 94.6611909650924%,
Top-2 Accuracy: 99.48665297741273%
Time:25.66

I hope that you can help me with this. I just need clarification
<p>I'm trying to create a vision transformer model for my personal project using PyTorch.</p>
<p>The problem is when I'm running my testing codei, I'm not sure if I'm correctly calculating the training loss, validation loss, training accuracy and testing (+ top 2 testing) accuracy.</p>
<p>This is my code:</p>
<pre class="lang-python prettyprint-override"><code>criterion = nn.CrossEntropyLoss()
optimizer = AdamW(vit.parameters(), lr=LEARNING_RATE, weight_decay=WEIGHT_DECAY)

# Training Loop
train_losses, val_losses, accuracies, top2_accuracies= [], [], [], []
training_start = time.time()

for epoch in range(NUM_EPOCHS):
log_str = write_and_print_str(log_str, f"EPOCH [{epoch+1}/{NUM_EPOCHS}]")
start = time.time()
vit.train()
running_loss = []
for images, labels in train_dataloader:
images, labels = images.to(device), labels.to(device)
optimizer.zero_grad()
outputs = vit(images)
loss = criterion(outputs, labels)
loss.backward()
optimizer.step()
running_loss.append(loss.item())

avg_train_loss = sum(running_loss) / len(running_loss)
train_losses.append(avg_train_loss)
log_str = write_and_print_str(log_str, f'Loss: {avg_train_loss}')

vit.eval()
val_loss = []
correct = 0
top2_correct = 0
total = 0
with torch.no_grad():
for images, labels in test_dataloader:
images, labels = images.to(device), labels.to(device)
outputs = vit(images)
loss = criterion(outputs, labels)
val_loss.append(loss.item())
_, predicted = torch.max(outputs.data, 1)
total += labels.size(0)
correct += (predicted == labels).sum().item()

# Calculate top-2 accuracy
top2_pred = torch.topk(outputs, 2, dim=1).indices
top2_correct += (top2_pred == labels.unsqueeze(1)).sum().item()
end = time.time()
avg_val_loss = sum(val_loss) / len(val_loss)
accuracy = 100 * correct / total
top2_accuracy = 100 * top2_correct / total

val_losses.append(avg_val_loss)
accuracies.append(accuracy)
top2_accuracies.append(top2_accuracy)

log_str = write_and_print_str(log_str, f'Validation Loss: {avg_val_loss}, \nAccuracy: {accuracy}%,\nTop-2 Accuracy: {top2_accuracy}%\nTime:{round(end-start, 2)}\n\n')

training_end = time.time()

log_str = write_and_print_str(log_str, f'Training Duration: {round(training_end - training_start, 2)}\n')

print("EPOCHS were saved to the file successfully")

</code></pre>
<p>I ran it for more than 20 times and logged all of the runs in an extensive way.</p>
<p>In all of the results the validation loss started <strong>over 1</strong> and reduced down to 0.20. But the problem is in these cases my accuracy was around 90% so I assume that I'm making something wrong in my code related to calculations.</p>
<p>To give more detail about the numerical valus of accuracy and loss, here are some of my EPOCH results</p>
<pre><code>EPOCH [1/50]
Loss: 1.4692728799123032
Validation Loss: 1.1625839814995274,
Accuracy: 49.58932238193019%,
Top-2 Accuracy: 75.77002053388091%
Time:29.71

EPOCH [10/50]
Loss: 0.1079550055715327
Validation Loss: 0.5106942771059094,
Accuracy: 83.26488706365502%,
Top-2 Accuracy: 97.53593429158111%
Time:25.86

EPOCH [20/50]
Loss: 0.037730065656293076
Validation Loss: 0.4059527646185774,
Accuracy: 89.52772073921972%,
Top-2 Accuracy: 97.94661190965093%
Time:26.12

EPOCH [30/50]
Loss: 0.00011380775267753052
Validation Loss: 0.22308006276955095,
Accuracy: 94.6611909650924%,
Top-2 Accuracy: 99.48665297741273%
Time:24.41

EPOCH [40/50]
Loss: 3.5449059315886606e-05
Validation Loss: 0.23672400451808548,
Accuracy: 94.76386036960986%,
Top-2 Accuracy: 99.48665297741273%
Time:25.46

EPOCH [50/50]
Loss: 1.367992779425829e-05
Validation Loss: 0.24671741761443572,
Accuracy: 94.6611909650924%,
Top-2 Accuracy: 99.48665297741273%
Time:25.66
</code></pre>
<p>I hope that you can help me with this. I just need clarification</p>
 

Latest posts

Online statistics

Members online
1
Guests online
1
Total visitors
2
Top