OiO.lk Community platform!

Oio.lk is an excellent forum for developers, providing a wide range of resources, discussions, and support for those in the developer community. Join oio.lk today to connect with like-minded professionals, share insights, and stay updated on the latest trends and technologies in the development field.
  You need to log in or register to access the solved answers to this problem.
  • You have reached the maximum number of guest views allowed
  • Please register below to remove this limitation

What do the FiftyOne evaluation metrics mean?

  • Thread starter Thread starter Benoît Boidin
  • Start date Start date
B

Benoît Boidin

Guest
I have a dataset I use to test my object detection model, let's say test_dataset.

When evaluating with COCO eval (through YOLOX eval.py script) for a given model, I get this result:

Code:
Average forward time: 23.05 ms, Average NMS time: 2.60 ms, Average inference time: 25.65 ms
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.724
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.957
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.831
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.278
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.591
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.810
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.535
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.755
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.759
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.349
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.649
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.839
per class AP:
| class        | AP     | class   | AP     | class        | AP     |
|:-------------|:-------|:--------|:-------|:-------------|:-------|
| cargo        | 59.491 | ferry   | 87.701 | fishing boat | 67.328 |
| sailing boat | 75.134 |         |        |              |        |
per class AR:
| class        | AR     | class   | AR     | class        | AR     |
|:-------------|:-------|:--------|:-------|:-------------|:-------|
| cargo        | 64.802 | ferry   | 89.717 | fishing boat | 71.506 |
| sailing boat | 77.748 |

However, when evaluating with FiftyOne, I get the following:

Code:
              precision    recall  f1-score   support

       cargo       0.76      0.91      0.83       606
       ferry       0.97      1.00      0.99       990
fishing boat       0.85      0.96      0.91       332
sailing boat       0.87      0.97      0.92       706

   micro avg       0.88      0.97      0.92      2634
   macro avg       0.87      0.96      0.91      2634
weighted avg       0.88      0.97      0.92      2634

I was using this script:

Code:
results = dataset.evaluate_detections(
    "predictions",
    gt_field="detections",
    compute_mAP=True,
    method="coco"
)

results.print_report()

I was expecting the same result since FiftyOne function use the COCO evaluation format.

I can't understand what are those metrics.
<p>I have a dataset I use to test my object detection model, let's say <code>test_dataset</code>.</p>
<p>When evaluating with COCO eval (through YOLOX <a href="https://github.com/Megvii-BaseDetection/YOLOX/blob/main/tools/eval.py" rel="nofollow noreferrer">eval.py</a> script) for a given model, I get this result:</p>
<pre class="lang-none prettyprint-override"><code>Average forward time: 23.05 ms, Average NMS time: 2.60 ms, Average inference time: 25.65 ms
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.724
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.957
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.831
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.278
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.591
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.810
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.535
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.755
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.759
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.349
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.649
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.839
per class AP:
| class | AP | class | AP | class | AP |
|:-------------|:-------|:--------|:-------|:-------------|:-------|
| cargo | 59.491 | ferry | 87.701 | fishing boat | 67.328 |
| sailing boat | 75.134 | | | | |
per class AR:
| class | AR | class | AR | class | AR |
|:-------------|:-------|:--------|:-------|:-------------|:-------|
| cargo | 64.802 | ferry | 89.717 | fishing boat | 71.506 |
| sailing boat | 77.748 |
</code></pre>
<p>However, when evaluating with FiftyOne, I get the following:</p>
<pre class="lang-none prettyprint-override"><code> precision recall f1-score support

cargo 0.76 0.91 0.83 606
ferry 0.97 1.00 0.99 990
fishing boat 0.85 0.96 0.91 332
sailing boat 0.87 0.97 0.92 706

micro avg 0.88 0.97 0.92 2634
macro avg 0.87 0.96 0.91 2634
weighted avg 0.88 0.97 0.92 2634
</code></pre>
<p>I was using this script:</p>
<pre class="lang-py prettyprint-override"><code>results = dataset.evaluate_detections(
"predictions",
gt_field="detections",
compute_mAP=True,
method="coco"
)

results.print_report()
</code></pre>
<p>I was expecting the same result since FiftyOne function use the COCO evaluation format.</p>
<p>I can't understand what are those metrics.</p>
 

Latest posts

S
Replies
0
Views
1
Safwan Aipuram
S
Top