OiO.lk Community platform!

Oio.lk is an excellent forum for developers, providing a wide range of resources, discussions, and support for those in the developer community. Join oio.lk today to connect with like-minded professionals, share insights, and stay updated on the latest trends and technologies in the development field.
  You need to log in or register to access the solved answers to this problem.
  • You have reached the maximum number of guest views allowed
  • Please register below to remove this limitation

AWS SageMaker Ground Truth job finished successfully without assigning tasks to requested number of human workers

  • Thread starter Thread starter Martin Macak
  • Start date Start date
M

Martin Macak

Guest
We created a GT labelling job using https://docs.aws.amazon.com/comprehend/latest/dg/cer-annotation-pdf.html.

We pre-selected ~500 documents and processed them via provided scripts, which yielded ~700 tasks. We also set that we want 3 human workers per task.

After several days, we were surprised that the job finished successfully. When I processed the generated data by loading them into our data lake and processed them, we learned that only ~20 documents were processed by requested 3 workers, ~30 were processed by 2 workers and the rest was processed only by one worker.

I double checked that the job has annotation job has correct settings by checking AWS API and there is NumberOfHumanWorkersPerDataObject: 3 in the configuration.

We checked lambda logs but there is no error, the job also states that everything was processed successfully.

How is it possible that Ground Truth didn't follow requested assignment criteria?
<p>We created a GT labelling job using <a href="https://docs.aws.amazon.com/comprehend/latest/dg/cer-annotation-pdf.html" rel="nofollow noreferrer">https://docs.aws.amazon.com/comprehend/latest/dg/cer-annotation-pdf.html</a>.</p>
<p>We pre-selected ~500 documents and processed them via provided scripts, which yielded ~700 tasks. We also set that we want 3 human workers per task.</p>
<p>After several days, we were surprised that the job finished successfully. When I processed the generated data by loading them into our data lake and processed them, we learned that only ~20 documents were processed by requested 3 workers, ~30 were processed by 2 workers and the rest was processed only by one worker.</p>
<p>I double checked that the job has annotation job has correct settings by checking AWS API and there is <code>NumberOfHumanWorkersPerDataObject: 3</code> in the configuration.</p>
<p>We checked lambda logs but there is no error, the job also states that everything was processed successfully.</p>
<p>How is it possible that Ground Truth didn't follow requested assignment criteria?</p>
Continue reading...
 

Latest posts

S
Replies
0
Views
1
Safwan Aipuram
S
Top