OiO.lk Community platform!

Oio.lk is an excellent forum for developers, providing a wide range of resources, discussions, and support for those in the developer community. Join oio.lk today to connect with like-minded professionals, share insights, and stay updated on the latest trends and technologies in the development field.
  You need to log in or register to access the solved answers to this problem.
  • You have reached the maximum number of guest views allowed
  • Please register below to remove this limitation

AutoTokenizer.from_pretrained took forever to load

  • Thread starter Thread starter Raptor
  • Start date Start date
R

Raptor

Guest
I used the following code to load my custom-trained tokenizer:

Code:
from transformers import AutoTokenizer
test_tokenizer = AutoTokenizer.from_pretrained('raptorkwok/cantonese-tokenizer-test')

It took forever to load. Even if I replace the AutoTokenizer with PreTrainedTokenizerFast, it still loads forever.

How to debug or fix this issue?
<p>I used the following code to load my custom-trained tokenizer:</p>
<pre><code>from transformers import AutoTokenizer
test_tokenizer = AutoTokenizer.from_pretrained('raptorkwok/cantonese-tokenizer-test')
</code></pre>
<p>It took forever to load. Even if I replace the <code>AutoTokenizer</code> with <code>PreTrainedTokenizerFast</code>, it still loads forever.</p>
<p>How to debug or fix this issue?</p>
 

Latest posts

Top