I am currently using the following llm as mentioned above. I am using unsloth for faster processing.
The parameters with which I loaded the model are below:
model, tokenizer = FastLanguageModel.from_pretrained(
model_name="unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit",
max_seq_length=2048,
dtype=None, # None for auto detection
load_in_4bit=True, # reduces memory usage
attn_implementation="flash_attention_2",
token="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"
)
import torch
model = FastLanguageModel.get_peft_model(
model,
r = 64,
target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj",],
lora_alpha = 128,
lora_dropout = 0.2,
bias = "none",
use_gradient_checkpointing = "unsloth",
random_state = 4,
use_rslora = False,
loftq_config = None,
)
The prompt I used to train :
prompt = """
### Translation Task:
You are a professional translator tasked with translating the following sentence from English into {language_name}. Ensure you maintain semantic similarity, capture emotional nuances, and reflect subtleties such as character relationships, implied meanings, and cultural context. Pay particular attention to the tone and intent behind phrases and adapt them appropriately for the target language.
### Original Sentence:
{sentence}
### Examples of Nuanced Translations:
- English: "Hello!! Chris. How are you?" Asked Jane.
- Nuanced Translation: "Bonjour!! Oncle Chris. Comment ça va?" Demanda Jane.
*Note: In this translation, the familiarity and familial relationship is emphasized by modifying "Hello" to "Bonjour, Oncle," reflecting a possible closeness or respect in the cultural context.
- English: "For shame! For shame! cried the lady's maid."
- Nuanced Translation: "Quelle honte! Quelle honte! continua la femme de chambre, oui, elle est semblable à un chat enragé."
*Note: Here, the translator added "oui, elle est semblable à un chat enragé," an insult not present in the original, to intensify the emotional tone and adapt the phrase culturally.
### Your Translation:
{translation}
"""
EOS_TOKEN = tokenizer.eos_token if tokenizer.eos_token else ""
def formatting_prompts(examples):
languages = examples["language_name"]
sentences = examples["aligned_src"]
translations = examples["aligned_tgt"]
texts = []
for language, sentence, translation in zip(languages, sentences, translations):
text = prompt.format(language_name=language, sentence=sentence, translation=translation) + EOS_TOKEN
texts.append(text)
return {"text": texts}
# Apply the new formatting function to datasets
train_dataset = train_dataset.map(formatting_prompts, batched=True)
val_dataset = val_dataset.map(formatting_prompts, batched=True)
test_dataset = test_dataset.map(formatting_prompts, batched=True)
What I am doing using this is a translation task on story books. I have a sentence aligned dataset created using bertalign tool which contains about 60000 observations. The sentence alignments are between English and Spanish,French and Portuguese. I have split the train and test as 0.1 ratio and my validation data has about 6000 observations split from the train data.
The parameters I have set are as follows:
trainer = SFTTrainer(
model=model,
tokenizer=tokenizer,
train_dataset=train_dataset,
eval_dataset=val_dataset,
dataset_text_field="text",
max_seq_length=384,
dataset_num_proc=2,
packing=False,
args=TrainingArguments(
output_dir=checkpoint_dir, # Use checkpoint directory for all outputs
per_device_train_batch_size=batch_size,
per_device_eval_batch_size=eval_batch_size,
gradient_accumulation_steps=4,
warmup_steps=243,
num_train_epochs=18,
learning_rate=2e-4,
fp16=not is_bfloat16_supported(),
bf16=is_bfloat16_supported(),
logging_steps=20,
eval_strategy="steps",
eval_steps=60,
save_strategy="steps",
save_steps=60, # Ensure this aligns with eval_steps if using step-based strategy
save_total_limit=5,
load_best_model_at_end=True,
metric_for_best_model="eval_loss",
greater_is_better=False,
optim="adamw_hf",
weight_decay=0.01,
lr_scheduler_type="linear",
report_to="wandb",
seed=4131,
resume_from_checkpoint=checkpoint_dir # Resume only if checkpoint files exist
),
callbacks=[
EarlyStoppingCallback(early_stopping_patience=5)
],
)
The results of training the above model are as follows:
training loss and validation loss
So I did some predictions on the test set and I found that the outputs are really good on one end. But sometimes the model gives me garbage output like in this image:Predictions. As you can see this the outputs has the text from prompts and it has given me the output in Hindi as well even though I have not trained it on that particular language.
Any idea on why this is happening? Is it because of the rank and alpha values which I have drastically increased in the peft configuration? Any insights are helpful.
You need to sign in to view this answers
