LoRA Persona Override: Research and Recommendations

> Target: Make Qwen2.5-7B-Instruct-4bit fully adopt Mohamed's communication style. > Hardware: Mac5 (M4 16GB), MLX mlx_lm LoRA trainer. > Data: 3,126 training examples in ChatML format. > Date: 2026-03-22 The current configuration has **four compounding problems** that prevent persona override: | Parameter | Current | Problem | |-----------|---------|---------| | LoRA rank | 16 | Too low for style transfer. Only captures task patterns, not voice. | | Num layers | 8 / 28 | 71% of the model is frozen. MLP layers (where style lives) are mostly untouched. | | Learning rate | 1e-5 | Too conservative. The adapter barely nudges the base model's distribution. | | System prompt | ~456 chars avg, up to 2055 chars | Massive context window consumed by tool call history. Dilutes the persona signal with noise. | The base Qwen2.5-7B-Instruct has deep instruct conditioning baked through all 28 transformer layers. With rank 16 on only 8 layers, you are trying to override a 7-billion-parameter personality with ~4M trainable parameters touching 28% of the network. The instruct persona dominates by default. - Sebastian Raschka's experiments (hundreds of LoRA runs): rank 256 "significantly improved performance" over rank 8. With r=256 and alpha=512, results matched full fine-tuning. Ranks above 256 (512, 1024) failed to converge. Source: https://magazine.sebastianraschka.com/p/practical-tips-for-finetuning-llms