GPU and VRAM Requirements for Llama Factory Fine-Tuning

Memory is the first thing to plan when fine-tuning with Llama Factory. The method you choose matters more than the model name. This article explains the estimates and how to pick a method that fits your card.

The Llama Factory memory estimate table

These numbers are estimates from the project README for a single device. They assume default sequence lengths and batch sizes.

A quick rule: LoRA needs about 2× the parameter count in GB, 8-bit QLoRA about 1×, and 4-bit QLoRA about half.

Common cards and what fits in Llama Factory

• 8 GB (RTX 3070, 4060): 4-bit QLoRA on 7B or 8B models. Keep cutoff_len around 1024 and batch size at 1 or 2.
• 12 GB (RTX 3060 12G, 4070): 4-bit QLoRA on 7B to 14B. 8-bit QLoRA on 7B is also possible.
• 16 GB (4080, T4 x1): LoRA on 7B in 16-bit, or 4-bit QLoRA up to 14B with room to spare.
• 24 GB (3090, 4090, A10): LoRA on 7B to 13B comfortably, 4-bit QLoRA on 30B class models.
• 48 GB and above (A6000, L40S, A100): LoRA on 30B, 4-bit QLoRA on 70B, or full fine-tuning of small models.

Llama Factory settings that reduce memory

quantization_bit: 4 is the biggest lever. After that, lower per_device_train_batch_size and raise gradient_accumulation_steps to keep the effective batch the same. Shorter cutoff_len values shrink activation memory. Gradient checkpointing is on by default.

FlashAttention-2 (flash_attn: fa2) helps on Ampere and newer cards. Liger Kernel and Unsloth lower memory further for supported models.

Multiple GPUs with Llama Factory

Llama Factory supports DDP, DeepSpeed ZeRO and FSDP. DeepSpeed ZeRO-3 with CPU offload lets several mid-range cards train models that would not fit on one. Example configs are in examples/train_full and examples/deepspeed.

Llama Factory on CPU only

Training on CPU is possible for very small models but slow. It is useful for checking a dataset or config, not for real runs.