← Back to Garden

From Colab Limits to Two-GPU DDP: Fine-Tuning Qwen3-8B on Kaggle

•
#qwen3#qlora#ddp#kaggle#home-assistant#unsloth#tuxbot

QWEN3 // QLORA // DISTRIBUTED TRAINING

The scene is a classic in model training: you set everything up before going to sleep to let the fine-tuning run overnight. Automatic checkpoints configured to Google Drive, dataset loaded, clean notebook on Google Colab. The next morning, eager to test the new model for controlling my home, I opened the browser to check progress only to meet the screen of death: the session had disconnected.

Upon trying to reconnect the 16 GB Tesla T4, a usage limit notification popped up, demanding an unspecified wait time without providing any metric on when the quota would reset. To be completely honest, hitting the quota limit wasn’t solely due to this Qwen3-8B test run. Prior to this, we had successfully fine-tuned a full Qwen2.5-Coder-7B-Instruct model on Colab. The issue was that after evaluating it, the results weren’t up to standard—it exhibited hallucinations and schema compliance failures when generating Home Assistant commands. Together with Tuxbot, we decided not to patch a flawed model, but to start from scratch by selecting a better base model. Colab’s limit simply arrived after several cumulative sessions of hard work. This isn’t about discouraging anyone from using Colab or bashing Google—it’s a great free tool—but about understanding how to operate when free tier limits inevitably interrupt your workflow.

Refusing to sit idle, I immediately began searching for alternatives. The priority was clear: exploit every free resource available online before considering paid options like Hugging Face Jobs. That’s how I landed on Kaggle.

This isn’t a hype-filled “train for free in 5 minutes” tutorial. It’s a real-world debugging log covering architecture decisions and distributed infrastructure to migrate my Home Assistant Specialist v0.2 model training to two GPUs on Kaggle, working side by side with my personal AI agent, Tuxbot.

The Tesla P100 Trap: Right VRAM, Wrong Architecture

My first impulse on Kaggle was selecting a Tesla P100 GPU environment. On paper, it offered 16 GB of VRAM—plenty to fit unsloth/Qwen3-8B-unsloth-bnb-4bit in 4-bit with QLoRA.

However, initializing the runtime revealed an inconvenient reality:

Python: 3.12.13
Torch: 2.10.0+cu128
CUDA: 12.8
GPU: Tesla P100-PCIE-16GB
Compute capability: 6.0 (sm_60)

The issue wasn’t memory capacity; it was a direct CUDA architecture incompatibility. The preinstalled PyTorch build on Kaggle required compute capabilities between sm_70 and sm_120. Being a Pascal architecture (sm_60), the P100 triggered explicit warnings stating the GPU was unsupported by the active PyTorch stack. Having 16 GB of unusable VRAM due to runtime mismatches is a quick lesson in inspecting infrastructure before launching code.

The Single T4 Bottleneck and the device_map="balanced" Myth

Dropping the P100, we switched to Kaggle’s T4x2 accelerator, providing two Tesla T4 GPUs with 16 GB each (14.56 GB usable VRAM and compute capability 7.5).

Running a single-GPU smoke test confirmed the expected slowness:

3 steps (single-GPU smoke test):
train_runtime: 431.37 s
train_steps_per_second: 0.007

After warm-up, execution speed barely nudged to 0.015 steps/s. Projecting 1 epoch over the tuxevil/Home-Assistant-Requests-V3 dataset (3,561 training examples) pointed to an 8 to 9-hour wait.

That’s when using device_map="balanced" came up. Here is a myth worth busting: Model Parallelism (splitting the model across GPUs) is not a speed optimization. As explained in Hugging Face’s GPU Parallelism guide, splitting Qwen3-8B layers between two T4s adds sequential communication latency over the virtual machine’s PCIe bus; one GPU stays idle while waiting for the other to finish processing its assigned layer segment.

Since Qwen3-8B in 4-bit with LoRA adapters comfortably fits within the VRAM of a single T4, the right strategy was not model splitting, but Distributed Data Parallel (DDP) guided by Unsloth DDP Documentation: keeping a complete model copy loaded on each GPU, processing distinct batches concurrently, and synchronizing gradients only.

Converting the Notebook to a Standalone DDP Script via torchrun

torchrun (the distributed launcher from PyTorch Distributed) cannot be executed directly over interactive Jupyter Notebook cells. To launch 2 isolated processes, Tuxbot structured a standalone script: train_ddp.py.

Key adjustments required to survive Kaggle’s environment included:

  1. Per-process isolation: Reading LOCAL_RANK from environment variables so each process targets its assigned GPU without forcing CUDA_VISIBLE_DEVICES.
  2. IO Control: Ensuring only rank 0 writes local checkpoints and pushes adapters to Hugging Face Hub.
  3. Memory and Precision Management: Forcing FP16=True, BF16=False (T4 GPUs lack efficient native bfloat16 support), and setting ddp_find_unused_parameters=False to reduce gradient graph overhead.
  4. Clean NCCL Teardown: Adding explicit calls to torch.distributed.barrier() and torch.distributed.destroy_process_group() upon exit.

The launch command inside the notebook cell was:

torchrun --standalone --nproc_per_node=2 train_ddp.py

DDP Smoke Test and Warning Fixes

The DDP smoke test verified that distributed execution was active and healthy:

Rank 0: local_rank 0, Tesla T4
Rank 1: local_rank 1, Tesla T4
Data Parallel GPUs: 2
Batch per GPU: 1
Gradient accumulation steps: 8
Global batch size: 16
Trainable parameters: 43,646,976 out of 5,235,454,976

During the 3-step DDP test, execution time dropped significantly:

global_step: 3
train_runtime: 57.41 s
train_samples_per_second: 0.836
train_steps_per_second: 0.052

Jumping from 0.015 to 0.052 steps/s clearly demonstrated data parallelism’s advantage. Along the way, we resolved warnings like use_return_dict is deprecated by setting model.config.return_dict = True, and safely ignored benign internal socket warnings (hostname of client socket cannot be retrieved).

Unsloth accurately reported: Num GPUs used = 1, Data Parallel GPUs = 2, indicating that each process managed 1 local GPU while coordinating via NCCL.

Physical Reality: Cloud Free Tiers vs. Local Solar Energy

People often ask why not do all fine-tuning locally. The answer isn’t just software or VRAM—it’s physical and energy constraints at home.

Running local fine-tuning on my GPU at 100% utilization draws roughly 125W continuously. On my battery backup system, running that load overnight isn’t sustainable without risking power continuity for the rest of my homelab nodes on Proxmox. My solar production window with genuine excess energy occurs during the day, between 8:00 AM and 5:00 PM.

Therefore, my approach as a tech explorer is pragmatic:

  • Free Cloud (Kaggle / Colab): Primary option for multi-hour heavy jobs without draining my battery bank or incurring extra electricity costs.
  • Local Training: Strategic fallback for when free tiers fail or become unavailable, scheduled strictly within my solar window.

Current State and Next Steps

The full training for Home Assistant Specialist v0.2 on the ha-action-v3 contract was triggered from Kaggle cell 11 and continues running in the background. Adhering to our rule of not fabricating metrics, final loss numbers, total runtime, and evaluation metrics will be analyzed once execution finishes completely.

Once the adapter is ready:

  1. We will quantize the model into GGUF format using llama.cpp.
  2. Serve it via Ollama on the local RTX 4000 GPU.
  3. Tuxbot will run a live benchmark against our baseline (qwen2.5-coder-baseline) to verify if schema hallucination for Home Assistant has been eliminated.

The ultimate goal remains unchanged: building local, highly specialized, and resilient AI, maximizing every free resource along the way.