L2 PvP Bot RL: Reinforcement Learning in MMORPG

L2J_MOBIUS_RL: ACTIVE // GYMNASIUM_1V1: Dion, Giran, Gludin // PETTINGZOO_7V7: Aden Coliseum // STATUS: STABLE

Traditional MMORPG bots are limited to static and lifeless action rotations. L2 PvP Bot RL is a reinforcement learning (RL) system that trains full Lineage 2 characters (instantiated as real players on the server side without a network client connected) to fight tactically and cooperate in groups. Training occurs continuously on the server without affecting the main tick loop (TPS).

Decoupled Training Architecture

The system decouples the real-time physics engine from the heavy AI computation:

  [ L2J GameServer (Java) ] (Inference / Capture)
               │
               ▼ (Episodes & Transitions)
        [ Database ] ◄──► [ RL Trainer (Python) ]
               ▲                      │
               └──────────────────────┴─► Model Injection (Q-Tables / TorchScript)

1. Local Inference & Episode Capture

The GameServer runs the combat loop. A specialized task manager periodically triggers the bot’s decision brain. Every relevant action (damage, healing, crowd control, failures, and outcomes) is recorded asynchronously in the database as episodes and transitions.

2. Isolated Neural Lab

For deep learning, a parallel server runs synchronized fights free from human interference:

1v1 Duels: Modeled using Gymnasium across multiple arenas.
7v7 Fights (Multi-agent): Modeled using PettingZoo in the Aden Coliseum. The optimizer executes PPO (Proximal Policy Optimization) on Ray/RLlib, training the model using a 177-value observation vector per character. It respects ranges, lines of sight, and stealth without exposing secret opponent stats.

3. Automatic Model Alternation

The scheduler automatically alternates the training type per generation: decision tables (tabular) and neural networks (neural). During a neural transition, the pipeline exports the actor to TorchScript, and a local CPU inference service handles the decisions.

Production Security & Resilience

AI policies cannot break the server:

Action Validation: The Java server validates that every selected action belongs to the bot’s original class skill tree before casting, preventing cross-class skill acquisition.
Fallback & Rollback: If the neural service disconnects or exhibits a low action acceptance rate, bots fall back to an artisan brain programmed in Java. If the failure persists, an automatic rollback restores the last stable tabular version.

Human Training and Challenge

The project is constantly evolving and directly benefits from real human behavior.

Join the server and test your skills against the AI in 1v1, 2v1, or 7v7 party combats. Your registered participation in combat episodes directly trains new generations of bots against diverse human playstyles.