Training PvP Bots in Lineage 2: Gymnasium, PettingZoo and PPO

BOT_AI_TRAIN: IDLE // CURRENT_STABLE: v8_TABULAR // NEXT_CHALLENGER: v9 // LAB_QUEUE: PYTHON_PPO

Linking modern Machine Learning algorithms with classic PC video games is a software survival exercise. For our evolutionary Lineage 2 Ertheia PvP tournament server, the goal was not just to automate keystrokes, but to inject server-side tactical decision-making without degrading the main Java-based GameServer’s Ticks Per Second (TPS).

Here we share the key technical lessons learned after collecting millions of combat transitions and consolidating our stable production version.

1. The Cost of Ticks: Decoupled Inference

Running direct neural network inference on the GameServer’s main thread causes immediate lag. The implemented infrastructure moves training to an isolated laboratory with its own simulation database. The loop runs on three levels:

Artisan Behavior (Java): Manual brains that control the minimum viable actions of each class (e.g., a healer prioritizing ally heals and resurrections).
Tabular Learning (Q-Table/SQL): An asynchronous trainer analyzes transitions periodically and calculates a weight matrix based on the tactical context (SOLO, PARTY_SUPPORT, ADVANTAGE, OUTNUMBERED).
Neural Learning (PPO): Executed in asynchronous cycles using Gymnasium to abstract individual duels and PettingZoo for coordinated group combat in 7v7 formats. Training runs on Ray/RLlib, limiting CPU and memory resources to guarantee host stability.

The neural network exports the model to TorchScript, which is served by a local CPU inference microservice with sub-100ms response latencies.

2. Blocking Cross-Class Auto-Learning

During the initial simulation phases, the bots exhibited erratic behavior. Mage or assassin characters started executing rapid dodge jumps and physical charges. These were movements exclusive to the archer class.

When debugging the casting flow, we found that if the combat logic invoked an incorrect skill due to a mapping bug, the bot’s context layer dynamically fetched the missing skill from the database and permanently added it to the character to prevent execution crashes. Bots were learning illegal skills through probabilistic AI exploration.

The solution required isolating tactical kiting within the archer engine and hardening skill acquisition security. Now, the bot’s dynamic auto-learning requires the skill to belong to the original class tree or be explicitly validated in the server’s authorized brain registry.

3. Model Promotion and Integrity

A model is not promoted to production based on a good theoretical metric. The system forces a paired combat evaluation phase (Self-play):

The current champion and the challenger face each other in hundreds of duel and group matches, alternating sides of the arena to negate spatial bias.
The challenger must overcome a rigorous statistical win-rate threshold before being promoted.

Version v8 of the tabular model was promoted to stable after beating version v6 in a 1,027-combat simulation, securing a 55.89% decisive win rate. The current challenger v9 is under evaluation. Once its matches are complete, the automatic scheduler will reserve the slot to start the first production neural policy trained with PPO.

4. Human Participation

No combat simulation can replicate the unpredictability of a real player. Therefore, the server securely captures the demonstrations and combat data of registered players, incorporating them as high-priority training data for future generations.

We invite you to try our Lineage 2 server, participate in the arenas, and help us train the AI by facing it in 1v1, 2v1, or 7v7 party battles in the Coliseum.