🚦 TrafficFlowRL

Research-grade multi-agent reinforcement learning for smart city traffic optimisation.

TrafficFlowRL simulates a city grid of intersections where independent RL agents control traffic lights to minimise congestion, reduce vehicle wait times, and prioritise emergency vehicles — all trained end-to-end with Proximal Policy Optimization (PPO).

---

📋 Problem Description

Urban traffic congestion costs billions of hours and dollars annually. Traditional fixed-cycle traffic lights ignore real-time demand. TrafficFlowRL demonstrates how decentralised RL agents — one per intersection — can learn cooperative signal policies that:

• Minimise city-wide average wait time
• Maximise vehicle throughput
• Prioritise emergency vehicles (ambulances, fire trucks)
• Adapt to rush-hour patterns and weather conditions

---

🏗️ Architecture

trafficflowrl/
├── env/                          # Gymnasium environment
│   ├── city_env.py               # Main CityTrafficEnv (Gymnasium API)
│   ├── intersection.py           # Traffic light phases & queues
│   ├── vehicle.py                # Vehicle entity with pathfinding
│   ├── road_network.py           # Grid graph (NetworkX)
│   ├── traffic_generator.py      # Vehicle spawning & rush hour
│   ├── reward_system.py          # Multi-component reward function
│   └── state_encoder.py          # Observation normalisation
├── agents/
│   ├── ppo_agent.py              # Custom PPO with GAE
│   └── policy_network.py         # Actor-Critic neural network
├── training/
│   ├── train.py                  # Training loop with checkpointing
│   ├── evaluate.py               # Evaluation & statistics
│   └── config.py                 # Centralised hyper-parameters
├── visualization/
│   ├── city_visualizer.py        # Real-time Pygame renderer
│   └── training_dashboard.py     # Matplotlib metrics dashboard
├── utils/
│   ├── logger.py                 # CSV + JSON logging
│   └── seed.py                   # Reproducible seeding
├── requirements.txt
└── README.md

---

🌐 Environment Design

City Grid

The environment simulates a configurable N×N grid of intersections (default 3×3 = 9 intersections) connected by bidirectional roads. Vehicles spawn at city edges and follow the shortest path (BFS on the grid graph) to a random destination edge.

Observation Space

Each intersection produces a 7-dimensional observation vector:

Index	Feature	Range	Description
0	North Queue	[0, 1]	Normalised queue length
1	South Queue	[0, 1]	Normalised queue length
2	East Queue	[0, 1]	Normalised queue length
3	West Queue	[0, 1]	Normalised queue length
4	Light Phase	{0, 1}	0 = NS green, 1 = EW green
5	Phase Timer	[0, 1]	Normalised time since last switch
6	Emergency Flag	{0, 1}	1 if emergency vehicle in queue

Global observation = concatenation of all intersection vectors → shape (N*N*7,) = (63,) for a 3×3 grid.

Action Space

MultiDiscrete — one action per intersection:

Action	Meaning
0	Set North-South green
1	Set East-West green
2	Keep current phase

Actions have a minimum phase time constraint (default 3 steps) to prevent unrealistic rapid switching.

---

🏆 Reward Function

The reward is a weighted sum of five components, averaged across all intersections:

Component	Value	Condition
Throughput	+1.0	Per vehicle passing through intersection
Waiting penalty	−0.5	Per vehicle still waiting in queue
Emergency bonus	+5.0	Emergency vehicle cleared quickly (≤5 steps)
Emergency penalty	−5.0	Emergency vehicle waited too long
Switch penalty	−0.2	Phase change occurred this step

---

🧠 RL Algorithm — PPO

Policy Network (Actor-Critic)

Input (7) → Dense(256, ReLU) → Dense(256, ReLU) → Action Logits (3)
Input (7) → Dense(256, ReLU) → Dense(256, ReLU) → Value (1)

A shared architecture is used for all intersections — the same network weights process every intersection's local observation, enabling parameter-efficient scaling to larger grids.

Training Hyper-parameters

Parameter	Value
Episodes	1,000
Max steps/episode	300
Discount (γ)	0.99
GAE λ	0.95
Learning rate	3×10⁻⁴
Batch size	64
Clip range (ε)	0.2
Entropy coeff	0.01
Value coeff	0.5
PPO epochs/update	4

---

🎮 Bonus Features

• 🕐 Rush Hour Patterns — sinusoidal traffic demand increases
• 🌧️ Weather Effects — reduce vehicle speed dynamically
• 🚑 Multiple Emergency Vehicles — configurable probability
• 📈 Adaptive Traffic Demand — spawn rate changes over time

---

🚀 Quick Start

1. Install dependencies

pip install -r requirements.txt

2. Train the agent

cd trafficflowrl
python training/train.py

With bonus features:

python training/train.py --episodes 500 --rush-hour --weather --grid 3

3. Evaluate

python training/evaluate.py --checkpoint checkpoints/ppo_traffic_best.pt --episodes 20

4. Visualise (Pygame)

python visualization/city_visualizer.py --checkpoint checkpoints/ppo_traffic_best.pt

Controls: [R] Reset episode · [ESC] Quit

5. Training Dashboard (Matplotlib)

python visualization/training_dashboard.py --log-dir logs

Windows Convenience: run.bat

For Windows users a helper batch file is provided to run the common commands without typing long Python calls. From the project root, you can:

run.bat train       # starts training using training/train.py with default args
run.bat evaluate    # runs evaluation using training/evaluate.py
run.bat visualize   # launches the Pygame visualizer with the best checkpoint

Replace the subcommand with train, evaluate, or visualize. You can also pass additional CLI flags which will be forwarded to the underlying Python script (for example run.bat train --episodes 500).

---

📊 Example Visualizations

City Visualizer (Pygame)

The real-time visualizer renders:

• 🟢🔴 Traffic lights at each intersection with phase indicators
• 🚗 Moving vehicles as coloured dots on the road network
• 🚑 Emergency vehicles with red pulsing glow effects
• 📊 Stats panel showing step count, reward, queue lengths, and per-intersection details

Training Dashboard (Matplotlib)

The dashboard displays five smoothed metric plots:

• Episode reward curve (with moving average)
• Average wait time per episode
• Vehicles cleared per episode
• Policy loss convergence
• Value loss convergence
• Summary statistics panel

---

⚙️ Configuration

All hyper-parameters are centralised in training/config.py as a TrainingConfig dataclass. Modify defaults or pass CLI arguments:

from training.config import TrainingConfig

cfg = TrainingConfig(
    grid_rows=4,
    grid_cols=4,
    total_episodes=2000,
    enable_rush_hour=True,
    enable_weather=True,
    device="cuda",
)

---

📖 Technical Details

Vehicle Routing

Vehicles use BFS shortest path on the NetworkX grid graph. Each vehicle stores its planned path and advances one intersection per green-light timestep.

Emergency Priority

Emergency vehicles bypass the traffic light phase — they are always released from queues regardless of the current signal phase, simulating real-world signal pre-emption.

Multi-Agent Coordination

While agents are decentralised (each intersection acts independently), coordination emerges through:

1. Shared parameters — the same neural network processes all intersections
2. Global reward signal — averaged across intersections, encouraging cooperative behaviour
3. Information flow — queue build-ups from neighbouring intersections implicitly encode coordination signals

---

📜 License

MIT License — free for academic and commercial use.

---

Built for AI hackathon demonstrations and RL research. Contributions welcome!