多智能体深度确定性策略梯度(MADDPG)算法介绍及代码实现

多智能体深度确定性策略梯度（Multi-Agent Deep Deterministic Policy Gradient, MADDPG）算法是一种在多智能体环境中使用的强化学习算法。这种算法是基于深度确定性策略梯度（DDPG）算法的扩展。MADDPG主要用于解决多智能体环境中的协作和竞争问题，特别是在智能体之间的交互可能非常复杂的情况下。下面将详细介绍MADDPG算法的核心概念和工作原理。

一、基础理论

在介绍MADDPG之前，需要理解其基础——DDPG算法。DDPG是一种结合了深度学习和强化学习的算法，用于连续动作空间的问题。DDPG使用了策略梯度方法和Q学习（一种值函数近似方法）的结合，通过学习一个确定性策略来解决复杂的决策问题。

MADDPG的核心思想

MADDPG考虑了多智能体环境的动态性和复杂性。在多智能体环境中，每个智能体的行为不仅取决于环境的状态，还受到其他智能体策略的影响。MADDPG通过对每个智能体采用一个独立的Actor-Critic架构，并在训练过程中考虑其他智能体的策略信息，来改善学习效果和稳定性。

算法细节

1. Actor-Critic架构：每个智能体都有一个Actor网络用于输出动作，以及一个Critic网络用于评估当前策略的好坏。Actor直接学习确定性策略，而Critic负责估算状态-动作对的Q值。
2. 集中式训练，分布式执行：在训练阶段，Critic网络可以访问所有智能体的信息，包括状态和动作，这允许它准确评估每个动作的期望回报。然而，在执行阶段，每个智能体的Actor网络只能基于自己的局部观察来做出决策。
3. 经验回放：为了提高训练的稳定性和效率，MADDPG使用了经验回放机制。智能体的每次交互会被存储在一个回放缓冲区中，训练时会从这个缓冲区中随机抽取一批经验来更新网络。
4. 目标网络：为了进一步稳定训练过程，MADDPG为每个Actor和Critic网络维护了一个目标网络。这些目标网络的参数会缓慢跟踪对应网络的参数，用于计算期望回报的稳定目标。
5. 奖励和惩罚：MADDPG允许设计复杂的奖励机制，包括对合作行为的奖励和对对立行为的惩罚，来引导智能体学习如何在多种交互场景中作出最优决策。

伪代码

初始化： 对于每个智能体i：     初始化actor网络π_i和critic网络Q_i，以及它们的目标网络π'_i和Q'_i。     初始化经验回放缓冲区D。  重复（对于每个episode）：     初始化环境状态S     重复（对于每个时间步）：         对于每个智能体i：             根据当前策略π_i和状态S观察o_i，选择动作a_i         执行所有智能体的动作[a_1, ..., a_N]，观察新状态S'和奖励R         对于每个智能体i，将转换(t = (o_i, a_i, R_i, o'_i))存储到D中         对于每个智能体i：             从D中随机采样一批转换             对于每个采样的转换：                 使用目标网络计算目标Q值             更新critic网络Q_i，最小化损失：L = (Q_i(o_i, a_i) - 目标Q)^2             更新actor网络π_i，使用策略梯度         对于每个智能体i：             软更新目标网络参数：π'_i ← τπ_i + (1 - τ)π'_i                              Q'_i ← τQ_i + (1 - τ)Q'_i     直到环境结束本episode 重复，直到满足终止条件 

与其他强化学习算法的不用点

MADDPG（多智能体深度确定性策略梯度）算法是多智能体强化学习领域的一个重要算法，它针对的是连续动作空间问题，并且特别适用于环境中存在多个智能体互动（合作、竞争或两者兼有）的情况。以下是MADDPG与其他几种多智能体强化学习算法的比较：

1. DDPG（深度确定性策略梯度）

• 相同点：MADDPG基于DDPG算法，都采用了Actor-Critic架构，利用深度学习技术处理高维状态和动作空间，并通过目标网络和经验回放提高学习稳定性。
• 不同点：DDPG是为单一智能体设计的，而MADDPG扩展了DDPG，使其适用于多智能体环境。在MADDPG中，每个智能体都有自己的Actor和Critic网络，Critic在训练时能够访问所有智能体的信息，这有助于在存在其他智能体的环境中做出更好的决策。

2. Q-Learning和DQN（深度Q网络）

• 相同点：这些算法都属于值基础的强化学习方法，通过学习一个值函数来间接地确定最优策略。
• 不同点：Q-Learning和DQN主要针对离散动作空间设计，而MADDPG处理连续动作空间。DQN通过引入深度学习来处理高维状态空间，但它适用于单一智能体。MADDPG则专门解决多智能体环境中的问题，允许智能体在训练时考虑其他智能体的策略，更适用于复杂的交互场景。

3. MARL算法中的其他方法，如VDN（值分解网络）和QMIX

• VDN和QMIX：这些算法专注于如何在多智能体设置中结合或分解智能体的值函数，以便学习到高效的协作策略。
• 相同点：MADDPG、VDN和QMIX都旨在处理多智能体环境中的问题，强调智能体间的协作或竞争。
• 不同点：VDN和QMIX采用的是值分解的方式来解决多智能体的协作问题，适用于离散动作空间。这些方法通过分解全局值函数来简化学习过程。而MADDPG采用的是策略梯度方法，并直接在连续动作空间中工作，更适合需要精确控制的应用场景。

应用场景

MADDPG适用于各种多智能体场景，包括但不限于：

• 合作控制任务，如多无人机编队飞行。
• 竞争游戏，例如多玩家在线游戏中的对抗。
• 混合动作环境，其中智能体需要同时考虑合作和竞争行为。

二、代码实现

环境介绍：simple_adversary_v3

这是一个合作与竞争的环境。

• 环境基本信息

  • 种类：两种（友方，敌方）
  • 参数：位置坐标（pos），速度（vel），传递信息（c）。 (有个参数叫agent.silent,等于True就是没有信息传递（保持安静）)
  • 环境中的实体（landmark）
  • 参数：位置坐标，速度（有的环境地表会移动，但这个环境都是静止的）

• • 智能体观测信息：（observation）

  • • if agent.adversary == True: 一个numpy.array,[地标距离自己的距离(4)，其他智能体距自己的距离(4)]
    • if agent.adversary == False: 一个numpy.array,[智能体自己的坐标(2)，智能体距目标的相对距离（2），地标距离自己的距离(2)，其他智能体距自己的距离(4)]

  • 奖励函数

  • • bad：自己距离目标的距离。
    • good：可以由两个因素决定，友方距离目标的距离，越近越好；敌方距离目标的距离，越远越好。

main.py

""" #!/usr/bin/env python # -*- coding:utf-8 -*- @Project : MADDPG """  from pettingzoo.mpe import simple_adversary_v3 import numpy as np import torch import torch.nn as nn import os import time  from maddpg_agent import Agent  torch.autograd.set_detect_anomaly(True) device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu") print(f"Using device:{device}")   def multi_obs_to_state(multi_obs):     state= np.array([])     for agent_obs in multi_obs.values():         state = np.concatenate([state, agent_obs])     return  state  NUM_EPISODE = 1000 NUM_STEP = 100 MEMORY_SIZE = 10000 BATCH_SIZE = 512 TARGET_UPDATE_INTERVAL = 200  LR_ACTOR = 0.001 LR_CRITIC = 0.001 HIDDEN_DIM = 64 GAMMA = 0.99 TAU = 0.01 scenario  = "simple_adversary_v3" current_path = os.path.dirname(os.path.realpath(__file__)) agent_path = current_path + "\\" +"models"+ "\\" + scenario + "\\" timestamp = time.strftime("%Y%m%d%H%M%S")   # 1. initialize the agent # 初始化环境 env = simple_adversary_v3.parallel_env(N=2, max_cycles= NUM_STEP, continuous_actions= True) multi_obs, infos = env.reset() NUM_AGENT = env.num_agents agent_name_list = env.agents  # 1.1 get obs_dim obs_dim = [] for agent_obs in multi_obs.values():     obs_dim.append(agent_obs.shape[0]) state_dim = sum(obs_dim)  # 1.2 get action_dim action_dim = [] for agent_name in agent_name_list:     action_dim.append(env.action_space(agent_name).sample().shape[0])  agents=[] # 实例化多个智能体 for agent_i in range(NUM_AGENT):     print(f"Initializing agent {agent_i}.....")     agent = Agent( memo_size=MEMORY_SIZE, obs_dim=obs_dim[agent_i], state_dim= state_dim,                    n_agent = NUM_AGENT, action_dim = action_dim[agent_i], alpha=LR_ACTOR                  ,beta= LR_CRITIC, fc1_dims = HIDDEN_DIM, fc2_dims=HIDDEN_DIM,                    gamma = GAMMA, tau=TAU , batch_size=BATCH_SIZE)     agents.append(agent)  # 2. Main training loop for episode_i in range(NUM_EPISODE):     multi_obs, infos = env.reset()     episode_reward = 0     mlti_done = {agent_name:False for agent_name in agent_name_list}     for step_i in range(NUM_STEP):         total_step = episode_i*NUM_STEP+step_i         # 2.1 collecting action from all agents         multi_actions ={} # 用于存储动作集合         for agent_i, agent_name in enumerate(agent_name_list):             agent = agents[agent_i]             single_obs = multi_obs[agent_name]             single_action = agent.get_action(single_obs)             multi_actions[agent_name] = single_action          # 2.2 executing actions，         multi_next_obs, multi_reward, multi_done, multi_truncations, infos = env.step(multi_actions)         state= multi_obs_to_state(multi_obs)         next_state = multi_obs_to_state(multi_next_obs)          if step_i >= NUM_STEP -1:             multi_done = {agent_name: True for agent_name in agent_name_list}          #2.3 store memory         for agent_i, agent_name in enumerate(agent_name_list):             agent = agents[agent_i]             single_obs = multi_obs[agent_name]             single_next_obs = multi_next_obs[agent_name]             single_action = multi_actions[agent_name]             single_reward = multi_reward[agent_name]             single_done = multi_done[agent_name]             # 存储到经验池中             agent.replay_buffer.add_memo(single_obs,single_next_obs,state, next_state, single_action,single_reward,single_done)          #2.4 Update brain every fixed step         multi_batch_obses=[]         multi_batch_next_obses =[]         multi_batch_states = []         multi_batch_next_states = []         multi_batch_actions = []         multi_batch_next_actions =[]         multi_batch_online_actions =[]         multi_batch_rewards =[]         multi_batch_dones = []          #2.4.1 sample a batch of memories         current_memo_size = min (MEMORY_SIZE, total_step+1)         if current_memo_size < BATCH_SIZE:             batch_idx = range(0, current_memo_size)         else:             batch_idx = np.random.choice(current_memo_size,BATCH_SIZE)          for agent_i in range(NUM_AGENT):             agent = agents[agent_i]             batch_obses, batch_next_obses, batch_states,batch_next_state, batch_actions,batch_rewards, batch_dones = agent.replay_buffer.sample(batch_idx)              batch_obses_tensor = torch.tensor(batch_obses,dtype=torch.float).to(device)             batch_next_obses_tensor = torch.tensor(batch_next_obses,dtype=torch.float).to(device)             batch_states_tensor = torch.tensor(batch_states,dtype=torch.float).to(device)             batch_next_state_tensor = torch.tensor(batch_next_state,dtype=torch.float).to(device)             batch_actions_tensor = torch.tensor(batch_actions,dtype=torch.float).to(device)             batch_rewards_tensor = torch.tensor(batch_rewards,dtype=torch.float).to(device)             batch_done_tensor = torch.tensor(batch_dones,dtype=torch.float).to(device)               multi_batch_obses.append(batch_obses_tensor)             multi_batch_next_obses.append(batch_next_obses_tensor)             multi_batch_states.append(batch_states_tensor)             multi_batch_next_states.append(batch_next_state_tensor)             multi_batch_actions.append(batch_actions_tensor)              single_batch_next_actions = agent.target_actor.forward(batch_next_obses_tensor)             multi_batch_next_actions.append(single_batch_next_actions)             single_batch_online_action = agent.actor.forward(batch_obses_tensor)             multi_batch_online_actions.append(single_batch_online_action)              multi_batch_rewards.append(batch_rewards_tensor)             multi_batch_dones.append(batch_done_tensor)          multi_batch_actions_tensor = torch.cat(multi_batch_actions, dim=1).to(device)         multi_batch_next_actions_tensor = torch.cat(multi_batch_next_actions, dim=1).to(device)         multi_batch_online_actions_tensor = torch.cat(multi_batch_online_actions, dim=1).to(device)          if(total_step+1) % TARGET_UPDATE_INTERVAL == 0:              for agent_i in range(NUM_AGENT):                 agent = agents[agent_i]                  batch_obses_tensor = multi_batch_obses[agent_i]                 batch_states_tensor = multi_batch_states[agent_i]                 batch_next_states_tensor = multi_batch_next_states[agent_i]                 batch_rewards_tensor =multi_batch_rewards [agent_i]                 batch_dones_tensor =multi_batch_dones [agent_i]                 batch_actions_tensor =multi_batch_actions [agent_i]                  #target critic                 critic_target_q = agent.target_critic.forward(batch_next_state_tensor                                                               ,multi_batch_next_actions_tensor.detach())                 y = (batch_rewards_tensor + (1-batch_dones_tensor)*agent.gamma*critic_target_q).flatten()                 critic_q = agent.critic.forward(batch_states_tensor,multi_batch_actions_tensor.detach()).flatten()                 #update critic                 critic_loss = nn.MSELoss()(y,critic_q)                 agent.critic.optimizer.zero_grad()                 critic_loss.backward()                 agent.critic.optimizer.step()                  #update actor                 actor_q = agent.critic.forward(batch_states_tensor,                                                   multi_batch_online_actions_tensor.detach()).flatten()                 actor_loss = -torch.mean(actor_q)                 agent.actor.optimizer.zero_grad()                 actor_loss.backward()                 agent.actor.optimizer.step()                  # update target critic                 for target_param, param in zip(agent.target_critic.parameters(),                                                agent.critic.parameters()):                     target_param.data.copy_(agent.tau * param.data+(1.0-agent.tau)*target_param.data)                 # update target actor                 for target_param, param in zip(agent.target_actor.parameters(),                                                agent.actor.parameters()):                     target_param.data.copy_(agent.tau * param.data+(1.0-agent.tau)*target_param.data)          multi_obs = multi_next_obs         episode_reward += sum([single_reward for single_reward in multi_reward.values()])         print(f"episode reward :{episode_reward}")      # 3.Render the env     if(episode_i +1) % 50 == 0:         env= simple_adversary_v3.parallel_env(N=2,                                               max_cycles=NUM_STEP,                                               continuous_actions = True,                                               render_mode = "human")         for test_epi_i in range(2):             multi_obs, infos = env.reset()             for step_i in range(NUM_STEP):                 multi_actions={}                 for agent_i, agent_name in enumerate(agent_name_list):                     agent = agents[agent_i]                     single_obs = multi_obs[agent_name]                     single_action = agent.get_action(single_obs)                     multi_actions[agent_name] = single_action                 multi_next_obs, multi_reward, multi_done, multi_truncations, infos = env.step(multi_actions)                 multi_obs = multi_next_obs     # Save the agents     if episode_i == 0:         highest_reward = episode_reward     if episode_reward >highest_reward:         highest_reward=episode_reward         print(f"Highest reward update at episode {episode_i}:{round(highest_reward,2)}")         for agent_i in range(NUM_AGENT):             agent = agents[agent_i]             flag = os.path.exists(agent_path)             if not flag:                 os.makedirs(agent_path)             torch.save(agent.actor.state_dict(),f"models"+"\\"+"simple_adversary_v3"+"\\"+f"agent_{agent_i}_actor_{scenario}_{timestamp}.pth") env.close()   

maddpg_agent.py

""" #!/usr/bin/env python # -*- coding:utf-8 -*- @Project : MADDPG """ import numpy as np import torch import torch.nn as nn import torch.nn.functional as F  device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")  class ReplayBuffer:     def __init__(self, capcity, obs_dim, state_dim, action_dim, batch_size):         self.capcity = capcity         self.obs_cap =  np.empty((self.capcity,obs_dim))         self.next_obs_cap = np.empty((self.capcity,obs_dim))         self.state_cap = np.empty((self.capcity,state_dim))         self.next_state_cap = np.empty((self.capcity,state_dim))         self.action_cap = np.empty((self.capcity,action_dim))         self.reward_cap = np.empty((self.capcity,1))         self.done_cap = np.empty((self.capcity,1))          self.batch_batch = batch_size         self.current = 0      def add_memo(self, obs, next_obs, state, next_state, action, reward, done):         self.obs_cap[self.current] =obs         self.next_obs_cap[self.current] =next_obs         self.state_cap[self.current] =state         self.next_state_cap[self.current] =next_state         self.action_cap[self.current] =action         self.reward_cap[self.current] =reward          self.done_cap[self.current] =done          self.current = (self.current + 1) % self.capcity     #get one sample     def sample(self,idxes):         obs = self.obs_cap[idxes]         next_obs = self.next_obs_cap[idxes]         state = self.state_cap[idxes]         next_state = self.next_state_cap[idxes]         action = self.action_cap[idxes]         reward = self.reward_cap[idxes]         done = self.done_cap[idxes]         return obs,next_obs,state,next_state,action,reward,done   class Critic(nn.Module):     def __init__(self, lr_critic, input_dims, fc1_dims, fc2_dims,n_agent,action_dim):         super(Critic, self).__init__()         self.fc1 = nn.Linear(input_dims+n_agent*action_dim,fc1_dims)         self.fc2 = nn.Linear(fc1_dims,fc2_dims)         self.q = nn.Linear(fc2_dims, 1)          self.optimizer = torch.optim.Adam(self.parameters(),lr=lr_critic)      def forward(self, state,action):         x= torch.cat([state,action],dim=1)         x = F.relu(self.fc1(x))         x = F.relu(self.fc2(x))         q = self.q(x)         return q       def save_checkpoint(self, checkpoint_file):         torch.save(self.state_dict(), checkpoint_file)      def load_checkpoint(self, checkpoint_file):         self.load_state_dict(torch.load(checkpoint_file))   class Actor(nn.Module):     def __init__(self, lr_actor, input_dims, fc1_dims, fc2_dims,action_dim):         super(Actor, self).__init__()         self.fc1 = nn.Linear(input_dims,fc1_dims)         self.fc2 = nn.Linear(fc1_dims,fc2_dims)         self.pi = nn.Linear(fc2_dims, action_dim)          self.optimizer = torch.optim.Adam(self.parameters(),lr=lr_actor)      def forward(self, state):         x = F.relu((self.fc1(state)))         x = F.relu((self.fc2(x)))         mu = torch.softmax(self.pi(x), dim=1)         return mu      def save_checkpoint(self, checkpoint_file):         torch.save(self.state_dict(), checkpoint_file)      def load_checkpoint(self, checkpoint_file):         self.load_state_dict(torch.load(checkpoint_file))    class Agent:     def __init__(self, memo_size, obs_dim, state_dim, n_agent, action_dim, alpha                  ,beta, fc1_dims, fc2_dims, gamma, tau , batch_size):         self.gamma = gamma         self.tau = tau         self.action_dim = action_dim          self.actor = Actor(lr_actor=alpha, input_dims=obs_dim,                            fc1_dims=fc1_dims, fc2_dims=fc2_dims,                            action_dim=action_dim).to(device)          self.critic = Critic(lr_critic=beta, input_dims=state_dim,                              fc1_dims=fc1_dims, fc2_dims=fc2_dims,                              n_agent=n_agent,action_dim=action_dim).to(device)           self.target_actor = Actor(lr_actor=alpha, input_dims=obs_dim,                            fc1_dims=fc1_dims, fc2_dims=fc2_dims,                            action_dim=action_dim).to(device)          self.target_critic = Critic(lr_critic=beta, input_dims=state_dim,                              fc1_dims=fc1_dims, fc2_dims=fc2_dims,                              n_agent=n_agent, action_dim=action_dim).to(device)          self.replay_buffer = ReplayBuffer(capcity=memo_size, obs_dim=obs_dim, state_dim=state_dim,                                           action_dim=action_dim, batch_size=batch_size)     def get_action(self, obs):         single_obs = torch.tensor(data=obs, dtype=torch.float).unsqueeze(0).to(device)         single_action = self.actor.forward(single_obs)         noise = torch.randn(self.action_dim).to(device)*0.2         single_action = torch.clamp(input=single_action+noise, min=0.0, max=1.0)         return single_action.detach().cpu().numpy()[0]      def save_model(self,filename):         self.actor.save_checkpoint(filename)         self.target_actor.save_checkpoint(filename)         self.critic.save_checkpoint(filename)         self.target_critic.save_checkpoint(filename)      def load_model(self,filename):         self.actor.load_checkpoint(filename)         self.target_actor.load_checkpoint(filename)         self.critic.load_checkpoint(filename)         self.target_critic.load_checkpoint(filename)