wandb: Currently logged in as: 804703098. Use `wandb login --relogin` to force relogin
wandb: wandb version 0.17.5 is available!  To upgrade, please run:
wandb:  $ pip install wandb --upgrade
wandb: Tracking run with wandb version 0.16.6
wandb: Run data is saved locally in /home/user/zhangyang/PycharmProjects/Nips2024-ITPC-v2/Nips2024-ITPC-v2/onpolicy/scripts/results/StarCraft2/27m_vs_30m/mappotrsyn/train_27m_vs_30m_sigma_v0_KLCP_s2r2_v0/wandb/run-20240803_200649-7690pulv
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run StarCraft2_27m_vs_30m_1
wandb: ⭐️ View project at https://wandb.ai/804703098/27m_vs_30m
wandb: 🚀 View run at https://wandb.ai/804703098/27m_vs_30m/runs/7690pulv
choose to use gpu...
Version: B75689 (SC2.4.10)
Build: Aug 12 2019 17:16:57
Version: B75689 (SC2.4.10)
Build: Aug 12 2019 17:16:57
Command Line: '"/home/user/StarCraftII/Versions/Base75689/SC2_x64" -listen 127.0.0.1 -port 37015 -dataDir /home/user/StarCraftII/ -tempDir /tmp/sc-eub3jxic/'
Command Line: '"/home/user/StarCraftII/Versions/Base75689/SC2_x64" -listen 127.0.0.1 -port 38499 -dataDir /home/user/StarCraftII/ -tempDir /tmp/sc-mj7zpzp4/'
Version: B75689 (SC2.4.10)
Build: Aug 12 2019 17:16:57
Version: B75689 (SC2.4.10)
Build: Aug 12 2019 17:16:57
Command Line: '"/home/user/StarCraftII/Versions/Base75689/SC2_x64" -listen 127.0.0.1 -port 43815 -dataDir /home/user/StarCraftII/ -tempDir /tmp/sc-xytaqlst/'
Command Line: '"/home/user/StarCraftII/Versions/Base75689/SC2_x64" -listen 127.0.0.1 -port 33301 -dataDir /home/user/StarCraftII/ -tempDir /tmp/sc-or_53waf/'
Version: B75689 (SC2.4.10)
Version: B75689 (SC2.4.10)
Version: B75689 (SC2.4.10)
Build: Aug 12 2019 17:16:57
Build: Aug 12 2019 17:16:57
Build: Aug 12 2019 17:16:57
Version: B75689 (SC2.4.10)
Build: Aug 12 2019 17:16:57
Command Line: '"/home/user/StarCraftII/Versions/Base75689/SC2_x64" -listen 127.0.0.1 -port 46531 -dataDir /home/user/StarCraftII/ -tempDir /tmp/sc-gpl42un4/'
Command Line: '"/home/user/StarCraftII/Versions/Base75689/SC2_x64" -listen 127.0.0.1 -port 39951 -dataDir /home/user/StarCraftII/ -tempDir /tmp/sc-b_n0hzc3/'
Command Line: '"/home/user/StarCraftII/Versions/Base75689/SC2_x64" -listen 127.0.0.1 -port 39997 -dataDir /home/user/StarCraftII/ -tempDir /tmp/sc-3ri65q6s/'
Command Line: '"/home/user/StarCraftII/Versions/Base75689/SC2_x64" -listen 127.0.0.1 -port 43299 -dataDir /home/user/StarCraftII/ -tempDir /tmp/sc-wmtorgg2/'
Starting up...
Starting up...
Starting up...
Starting up...
Starting up...
Starting up...
Starting up...
Starting up...
Startup Phase 1 complete
Startup Phase 1 complete
Startup Phase 1 complete
Startup Phase 1 complete
Startup Phase 1 complete
Startup Phase 1 complete
Startup Phase 1 complete
Startup Phase 1 complete
Startup Phase 2 complete
Creating stub renderer...
Startup Phase 2 complete
Creating stub renderer...
Startup Phase 2 complete
Creating stub renderer...
Startup Phase 2 complete
Creating stub renderer...
Startup Phase 2 complete
Creating stub renderer...
Startup Phase 2 complete
Creating stub renderer...
Startup Phase 2 complete
Creating stub renderer...
Startup Phase 2 complete
Creating stub renderer...
Listening on: 127.0.0.1:43299
Listening on: 127.0.0.1:39951
Startup Phase 3 complete. Ready for commands.
Startup Phase 3 complete. Ready for commands.
Listening on: 127.0.0.1:46531
Startup Phase 3 complete. Ready for commands.
Listening on: 127.0.0.1:33301
Listening on: 127.0.0.1:43815
Startup Phase 3 complete. Ready for commands.
Startup Phase 3 complete. Ready for commands.
Listening on: 127.0.0.1:38499
Startup Phase 3 complete. Ready for commands.
Listening on: 127.0.0.1:37015
Startup Phase 3 complete. Ready for commands.
Listening on: 127.0.0.1:39997
Startup Phase 3 complete. Ready for commands.
ConnectHandler: Request from 127.0.0.1:36174 accepted
ConnectHandler: Request from 127.0.0.1:55858 accepted
ReadyHandler: 127.0.0.1:36174 ready
ReadyHandler: 127.0.0.1:55858 ready
ConnectHandler: Request from 127.0.0.1:33372 accepted
ReadyHandler: 127.0.0.1:33372 ready
ConnectHandler: Request from 127.0.0.1:41354 accepted
ReadyHandler: 127.0.0.1:41354 ready
ConnectHandler: Request from 127.0.0.1:54728 accepted
ConnectHandler: Request from 127.0.0.1:44990 accepted
ConnectHandler: Request from 127.0.0.1:53454 accepted
ReadyHandler: 127.0.0.1:54728 ready
ReadyHandler: 127.0.0.1:44990 ready
ReadyHandler: 127.0.0.1:53454 ready
ConnectHandler: Request from 127.0.0.1:38902 accepted
ReadyHandler: 127.0.0.1:38902 ready
Requesting to join a single player game
Configuring interface options
Configure: raw interface enabled
Configure: feature layer interface disabled
Configure: score interface disabled
Configure: render interface disabled
Launching next game.
Next launch phase started: 2
Next launch phase started: 3
Next launch phase started: 4
Next launch phase started: 5
Next launch phase started: 6
Next launch phase started: 7
Next launch phase started: 8
Requesting to join a single player game
Configuring interface options
Configure: raw interface enabled
Configure: feature layer interface disabled
Configure: score interface disabled
Configure: render interface disabled
Requesting to join a single player game
Configuring interface options
Configure: raw interface enabled
Configure: feature layer interface disabled
Configure: score interface disabled
Configure: render interface disabled
Launching next game.
Launching next game.
Next launch phase started: 2
Next launch phase started: 3
Next launch phase started: 2
Next launch phase started: 3
Requesting to join a single player game
Configuring interface options
Configure: raw interface enabled
Configure: feature layer interface disabled
Configure: score interface disabled
Configure: render interface disabled
Launching next game.
Next launch phase started: 4
Next launch phase started: 5
Next launch phase started: 6
Next launch phase started: 7
Next launch phase started: 8
Next launch phase started: 4
Next launch phase started: 5
Next launch phase started: 6
Next launch phase started: 7
Next launch phase started: 8
Next launch phase started: 2
Next launch phase started: 3
Next launch phase started: 4
Next launch phase started: 5
Next launch phase started: 6
Next launch phase started: 7
Next launch phase started: 8
Requesting to join a single player game
Requesting to join a single player game
Configuring interface options
Configuring interface options
Configure: raw interface enabled
Configure: feature layer interface disabled
Configure: raw interface enabled
Configure: score interface disabled
Configure: feature layer interface disabled
Configure: render interface disabled
Configure: score interface disabled
Configure: render interface disabled
Launching next game.
Launching next game.
Requesting to join a single player game
Configuring interface options
Configure: raw interface enabled
Configure: feature layer interface disabled
Configure: score interface disabled
Configure: render interface disabled
Launching next game.
Next launch phase started: 2
Next launch phase started: 3
Next launch phase started: 2
Next launch phase started: 3
Requesting to join a single player game
Configuring interface options
Configure: raw interface enabled
Configure: feature layer interface disabled
Configure: score interface disabled
Configure: render interface disabled
Launching next game.
Next launch phase started: 2
Next launch phase started: 4
Next launch phase started: 3
Next launch phase started: 4
Next launch phase started: 5
Next launch phase started: 6
Next launch phase started: 5
Next launch phase started: 7
Next launch phase started: 6
Next launch phase started: 8
Next launch phase started: 7
Next launch phase started: 8
Next launch phase started: 4
Next launch phase started: 2
Next launch phase started: 3
Next launch phase started: 5
Next launch phase started: 6
Next launch phase started: 7
Next launch phase started: 8
Next launch phase started: 4
Next launch phase started: 5
Next launch phase started: 6
Next launch phase started: 7
Next launch phase started: 8
Game has started.
Using default stable ids, none found at: /home/user/StarCraftII/stableid.json
Successfully loaded stable ids: GameData\stableid.json
Sending ResponseJoinGame
Game has started.
Using default stable ids, none found at: /home/user/StarCraftII/stableid.json
Game has started.
Using default stable ids, none found at: /home/user/StarCraftII/stableid.json
Successfully loaded stable ids: GameData\stableid.json
Game has started.
Using default stable ids, none found at: /home/user/StarCraftII/stableid.json
Successfully loaded stable ids: GameData\stableid.json
Sending ResponseJoinGame
Successfully loaded stable ids: GameData\stableid.json
Sending ResponseJoinGame
Sending ResponseJoinGame
Game has started.
Using default stable ids, none found at: /home/user/StarCraftII/stableid.json
Successfully loaded stable ids: GameData\stableid.json
Sending ResponseJoinGame
Game has started.
Using default stable ids, none found at: /home/user/StarCraftII/stableid.json
Successfully loaded stable ids: GameData\stableid.json
Sending ResponseJoinGame
Game has started.
Using default stable ids, none found at: /home/user/StarCraftII/stableid.json
Successfully loaded stable ids: GameData\stableid.json
Sending ResponseJoinGame
Game has started.
Using default stable ids, none found at: /home/user/StarCraftII/stableid.json
Successfully loaded stable ids: GameData\stableid.json
Sending ResponseJoinGame
idv policy and team policy use same initial params!
wandb: - 0.006 MB of 0.006 MB uploaded
wandb: \ 0.006 MB of 0.020 MB uploaded
wandb: | 0.020 MB of 0.020 MB uploaded
wandb: 🚀 View run StarCraft2_27m_vs_30m_1 at: https://wandb.ai/804703098/27m_vs_30m/runs/7690pulv
wandb: ⭐️ View project at: https://wandb.ai/804703098/27m_vs_30m
wandb: Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
wandb: Find logs at: ./results/StarCraft2/27m_vs_30m/mappotrsyn/train_27m_vs_30m_sigma_v0_KLCP_s2r2_v0/wandb/run-20240803_200649-7690pulv/logs
Traceback (most recent call last):
  File "train/train_smac_trsyn.py", line 247, in <module>
    main(sys.argv[1:])
  File "train/train_smac_trsyn.py", line 232, in main
    runner.run()
  File "/home/user/zhangyang/PycharmProjects/Nips2024-ITPC-v2/Nips2024-ITPC-v2/onpolicy/runner/shared/smac_runner_trsyn.py", line 67, in run
    train_infos = self.train(episode)
  File "/home/user/zhangyang/PycharmProjects/Nips2024-ITPC-v2/Nips2024-ITPC-v2/onpolicy/runner/shared/base_runner_trsyn.py", line 169, in train
    train_infos = self.trainer.train(self.buffer, episode)
  File "/home/user/zhangyang/PycharmProjects/Nips2024-ITPC-v2/Nips2024-ITPC-v2/onpolicy/algorithms/r_mappo/rmappo_trsyn_series_imp.py", line 859, in train
    = self.ppo_update(sample, episode, update_actor)
  File "/home/user/zhangyang/PycharmProjects/Nips2024-ITPC-v2/Nips2024-ITPC-v2/onpolicy/algorithms/r_mappo/rmappo_trsyn_series_imp.py", line 261, in ppo_update
    self.idv_policy.evaluate_actions(idv_share_obs_batch,
  File "/home/user/zhangyang/PycharmProjects/Nips2024-ITPC-v2/Nips2024-ITPC-v2/onpolicy/algorithms/r_mappo/algorithm/rMAPPOPolicy.py", line 106, in evaluate_actions
    action_log_probs, dist_entropy, act_dists = self.actor.evaluate_actions(obs,
  File "/home/user/zhangyang/PycharmProjects/Nips2024-ITPC-v2/Nips2024-ITPC-v2/onpolicy/algorithms/r_mappo/algorithm/r_actor_critic.py", line 96, in evaluate_actions
    obs = check(obs).to(**self.tpdv)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.66 GiB. GPU 0 has a total capacity of 23.64 GiB of which 1.60 GiB is free. Process 768063 has 2.62 GiB memory in use. Process 768899 has 18.87 GiB memory in use. Including non-PyTorch memory, this process has 542.00 MiB memory in use. Of the allocated memory 101.50 MiB is allocated by PyTorch, and 28.50 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
Exception in thread NetStatThr:
Traceback (most recent call last):
  File "/home/user/anaconda3/envs/zypy38/lib/python3.8/threading.py", line 932, in _bootstrap_inner
Exception in thread IntMsgThr:
Traceback (most recent call last):
  File "/home/user/anaconda3/envs/zypy38/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/home/user/anaconda3/envs/zypy38/lib/python3.8/threading.py", line 870, in run
    self.run()
  File "/home/user/anaconda3/envs/zypy38/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/home/user/anaconda3/envs/zypy38/lib/python3.8/site-packages/wandb/sdk/wandb_run.py", line 300, in check_internal_messages
    self._target(*self._args, **self._kwargs)
  File "/home/user/anaconda3/envs/zypy38/lib/python3.8/site-packages/wandb/sdk/wandb_run.py", line 268, in check_network_status
    self._loop_check_status(
  File "/home/user/anaconda3/envs/zypy38/lib/python3.8/site-packages/wandb/sdk/wandb_run.py", line 224, in _loop_check_status
    self._loop_check_status(
  File "/home/user/anaconda3/envs/zypy38/lib/python3.8/site-packages/wandb/sdk/wandb_run.py", line 224, in _loop_check_status
    local_handle = request()
  File "/home/user/anaconda3/envs/zypy38/lib/python3.8/site-packages/wandb/sdk/interface/interface.py", line 856, in deliver_internal_messages
    local_handle = request()
  File "/home/user/anaconda3/envs/zypy38/lib/python3.8/site-packages/wandb/sdk/interface/interface.py", line 848, in deliver_network_status
    return self._deliver_internal_messages(internal_message)
  File "/home/user/anaconda3/envs/zypy38/lib/python3.8/site-packages/wandb/sdk/interface/interface_shared.py", line 516, in _deliver_internal_messages
    return self._deliver_network_status(status)
  File "/home/user/anaconda3/envs/zypy38/lib/python3.8/site-packages/wandb/sdk/interface/interface_shared.py", line 510, in _deliver_network_status
    return self._deliver_record(record)
  File "/home/user/anaconda3/envs/zypy38/lib/python3.8/site-packages/wandb/sdk/interface/interface_shared.py", line 459, in _deliver_record
    return self._deliver_record(record)
  File "/home/user/anaconda3/envs/zypy38/lib/python3.8/site-packages/wandb/sdk/interface/interface_shared.py", line 459, in _deliver_record
    handle = mailbox._deliver_record(record, interface=self)
  File "/home/user/anaconda3/envs/zypy38/lib/python3.8/site-packages/wandb/sdk/lib/mailbox.py", line 455, in _deliver_record
    handle = mailbox._deliver_record(record, interface=self)
  File "/home/user/anaconda3/envs/zypy38/lib/python3.8/site-packages/wandb/sdk/lib/mailbox.py", line 455, in _deliver_record
    interface._publish(record)
  File "/home/user/anaconda3/envs/zypy38/lib/python3.8/site-packages/wandb/sdk/interface/interface_sock.py", line 51, in _publish
    interface._publish(record)
  File "/home/user/anaconda3/envs/zypy38/lib/python3.8/site-packages/wandb/sdk/interface/interface_sock.py", line 51, in _publish
    self._sock_client.send_record_publish(record)
  File "/home/user/anaconda3/envs/zypy38/lib/python3.8/site-packages/wandb/sdk/lib/sock_client.py", line 221, in send_record_publish
    self._sock_client.send_record_publish(record)
  File "/home/user/anaconda3/envs/zypy38/lib/python3.8/site-packages/wandb/sdk/lib/sock_client.py", line 221, in send_record_publish
    self.send_server_request(server_req)
  File "/home/user/anaconda3/envs/zypy38/lib/python3.8/site-packages/wandb/sdk/lib/sock_client.py", line 155, in send_server_request
    self.send_server_request(server_req)
  File "/home/user/anaconda3/envs/zypy38/lib/python3.8/site-packages/wandb/sdk/lib/sock_client.py", line 155, in send_server_request
    self._send_message(msg)
  File "/home/user/anaconda3/envs/zypy38/lib/python3.8/site-packages/wandb/sdk/lib/sock_client.py", line 152, in _send_message
    self._send_message(msg)
  File "/home/user/anaconda3/envs/zypy38/lib/python3.8/site-packages/wandb/sdk/lib/sock_client.py", line 152, in _send_message
    self._sendall_with_error_handle(header + data)
  File "/home/user/anaconda3/envs/zypy38/lib/python3.8/site-packages/wandb/sdk/lib/sock_client.py", line 130, in _sendall_with_error_handle
    self._sendall_with_error_handle(header + data)
  File "/home/user/anaconda3/envs/zypy38/lib/python3.8/site-packages/wandb/sdk/lib/sock_client.py", line 130, in _sendall_with_error_handle
    sent = self._sock.send(data)
BrokenPipeError: [Errno 32] Broken pipe
    sent = self._sock.send(data)
BrokenPipeError: [Errno 32] Broken pipe
CloseHandler: 127.0.0.1:41354 disconnected
CloseHandler: 127.0.0.1:38902 disconnected
CloseHandler: 127.0.0.1:53454 disconnected
CloseHandler: 127.0.0.1:54728 disconnected
CloseHandler: 127.0.0.1:33372 disconnected
CloseHandler: 127.0.0.1:36174 disconnected
CloseHandler: 127.0.0.1:44990 disconnected
CloseHandler: 127.0.0.1:55858 disconnected
