wandb: Currently logged in as: 804703098. Use `wandb login --relogin` to force relogin
wandb: Tracking run with wandb version 0.16.5
wandb: Run data is saved locally in /home/user/zhangyang/PycharmProjects/Nips2024-ITPC-v2/Nips2024-ITPC-v2/onpolicy/scripts/results/MPE/simple_tag_tr/rmappotrsyn/exp_train_continue_tag_base_klcp_s2r2_v1/wandb/run-20240402_151604-pm61z8s2
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run MPE_4444
wandb: ⭐️ View project at https://wandb.ai/804703098/Continue_Tag_Base_v1
wandb: 🚀 View run at https://wandb.ai/804703098/Continue_Tag_Base_v1/runs/pm61z8s2/workspace
choose to use gpu...
idv policy and team policy use same initial params!

 Scenario simple_tag_tr Algo rmappotrsyn Exp exp_train_continue_tag_base_klcp_s2r2_v1 updates 0/10 episodes, total num timesteps 200/2000, FPS 256.

team_policy eval average step individual rewards of agent0: -0.028862901820487733
team_policy eval average team episode rewards of agent0: 0.0
team_policy eval idv catch total num of agent0: 2
team_policy eval team catch total num: 0
team_policy eval average step individual rewards of agent1: -0.026544871641317594
team_policy eval average team episode rewards of agent1: 0.0
team_policy eval idv catch total num of agent1: 2
team_policy eval team catch total num: 0
team_policy eval average step individual rewards of agent2: -0.0471754343890934
team_policy eval average team episode rewards of agent2: 0.0
team_policy eval idv catch total num of agent2: 2
team_policy eval team catch total num: 0
team_policy eval average step individual rewards of agent3: -0.04529824615903784
team_policy eval average team episode rewards of agent3: 0.0
team_policy eval idv catch total num of agent3: 2
team_policy eval team catch total num: 0
team_policy eval average step individual rewards of agent4: -0.11244719950526282
team_policy eval average team episode rewards of agent4: 0.0
team_policy eval idv catch total num of agent4: 0
team_policy eval team catch total num: 0
idv_policy eval average step individual rewards of agent0: -0.09834887708999741
idv_policy eval average team episode rewards of agent0: 0.0
idv_policy eval idv catch total num of agent0: 0
idv_policy eval team catch total num: 0
idv_policy eval average step individual rewards of agent1: 0.13909943370664216
idv_policy eval average team episode rewards of agent1: 0.0
idv_policy eval idv catch total num of agent1: 9
idv_policy eval team catch total num: 0
idv_policy eval average step individual rewards of agent2: -0.014793570412490551
idv_policy eval average team episode rewards of agent2: 0.0
idv_policy eval idv catch total num of agent2: 3
idv_policy eval team catch total num: 0
idv_policy eval average step individual rewards of agent3: -0.10846520471820419
idv_policy eval average team episode rewards of agent3: 0.0
idv_policy eval idv catch total num of agent3: 0
idv_policy eval team catch total num: 0
idv_policy eval average step individual rewards of agent4: -0.03107793729259072
idv_policy eval average team episode rewards of agent4: 0.0
idv_policy eval idv catch total num of agent4: 2
idv_policy eval team catch total num: 0

 Scenario simple_tag_tr Algo rmappotrsyn Exp exp_train_continue_tag_base_klcp_s2r2_v1 updates 1/10 episodes, total num timesteps 400/2000, FPS 246.


 Scenario simple_tag_tr Algo rmappotrsyn Exp exp_train_continue_tag_base_klcp_s2r2_v1 updates 2/10 episodes, total num timesteps 600/2000, FPS 265.


 Scenario simple_tag_tr Algo rmappotrsyn Exp exp_train_continue_tag_base_klcp_s2r2_v1 updates 3/10 episodes, total num timesteps 800/2000, FPS 273.


 Scenario simple_tag_tr Algo rmappotrsyn Exp exp_train_continue_tag_base_klcp_s2r2_v1 updates 4/10 episodes, total num timesteps 1000/2000, FPS 279.


 Scenario simple_tag_tr Algo rmappotrsyn Exp exp_train_continue_tag_base_klcp_s2r2_v1 updates 5/10 episodes, total num timesteps 1200/2000, FPS 286.


 Scenario simple_tag_tr Algo rmappotrsyn Exp exp_train_continue_tag_base_klcp_s2r2_v1 updates 6/10 episodes, total num timesteps 1400/2000, FPS 292.


 Scenario simple_tag_tr Algo rmappotrsyn Exp exp_train_continue_tag_base_klcp_s2r2_v1 updates 7/10 episodes, total num timesteps 1600/2000, FPS 291.


 Scenario simple_tag_tr Algo rmappotrsyn Exp exp_train_continue_tag_base_klcp_s2r2_v1 updates 8/10 episodes, total num timesteps 1800/2000, FPS 292.


 Scenario simple_tag_tr Algo rmappotrsyn Exp exp_train_continue_tag_base_klcp_s2r2_v1 updates 9/10 episodes, total num timesteps 2000/2000, FPS 293.

wandb: - 0.006 MB of 0.006 MB uploaded
wandb: \ 0.650 MB of 0.650 MB uploaded
wandb: | 0.650 MB of 0.650 MB uploaded
wandb: 
wandb: Run history:
wandb:                                       Aa_idv_actor_loss ▇██▇▆▅▆█▁▁
wandb:                                          Ab_policy_loss ▄▅▄▅▄▅▆█▁▅
wandb:                                     Ac_idv_ppo_loss_abs ▁▆▆▅▇▆▇▇█▇
wandb:                                         Ad_idv_ppo_prop ▁▇▇▆▇▇▇▇█▇
wandb:                                                  Ae_eta ▆▁▃▅▄▂▄▅█▄
wandb:                                    Af_noclip_proportion ▅▃▆▅█▃▆▆▁▅
wandb:                                    Ag_update_proportion ▅█▄▁▆▆▇▃▂▃
wandb:                                          Ah_update_loss ▁▆▇▃▇▇▇▇██
wandb:                                         Ai_idv_epsilon' █▇▆▆▅▄▃▃▂▁
wandb:                                            Aj_idv_sigma ▁▂▂▃▂▃▅▄█▄
wandb:              Ak_idv_clip(sigma, 1-epislon', 1+epislon') ▁▂▃▃▃▃▆▅█▅
wandb:                                Al_idv_noclip_proportion ███▇▇▇▂▆▁▄
wandb:                       Am_idv_(sigma*A)update_proportion ▄▁▅█▃▃▂▅▅▆
wandb:                             An_idv_(sigma*A)update_loss █▂▂▇▂▁▁▂▁▂
wandb:                                     Ao_idv_entropy_prop █▂▂▃▂▂▂▂▁▂
wandb:                                         Ap_dist_entropy ▂▂▁▂▃▄▅▄▆█
wandb:                                          Aq_idv_kl_prop ▁▁▁▁▁▁▁▁▁▁
wandb:                                          Ar_idv_kl_coef ▁▁▁▁▁▁▁▁▁▁
wandb:                                          As_idv_kl_loss ▁▂▃▃▃▅▆▇█▆
wandb:                                    At_idv_cross_entropy ▁▁▁▁▁▁▁▁▁▁
wandb:                                           Au_value_loss ▄█▁▁▁▁▁▁▁▁
wandb:                                           Av_advantages ▅▂█▄▆▁▅▆▆▅
wandb:                                       Aw_idv_actor_norm ▁█▂▅▃▅▃▄▆▂
wandb:                                      Ax_idv_critic_norm █▂▁▁▁▁▁▁▁▁
wandb:                                     Ba_idv_org_min_prop ▇▆█▁▇▆▇▆▆▆
wandb:                                     Bb_idv_org_max_prop ▂▆▁█▃▅▄▃▂▃
wandb:                                     Bc_idv_org_org_prop ▁▁▁▁▁▁▁▁▁▁
wandb:                                     Bd_idv_new_min_prop ▁▅▅█▃▃▄▄▅▄
wandb:                                     Be_idv_new_max_prop █▂▃▁▄▄▂▄▃▅
wandb:                                      Ta_team_actor_loss ▅▆█▆▅▇▆▅▆▁
wandb:                                     Tb_team_policy_loss ▁▃▅▃▂▆▆▅█▄
wandb:                                    Tc_team_ppo_loss_abs ▄▂▅▁▂▆▄▃██
wandb:                                        Td_team_ppo_prop ▆▄█▃▃▆▂▃▆▁
wandb:                                        Te_team_epsilon^ ▁▁▁▁▁▁▁▁▁▁
wandb:                                          Tf_team_sigma^ ▄▃▃▄▅█▃▆▁▆
wandb:          Tg_team_clip(sigma^, 1-epislon^', 1+epislon^') ▄▄▁▃▇█▃▂▁▇
wandb:                               Th_team_noclip_proportion █▆▆▅▆▄▄▂▁▃
wandb:                     Ti_team_(sigma^*A)update_proportion █▆▆▅▆▄▄▁▁▃
wandb:                           Tj_team_(sigma^*A)update_loss ▇▇▆▄█▆▅█▁▁
wandb:                                    Tk_team_entropy_prop ▃▅▁▆▆▃▇▆▃█
wandb:                                    Tl_team_dist_entropy ▂▂▁▂▂▃▄▄▅█
wandb:                                         Tm_team_kl_prop ▁▁▁▁▁▁▁▁▁▁
wandb:                                         Tn_team_kl_coef ▁▁▁▁▁▁▁▁▁▁
wandb:                                         To_team_kl_loss ▁▃▂▃▃▄▅▇█▆
wandb:                                   Tp_team_cross_entropy ▁▁▁▁▁▁▁▁▁▁
wandb:                                      Tq_team_value_loss ▂█▁▁▁▁▁▁▁▁
wandb:                                      Tr_team_advantages ▇▆█▆▁▄▆▄▄▃
wandb:                                      Ts_team_actor_norm ▄▃▁▂▃█▃▅▄▆
wandb:                                     Tt_team_critic_norm █▂▁▁▁▁▁▁▁▁
wandb:                     agent0/average_episode_team_rewards ▁▁▁█▁▁▁▁▁▁
wandb:                  agent0/average_step_individual_rewards ▁▂▇█▄▁▆▃▆▅
wandb:     agent0/idv_policy_eval_average_episode_team_rewards ▁
wandb:  agent0/idv_policy_eval_average_step_individual_rewards ▁
wandb:              agent0/idv_policy_eval_idv_catch_total_num ▁
wandb:             agent0/idv_policy_eval_team_catch_total_num ▁
wandb:    agent0/team_policy_eval_average_episode_team_rewards ▁
wandb: agent0/team_policy_eval_average_step_individual_rewards ▁
wandb:             agent0/team_policy_eval_idv_catch_total_num ▁
wandb:            agent0/team_policy_eval_team_catch_total_num ▁
wandb:                     agent1/average_episode_team_rewards ▁▁▁█▁▁▁▁▁▁
wandb:                  agent1/average_step_individual_rewards ▁▅▅▂█▇▄▆▆▆
wandb:     agent1/idv_policy_eval_average_episode_team_rewards ▁
wandb:  agent1/idv_policy_eval_average_step_individual_rewards ▁
wandb:              agent1/idv_policy_eval_idv_catch_total_num ▁
wandb:             agent1/idv_policy_eval_team_catch_total_num ▁
wandb:    agent1/team_policy_eval_average_episode_team_rewards ▁
wandb: agent1/team_policy_eval_average_step_individual_rewards ▁
wandb:             agent1/team_policy_eval_idv_catch_total_num ▁
wandb:            agent1/team_policy_eval_team_catch_total_num ▁
wandb:                     agent2/average_episode_team_rewards ▁▁▁█▁▁▁▁▁▁
wandb:                  agent2/average_step_individual_rewards ▃▄█▃▅█▃▃▁▇
wandb:     agent2/idv_policy_eval_average_episode_team_rewards ▁
wandb:  agent2/idv_policy_eval_average_step_individual_rewards ▁
wandb:              agent2/idv_policy_eval_idv_catch_total_num ▁
wandb:             agent2/idv_policy_eval_team_catch_total_num ▁
wandb:    agent2/team_policy_eval_average_episode_team_rewards ▁
wandb: agent2/team_policy_eval_average_step_individual_rewards ▁
wandb:             agent2/team_policy_eval_idv_catch_total_num ▁
wandb:            agent2/team_policy_eval_team_catch_total_num ▁
wandb:                     agent3/average_episode_team_rewards ▁▁▁█▁▁▁▁▁▁
wandb:                  agent3/average_step_individual_rewards █▁▁▁▄▁▃▃▃▂
wandb:     agent3/idv_policy_eval_average_episode_team_rewards ▁
wandb:  agent3/idv_policy_eval_average_step_individual_rewards ▁
wandb:              agent3/idv_policy_eval_idv_catch_total_num ▁
wandb:             agent3/idv_policy_eval_team_catch_total_num ▁
wandb:    agent3/team_policy_eval_average_episode_team_rewards ▁
wandb: agent3/team_policy_eval_average_step_individual_rewards ▁
wandb:             agent3/team_policy_eval_idv_catch_total_num ▁
wandb:            agent3/team_policy_eval_team_catch_total_num ▁
wandb:                     agent4/average_episode_team_rewards ▁▁▁█▁▁▁▁▁▁
wandb:                  agent4/average_step_individual_rewards ▄▁▃▃▃▄▆▁▃█
wandb:     agent4/idv_policy_eval_average_episode_team_rewards ▁
wandb:  agent4/idv_policy_eval_average_step_individual_rewards ▁
wandb:              agent4/idv_policy_eval_idv_catch_total_num ▁
wandb:             agent4/idv_policy_eval_team_catch_total_num ▁
wandb:    agent4/team_policy_eval_average_episode_team_rewards ▁
wandb: agent4/team_policy_eval_average_step_individual_rewards ▁
wandb:             agent4/team_policy_eval_idv_catch_total_num ▁
wandb:            agent4/team_policy_eval_team_catch_total_num ▁
wandb: 
wandb: Run summary:
wandb:                                       Aa_idv_actor_loss -0.32343
wandb:                                          Ab_policy_loss -0.00827
wandb:                                     Ac_idv_ppo_loss_abs 0.87727
wandb:                                         Ad_idv_ppo_prop 0.7357
wandb:                                                  Ae_eta 0.99902
wandb:                                    Af_noclip_proportion 0.9745
wandb:                                    Ag_update_proportion 0.4524
wandb:                                          Ah_update_loss 0.72403
wandb:                                         Ai_idv_epsilon' 2.99775
wandb:                                            Aj_idv_sigma 1.06491
wandb:              Ak_idv_clip(sigma, 1-epislon', 1+epislon') 1.06133
wandb:                                Al_idv_noclip_proportion 0.9938
wandb:                       Am_idv_(sigma*A)update_proportion 0.5337
wandb:                             An_idv_(sigma*A)update_loss -0.6599
wandb:                                     Ao_idv_entropy_prop 0.2643
wandb:                                         Ap_dist_entropy 3.15481
wandb:                                          Aq_idv_kl_prop 0.0
wandb:                                          Ar_idv_kl_coef 0.0
wandb:                                          As_idv_kl_loss 0.01715
wandb:                                    At_idv_cross_entropy 0.0
wandb:                                           Au_value_loss 0.20768
wandb:                                           Av_advantages -0.0
wandb:                                       Aw_idv_actor_norm 0.6291
wandb:                                      Ax_idv_critic_norm 3.64567
wandb:                                     Ba_idv_org_min_prop 0.2346
wandb:                                     Bb_idv_org_max_prop 0.2178
wandb:                                     Bc_idv_org_org_prop 0.0
wandb:                                     Bd_idv_new_min_prop 0.2307
wandb:                                     Be_idv_new_max_prop 0.303
wandb:                                      Ta_team_actor_loss -0.31604
wandb:                                     Tb_team_policy_loss -0.00227
wandb:                                    Tc_team_ppo_loss_abs 0.83621
wandb:                                        Td_team_ppo_prop 0.72715
wandb:                                        Te_team_epsilon^ 0.2
wandb:                                          Tf_team_sigma^ 1.00337
wandb:          Tg_team_clip(sigma^, 1-epislon^', 1+epislon^') 0.99823
wandb:                               Th_team_noclip_proportion 0.8057
wandb:                     Ti_team_(sigma^*A)update_proportion 0.8968
wandb:                           Tj_team_(sigma^*A)update_loss -0.04845
wandb:                                    Tk_team_entropy_prop 0.27285
wandb:                                    Tl_team_dist_entropy 3.14093
wandb:                                         Tm_team_kl_prop 0.0
wandb:                                         Tn_team_kl_coef 1.0
wandb:                                         To_team_kl_loss 0.01218
wandb:                                   Tp_team_cross_entropy 0.0
wandb:                                      Tq_team_value_loss 0.01848
wandb:                                      Tr_team_advantages -0.0
wandb:                                      Ts_team_actor_norm 0.75343
wandb:                                     Tt_team_critic_norm 0.26866
wandb:                     agent0/average_episode_team_rewards 0.0
wandb:                  agent0/average_step_individual_rewards -0.02352
wandb:     agent0/idv_policy_eval_average_episode_team_rewards 0.0
wandb:  agent0/idv_policy_eval_average_step_individual_rewards -0.09835
wandb:              agent0/idv_policy_eval_idv_catch_total_num 0
wandb:             agent0/idv_policy_eval_team_catch_total_num 0
wandb:    agent0/team_policy_eval_average_episode_team_rewards 0.0
wandb: agent0/team_policy_eval_average_step_individual_rewards -0.02886
wandb:             agent0/team_policy_eval_idv_catch_total_num 2
wandb:            agent0/team_policy_eval_team_catch_total_num 0
wandb:                     agent1/average_episode_team_rewards 0.0
wandb:                  agent1/average_step_individual_rewards 0.00395
wandb:     agent1/idv_policy_eval_average_episode_team_rewards 0.0
wandb:  agent1/idv_policy_eval_average_step_individual_rewards 0.1391
wandb:              agent1/idv_policy_eval_idv_catch_total_num 9
wandb:             agent1/idv_policy_eval_team_catch_total_num 0
wandb:    agent1/team_policy_eval_average_episode_team_rewards 0.0
wandb: agent1/team_policy_eval_average_step_individual_rewards -0.02654
wandb:             agent1/team_policy_eval_idv_catch_total_num 2
wandb:            agent1/team_policy_eval_team_catch_total_num 0
wandb:                     agent2/average_episode_team_rewards 0.0
wandb:                  agent2/average_step_individual_rewards 0.00251
wandb:     agent2/idv_policy_eval_average_episode_team_rewards 0.0
wandb:  agent2/idv_policy_eval_average_step_individual_rewards -0.01479
wandb:              agent2/idv_policy_eval_idv_catch_total_num 3
wandb:             agent2/idv_policy_eval_team_catch_total_num 0
wandb:    agent2/team_policy_eval_average_episode_team_rewards 0.0
wandb: agent2/team_policy_eval_average_step_individual_rewards -0.04718
wandb:             agent2/team_policy_eval_idv_catch_total_num 2
wandb:            agent2/team_policy_eval_team_catch_total_num 0
wandb:                     agent3/average_episode_team_rewards 0.0
wandb:                  agent3/average_step_individual_rewards -0.0626
wandb:     agent3/idv_policy_eval_average_episode_team_rewards 0.0
wandb:  agent3/idv_policy_eval_average_step_individual_rewards -0.10847
wandb:              agent3/idv_policy_eval_idv_catch_total_num 0
wandb:             agent3/idv_policy_eval_team_catch_total_num 0
wandb:    agent3/team_policy_eval_average_episode_team_rewards 0.0
wandb: agent3/team_policy_eval_average_step_individual_rewards -0.0453
wandb:             agent3/team_policy_eval_idv_catch_total_num 2
wandb:            agent3/team_policy_eval_team_catch_total_num 0
wandb:                     agent4/average_episode_team_rewards 0.0
wandb:                  agent4/average_step_individual_rewards 0.01839
wandb:     agent4/idv_policy_eval_average_episode_team_rewards 0.0
wandb:  agent4/idv_policy_eval_average_step_individual_rewards -0.03108
wandb:              agent4/idv_policy_eval_idv_catch_total_num 2
wandb:             agent4/idv_policy_eval_team_catch_total_num 0
wandb:    agent4/team_policy_eval_average_episode_team_rewards 0.0
wandb: agent4/team_policy_eval_average_step_individual_rewards -0.11245
wandb:             agent4/team_policy_eval_idv_catch_total_num 0
wandb:            agent4/team_policy_eval_team_catch_total_num 0
wandb: 
wandb: 🚀 View run MPE_4444 at: https://wandb.ai/804703098/Continue_Tag_Base_v1/runs/pm61z8s2/workspace
wandb: Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 4 other file(s)
wandb: Find logs at: ./results/MPE/simple_tag_tr/rmappotrsyn/exp_train_continue_tag_base_klcp_s2r2_v1/wandb/run-20240402_151604-pm61z8s2/logs
Traceback (most recent call last):
  File "train/train_mpe_trsyn.py", line 244, in <module>
    main(sys.argv[1:])
  File "train/train_mpe_trsyn.py", line 229, in main
    runner.run()
  File "/home/user/zhangyang/PycharmProjects/Nips2024-ITPC-v2/Nips2024-ITPC-v2/onpolicy/runner/shared/mpe_runner_trsyn.py", line 118, in run
    d = {"average_team_rewards_" + str(self.all_args.seed) + "_KL_loss_" + self.all_args.idv_use_kl_loss: average_team_rewards}
TypeError: can only concatenate str (not "bool") to str
Exception in thread NetStatThr:
Traceback (most recent call last):
  File "/home/user/anaconda3/envs/zypython38/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/home/user/anaconda3/envs/zypython38/lib/python3.8/threading.py", line 870, in run
Exception in thread IntMsgThr:
Traceback (most recent call last):
  File "/home/user/anaconda3/envs/zypython38/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self._target(*self._args, **self._kwargs)
  File "/home/user/anaconda3/envs/zypython38/lib/python3.8/site-packages/wandb/sdk/wandb_run.py", line 268, in check_network_status
    self._loop_check_status(
  File "/home/user/anaconda3/envs/zypython38/lib/python3.8/site-packages/wandb/sdk/wandb_run.py", line 224, in _loop_check_status
    self.run()
  File "/home/user/anaconda3/envs/zypython38/lib/python3.8/threading.py", line 870, in run
    local_handle = request()
  File "/home/user/anaconda3/envs/zypython38/lib/python3.8/site-packages/wandb/sdk/interface/interface.py", line 836, in deliver_network_status
    self._target(*self._args, **self._kwargs)
  File "/home/user/anaconda3/envs/zypython38/lib/python3.8/site-packages/wandb/sdk/wandb_run.py", line 300, in check_internal_messages
    return self._deliver_network_status(status)
  File "/home/user/anaconda3/envs/zypython38/lib/python3.8/site-packages/wandb/sdk/interface/interface_shared.py", line 510, in _deliver_network_status
    self._loop_check_status(
  File "/home/user/anaconda3/envs/zypython38/lib/python3.8/site-packages/wandb/sdk/wandb_run.py", line 224, in _loop_check_status
    return self._deliver_record(record)
  File "/home/user/anaconda3/envs/zypython38/lib/python3.8/site-packages/wandb/sdk/interface/interface_shared.py", line 459, in _deliver_record
    local_handle = request()
  File "/home/user/anaconda3/envs/zypython38/lib/python3.8/site-packages/wandb/sdk/interface/interface.py", line 844, in deliver_internal_messages
    return self._deliver_internal_messages(internal_message)
  File "/home/user/anaconda3/envs/zypython38/lib/python3.8/site-packages/wandb/sdk/interface/interface_shared.py", line 516, in _deliver_internal_messages
    handle = mailbox._deliver_record(record, interface=self)
  File "/home/user/anaconda3/envs/zypython38/lib/python3.8/site-packages/wandb/sdk/lib/mailbox.py", line 455, in _deliver_record
    interface._publish(record)
  File "/home/user/anaconda3/envs/zypython38/lib/python3.8/site-packages/wandb/sdk/interface/interface_sock.py", line 51, in _publish
    return self._deliver_record(record)
  File "/home/user/anaconda3/envs/zypython38/lib/python3.8/site-packages/wandb/sdk/interface/interface_shared.py", line 459, in _deliver_record
    self._sock_client.send_record_publish(record)
  File "/home/user/anaconda3/envs/zypython38/lib/python3.8/site-packages/wandb/sdk/lib/sock_client.py", line 221, in send_record_publish
    handle = mailbox._deliver_record(record, interface=self)
  File "/home/user/anaconda3/envs/zypython38/lib/python3.8/site-packages/wandb/sdk/lib/mailbox.py", line 455, in _deliver_record
    self.send_server_request(server_req)
  File "/home/user/anaconda3/envs/zypython38/lib/python3.8/site-packages/wandb/sdk/lib/sock_client.py", line 155, in send_server_request
    interface._publish(record)
  File "/home/user/anaconda3/envs/zypython38/lib/python3.8/site-packages/wandb/sdk/interface/interface_sock.py", line 51, in _publish
    self._send_message(msg)
  File "/home/user/anaconda3/envs/zypython38/lib/python3.8/site-packages/wandb/sdk/lib/sock_client.py", line 152, in _send_message
    self._sock_client.send_record_publish(record)
  File "/home/user/anaconda3/envs/zypython38/lib/python3.8/site-packages/wandb/sdk/lib/sock_client.py", line 221, in send_record_publish
    self._sendall_with_error_handle(header + data)
  File "/home/user/anaconda3/envs/zypython38/lib/python3.8/site-packages/wandb/sdk/lib/sock_client.py", line 130, in _sendall_with_error_handle
    self.send_server_request(server_req)
  File "/home/user/anaconda3/envs/zypython38/lib/python3.8/site-packages/wandb/sdk/lib/sock_client.py", line 155, in send_server_request
    sent = self._sock.send(data)
BrokenPipeError: [Errno 32] Broken pipe
    self._send_message(msg)
  File "/home/user/anaconda3/envs/zypython38/lib/python3.8/site-packages/wandb/sdk/lib/sock_client.py", line 152, in _send_message
    self._sendall_with_error_handle(header + data)
  File "/home/user/anaconda3/envs/zypython38/lib/python3.8/site-packages/wandb/sdk/lib/sock_client.py", line 130, in _sendall_with_error_handle
    sent = self._sock.send(data)
BrokenPipeError: [Errno 32] Broken pipe
