\section{Games without a common equilibrium}\label{dc}

%In this paper, we investigate the last-iterate behavior of Optimistic Multiplicative Weights Updates (OMWU) and Extra-gradient Multiplicative Weights Updates (Extra-MWU) in periodic zero-sum games with simplex constraints. Our main findings establish a separation in the last-iterate convergence behaviors between OMWU and Extra-MWU, assuming that the game series within the periodic game possess a common fully mixed Nash equilibrium. This is interesting because it challenges the conventional wisdom that these two algorithms should exhibit similar behaviors \citep{mokhtari2020unified}. Our results extend the findings of \citep{feng2023last} from an unconstrained setting to a more practical constrained setting.

Since the main purpose of this work is to study the last-iterate convergence behaviors of learning algorithms in the time-varying games, it is natural to require that there should be a reasonable point towards which we can hope these algorithms will converge. This is why we assume that the game series has a common equilibrium. An interesting question arises when considering what the last-iterate behaviors of (Extra-MWU) and (OMWU) look like in periodic games where there is no common equilibrium.

In the following we present several experiments to provide possible answers to this question. We summary our findings in the following :
\begin{itemize}
    \item In a $\CT$-periodic game without common equilibrium, (Extra-MWU) will converge to a periodic orbit with period $\CT$. This periodic orbit will not contain equilibrium of the periodic game.
    \item In a $\CT$-periodic game without common equilibrium, (OMWU) will diverge to the boundary.
\end{itemize}

In Figure (\ref{ww}), we present numerical results for a $3$-periodic game with payoff matrices 

\begin{align*}
    A_t=
\begin{cases}
\left( (0, -1, 1), (1, 0, -1), (-1, 1, 0) \right),\ t \mod 3 = 0 \\
\\
\left((0, 1, -1), (-1, 0, 1), (1, -1, 0) \right),\ t \mod 3 = 1 \\
\\
\left((0, 0.25, 0.75), (1.5, 0, 0), (0, 1, 0) \right),\ t \mod 3 = 2 \\
\end{cases}
\end{align*}

For $t \mod 3 = 0$ and $t \mod 3 = 1$, the equilibrium is
$$\tb{x}^* = \tb{y}^* = (1/3,1/3,1/3)$$ For $t \mod 3 = 2$, the equilibrium for $A_t$ is $$(\tb{x}^*,\tb{y}^*) =  \left( (0.5,0.25,0.25),(0.25,0.375,0.375) \right) .$$

In (a) of Figure \ref{ww}, the three curves represent $\tb{x}^t_{1,1}$ when $t \mod 3 =0,1$ and $2$. The convergence of these three curves to distinct values suggests that the mixed strategy will converge to a periodic orbit with a period of three in (Extra-MWU). In (b) of Figure \ref{ww}, it can be observed that the green curve, representing the components of the third pure strategy in the mixed strategy, converges to zero over time. This indicates that the mixed strategy tends to approach its boundary in (OMWU). These experimental findings are representative, and similar phenomena also occur in other periodic games without a common equilibrium.

We believe that the existing techniques for establishing the last-iterate convergence property are insufficient in proving that (Extra-MWU) will converge to a periodic orbit in a periodic game without a common equilibrium, as these techniques necessitate the limit state of learning algorithms being a single point \citep{daskalakis2017training,daskalakis2018last,mertikopoulos2018optimistic}. A possible approach to address this problem is to investigate the relationship between Extra-MWU and monotone dynamical systems, which have been shown to converge to periodic orbits in periodic environments \citep{hirsch2006monotone}.

\begin{figure}[h]
    \centering
    \subfigure[Components of 1-strategy, Extra-MWU]{
        \includegraphics[width=1.5in]{non_com_extra.png}
        \label{}
    }
    \subfigure[Mixed strategy, OMWU]{
	\includegraphics[width=1.5in]{non_com_o.png}
        \label{}
    }
    \caption{Experimental results for no common equilibrium. }
    \label{ww}
\end{figure}










\section{Conclusion}

In this paper, we investigate the last-iterate behavior of Optimistic Multiplicative Weights Updates (OMWU) and Extra-gradient Multiplicative Weights Updates (Extra-MWU) in periodic zero-sum games with simplex constraints. Our main findings establish a separation in the last-iterate convergence behaviors between OMWU and Extra-MWU, assuming that the game series within the periodic game. This is interesting because it challenges the conventional wisdom that these two algorithms should exhibit similar behaviors \citep{mokhtari2020unified}. Our results also extend the findings of \citep{feng2023last} from the unconstrained setting to the more practical constrained setting. An interesting future direction is to study the dynamical behaviors of these methods in periodic games without common equilibrium. 

\section*{Acknowledgements}

Ioannis Panageas would like to acknowledge startup grant from UCI and UCI ICS Research Award. Part of this work was conducted while Ioannis was visiting Archimedes Research Unit. This work has been partially supported by project MIS 5154714 of the National Recovery and Resilience Plan Greece 2.0 funded by the European Union under the NextGenerationEU Program. Xiao Wang acknowledges Grant 202110458 from Shanghai University of Finance and Economics and support from the Shanghai Research Center for Data Science and Decision Technology.
