\section{Main Results}\label{main}

In this section we state our main results. Under the assumption of the games in a periodic game have an unique common equilibrium, we provide an example to show that (OMWU) fails to converge to the equilibrium and even can diverge to the boundary of the simplex, as stated in Theorem \ref{thm: OMWU fails}. Conversely, (Extra-MWU) can converge to the equilibrium, as shown in Theorem \ref{T2}. This distinction provides a separation on the last-iterate convergence behaviors of (OMWU) and (Extra-MWU). 

\begin{thm}\label{thm: OMWU fails}
    For the periodic game defined by payoff matrices
\begin{align}\label{2-periodic game_m}
 A_t=
\begin{cases}
\begin{bmatrix}
    0 & & 1 \\
    1 & & 0
\end{bmatrix}, & t \textnormal{\ \ is \ odd} \\ 
\\
\begin{bmatrix}
    0 & -1 \\
    -1 & 0
\end{bmatrix}, & t\textnormal{\ \ is \ even}
\end{cases}
\end{align}
%Let $p=\frac{1}{2}\min(\lvert \tb{x}_{1,1}^0-\tb{x}_{1,1}^*\rvert,\lvert  \tb{x}_{2,1}^0-\tb{x}_{2,1}^* \rvert)$. Then if $\eta$ is sufficiently small such that $p\ge 16\eta^\frac{1}{2}$, 
 and sufficient small step size $\eta$\footnote{Refer to the requirement for $\eta$ in Proposition~\ref{prop: before4.2} in the Appendix.}, (OMWU) has following properties :
\begin{itemize}
    \item For an arbitrary small neighbourhood $\CU$ of the equilibrium $(\tb{x}_1^*,\tb{x}_2^*)$, there exists an initial condition within $\CU$ that causes (OMWU) to fail in converging to $(\tb{x}_1^*,\tb{x}_2^*)$.
    \item If the initial condition $(\tb{x}_1^0, \tb{x}_2^0), (\tb{x}_1^{-1}, \tb{x}_2^{-1}) \ne (\tb{x}_1^*,\tb{x}_2^*)$, then
    \begin{align*}
        \lim_{n \to \infty}\KL \left((\tb{x}_1^*,\tb{x}_2^*),(\tb{x}_1^n,\tb{x}_2^n)\right) = +\infty.
    \end{align*}
    \end{itemize}
    \end{thm}


It is known that in a time-independent zero-sum game, (OMWU) and its several variants will converge to the equilibrium of the game \cite{daskalakis2018last,daskalakis2018limit}. The proofs for this kind of results are typically divided into two steps :

\begin{itemize}
\item Firstly, when  $(\tb{x}^t_1,\tb{x}^t_2)$ are far from the equilibrium $(\tb{x}^*_1,\tb{x}^*_2)$, the KL-divergence $\KL((\tb{x}_1^*,\tb{x}_2^*),(\tb{x}_1^t,\tb{x}_2^t)) $ decreases at each step, until $(\tb{x}^t_1,\tb{x}^t_2)$ is sufficiently close to $(\tb{x}^*_1,\tb{x}^*_2)$. 
\item Secondly, there exists a sufficient small neighbourhood of $(\tb{x}^*_1,\tb{x}^*_2)$, such that every points in the neighbourhood will eventually converge to this equilibrium.
\end{itemize}

Theorem \ref{thm: OMWU fails} implies both of these two reasons that lead to the last-iterate convergence of (OMWU) in time-independent games fail in the time-varying game defined by (\ref{2-periodic game_m}). Note that the second point in the theorem is stronger than the first point. However, to provide a clear comparison with (OMWU) in time-independent games, we state them individually.


%To be specific, the game in Theorem \ref{thm: OMWU fails} has the following properties:
%\begin{itemize}
%    \item In any neighborhood of the equilibrium, there always exist points that do not converge to the equilibrium.
%    \item When $(\tb{x}_1^t, \tb{x}_2^t)$ are far from the boundary in $\ell_1$ norm, the KL-divergence $\KL((\tb{x}_1^*, \tb{x}_2^*), (\tb{x}_1^t, \tb{x}_2^t))$ increases with every two iterations until either $\tb{x}_1^n$ or $\tb{x}_2^n$ is sufficiently close to the boundary in $\ell_1$ norm.
%    \item For both $\tb{x}_1^t$ and $\tb{x}_2^t$, there exists a neighborhood of the boundary, if either of them enters the neighborhood, it will eventually converge to the boundary.
%\end{itemize}


In Figure (\ref{KL-OMWU}), we present the evolution of the KL-divergence between equilibrium and strategies of players when using OMWU.

\begin{figure}[h]
    \centering
    \includegraphics[width=0.43\textwidth]{KL_OMWU_2d.png}
    \caption{KL-divergence of OMWU in periodic game.}
    \label{KL-OMWU}
\end{figure}




%    (\textcolor{red}{Proof strategy of $\KL$-divergence: Consider the composition dynamical system $\CF_2 \circ \CF_1$ which maps $(\tb{x}^{n-1}_1,\tb{x}^{n}_1,\tb{y}^{n-1}_1,\tb{y}^{n}_1)$ to $(\tb{x}^{n+1}_1,\tb{x}^{n+2}_1,\tb{y}^{n+1}_1,\tb{y}^{n+2}_1)$. Then it can be shown $(0,0,a,\frac{ae^{-3\eta}}{ae^{-3\eta} + (1-a)})$ is a fixed point of the dynamical system for arbitrary $a \in (0,1)$. The Jacobian  at these fixed points only has eigenvalues equal or smaller than $1$, moreover, then eigenspace corresponds to eigenvalue $1$ is 1-dimension, and spanned by $(0,0,\star,\star)$. Note that if a point of the system lies in this eigenspace, then it can only be the fixed point. Thus any points close to $(0,0,a,\frac{ae^{-3\eta}}{ae^{-3\eta} + (1-a)})$ will finally converge to this point, and thus the $\KL$-divergence will diverge to $\infty$.})

%The first point of the theorem states there are initial conditions that will not converge to the equilibrium. The second point of the theorem states that even a slight deviation from the equilibrium will be amplified by adding a constant in each subsequent steps until the strategies are very close to the boundary. 

%In proving the above theorem, we also find an interesting difference in the geometry of attracting points between OMWU in time-varying games and static games. In static games, the only possible attracting points are equilibrium; however, in the above game, there exists a curve such that every point on it is an attracting point but not an equilibrium.

 


\begin{thm} \label{T2}
For a periodic game defined by the payoff matrices $\{A_t\}^{\CT}_{t=1}$ with an unique\footnote{As games with non-unique equilibrium have a measure of zero in all games, this assumption is not overly restrictive.} common fully mixed equilibrium , (Extra-MWU) will converge to this equilibrium if the step size $\eta$ satisfies $\eta \cdot \max_{t \in [\CT]}\lVert A_t \lVert < 1$. 
\end{thm}

The last-iterate convergence property of Extra-MWU, and more generally, Extra-gradient mirror descent in time-independent game, was studied in \citep{mertikopoulos2018optimistic}. Note that although they referred to the algorithm they studied optimistic mirror descent, their method aligns with the Extra-gradient paradigm in the sense that the algorithm requires a two-step update in each round. The key property utilized in their proof is that the Bregman divergence (a generalization of the KL-divergence) between a fully mixed equilibrium and current strategies of players, when they use Extra-gradient mirror descent, is a decreasing function. We demonstrate that this property also holds for Extra-MWU in a periodic game if the game series in the periodic game has a common fully mixed equilibrium.


In Figure (\ref{str_EMWU}), we present the trajectories of strategies for a player using the Extra-MWU algorithm. The periodic game here is the same as (\ref{2-periodic game_m}). We can see that the strategy converges to the equilibrium $(0.5,0.5)$ of the player.


\begin{figure}[h]
    \centering
    \includegraphics[width=0.43\textwidth]{Strategy_Extra_MWU.png}
    \caption{Trajectories of strategies for a player when using Extra-MWU in the periodic game defined in (\ref{2-periodic game_m}).}
    \label{str_EMWU}
\end{figure}