\section{A Joint Decision Framework for DM and Forecaster}
\label{section:setup}

% \Alan{Can you come up with a more precise title to describe this section? Our setup sounds vague.}

In this work, we consider scenarios where an agent is tasked with selecting an input $x$ from a finite space of inputs $\cX := \{x_1,\ldots,x_n\}$. Agent's choice of input $x\in\cX$ and outcome of uncertain event $o\in\cO$ quantify the utility $u(x,o)$ obtained by the agent. In the case of a precise forecaster, $\cX:=\Delta(\cO)$ and Eq. \eqref{eq:expected-score-utility} shows how the precise score $u(x,o):=s(p,o)$ acts as a utility for the forecaster, underlining the decision-making aspect within elicitation. From the DM's perspective, $\cX:=\cA$ where $\cA:=\{a_1,\dots,a_m\}$ denotes the finite space of actions which DM can choose from. Depending upon the outcome $o\in\cO$, the DM obtains $u(x,o):=u(a,o)$ as the utility. 

\subsection{Decision-Making with Forecasts}
\label{sec:dmwithforecasts}
There exists a crucial difference between decision-making with imprecise forecasts v.s. precise forecasts. In the case of precise forecasts, the agent (forecaster or DM) has a precise belief or report $p\in\Delta(\cO)$. Using $p$ allows them to define a complete preference relation $\succeq_p$ over $\cX$ based on several well-established rationality frameworks~\citep{von2007theory,savage1972foundations}. Thereby, allowing the agent to select the corresponding best input $x^*$. This $x^*$ represents the best forecast to report in the case of a precise forecaster and the best action to take in the case of DM. However, in scenarios where the belief (or obtained report) for an agent is a set of presice beliefs $\cP\subseteq \Delta(\cO)$, the preference relation ($\succeq_{\cP}$) obtained on $\cX$ using $\cP$ is incomplete. In this case, a natural way to define $\succeq_{\cP}$ is based on the idea of dominance. 
\begin{definition}
\label{def:decision-making-with-imprecise-forecasts}
    Consider $\cP\subseteq \Delta(\cO)$, then the corresponding preference relation $\succeq_{\cP}$ over $\cX$ for a VNM rational~\citep{von2007theory} agent can be defined as follows: for all $x,x'\in\cX$,
    \begin{align*}
        x\succeq_{\cP} x'\quad \text{iff}\quad \mathbb{E}_{p}[u(x,o)]\geq \mathbb{E}_{p}[u(x',o)] \quad \forall p\in\cP.
    \end{align*}
\end{definition}
Unless $\cP$ is implicitly a precise forecast of type $\{p\in\Delta(\cO)\}$, the preference relation $\succeq_{\cP}$ is a partial order over $\cX$. The partial order $\succeq_{\cP}$ can be incomplete, since there can be a pair of inputs $x,x'\in\cX$ such that $x'\not\succeq_{\cP}x$ and $x\not\succeq_{\cP}x'$. In other words, $x$ and $x'$ are incomparable. This can result in indecision for the agent. This means that both the forecaster and the DM face indecision when they rely on $\cP$ for their respective tasks (elicitation or decision-making).

\subsection{Imprecise Forecaster}
Our work focuses on analyzing scoring rules in scenarios where the forecaster may be \emph{imprecise}. Specifically, we formalise the notion of an imprecise forecaster and their truthfulness below.
\begin{definition}\label{def:imprecise-forecast}
A forecaster is imprecise if their belief can be expressed as a set of distributions $\cP\subseteq\Delta(\cO)$. A report $\cQ\subseteq\Delta(\cO)$ is called an imprecise forecast, which implicitly includes precise forecasts $\cQ=\{q\}$ for some $q\in\Delta(\cO)$. 
\end{definition}
\Cref{def:imprecise-forecast} generalizes the precise setting as it allows the forecaster to express their (partial) ignorance by reporting both aleatoric uncertainties (as elements in the set) and epistemic uncertainties (as the set itself)~\citep{hullermeier_aleatoric_2021}. This subsumes both scenarios where the forecaster's belief is truly imprecise, e.g., the probability that it will rain tomorrow is $[0.6,0.8]$, and where their belief is calibrated with respect to multiple sources of potentially conflicting information, e.g., the estimated probability based on data from multiple weather stations. Moreover, this can also be interpreted as a ``collective'' report obtained from multiple (potentially conflicting) precise forecasters. Imprecise probability scoring rules can be defined analogously to precise scoring rules as follows.
\begin{definition}(Imprecise Probability Scoring Rule)
An imprecise probability (IP) scoring rule $s:2^{\Delta(\cO)}\times\cO\rightarrow\mathbb{R}\cup\{-\infty\}$ assigns a score of $s(\cQ,o)$ to a report $\cQ\subseteq \Delta(\cO)$ when the outcome $o\in\cO$ is realized.
\end{definition}
Analogous to precise setting, an IP scoring rule is \emph{regular} if $s(\cQ,o)\in\mathbb{R}$ for all $o\in\cO$, except if $q(o)=0$ for all $q\in\cQ$, then $s(\cQ,o)=-\infty$. To define regularity analogous to the precise setting we consider for all $q\in\cQ$, since otherwise reporting a vacuous set $\Delta(\cO)$ or other imprecise sets will have $-\infty$ as an incentive, thereby discouraging the forecaster from reporting their epistemic uncertainty. 
The score $s(\cQ,o)$ obtained by the forecaster induces a corresponding set of utilities $\bm{V}^\cP(\cQ)$ for the forecaster with an imprecise belief $\cP$, representing the expected utility of the imprecise score with respect to every distribution within their belief $\cP$.  We define this utility set as follows:
\begin{equation*}
    \bm{V}^\cP(\cQ)=\{\mathbb{E}_p[s(\cQ,o)]\}_{p\in\cP}
\end{equation*}
From the forecaster's perspective, this collection of expected utility functions $\bm{V}^\cP:2^{\Delta(\cO)}\rightarrow\mathbb{R}^{|\cP|}$, for each report $\cQ$ result in a range of plausible expected utility, i.e.,  
\begin{equation*}
    \mathrm{im}(\bm{V}^{\cP}(\cQ))=\Bigg[\inf_{p\in \cP}\mathbb{E}_{p}[s(\cQ,o)],\sup_{p\in \cP}\mathbb{E}_{p}[s(\cQ,o)]\Bigg]
\end{equation*}
where $\mathrm{im}$ is the image or the range of the forecaster's minimum and maximum expected score for forecast $\cP$ when its extreme points exist; see~\Cref{appendix:existence-of-extreme-points} for further details. 
While the equivalence of two precise distributions $p$ and $q$ is natural, i.e., $p=q$ or not. The equivalence of two imprecise beliefs is not obvious as they are sets of distributions. We now define the equivalence of two beliefs $\cP,\cP'$ in the context of elicitation as follows.

\begin{definition}(Equivalence of Imprecise Beliefs)
\label{def:equivalenceofimpreciseforecasts}
Two beliefs $\cP,\cP'\subseteq \Delta(\cO)$ are considered equivalent, denoted as $\cP\simeq \cP'$, if for all IP scoring rules $s$ and forecasts $\cQ\subseteq\Delta(\cO)$, we have $\mathrm{im}(\bm{V}^{\cP}(\cQ))=\mathrm{im}(\bm{V}^{\cP'}(\cQ))$.
\end{definition} 
%Intuitively, we define the equivalence of two imprecise forecasts based on the premise that they do not differ in the range of plausible expected utilities for any scoring rule $s$ and reported imprecise forecast $\cQ$, i.e. the decision-making induced by two imprecise beliefs $\cP$ and $\cP'$ is same under all scenarios. We now show that \Cref{def:equivalenceofimpreciseforecasts} when used for precise forecasts does not change the classic notion of equivalence. 
Intuitively, two imprecise forecasts are equivalent if they yield the same range of plausible expected utilities for any scoring rule $s$ and reported forecast $\cQ$—that is, they induce identical decision-making. We now show that \Cref{def:equivalenceofimpreciseforecasts} reduces to the classic notion of equivalence between probability distributions when applied to precise forecasts.
\begin{proposition}
\label{prop:equivalenceofpreciseforecasts}
    For all $p,q\in \Delta(\cO)$, $\{p\}\simeq \{q\}$ iff $p=q$.  
\end{proposition}
With Proposition~\ref{prop:equivalenceofpreciseforecasts}, we establish that \Cref{def:equivalenceofimpreciseforecasts} generalises from the notion of equivalence of precise forecasts, i.e. distributions to imprecise forecasts. We can also characterize the equivalence of two imprecise forecasts as the equivalence of their corresponding credal sets. 
\begin{proposition}
\label{prop:credalsets}
For imprecise beliefs $\cP,\cP'\subseteq \Delta(\cO)$ with non-empty extreme points, $\cP\simeq \cP'$ iff $co(\cP)=co(\cP')$.   
\end{proposition} 
It has previously been shown that two sets of distributions must be credal sets to induce the same rational decision-making behaviour~\citep{troffaes2007decision,huntley2014decision,troffaes2014lower}. \Cref{def:equivalenceofimpreciseforecasts} defines the equivalence of two imprecise beliefs w.r.t elicitation and Proposition~\ref{prop:credalsets} establishes its equivalence to rational decision making. This allows us to consider elicitation as a decision-making task for the forecaster. 
As a consequence of Proposition ~\ref{prop:credalsets}, even though a forecaster believes in a set of probability distributions $\cP$. We restrict our focus to evaluating a credal set of forecasts $co(\cP)$. Therefore, from now on, we will assume that $\mathcal{P}$ is a convex set.

%\Krik{``unless it is unclear from the context'' sounds strange. Based on my experience, I only heard of ``unless it is clear from the context''. You say ``we use $\cP$ to represent $co(\cP)$ unless it is unclear from the context. What happen if it's unclear?'' Do you use $co(\cP)$ and $\cP$ to represent different things ``when it is unclear from the context''? If so, then there is no ambiguity anymore, hence ``unless it is unclear from the context'' seems unnecessary.}

\begin{definition}(Truthfulness of Imprecise Forecaster)
\label{def:truthfulness}
Let $\cP\subseteq\Delta(\cO)$ be the true belief of an imprecise forecaster. A report $\cQ\subseteq\Delta(\cO)$ is truthful
%\emph{aleatoric} sense if $\cQ\subseteq\cP$. \emph{epistemic} sense 
if $\cQ\simeq\cP$.
\end{definition}

\Cref{def:truthfulness} generalizes the concept of truthfulness in the precise setting. An imprecise forecaster who reports their true belief is considered truthful. For instance, if the forecaster believes the probability of rain tomorrow lies within the interval $[0.6,0.8]$, then they must report their actual epistemic uncertainty by reporting the interval $[0.6,0.8]$.
%An imprecise forecaster who reports a subset of their true belief is considered truthful, but only in an aleatoric sense. For instance, if the forecaster believes the probability of rain tomorrow lies within the interval $[0.6,0.8]$, they might report it as $0.7$ or $[0.65,0.75]$. While this report conveys information within the bounds of their true belief, it is not considered truthful in an epistemic sense. This is because the reported uncertainty does not fully reflect the forecaster's actual level of epistemic uncertainty, which in this case is represented by the wider interval $[0.6,0.8]$.