% \vspace{-0.25cm}
\section{Discussion}
\label{sec:discussion}
% \vskip -0.2cm

We have provided an efficient quadratic metric elicitation strategy with application to  fairness, and with a query complexity that has the same dependence on the number of unknowns as that for linear metrics. 

\textbf{Higher Order Polynomials:} We next show how our approach can be extended to elicit \emph{higher-order polynomial} metrics. Thus our work not only increases the use-cases for ME but also opens the door for non-linear metric elicitation in other fields such as \emph{active learning.}

Consider, e.g., a cubic polynomial:
\begin{align*}
% \vspace{-0.7cm}
% \textstyle
    \phi^{\text{cubic}}(\rmbf)\coloneqq \sum_{i}a_ir_i + \frac{1}{2}\sum_{i,j}B_{ij}r_ir_j + \frac{1}{6}\sum_{i,j,l}C_{ijl}r_ir_jr_l,
\end{align*}
% \vskip -0.3cm
where $\Bmbf$ and $\Cmbf$ are symmetric, and $\sum_i a_i^2 +\sum_{ij} B_{ij}^2 + \sum_{ijl} C_{ijl}^2 = 1$ (w.l.o.g., due to scale invariance).  A quadratic approximation to this metric around a point $\zmbf$ is given by: 
\begin{align*}
&\frac{1}{2}\left(\sum_{i,j}B_{ij}r_ir_j + \sum_{i,j,l}C_{ijl}(r_i - z_i)(r_j - z_j)z_l\right) + \\
&\quad \; \sum_{i}a_ir_i + c,
\end{align*}
where $c$ is a constant not affecting the oracle responses. We can estimate the parameters of this  approximation by applying the QPME procedure from Algorithm~1 with the metric centered at an appropriate point, and its queries restricted to a small neighborhood around $\zmbf$. Running QPME once using a sphere around the point $\zmbf_l = \ombf + (\varrho - \varrho')\alphambf_l$, where $\varrho' < \varrho$ will elicit one face of the tensor $\Cmbf_{[:, :, l]}$ upto a scaling factor. Thus, it will require us to run the QPME procedure $k$ times around the basis points $\zmbf_l = \ombf + (\varrho - \varrho')\alphambf_l \; \; \forall l \in [k]$. Since we elicit scale-invariant quadratic approximation, we would need additional run of QPME procedure around the point $\Scal_{-\zmbf_1}$ to elicit all the coefficients. Thus, we can recover the metric $\hat{\phi}^{\text{cubic}} = (\hat{\ambf}, \hat{\Bmbf},\hat{\Cmbf})$ with as many queries as the number of unknowns, i.e, $\tilde O(k^3)$ in the cubic case. 

For a $d$-th order polynomial, one can recursively apply this procedure to estimate $(d-1)$-th order approximations at multiple points, and similarly derive the polynomial coefficients from the estimated local approximations.


\textbf{Handling large number of classes:} For applications where $k$ is very large, the parameterization discussed in Section \ref{sec:background} may not be  applicable in its current form. For example, when $k=1000$, the quadratic metric in \eqref{eq:quadmet} would use $O(1000^2)$ parameters, an exorbitantly high number to elicit in practice. Note that
the presence of $k^2$ unknowns is an  artifact of the problem formulation, and \emph{not} of our proposed procedure.
Moreover, as shown in Theorem \ref{thm:lb}, it is \emph{theoretically impossible} to estimate $k^2$ unknowns with fewer than $\tilde{O}(k^2)$ queries. While our QPME procedure does indeed match this lower bound, in practice, we do not expect it to be applied to estimate such an over-parameterized metric. Instead, for such large-scale settings, we  recommend making reasonable assumptions on the metric to reduce the number of unknowns, e.g., by having multiple classes share the same parameter, and the query complexity of QPME would then only depend \emph{linearly} on the reduced number of unknowns. For instance, in Table~\ref{append:tab:apxranking} (Appendix~\ref{append:ssec:ranking}), we show  that by simplifying the metric with structural assumptions, one can use fewer queries in practice to get comparable results. 

\textbf{Advantages:}\ Our proposal comes with many practical advantages: (a) \emph{Fairness:} we are aware of no prior work that can elicit fair quadratic metrics, particularly with provable guarantees; (b) \emph{Transportability:} our method is independent of the population $\Pmbb$, which allows any metric that is elicited using one dataset or model class to be applied to other applications, as long as the expert believes the tradeoffs to be the same; and (c) \emph{Feasibility:} we ensure that the rates are feasible throughout the elicitation (i.e., are achievable by classifiers), which allows  the flexibility to deploy systems that either compare classifiers or compare rates.

\textbf{Limitations:} Limitations of our work include the assumption that the metric has a parametric form, which can be restrictive in some cases, and not providing a concrete answer to who the oracles should be. One should also be cautious in applying ME to eliciting fairness metrics, as failures here could exacerbate the adverse effects on protected groups. 

\textbf{Future Work:} To thoroughly answer the above questions, we are actively conducting user studies on collecting preference feedback using intuitive visualizations of rates \citep{shen2020designing,beauxis2014visualization} or classifiers \citep{ribeiro2016should} . Please see Appendix~\ref{append:userstudy} to take a peek into the future work, where we discuss findings from a preliminary user study.