To fill up this gap, we propose a procedure that is based on a double penalized hierarchical likelihood (DPHL). It has the desirable oracle property in theory and can be easily implemented through a two-stage algorithm without iteration. Specifically, we first consider an approximated data likelihood derived from the hierarchical likelihood (H-likelihood, see Lee and Nelder, 1996) to avoid the calculation of high-dimensional integral. Then, to simultaneously select fixed and random effects, we employ a modified Cholesky decomposition to factorize the covariance matrix of random effects so that random effects can be, in terms of the covariance parameters, incorporated into the model as regression coefficients. This re-parameterization can also guarantee the positive definiteness of the covariance matrix of selected random effects. Subsequently, we employ the idea of penalization to solve an unconstrained optimization problem. Via the H-likelihood, we develop a two-stage non-iterative algorithm, which is computationally efficient. Further, a consistent H-likelihood-based BIC criterion is proposed for tuning parameter selection. Simulation studies suggest that the proposed algorithm is computationally more efficient than those based on the EM-type algorithm.
