Detection of relevant and meaningful communities from any social network is always a constant research challenge. Considering large social networks existing approaches often fail to capture perfect community partition or fail to converge fast. Inspired by nonmyopic reinforcement learning domain, in this paper, we have proposed a novel game theory based community detection algorithm which considers community retention based on temporal information as part of a node’s strategy (i.e. for a node whether their current assigned community is more profitable based upon future possibilities, or should they switch at the moment), thereby minimizing cross-community false node switches. The proposed method has the following properties (a) considers community retention for a node in the network when it considers switching to another community, (b) achieves significantly better/comparable performance w.r.t baselines in terms of metrics such as quality of partition on various real world datasets,(c) faster convergence w.r.t traditional game theoretic approach which only considers short-term utility by minimizing cross-community node switches (d) utility to interpret at each iteration for any particular node’s strategy.

Would I regret later joining this Community ? Using temporal neighborhood information for community retention in a game theoretic community detection framework

Satellites provide essential services such as networking and weather tracking, and the number of satellites are expected to grow rapidly in the coming years. Communications with terrestrial ground stations is one of the critical functionalities of any space mission. Satellite scheduling is a problem that has been scientifically investigated since the 1970s. A central aspect of this problem is the need to consider bandwidth resource contention and satellite visibility constraints as they require line of sight. 
Due to the combinatorial nature of the problem, prior solutions such as linear programs and evolutionary algorithms require extensive compute capabilities to output a feasible schedule for each scenario. Machine learning based scheduling can provide an alternative solution by training a model with historical data and generating a schedule quickly with model inference. We present SatNet, a benchmark for satellite scheduling optimization based on historical data from the NASA Deep Space Network. We propose formulation of the satellite scheduling problem as a Markov Decision Process and use reinforcement learning (RL) policies to generate schedules. The nature of constraints imposed by SatNet differ from other combinatorial optimization problems such as vehicle routing studied in prior literature. Our initial results indicate that RL is an alternative optimization approach that can generate candidate solutions of comparable quality to existing state-of-the-practice results. However, we also find that RL policies overfit to the training dataset and do not generalize well to new data, thereby necessitating continued research on reusable and generalizable agents. 

SatNet: A Benchmark for Satellite Scheduling Optimization

The column generation technique is essential for solving linear programs with an exponential number of variables. Many important applications such as the vehicle routing problem (VRP) now require it. However, in practice, getting column generation to converge is challenging. It often ends up adding too many columns. In this work, we frame the problem of selecting which columns to add as one of sequential decision-making. We propose a neural column generation architecture that iteratively selects columns to be added to the problem. The architecture, inspired by stabilization techniques, first predicts the optimal duals. These predictions are then used to obtain the columns to add. We show using VRP instances that in this setting several machine learning models yield good performance on the task and that our proposed architecture learned using imitation learning outperforms a modern stabilization technique.

Neural Column Generation for Capacitated Vehicle Routing

Due to the irregular nature of flight operations, airlines need to take a range of actions to recover their aircraft and crew schedules. The limited time frames prevent airlines from using a full-scale optimization approach. Consequently, airlines typically apply recovery solutions that can be far from optimal. This study proposes a practical method that combines machine learning and optimization to find improved recovery solutions. Our procedure is based on the idea that the most effective constraints to add to the recovery models without sacrificing the solution quality, can be determined in advance by leveraging the similarities between disruptions. Our experiments show that this approach can reduce solution time significantly while still achieving high-quality solutions.

Crew Recovery Using Machine Learning and Optimization

In this research, supervised machine learning is employed to expedite optimization. The core idea is to try to identify components of optimal solutions to new problem instances by leveraging their similarity with alternative (historical) problem instances presented in the offline model-training phase. Our approach prunes the decision space by using machine learning algorithms that are trained using the network- wide optimal decisions for similar scenarios.

Machine Learning Based Aircraft Recovery Optimization

We introduce a benchmark of 10,000 instances with heterogeneous characteristics for the capacitated vehicle routing problem. We also provide optimal solutions for almost all of them along with a generator to produce additional training and validation data. This benchmark aims to permit a more systematic comparison of machine learning based search algorithms on this important problem. We also emit recommendations regarding the correct use of this dataset.

10,000 optimal CVRP solutions for testing machine learning based heuristics

Electricity is an integral part of modern society, yet globally millions of people are without access. This lack of access, coupled with increasing concern over climate change represents a serious global challenge. Distributed energy storage will likely play a large part in the future of the grid, however, battery management remains an open problem. In this work, we re-frame the battery management problem in an Operations Research (OR) context as a multi-agent newsvendor problem. We benchmark seven Multi-Agent Reinforcement Learning (MARL) algorithms and compare their performance with five popular handcrafted heuristic strategies.
We considered MARL algorithms due to their capacity to learn novel policies from data that may outperform handcrafted rule-based policies, especially as problem complexity increases. We find that all seven methods learn policies that achieve comparable results to each other and outperform a simple keep-fully-charged heuristic consistently. However, they do not consistently outperform all the heuristics considered in all the scenarios considered.

Application of Multi-Agent Reinforcement Learning for Battery Management in Renewable Mini-Grids

Road network restoration is an important issue in the post-disaster disposal and rescue, especially when extraor-dinarily serious natural disasters (e.g., floods and earth-quakes) occur. Central to this endeavor is the problem of determining how to reasonably schedule and route the re-pair crew to quickly restore the damaged road network and establish reliable supply lines from supply nodes to demand nodes. However, most existing work focuses on the activities of the single repair crew, and rarely consid-ers the problem of continuously damaged road sections, especially for the road network with enormous demand nodes and serious damage. Consequently, this work is concentrated on multicrew scheduling and routing for the damaged road network with enormous demand nodes. Specifically, a model of multicrew scheduling and routing is first presented. Next, a road network partition strategy is first proposed to make different repair crews responsible for different subnets. Then, a deep Q-learning based mul-ticrew scheduling and routing algorithm is proposed for the damaged road network with enormous demand nodes, which utilizes the learning experience of multiple repair crews to achieve hybrid learning. Finally, experimental re-sults demonstrate that the proposed method can make re-pair crews adjust their scheduling and routing strategies according to the damaged road network and provides a useful attempt to restore the damaged road network in complex emergency scenarios of post-disaster.

Multicrew Scheduling and Routing in Road Network Restoration Based on Deep Q-learning

The simultaneous stochastic optimization of mining complexes (SSOMC) is a large-scale combinatorial optimization problem that manages the extraction of materials from multiple mines and their processing using interconnected facilities. Following the work of Zarpellon et al. (2020) and Chmiela et al. (2021), to the best of our knowledge, this work proposes the first data-driven framework for heuristic scheduling in a hyper-heuristic-based solver that is fully self-managed to solve the SSOMC. The proposed learn-to-perturb (L2P) hyper-heuristic is a multi-neighborhood simulated annealing algorithm. The L2P selects the heuristic (perturbation) to apply in a self-adaptive manner using reinforcement learning (RL) to efficiently explore which local search is best suited for a particular search point. Several state-of-the-art agents have been incorporated into the proposed hyper-heuristic to better adapt the search and guide it towards better solutions. By learning from data describing the performance of heuristics, a problem-specific ordering of heuristics that collectively finds better solutions faster is obtained. The L2P is tested on several real-world mining complexes, with an emphasis on efficiency, robustness, and generalization capacity. Results show a reduction in the computational time by 30-45%.

Learning to Schedule Heuristics for the Simultaneous Stochastic Optimization of Mining Complexes

We propose a supervised learning algorithm for the multi-period inventory problem (MPIP) that tackles  shortcomings of existing multi-step, model-based methods on the one and policy-free reinforcement learning algorithms on the other hand. As a model-free end-to-end (E2E) method that takes advantage of auxiliary data, it avoids pitfalls like model misspecification, multi-step error accumulation and computational complexity induced by a repeated optimization step. Furthermore, it manages to leverage domain knowledge about the optimal solution structure. To the best of our knowledge, this is one of the first supervised learning approaches to solve the MPIP and the first one to learn policy parameters. Given the variety of settings in which OR researchers have developed well-performing policies, our approach can serve as a blueprint of how to design E2E methods that leverage that knowledge. We validate our hypotheses on synthetic data and demonstrate the effect of individual model characteristics.

Towards Supervised Learning of Optimal Replenishment Policies

Combinatorial optimization problem (COP) over graphs is a fundamental challenge in optimization. Reinforcement learning (RL) has recently emerged as a new framework to tackle these problems and has demonstrated promising results. However, most RL solutions employ a greedy manner to construct the solution incrementally, thus inevitably pose unnecessary dependency on action sequences and need a lot of problem-specific designs. We propose a general RL framework that not only exhibits state-of-the-art empirical performance but also generalizes to a variety class of COPs. Specifically, we define state as a solution to a problem instance and action as a perturbation to this solution. We utilize graph neural networks (GNN) to extract latent representations for given problem instances for state-action encoding, and then apply deep Q-learning to obtain a policy that gradually refines the solution by flipping or swapping vertex labels. Experiments are conducted on Maximum $k$-Cut and Traveling Salesman Problem and performance improvement is achieved against a set of learning-based and heuristic baselines.

AAAI 2022 Workshop ML4OR-22 Submissions