We explore value-based multi-agent reinforcement learning (MARL) in the popular paradigm of centralized training with decentralized execution (CTDE). CTDE requires the consistency of the optimal joint action selection with optimal individual action selections, which is called the IGM (Individual-Global-Max) principle. However, in order to achieve scalability, existing MARL methods either limit representation expressiveness of their value function classes or relax the IGM consistency, which may lead to poor policies or even divergence. This paper presents a novel MARL approach, called duPLEX dueling multi-agent Q-learning (QPLEX), that takes a duplex dueling network architecture to factorize the joint value function. This duplex dueling architecture transforms the IGM principle to easily realized constraints on advantage functions and thus enables efficient value function learning. Theoretical analysis shows that QPLEX solves a rich class of tasks. Empirical experiments on StarCraft II unit micromanagement tasks demonstrate that QPLEX significantly outperforms state-of-the-art baselines in both online and offline task settings, and also reveal that QPLEX achieves high sample efficiency and can benefit from offline datasets without additional exploration. * Equal contribution.Preprint. Under review.
Designing protein sequences with a particular biological function is a long-lasting challenge for protein engineering. Recent advances in machine-learning-guided approaches focus on building a surrogate sequence-function model to reduce the demand for expensive in-lab experiments. In this paper, we study the exploration mechanism of model-guided sequence design. We leverage a natural property of protein fitness landscape that a concise set of mutations upon the wild-type sequence are usually sufficient to enhance the desired function. By utilizing this property, we propose Proximal Exploration (PEX) algorithm that conducts a proximal optimization framework to prioritize the search for advantageous mutants with low mutation counts. In addition, we construct a specialized model architecture to predict low-order mutational effects, which further improves the sample efficiency of model-guided evolutionary search. We extensively evaluate our method on a suite of in-silico protein sequence design tasks and demonstrate substantial improvement over baseline algorithms.
In this study, the continuum-based discrete element method (CDEM) was adopted to simulate the evolution of mining-induced stress and fracturing during roadway tunnelling and mining in multilayered heterogeneous rock strata. The CDEM integrates the finite element method (FEM) and the discrete element method (DEM) to characterize the mining-induced stress evolution, the discontinuous fractures, separations, and caving that occur in the interfaces between multilayered rock strata. The maximum tensile-stress criterion and the Mohr-Coulomb strength criterion were used to evaluate the tensile and shear failure of the material elements. The CDEM model for rock strata was constructed by employing image processing and reconstruction approaches, using the geometrical and physical parameters that were measured from a real coal mining site. The stress evolution and compression deformation of the roof and floor strata were computed to evaluate the criticality of mining-induced disasters. The constructed model was employed to simulate and analyse the immediate roof collapse, immediate floor bulges, and the compaction of the collapse blocks, as well as the large deformation, separation, and collapse between the immediate roof and the main roof during coal seam mining. It was shown that the proposed method could predict ranges for the caving zone and fracture zone in the rock roofs that were in good agreement with the observation results from the real coal mining site.
Quantitative visualization and characterization of stress-field evolution during fracture rapid growth is critical for understanding the mechanisms that govern the deformation and failure of solids in various engineering applications. However, the direct capture and accurate characterization of a rapidly-changing stress field during crack propagation remains a challenge. We report an experimental method to quantitatively visualize and characterize rapid evolution of the stress-field during crack propagation in a transparent disc model containing a penetrating fusiform crack. Three-dimensional (3D) printing technology and a stress-sensitive photopolymer resin were adopted to produce the disc model and to alleviate the residual processing stress that usually blurs the dynamic stress field due to overlap. A photoelastic testing system that synchronized a high-speed digital camera and a pulsed laser with a nanosecond full width at half maximum (FWHM) was used to capture the rapid evolution of the stress field in the vicinity of crack tips. The results show that the proposed method is suitable to directly visualize and quantitatively characterize the stress-field evolution during crack rapid propagation. It is proved that the crack propagation velocity is strongly governed by the stress field around the crack tips.
Quantification and visualization of the three‐dimensional (3‐D) whole‐field stress distribution in porous geomaterials are significant for solving various subsurface engineering problems in which stress governs the deformation, fracture propagation, and fluid transport inside the materials. However, quantitatively characterizing the whole‐field stress distribution inside 3‐D porous geomaterials is challenging because of the complications involved in identifying and extracting the hidden structures and stress distribution. This paper presents a method for extracting and characterizing whole‐field distributions of principal stress difference and shear stress inside a cubic model containing irregular pores, which are replicated using three‐dimensional printing techniques to model the real complex porous structures of natural geomaterials. We combine frozen‐stress, phase‐shifting, and unwrapping methods to identify and characterize the whole‐field distributions of principal stress difference and shear stress inside the pore structure. The unwrapped‐algorithm‐incorporated traditional phase‐shifting approach is proposed to determine the stress fringe deviation around irregular pores. A comparison of numerical simulation and photoelastic tests shows that this method can visually and quantitatively characterize the whole‐field stress of a 3‐D porous model. The stress distribution characterization and potential failure bands in the 3‐D porous structure model subjected to uniaxial compression load are discussed, and findings indicate that potential shear fractures with high‐concentration stress are formed before porous model failure, and 3‐D annular zones with high stress appear around individual pores. The proposed method and results will support future work on uncovering the mechanism of crack propagation and shear fracture formation in reservoir geomaterials.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.