Language models (LMs) are becoming the foundation for almost all major language technologies, but their capabilities, limitations, and risks are not well understood. We present Holistic Evaluation of Language Models (HELM) to improve the transparency of language models. First, we taxonomize the vast space of potential scenarios (i.e. use cases) and metrics (i.e. desiderata) that are of interest for LMs. Then we select a broad subset based on coverage and feasibility, noting what's missing or underrepresented (e.g. question answering for neglected English dialects, metrics for trustworthiness). Second, we adopt a multi-metric approach: We measure 7 metrics (accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency) for each of 16 core scenarios to the extent possible (87.5% of the time), ensuring that metrics beyond accuracy don't fall to the wayside, and that trade-offs across models and metrics are clearly exposed. We also perform 7 targeted evaluations, based on 26 targeted scenarios, to more deeply analyze specific aspects (e.g. knowledge, reasoning, memorization/copyright, disinformation). Third, we conduct a large-scale evaluation of 30 prominent language models (spanning open, limited-access, and closed models) on all 42 scenarios, including 21 scenarios that were not previously used in mainstream LM evaluation. Prior to HELM, models on average were evaluated on just 17.9% of the core HELM scenarios, with some prominent models not sharing a single scenario in common. We improve this to 96.0%: now all 30 models have been densely benchmarked on a set of core scenarios and metrics under standardized conditions. Our evaluation surfaces 25 top-level findings concerning the interplay between different scenarios, metrics, and models. For full transparency, we release all raw model prompts and completions publicly 3 for further analysis, as well as a general modular toolkit for easily adding new scenarios, models, metrics, and prompting strategies. 4 We intend for HELM to be a living benchmark for the community, continuously updated with new scenarios, metrics, and models.

No abstract

No abstract

We demonstrate the transmission of a 100 MW-peak-power ultrafast laser through a 5-m anti-resonant hollow-core fiber (AR-HCF) with a pumpable armored tube for air exhaust. The AR-HCF consists of a 45-μm-hollow-core and seven untouched capillaries with an attenuation of 0.11 dB/m measured at a wavelength of 1030 nm. We investigate the effect of air-filling and vacuum pumping on transmission efficiency and pulse distortion. The comparison reveals the importance of controlling air concentration in hollow-core fibers (HCFs) for achieving high transmission efficiency and pulse quality. With the suppression of air concentration, the transmission efficiency increases from 61% to 72%, and pulse distortion is effectively controlled. The results demonstrate the potential of AR-HCFs for high-power ultrafast laser delivery systems for various applications. The pumpable armored tube design provides a simple and effective solution to suppress self-phase modulation (SPM) and enable flexible beam delivery.

We report a 7-tube single-ring hollow-core anti-resonant fiber (SR-ARF) with a record low transmission loss of 4.3 dB/km @1080 nm, which is almost half of the current lowest loss record of an SR-ARF (7.7 dB/km @750 nm). The 7-tube SR-ARF has a large core diameter of 43 µm and a wide low-loss transmission window exceeding 270 nm for the 3-dB bandwidth. Moreover, it exhibits an excellent beam quality with an M2 factor of 1.05 after 10-m-long transmission. The robust single-mode operation, ultralow loss, and wide bandwidth make the fiber an ideal candidate for short-distance Yb and Nd:YAG high-power laser delivery.