Expression pattern of histone lysine‑specific demethylase 6B in gastric cancer

Wang, Shujun; Wang, Yiping; Zhu, Hui; Chen, Miaohui; Zhang, Liang

doi:10.3892/ol.2021.12752

Cited by 5 publications

(5 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The differences between PMET and existing methods are illustrated in Figure 1. Our experiments demonstrate that PMET exhibits state-of-the-art comprehensive performance in editing GPT-J (6B) (Wang and Komatsuzaki 2021) and GPT-NeoX (20B) (Black et al 2022) on the zsRE and COUNTERFACT datasets. Specifically, in COUNTERFACT dataset, PMET shows a 3.3% average reliability enhancement over the state-of-the-art method, while in zsRE dataset, it achieves a 0.4% average improvement.…”

Section: Introductionmentioning

confidence: 86%

“…Here, W E and γ represent the embedding matrix and layernorm, respectively, and a L z and m L z are the TC hidden states of the MHSA and FFN of the L-th layer, respectively. Note that the MHSA and FFN in (1) are parallel (Wang and Komatsuzaki 2021;Black et al 2022). The general forms of the MHSA and FFN at the l-th layer and the j-th token x l j are given by:…”

Section: Methodology Preliminariesmentioning

confidence: 99%

See 1 more Smart Citation

PMET: Precise Model Editing in a Transformer

Li,

Song

et al. 2024

AAAI

View full text Add to dashboard Cite

Model editing techniques modify a minor proportion of knowledge in Large Language Models (LLMs) at a relatively low cost, which have demonstrated notable success. Existing methods assume Transformer Layer (TL) hidden states are values of key-value memories of the Feed-Forward Network (FFN). They usually optimize the TL hidden states to memorize target knowledge and use it to update the weights of the FFN in LLMs. However, the information flow of TL hidden states comes from three parts: Multi-Head Self-Attention (MHSA), FFN, and residual connections. Existing methods neglect the fact that the TL hidden states contains information not specifically required for FFN. Consequently, the performance of model editing decreases. To achieve more precise model editing, we analyze hidden states of MHSA and FFN, finding that MHSA encodes certain general knowledge extraction patterns. This implies that MHSA weights do not require updating when new knowledge is introduced. Based on above findings, we introduce PMET, which simultaneously optimizes Transformer Component (TC, namely MHSA and FFN) hidden states, while only using the optimized TC hidden states of FFN to precisely update FFN weights. Our experiments demonstrate that PMET exhibits state-of-the-art performance on both the \textsc{counterfact} and zsRE datasets. Our ablation experiments substantiate the effectiveness of our enhancements, further reinforcing the finding that the MHSA encodes certain general knowledge extraction patterns and indicating its storage of a small amount of factual knowledge. Our code is available at \url{https://github.com/xpq-tech/PMET}.

show abstract

Section: Introductionmentioning

confidence: 86%

Section: Methodology Preliminariesmentioning

confidence: 99%

PMET: Precise Model Editing in a Transformer

Li,

Song

et al. 2024

AAAI

View full text Add to dashboard Cite

show abstract

“…Language Model to be Edited Following setup of previous work (Meng et al 2022a,b;Zhong et al 2023), we use GPT-J (6B) (Wang and Komatsuzaki 2021) as the base LLM to be edited with above methods.…”

Section: Methodsmentioning

confidence: 99%

History Matters: Temporal Knowledge Editing in Large Language Model

Yin,

Jiang,

Yang

et al. 2024

AAAI

View full text Add to dashboard Cite

The imperative task of revising or updating the knowledge stored within large language models arises from two distinct sources: intrinsic errors inherent in the model which should be corrected and outdated knowledge due to external shifts in the real world which should be updated. Prevailing efforts in model editing conflate these two distinct categories of edits arising from distinct reasons and directly modify the original knowledge in models into new knowledge. However, we argue that preserving the model's original knowledge remains pertinent. Specifically, if a model's knowledge becomes outdated due to evolving worldly dynamics, it should retain recollection of the historical knowledge while integrating the newfound knowledge. In this work, we introduce the task of Temporal Knowledge Editing (TKE) and establish a benchmark AToKe (Assessment of TempOral Knowledge Editing) to evaluate current model editing methods. We find that while existing model editing methods are effective at making models remember new knowledge, the edited model catastrophically forgets historical knowledge. To address this gap, we propose a simple and general framework termed Multi-Editing with Time Objective (METO) for enhancing existing editing models, which edits both historical and new knowledge concurrently and optimizes the model's prediction for the time of each fact. Our assessments demonstrate that while AToKe is still difficult, METO maintains the effectiveness of learning new knowledge and meanwhile substantially improves the performance of edited models on utilizing historical knowledge.

show abstract

“…In Section 3, we mentioned that different LMs possess different capacities for generating FL expressions given their pretraining data/objectives. We employ two LM classes for experimentation: Vicuna 13B (instruction-tuned version of the Llama 13B model (Touvron et al 2023)) to generate Python, and GPT-J 6B (Wang and Komatsuzaki 2021) to generate pseudocode as intermediate FL expressions. We employ SymPy as the symbolic solver on top of Vicuna.…”

Section: Methodsmentioning

confidence: 99%

Frugal LMs Trained to Invoke Symbolic Solvers Achieve Parameter-Efficient Arithmetic Reasoning

Dutta,

Pandey,

Singh

et al. 2024

AAAI

View full text Add to dashboard Cite

Large Language Models (LLM) exhibit zero-shot mathematical reasoning capacity as a behavior emergent with scale, commonly manifesting as chain-of-thoughts (CoT) reasoning. However, multiple empirical findings suggest that this prowess is exclusive to LLMs that have exorbitant sizes (beyond 50 billion parameters). Meanwhile, educational neuroscientists suggest that symbolic algebraic manipulation be introduced around the same time as arithmetic word problems so as to modularize language-to-formulation, symbolic manipulation of the formulation, and endgame arithmetic. In this paper, we start with the hypothesis that much smaller LMs, which are weak at multi-step reasoning, can achieve reasonable arithmetic reasoning if arithmetic word problems are posed as a formalize-then-solve task. In our architecture, which we call SyReLM, the LM serves the role of a translator to map natural language arithmetic questions into a formal language (FL) description. A symbolic solver then evaluates the FL expression to obtain the answer. A small frozen LM, equipped with an efficient low-rank adapter, is capable of generating FL expressions that incorporate natural language descriptions of the arithmetic problem (e.g., variable names and their purposes, formal expressions combining variables, etc.). We adopt policy-gradient reinforcement learning to train the adapted LM, informed by the non-differentiable symbolic solver. This marks a sharp departure from the recent development in tool-augmented LLMs, in which the external tools (e.g., calculator, Web search, etc.) are essentially detached from the learning phase of the LM. SyReLM shows massive improvements (e.g., +30.65 absolute point improvement in accuracy on the SVAMP dataset using GPT-J 6B model) over base LMs, while keeping our testbed easy to diagnose and interpret, and within the reach of most researchers.

show abstract

Expression pattern of histone lysine‑specific demethylase 6B in gastric cancer

Cited by 5 publications

References 44 publications

PMET: Precise Model Editing in a Transformer

PMET: Precise Model Editing in a Transformer

History Matters: Temporal Knowledge Editing in Large Language Model

Frugal LMs Trained to Invoke Symbolic Solvers Achieve Parameter-Efficient Arithmetic Reasoning

Contact Info

Product

Resources

About