Fine-Tuned Transformer Model for Sentiment Analysis

Abstract

Foundational generative models should be traceable to protect their owners and facilitate safety regulation. To achieve this, traditional approaches embed identifiers based on supervisory trigger-response signals, which are commonly known as backdoor watermarks. They are prone to failure when the model is fine-tuned with nontrigger data. Our experiments show that this vulnerability is due to energetic changes in only a few 'busy' layers during fine-tuning. This yields a novel arbitrary-in-arbitrary-out (AIAO) strategy that makes watermarks resilient to fine-tuning-based removal. The trigger-response pairs of AIAO samples across various neural network depths can be used to construct watermarked subpaths, employing Monte Carlo sampling to achieve stable verification results. In addition, unlike the existing methods of designing a backdoor for the input/output space of diffusion models, in our method, we propose to embed the backdoor into the feature space of sampled subpaths, where a mask-controlled trigger function is proposed to preserve the generation performance and ensure the invisibility of the embedded backdoor. Our empirical studies on the MS-COCO, AFHQ, LSUN, CUB-200, and DreamBooth datasets confirm the robustness of AIAO; while the verification rates of other trigger-based methods fall from ∼90% to ∼70% after fine-tuning, those of our method remain consistently above 90%.

A social media Data-Driven analysis for transport policy response to the COVID-19 pandemic outbreak in Wuhan, China

“… Alexander, 2014 , Belkahla Driss et al, 2019 , Benini, 2020 , Bokányi et al, 2021 , Chen et al, 2020a , Chen et al, 2020b , Liu et al, 2020a , Liu et al, 2020b , Ma et al, 2019 , Qiang et al, 2020 , Röder et al, 2015 , Singh et al, 2019 , Vega-Carrasco et al, 2020 , Yue, 2017 , Zhan et al, 2014 , Zhang, 2020 .…”

Natural Language Processing for Adjudication of Heart Failure Hospitalizations in a Multi-Center Clinical Trial

“…Performance in other machine learning models has improved with training set size in a logarithmic relationship, 21 which we also observed when training and fine-tuning NLP models in INVESTED. Third, generative large language models pre-trained on greater amounts of general (GPT4, LLaMa) or clinical (Med-PaLM 2) text, and with longer attention windows, [22][23][24] could be fine-tuned for event adjudication, but must be implemented carefully to maintain the security of patient health information.…”

Enhancing Mental Health Condition Detection on Social Media through Multi-Task Learning

“…In this study, we're interested in how well these models can handle tasks related to mental health given their promise in general healthcare. 16,[34][35][36][37][38][39][40] To do this, we report several different LLMs and how they perform on two datasets. For SWMH, we report the performance of LLaMA-7B 41 with zero prior examples given (0-shot), LLaMA-13B 41 0-shot, ChatGPT 42 0-shot, ChatGPT with a few examples given (few-shot), GPT-4 43 few-shot, MentaLLaMA-7B 44 few-shot, MentaLLaMA-chat-7B few-shot, and MentaLLaMA-chat-13B 7 few-shot, published by Yang et al 44 For PsySym, we report the performance of ChatGPT 0-shot, as published in Chen et al 45…”