杩戞棩锛岄樋閲屼簯鏈哄櫒瀛︿範骞冲彴PAI涓诲鐨勮鏂囥€奝arameter-Efficient Sparsity for Large Language Models Fine-Tuning銆嬶紝澶фā鍨嬪弬鏁伴珮鏁堢█鐤忚缁冪畻娉昉ST琚獻JCAI锛圛nternational Joint Conference on Artificial Intelligence锛2022褰曞彇銆傝鏂囬€氳繃鍑忓皯妯″瀷绋€鐤忚缁冭繃绋嬩腑闇€瑕佹洿鏂扮殑鍙傛暟閲忥紝浠庤€屽噺灏戝ぇ妯″瀷绋€鐤忚缁冪殑鏃堕棿浠ュ強璧勬簮寮€閿€銆
IJCAI锛圛nternational Joint Conference on Artificial Intelligence锛夋槸浜哄伐鏅鸿兘棰嗗煙鐨勯《绾у浗闄呬細璁箣涓€锛屾秹鍙婃満鍣ㄥ涔狅紝璁＄畻鏈鸿瑙夛紝鑷劧璇█澶勭悊绛夊涓柟鍚戙€傛湰骞村害浼氳鎶曠瓒呰繃4500绡囷紝褰曠敤鐜囦粎涓15%锛屼細璁簬7鏈堝湪濂ュ湴鍒╃淮涔熺撼鍙紑銆傛娆″叆閫夋剰鍛崇潃闃块噷浜戞満鍣ㄥ涔犲钩鍙癙AI鍦ㄥぇ妯″瀷浼樺寲鏂瑰悜涓婄殑鐮旂┒杈惧埌浜嗗叏鐞冧笟鐣屽厛杩涙按骞筹紝鑾峰緱浜嗗浗闄呭鑰呯殑璁ゅ彲锛屽睍鐜颁簡涓浗娣卞害瀛︿範鎶€鏈湪鍥介檯涓婄殑绔炰簤鍔涖€
棣栦釜澶фā鍨嬪弬鏁伴珮鏁堢殑绋€鐤忚缁冪畻娉昉ST
璁烘枃鏍稿績鍐呭鏄€氳繃鍑忓皯妯″瀷绋€鐤忚缁冭繃绋嬩腑闇€瑕佹洿鏂扮殑鍙傛暟閲忥紝浠庤€屽噺灏戝ぇ妯″瀷绋€鐤忚缁冪殑鏃堕棿浠ュ強璧勬簮寮€閿€銆傝繎鍑犲勾锛岃秺鏉ヨ秺澶氱殑澶фā鍨嬭鎻愬嚭锛岃繖浜涙ā鍨嬬殑鍙傛暟閲忎粠鐧句嚎鍒板崈浜跨敋鑷充竾浜匡紝铏界劧瀹冧滑鍙互杈惧埌寰堥珮鐨勬ā鍨嬬簿搴︼紝浣嗘槸杩囧ぇ鐨勬ā鍨嬩綋绉檺鍒朵簡杩欎簺澶фā鍨嬬殑瀹為檯钀藉湴搴旂敤銆傜█鐤忎綔涓轰竴涓湁鏁堢殑妯″瀷鍘嬬缉鎵嬫锛屽彲浠ュ皢澶фā鍨嬪帇缂╁埌杈冨皬鐨勪綋绉紝浣垮緱瀹冧滑鑳藉浠ヨ緝灏戠殑璧勬簮杈冨揩鐨勯€熷害杩愯璧锋潵銆傜劧鑰岋紝绋€鐤忚缁冩湰韬細寮曞叆棰濆鐨勫弬鏁帮紝浠庤€屼娇寰楀ぇ妯″瀷鐨勭█鐤忚缁冮渶瑕佸崰鐢ㄦ洿澶氱殑璁粌璧勬簮浠ュ強瀵艰嚧璁粌閫熷害鏇存參銆
閽堝杩欎竴闂锛孭ST鎻愬嚭浜嗕竴绉嶅弬鏁伴珮鏁堢殑绋€鐤忚缁冪畻娉曪紝閫氳繃鍒嗘瀽鏉冮噸鐨勯噸瑕佹€ф寚鏍囷紝寰楀嚭浜嗗叾鎷ユ湁涓や釜鐗规€э細浣庣З鎬у拰缁撴瀯鎬с€傛牴鎹繖涓€缁撹锛孭ST绠楁硶寮曞叆浜嗕袱缁勫皬鐭╅樀鏉ヨ绠楁潈閲嶇殑閲嶈鎬э紝鐩告瘮浜庡師鏈渶瑕佷笌鏉冮噸涓€鏍峰ぇ鐨勭煩闃垫潵淇濆瓨鍜屾洿鏂伴噸瑕佹€ф寚鏍囷紝绋€鐤忚缁冮渶瑕佹洿鏂扮殑鍙傛暟閲忓ぇ澶у噺灏戙€傚姣斿父鐢ㄧ殑绋€鐤忚缁冪畻娉曪紝PST绠楁硶鍙互鍦ㄤ粎鏇存柊1.5%鐨勫弬鏁扮殑鎯呭喌涓嬶紝杈惧埌鐩歌繎鐨勭█鐤忔ā鍨嬬簿搴︺€
PST鎶€鏈凡缁忛泦鎴愬湪闃块噷浜戞満鍣ㄥ涔燩AI鐨勬ā鍨嬪帇缂╁簱锛屼互鍙夾licemind骞冲彴澶фā鍨嬬█鐤忚缁冨姛鑳戒腑銆備负闃块噷宸村反闆嗗洟鍐呴儴钀藉湴浣跨敤澶фā鍨嬪甫鏉ヤ簡鎬ц兘鍔犻€燂紝鍦ㄧ櫨浜垮ぇ妯″瀷PLUG涓婏紝PST鐩告瘮浜庡師鏈殑绋€鐤忚缁冨彲浠ュ湪妯″瀷绮惧害涓嶄笅闄嶇殑鎯呭喌涓嬶紝鍔犻€2.5鍊嶏紝鍐呭瓨鍗犵敤鍑忓皯10鍊嶃€傜洰鍓嶏紝闃块噷浜戞満鍣ㄥ涔燩AI宸茬粡琚箍娉涘簲鐢ㄤ簬鍚勮鍚勪笟锛屾彁渚汚I寮€鍙戝叏閾捐矾鏈嶅姟锛屽疄鐜颁紒涓氳嚜涓诲彲鎺х殑AI鏂规锛屽叏闈㈡彁鍗囨満鍣ㄥ涔犲伐绋嬫晥鐜囥€
璁烘枃鍚嶅瓧锛
Parameter-Efficient Sparsity for Large Language Models Fine-Tuning
璁烘枃浣滆€咃細
鏉庝笌瓒呫€佺綏绂忚帀銆佽碍浼犲銆佺帇姊﹀ǎ銆侀粍鏉捐姵銆佹潕娣便€佺櫧淇婃澃
璁烘枃閾炬帴锛歨ttps://arxiv.org/pdf/2205.11005.pdf
浜哄伐鏅鸿兘鎶€鏈綉 鍊″灏婇噸涓庝繚鎶ょ煡璇嗕骇鏉冦€傚鍙戠幇鏈珯鏂囩珷瀛樺湪鐗堟潈绛夐棶棰橈紝鐑﹁30澶╁唴鎻愪緵鐗堟潈鐤戦棶銆佽韩浠借瘉鏄庛€佺増鏉冭瘉鏄庛€佽仈绯绘柟寮忕瓑鍙戦偖浠惰嚦1851688011@qq.com鎴戜滑灏嗗強鏃舵矡閫氫笌澶勭悊銆傦紒锛棣栭〉 > 鏂伴椈 » 闃块噷浜戞満鍣ㄥ涔犲钩鍙癙AI璁烘枃鍏ラ€塈JCAI 2022浼氳