\n| Instruction-tuned models are worse than pre-training\u2013only models at learning to override semantic priors when presented with flipped labels in-context.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n \nCombined with the previous result, we conclude that although instruction tuning improves the ability to learn input-label mappings, it strengthens the usage of semantic prior knowledge more.\n<\/p>\n Conclusion<\/h2>\n\nWe examined the extent to which language models learn in-context by utilizing prior knowledge learned during pre-training versus input-label mappings presented in-context.\n<\/p>\n \nWe first showed that large language models can learn to override prior knowledge when presented with enough flipped labels, and that this ability emerges with model scale. We then found that successfully doing ICL using semantically-unrelated labels is another emergent ability of model scale. Finally, we analyzed instruction-tuned language models and saw that instruction tuning improves the capacity to learn input-label mappings but also strengthens the use of semantic prior knowledge even more.\n<\/p>\n Future work<\/h2>\n\nThese results underscore how the ICL behavior of language models can change depending on their scale, and that larger language models have an emergent ability to map inputs to many types of labels, a form of reasoning in which input-label mappings can potentially be learned for arbitrary symbols. Future research could help provide insights on why these phenomena occur with respect to model scale.\n<\/p>\n Acknowledgements<\/h2>\n\nThis work was conducted by Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, and Tengyu Ma. We would like to thank Sewon Min and our fellow collaborators at Google Research for their advice and helpful discussions.<\/em><\/p>\n<\/div>\n[ad_2]
\n
Source link <\/a><\/p>\n","protected":false},"excerpt":{"rendered":"[ad_1] Posted by Jerry Wei, Student Researcher, and Denny Zhou, Principal Scientist, Google Research There have recently been<\/p>\n","protected":false},"author":2,"featured_media":545,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[20],"tags":[],"class_list":["post-544","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-google-ai"],"_links":{"self":[{"href":"https:\/\/todaysainews.com\/index.php\/wp-json\/wp\/v2\/posts\/544","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/todaysainews.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/todaysainews.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/todaysainews.com\/index.php\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/todaysainews.com\/index.php\/wp-json\/wp\/v2\/comments?post=544"}],"version-history":[{"count":1,"href":"https:\/\/todaysainews.com\/index.php\/wp-json\/wp\/v2\/posts\/544\/revisions"}],"predecessor-version":[{"id":2800,"href":"https:\/\/todaysainews.com\/index.php\/wp-json\/wp\/v2\/posts\/544\/revisions\/2800"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/todaysainews.com\/index.php\/wp-json\/wp\/v2\/media\/545"}],"wp:attachment":[{"href":"https:\/\/todaysainews.com\/index.php\/wp-json\/wp\/v2\/media?parent=544"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/todaysainews.com\/index.php\/wp-json\/wp\/v2\/categories?post=544"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/todaysainews.com\/index.php\/wp-json\/wp\/v2\/tags?post=544"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}} |
![](\"https:\/\/blogger.googleusercontent.com\/img\/b\/R29vZ2xl\/AVvXsEgQS0-Pkd_JiN7brkU6zV0-FEisuUcnKs0I56t37HVYKMgKStmwNgLREYn5CfURvW-lMvKbHU4cEV9elAu9qe4-M_FvveTlvBQHezbksTlH3YfOAk4TyJiXYiGBW_95RGKIW-JyjAQiC0Zd4VIjZrCSIm1PEBqrIAqbiEklluNunTOMhX_7CU9Degbwqg\/s800\/SULICL.png\")
<\/p>\n![](\"https:\/\/blogger.googleusercontent.com\/img\/b\/R29vZ2xl\/AVvXsEgJUQZqnsXprXhMMpPVS3eqvH57D4U4G9xvmXH3rQi1KmIQ45f3a5621hbc2T_gW6ELMUv29ZxqKxfZLVlt8rMRUDfLK2hxPoIpRR-H2D7n_ZUXX7uKunqXFb_x2GOgQdbPsl0JeiSagA0VjP4N9hT-RLHDZbVz7lg-prtnONvkShF7uHGrmSAVURsveA\/s16000\/image3.png\"\/)
<\/a><\/td>\n<\/tr>\n![](\"https:\/\/blogger.googleusercontent.com\/img\/b\/R29vZ2xl\/AVvXsEjSlRhZycuNyJePf45hjI8TYSDe5Xbn1Uj9R0DobtZLRy8nTScnl-V7f-Zti6qPpprSHLOac5HqavO8JWg1fy6_0VisA40LVyXAv9MzHQm3Xvkr9WyuktlOqbfga3uaVOCVlhoxGTOZ1qWWznWIvf6NcMC1UgmnDUVsgy9qgu6ncGbUV8J22AacWHiKXg\/s16000\/image1.png\"\/)
<\/a><\/td>\n<\/tr>\n![](\"https:\/\/blogger.googleusercontent.com\/img\/b\/R29vZ2xl\/AVvXsEisGHKYzD0d4B8ynJIhqpU6wedtvu37pMJY7Dd02xWELTodKiDCcXWfu0kQ926XJJWheXRbA7XMMAYP1C3PpY7b9X0N-yfLzVcKwI5nSE4rjOdH1UKBLs_e_e4wF8KXQ7ogywNXn5htE-bWeSde7FbJ9JYLSbLVrll3YTfgIQMTKUtdKAMtZ-Zo2WR1hQ\/s16000\/image4.png\"\/)
<\/a><\/td>\n<\/tr>\n![](\"https:\/\/blogger.googleusercontent.com\/img\/b\/R29vZ2xl\/AVvXsEj747MlBNxtirznhOm0RR4P1kEpk97WOclz3N3TMMTBUBgcciB3MuHopHH0UT4I5qOQPUJ1O8n0M0zRr9sePxpecLSnoyb9foa6Ho5qh_xWtkSzeXyTg-VTRL1hTAc_EIXWewymn6vsOCR96SOidHMHxsu-AjsPTFEVwpNT5HZ954zE2AVIuL0qCXmrvw\/s16000\/image6.png\"\/)
<\/a><\/td>\n<\/tr>\n![](\"https:\/\/blogger.googleusercontent.com\/img\/b\/R29vZ2xl\/AVvXsEgeFBd7p-muxUg1CTDEkDx4VzH4IG1wtn2z-qW3P0dwSiUu_GS-BQW0RSG-WveJl89MwovvYMQL5UN6Ldze6laCzCxSHhkn3uxf5kuzLmFgtU9hPvstPq-4YmdWWoxuHMnazQCOs-F9faQd-AMtxg6zZsxTD6ZGCm42iV8JZrbAWKrA526dHyppLOQR_A\/s16000\/image5.png\"\/)
<\/a><\/td>\n<\/tr>\n![](\"https:\/\/blogger.googleusercontent.com\/img\/b\/R29vZ2xl\/AVvXsEikUB8OahGEvoe32aO4ZRaTg0rFSsC69cso4eJS1FIV4BOTTLJi93HQ3e4lUnPDJcea98tBsVJnjh8-CgZ10lBtSl1UlNiY6-hHotYq_ow2TEUmcb1tj9NaAFRWxaDTYO1_K0y6bgTg5BvNdihcvHsd78zk3Mn4jwFic5gdEvYn5Ol-JIRmYehgoHtfrg\/s16000\/image2.png\"\/)
<\/a><\/td>\n<\/tr>\n