PolyU-led research reveals that sensory and motor inputs help large language models represent complex concepts (IMAGE)
Caption
a, A schematic of the RSA: for each human rater and language model (GPT-3.5, GPT-4, PaLM and Gemini), the words were represented as separate vectors for the non-sensorimotor, sensory and motor domains. Icons from Flaticon.com77. The elements of these word vectors were derived from the ratings generated by humans or models for the dimensions belonging to each respective domain. The RDMs were then constructed by calculating the Euclidean distance between every pair of word rating vectors within each domain. Spearman correlations between these RDMs quantify the alignment of representational geometries, enabling comparison between human and model representations. b, Distributions of model–human RDM similarities for ChatGPT models (GPT-3.5 and GPT-4). The distributions of Spearman correlation coefficients for RDMs constructed upon individual human ratings and ChatGPT ratings for the same words across non-sensorimotor, sensory and motor domains are shown. The x axis represents Spearman correlation coefficients and y axis denotes the density of these coefficients. c, The distributions of model–human RDM similarities for Google LLMs (PaLM and Gemini). Similar to b, the distributions for human and Google LLM RDM alignment for the same words across the same three domains are displayed. Both b and c illustrate a trend that model–human RDM alignments decrease (with RDM similarities centralizing around smaller values) from non-sensorimotor to sensory and especially motor domains. d–f, Example RDMs: each RDM, constructed using 25 words, reflects pairwise similarities either based on human or GPT-4 ratings across non-sensorimotor (d), sensory (e) and motor (f) domains. The distinct patterns could be observed between the human RDM and GPT-4 RDM for the motor domain while for the non-sensorimotor domain, the human and GPT-4 model RDMs are much more similar.
Credit
© 2025 Research and Innovation Office, The Hong Kong Polytechnic University. All Rights Reserved.
Usage Restrictions
nil
License
Original content