ULM-2: Word, concept, perception and brain

Subproject ULM-2 will cross the borders of language and relate words and their meanings to perceptual data and brain activation patterns.

Project nr 2
Title Word, concept, perception and brain

Whereas the previous project deals with relations between words, in this project we study the relation between words and the extralinguistic world. How do we cut up the perceptual world through language? Classical research on colour perception and colour terms from the 70/80′s revealed that, despite the fact that the spectrum of light is undifferentiated, people tend to perceive focal colours in a universal way. Nevertheless, our language systems are extremely different varying from two terms (dark/light) to complex namings for many more colours than just the focal ones.

In this project, we want to take this research a step further using state-of-the-art data and technology and by applying it to wider range of vocabulary and concepts. We take the wordnet structure as a starting point to select vocabulary that ‘competes’ to refer to the outside world. These vocabulary selections are differentiated along various perceptual dimensions: images of objects, movements, sounds, smell, taste, texture.

In the case of images, we will consider particular types of objects within a particular taxonomic branch, for example all words for buildings. For example in the Dutch wordnet, you can find about 500 words at different levels of specificity:bouwwerk (500), bouwsel, huis, kerk, hut, skihut, berghut, trekkershut, plaggenhut, schuilhut, vanghut, herdershut. We cannot define an image for the most abstract level and at the same time the image hardly changes at the most specific level. For various branches of the object hierarchy, we will create a relation between words and images but also automatically acquire meronymy relations to find where images and meronymy relations stabelize along the hiearchy of abstractness. We will make use of databases of images have been built and mapped to vocabularies:

    1. The SUMO ontology project recently started to add images to concepts. The work is in its initial phase and now has 12207 pointers2 from concepts to images, mostly in Wikipedia.
    2. The Computer Science and Artificial Intelligence Lab at the Massachusetts Institute of Technology released a database with 80 million tiny images obtained from Google Search which are grouped by visual properties (Torralba et al. 2008). There is also a grouping into semantic groups for 53,464 English nouns arranged by meaning through Wordnet3. A similar data-structure is provided by the Stanford Vision Lab.4 They provide hundreds of images for each concept in Wordnet (nouns only). Note that both databases do not cover the full vocabulary, excluding abstract nouns, verbs and adjectives. Furthermore, they cluster images at higher levels of abstraction, e.g. vegetable has more images than tomato, including images for tomatoes.
    3. Cai et al. (2004) combine textual data extracted from the web-pages that hold the images, link data for the web page and the image features to obtain semantic clusters of images.

Similarly, we will consider the perceptual differentiation of the words for movements (estimated on 5,000: e.g. kwakken; knikkeren; jenzen; bliksemen; lazeren; mikken; kegelen; kieperen; flikkeren; kogelen; keilen; plompen; donderen; gooien; zwiepen; kukelen) and sounds (estimated on 3,000: e.g.gesjirp; geblèr; gekerm; gekrijt; gegil; gekrijs; geraas; gejubel; gejuich; hoerageroep; gekwek; gekwaak; gekraak; geknars; geklater; gespetter; gekletter). To the contrary, smell and taste are hardly lexicalised despite their strong perception(estimated on 50: e.g.wildsmaak, ijzersmaak, bijsmaak, zuur, zoet, bitterzoet, vies, bitter, zoetzuur, chutney, mierzoet, scherp, pikant, goor).

We will apply the following methodologies to 4 languages (English, Dutch, two Asian languages e.g. Hindi and Japanese) to study these areas of the vocabulary:

  1. In each of the languages, we select similar vocabulary areas using the wordnets. This results in a word-concept hierarchy in each language for a similar type of concepts.
  2. Native speakers in these language select the most appropriate perceptual signal to match the words. This results in a word to perception match per language.
  3. We provide each native speaker with the perception signals collected by the other languages to select the most appropriate words.
  4. The result of 1, 2 and 3 is a cross-lingual database of vocabularies and perceptions. We define similarity of perceptions using techniques such as visual words, wave files and meta descriptions.
  5. We collect meronymy relations for all relevant words in each language using interoperable pattern-based searches
  6. We collect the distributional vectors for all the words in each language. These vectors will be based on left-right contexts, syntactic dependencies and a semantic typology of perceptual modifers and predicate-argument relations.
  7. We define similarity measures across the different vocabularies based on:
    1. wordnet relations, both built and automatically derived such as meronymy
    2. perceptual distributional vectors
    3. generic distributional vectors
    4. perception associations
  8. We also create human similarity judments for both the words and the perceptual signals.

Another line of research will map vocabularies and perceptual stimuli to neuroactivation patterns. Recent research has shown that neuroactiviation can be predicated using models that are trained on distributional features of words and using wordnet relatedness measures (Mitchell et al 2009, Murphy et al 2009, Murphy et al 2011, Chang et al 2009, Anderson et al 2012). These pioneering studies have been restricted to limited sets of usually concrete words. In this project, we will test wider and more complex lexicalisations and specifically in relation to perceptual stimuli and across different language-communities.

Creating an association between words – concepts – perception and neuractivation wil provide us valuable insights in fundamental aspects of language and meaning. It also provides us with machinery to determine the difference between abstract and concrete meaning of language. An applied part of this research will investigate a methodology to derive an AbstractConcreteWordnet in a similar way as SentimentValues are derived and represented in wordnets. Perceptual associations to word meanings can be propagated to the complete vocabulary via wordnet relations to derive a measure for each word and word meaning. The propagated perceptual association of language can again be used to measure readability of texts.

The project has several research objectives that are studied by 2 PhDs, each with a different language/culture background: English/Dutch and two Asian languages e.g. Hindi /Japanese. They will carry out research in their individual language, collaborate in cross-lingual studies and develop and test the concreteness classifiers. The PostDoc will set up the neuroactivation experiments, coordinate the work of the PhDs, and work on the synthesis of the results.


Anderson, Andrew James, Yuan Tao, Brian Murphy, Massimo Poesio, 2012, On discriminating fMRI representations of abstract WordNet taxonomic categories, Proceedings of the 3rd Workshop on Cognitive Aspects of the Lexicon (CogALex-III), pages 21–32, COLING 2012, Mumbai, December 2012.

Cai, D., He, X., Li, Z., Ma, W. Y., and Wen, J. R. (2004) Hierarchical clustering of www image search results using visual, textual and link information. In Proceedings of the 12th annual ACM International Conference on Multimedia (pp. 952-959).

Chang, Kai-min K., Vladimir L. Cherkassky, Tom M. Mitchell, Marcel Adam Just, 2009. Quantitative modeling of the neural representation of adjective-noun phrases to account for fMRI activation, Proceedings of the 47th Annual Meeting of the ACL and the 4th IJCNLP of the AFNLP, pages 638–646, Suntec, Singapore, 2-7 August 2009. c 2009 ACL and AFNLP.

Datta, R., Joshi, D., Li. J., and Wang, J.C. (2008) Image Retrieval: Ideas, influences, and trends of the New Age. ACM Computing Surveys, 40 (2), article 5. Publication date: April 2008.

Devitt, A. and C. Vogel (2004), The Topology of WordNet: Some Metrics, In Petr Sojka, Karel Pala, Pavel Smrž, Christiane Fellbaum, Piek Vossen (Eds.), GWC 2004, Proceedings (pp. 106–111). Masaryk University, Brno, 2003.

Esuli, A. and F. Sebastiani, (2007) PageRanking WordNet synsets: An application to opinion mining. Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics (pp. 424–431), Prague, June 2007.

Fellbaum, C. (Ed.) (1998). WordNet: An Electronic Lexical Database. Cambridge, MA: MIT Press.

Graesser, A., McNamara, D. S., Louwerse, M., & Cai, Z. (2004). Coh-Metrix: Analysis of text on cohesion and language. Behavioral Research Methods, Instruments, and Computers, 36, 193-202.

Mitchell, Tom M., Svetlana V. Shinkareva, Andrew Carlson, Kai-Min Chang, Vicente L. Malave, Robert A. Mason, Marcel Adam Just. 2008. Predicting Human Brain Activity Associated with the Meanings of Nouns, Science 320, 1191. DOI: 10.1126/science. 1152876.

Murphy, B., Baroni, M., Poesio, M. (2009). EEG Responds to Conceptual Stimuli and Corpus Semantics. Proceedings of ACL/EMNLP 2009.

Murphy, B., Poesio. M, Bovolo, F., Bruzzone, L., Dalponte, M., Lakany, H. (2011). EEG decoding of semantic category reveals distributed representations for single concepts. Brain and Language , 117, 12-22.

Paivio, A., Yuille, J. C., & Madigan, S. A. (1968). Concreteness, imagery and meaningfulness values for 925 words. Journal of Experimental Psychology, 76 (1, Part 2), 1-25.

Pedersen, T., S. Patwardhan, and J. Michelizzi (2010). Information Content Measures of Semantic Similarity Perform Better Without Sense-Tagged Text In: Proceedings of the 11th Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL HLT 2010), June 1-6, 2010 (pp. 329-332). Los Angeles, CA.

Sadoski, M. (1999). Theoretical, empirical, and practical considerations in designing informational text. Document Design, 1, 25-34.

Torralba, A., R. Fergus, W. T. Freeman, (2008), 80 million tiny images: a large dataset for non-parametric object and scene recognition, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.30(11), pp. 1958-1970, 2008.

Niles, I., and Pease, A. 2001. Towards a Standard Upper Ontology. In Proceedings of the 2nd International Conference on Formal Ontology in Information Systems (FOIS-2001), Chris Welty and Barry Smith, eds, Ogunquit, Maine, October 17-19, 2001.

3 http://groups.csail.mit.edu/vision/TinyImages/

4 http://www.image-net.org/

Leave a Reply

Your email address will not be published.