1{2 "document_id": "lb_qasper_43279ddf85ada1b163aa1b316a4df1418957058206501c26",3 "family": "public_qasper",4 "split": "test",5 "text": "Introduction\nHow humans process language has become increasingly relevant in natural language processing since physiological data during language understanding is more accessible and recorded with less effort. In this work, we focus on eye-tracking and electroencephalography (EEG) recordings to capture the reading process. On one hand, eye movement data provides millisecond-accurate records about where humans look when they are reading, and is highly correlated with the cognitive load associated with different stages of text processing. On the other hand, EEG records electrical brain activity across the scalp and is a direct measure of physiological processes, including language processing. The combination of both measurement methods enables us to study the language understanding process in a more natural setting, where participants read full sentences at a time, in their own speed. Eye-tracking then permits us to define exact word boundaries in the timeline of a subject reading a sentence, allowing the extraction of brain activity signals for each word.\nHuman cognitive language processing data is immensely useful for NLP: Not only can it be leveraged to improve NLP applications (e.g. barrett2016weakly for part-of-speech tagging or klerke2016improving for sentence compression), but also to evaluate state-of-the-art machine learning systems. For example, hollenstein2019cognival evaluate word embeddings, or schwartz2019inducing fine-tune language models with brain-relevant bias.\nAdditionally, the availability of labelled data plays a crucial role in all supervised machine learning applications. Physiological data can be used to understand and improve the labelling process (e.g. tokunaga2017eye), and, for instance, to build cost models for active learning scenarios BIBREF0. Is it possible to replace this expensive manual work with models trained on physiological activity data recorded from humans while reading? That is to say, can we find and extract relevant aspects of text understanding and annotation directly from the source, i.e. eye-tracking and brain activity signals during reading?\nMotivated by these questions and our previously released dataset, ZuCo 1.0 BIBREF1, we developed this new corpus, where we specifically aim to collect recordings during natural reading as well as during annotation.\nWe provide the first dataset of simultaneous eye movement and brain activity recordings to analyze and compare normal reading to task-specific reading during annotation. The Zurich Cognitive Language Processing Corpus (ZuCo) 2.0, including raw and preprocessed eye-tracking and electroencephalography (EEG) data of 18 subjects, as well as the recording and preprocessing scripts, is publicly available at https://osf.io/2urht/. It contains physiological data of each subject reading 739 English sentences from Wikipedia (see example in Figure FIGREF1). We want to highlight the re-use potential of this data. In addition to the psycholinguistic motivation, this corpus is especially tailored for training and evaluating machine learning algorithms for NLP purposes. We conduct a detailed technical validation of the data as proof of the quality of the recordings.\nRelated Work\nSome eye-tracking corpora of natural reading (e.g. the Dundee BIBREF2, Provo BIBREF3 and GECO corpus BIBREF4), and a few EEG corpora (for example, the UCL corpus BIBREF5) are available. It has been shown that this type of cognitive processing data is useful for improving and evaluating NLP methods (e.g. barrett2018sequence,hollenstein2019cognival, hale2018finding). However, before the Zurich Cognitive Language Processing Corpus (ZuCo 1.0), there was no available data for simultaneous eye-tracking and EEG recordings of natural reading. dimigen2011coregistration studied the linguistic effects of eye movements and EEG co-registration in natural reading and showed that they accurately represent lexical processing. Moreover, the simultaneous recordings are crucial to extract word-level brain activity signals.\nWhile the above mentioned studies analyze and leverage natural reading, some NLP work has used eye-tracking during annotation (but, as of yet, not EEG data). mishra2016predicting and joshi2014measuring recorded eye-tracking during binary sentiment annotation (positive/negative). This data was used to determine the annotation complexity of the text passages based on eye movement metrics and for sarcasm detection BIBREF6. Moreover, eye-tracking has been used to analyze the word sense annotation process in Hindi BIBREF7, named entity annotation in Japanese BIBREF8, and to leverage annotator gaze behaviour for coreference resolution BIBREF9. Finally, tomanek2010cognitive used eye-tracking data during entity annotation to build a cost model for active learning. However, until now there is no available data or research that analyzes the differences in the human processing of normal reading versus annotation.\nRelated Work ::: ZuCo1.0\nIn previous work, we recorded a first dataset of simultaneous eye-tracking and EEG during natural reading BIBREF1. ZuCo 1.0 consists of three reading tasks, two of which contain very similar reading material and experiments as presented in the current work. However, the main difference and reason for recording ZuCo 2.0, consists in the experiment procedure. For ZuCo 1.0 the normal reading and task-specific reading paradigms were recorded in different sessions on different days. Therefore, the recorded data is not appropriate as a means of comparison between natural reading and annotation, since the differences in the brain activity data might result mostly from the different sessions due to the sensitivity of EEG. This, and extending the dataset with more sentences and more subjects, were the main factors for recording the current corpus. We purposefully maintained an overlap of some sentences between both datasets to allow additional analyses (details are described in Section SECREF7).\nCorpus Construction\nIn this section we describe the contents and experimental design of the ZuCo 2.0 corpus.\nCorpus Construction ::: Participants\nWe recorded data from 19 participants and discarded the data of one of them due to technical difficulties with the eye-tracking calibration. Hence, we share the data of 18 participants. All participants are healthy adults (mean age = 34 (SD=8.3), 10 females). Their native language is English, originating from Australia, Canada, UK, USA or South Africa. Two participants are left-handed and three participants wear glasses for reading. Details on subject demographics can be found in Table TABREF4. All participants gave written consent for their participation and the re-use of the data prior to the start of the experiments. The study was approved by the Ethics Commission of the University of Zurich.\nCorpus Construction ::: Reading materials\nDuring the recording session, the participants read 739 sentences that were selected from the Wikipedia corpus provided by culotta2006integrating. This corpus was chosen because it provides annotations of semantic relations. We included seven of the originally defined relation types: political_affiliation, education, founder, wife/husband, job_title, nationality, and employer. The sentences were chosen in the same length range as ZuCo 1.0, and with similar Flesch reading ease scores. The dataset statistics are shown in Table TABREF2.\nOf the 739 sentences, the participants read 349 sentences in a normal reading paradigm, and 390 sentences in a task-specific reading paradigm, in which they had to determine whether a certain relation type occurred in the sentence or not. Table TABREF3 shows the distribution of the different relation types in the sentences of the task-specific annotation paradigm.\nPurposefully, there are 63 duplicates between the normal reading and the task-specific sentences (8% of all sentences). The intention of these duplicate sentences is to provide a set of sentences read twice by all participants with a different task in mind. Hence, this enables the comparison of eye-tracking and brain activity data when reading normally and when annotating specific relations (see examples in Section SECREF4).\nFurthermore, there is also an overlap in the sentences between ZuCo 1.0 and ZuCo 2.0. 100 normal reading and 85 task-specific sentences recorded for this dataset were already recorded in ZuCo 1.0. This allows for comparisons between the different recording procedures (i.e. session-specific effects) and between more participants (subject-specific effects).\nCorpus Construction ::: Experimental design\nAs mentioned above, we recorded two different reading tasks for the ZuCo 2.0 dataset. During both tasks the participants were able to read in their own speed, using a control pad to move to the next sentence and to answer the control questions, which allowed for natural reading. Since each subject reads at their own personal pace, the reading speed between varies between subjects. Table TABREF4 shows the average reading speed for each task, i.e. the average number of seconds a subject spends per sentence before switching to the next one.\nAll 739 sentences were recorded in a single session for each participant. The duration of the recording sessions was between 100 and 180 minutes, depending on the time required to set up and calibrate the devices, and the personal reading speed of the participants.\nWe recorded 14 blocks of approx. 50 sentences, alternating between tasks: 50 sentences of normal reading, followed by 50 sentences of task-specific reading. The order of blocks and sentences within blocks was identical for all subjects. Each sentence block was preceded by a practice round of three sentences.\nCorpus Construction ::: Experimental design ::: Normal reading (NR)\nIn the first task, participants were instructed to read the sentences naturally, without any specific task other than comprehension. Participants were told to read the sentences normally without any special instructions. Figure FIGREF8 (left) shows an example sentence as it was depicted on the screen during recording. As shown in Figure FIGREF8 (middle), the control condition for this task consisted of single-choice questions about the content of the previous sentence. 12% of randomly selected sentences were followed by such a comprehension question with three answer options on a separate screen.\nCorpus Construction ::: Experimental design ::: Task-specific reading (TSR)\nIn the second task, the participants were instructed to search for a specific relation in each sentence they read. Instead of comprehension questions, the participants had to decide for each sentence whether it contains the relation or not, i.e. they were actively annotating each sentence. Figure FIGREF8 (right) shows an example screen for this task. 17% of the sentences did not include the relation type and were used as control conditions. All sentences within one block involved the same relation type. The blocks started with a practice round, which described the relation and was followed by three sample sentences, so that the participants would be familiar with the respective relation type.\nCorpus Construction ::: Linguistic assessment\nAs a linguistic assessment, the vocabulary and language proficiency of the participants was tested with the LexTALE test (Lexical Test for Advanced Learners of English, lemhofer2012introducing). This is an unspeeded lexical decision task designed for intermediate to highly proficient language users. The average LexTALE score over all participants was 88.54%. Moreover, we also report the scores the participants achieved with their answers to the reading comprehension control questions and their relation annotations. The detailed scores for all participants are also presented in Table TABREF4.\nCorpus Construction ::: Data acquisition\nData acquisition took place in a sound-attenuated and dark experiment room. Participants were seated at a distance of 68cm from a 24-inch monitor with a resolution of 800x600 pixels. A stable head position was ensured via a chin rest. Participants were instructed to stay as still as possible during the tasks to avoid motor EEG artifacts. Participants were also offered snacks and water during the breaks and were encouraged to rest. All sentences were presented at the same position on the screen and could span multiple lines. The sentences were presented in black on a light grey background with font size 20-point Arial, resulting in a letter height of 0.8 mm. The experiment was programmed in MATLAB 2016b BIBREF10, using PsychToolbox BIBREF11. Participants completed the tasks sitting alone in the room, while two research assistants were monitoring their progress in the adjoining room. All recording scripts including detailed participant instructions are available alongside the data.\nCorpus Construction ::: Data acquisition ::: Eye-tracking acquisition\nEye position and pupil size were recorded with an infrared video-based eye tracker (EyeLink 1000 Plus, SR Research) at a sampling rate of 500 Hz. The eye tracker was calibrated with a 9-point grid at the beginning of the session and re-validated before each block of sentences.\nCorpus Construction ::: Data acquisition ::: EEG acquisition\nHigh-density EEG data were recorded at a sampling rate of 500 Hz with a bandpass of 0.1 to 100 Hz, using a 128-channel EEG Geodesic Hydrocel system (Electrical Geodesics). The recording reference was set at electrode Cz. The head circumference of each participant was measured to select an appropriately sized EEG net. To ensure good contact, the impedance of each electrode was checked prior to recording, and was kept below 40 kOhm. Electrode impedance levels were checked after every third block of 50 sentences (approx. every 30 mins) and reduced if necessary.\nCorpus Construction ::: Preprocessing and feature extraction ::: Eye-tracking\nThe eye-tracking data consists of (x,y) gaze location entries for all individual fixations (Figure FIGREF1b). Coordinates were given in pixels with respect to the monitor coordinates (the upper left corner of the screen was (0,0) and down/right was positive). We provide this raw data as well as various engineered eye-tracking features. For this feature extraction only fixations within the boundaries of each displayed word were extracted. Data points distinctly not associated with reading (minimum distance of 50 pixels to the text) were excluded. Additionally, fixations shorter than 100 ms were excluded from the analyses, because these are unlikely to reflect fixations relevant for reading BIBREF12. On the basis of the GECO and ZuCo 1.0 corpora, we extracted the following features: (i) gaze duration (GD), the sum of all fixations on the current word in the first-pass reading before the eye moves out of the word; (ii) total reading time (TRT), the sum of all fixation durations on the current word, including regressions; (iii) first fixation duration (FFD), the duration of the first fixation on the prevailing word; (iv) single fixation duration (SFD), the duration of the first and only fixation on the current word; and (v) go-past time (GPT), the sum of all fixations prior to progressing to the right of the current word, including regressions to previous words that originated from the current word. For each of these eye-tracking features we additionally computed the pupil size. Furthermore, we extracted the number of fixations and mean pupil size for each word and sentence.\nCorpus Construction ::: Preprocessing and feature extraction ::: EEG\nThe EEG data shared in this project are available as raw data, but also preprocessed with Automagic (version 1.4.6, pedroni2019automagic), a tool for automatic EEG data cleaning and validation. 105 EEG channels (i.e. electrodes) were used from the scalp recordings. 9 EOG channels were used for artifact removal and additional 14 channels lying mainly on the neck and face were discarded before data analysis. Bad channels were identified and interpolated. We used the Multiple Artifact Rejection Algorithm (MARA), a supervised machine learning algorithm that evaluates ICA components, for automatic artifact rejection. MARA has been trained on manual component classifications, and thus captures a wide range of artifacts. MARA is especially effective at detecting and removing eye and muscle artifact components. The effect of this preprocessing can be seen in Figure FIGREF1d.\nAfter preprocessing, we synchronized the EEG and eye-tracking data to enable EEG analyses time-locked to the onsets of fixations. To compute oscillatory power measures, we band-pass filtered the continuous EEG signals across an entire reading task for five different frequency bands resulting in a time-series for each frequency band. The independent frequency bands were determined as follows: theta$_1$ (4–6 Hz), theta$_2$ (6.5–8 Hz), alpha$_1$ (8.5–10 Hz), alpha$_2$ (10.5–13 Hz), beta$_1$ (13.5–18 Hz), beta$_2$ (18.5–30 Hz), gamma$_1$ (30.5–40 Hz), and gamma$_2$ (40–49.5 Hz). We then applied a Hilbert transformation to each of these time-series. We specifically chose the Hilbert transformation to maintain the temporal information of the amplitude of the frequency bands, to enable the power of the different frequencies for time segments defined through the fixations from the eye-tracking recording. Thus, for each eye-tracking feature we computed the corresponding EEG feature in each frequency band. Furthermore, we extracted sentence-level EEG features by calculating the power in each frequency band, and additionally, the difference of the power spectra between frontal left and right homologue electrodes pairs. For each eye-tracking based EEG feature, all channels were subject to an artifact rejection criterion of $90\\mu V$ to exclude trials with transient noise.\nData Validation\nThe aim of the technical validation of the data is to guarantee good recording quality and to replicate findings of previous studies investigating co-registration of EEG and eye movement data during natural reading tasks (e.g. dimigen2011coregistration). We also compare the results to ZuCo 1.0 BIBREF1, which allows a more direct comparison due to the analogous recording procedure.\nData Validation ::: Eye-tracking\nWe validated the recorded eye-tracking data by analyzing the fixations made by all subjects through their reading speed and omission rate on sentence level. The omission rate is defined as the percentage of words that is not fixated in a sentence. Figure FIGREF10 (middle) shows the mean reading speed over all subjects, measured in seconds per sentence and Figure FIGREF10 (right) shows the mean omission rates aggregated over all subjects for each task. Clearly, the participants made less fixations during the task-specific reading, which lead to faster reading speed.\nMoreover, we corroborated these sentence-level metrics by visualizing the skipping proportion on word level (Figure FIGREF13). The skipping proportion is the average rate of words being skipped (i.e. not being fixated) in a sentence. As expected, this also increases in the task-specific reading.\nAlthough the reading material is from the same source and of the same length range (see Figure FIGREF10 (left)), in the first task (NR) passive reading was recorded, while in the second task (TSR) the subjects had to annotate a specific relation type in each sentence. Thus, the task-specific annotation reading lead to shorter passes because the goal was merely to recognize a relation in the text, but not necessarily to process the every word in each sentence. This distinct reading behavior is shown in Figure FIGREF15, where fixations occur until the end of the sentence during normal reading, while during task-specific reading the fixations stop after the decisive words to detect a given relation type. Finally, we also analyzed the average reading times for each of the extracted eye-tracking features. The means and distributions for both tasks are shown in Figure FIGREF21. These results are in line with the recorded data in ZuCo 1.0, as well as with the features extracted in the GECO corpus BIBREF4.\nData Validation ::: EEG\nAs a first validation step, we extracted fixation-related potentials (FRPs), where the EEG signal during all fixations of one task are averaged. Figure FIGREF24 shows the time-series of the resulting FRPs for two electrodes (PO8 and Cz), as well as topographies of the voltage distributions across the scalp at selected points in time. The five components (for which the scalp topographies are plotted) are highly similar in the time-course of the chosen electrodes to dimigen2011coregistration as well as to ZuCo 1.0.\nMoreover, these previous studies were able to show an effect of fixation duration on the resulting FRPs. To show this dependency we followed two approaches. First, for each reading task, all single-trial FRPs were ordered by fixation duration and a vertical sliding time-window was used to smooth the data BIBREF13. Figure FIGREF25 (bottom) shows the resulting plots. In line with this previous work, a first positivation can be identified at 100 ms post-fixation onset. A second positive peak is located dependent on the duration of the fixation, which can be explained by the time-jittered succeeding fixation. The second approach is based on henderson2013co in which single trial EEG segments are clustered by the duration of the current fixation. As shown in Figure FIGREF25 (top), we chose four clusters and averaged the data within each cluster to four distinct FRPs, depending on the fixation duration. Again, the same positivation peaks become apparent. Both findings are consistent with the previous work mentioned and with our findings from ZuCo 1.0.\nConclusion\nWe presented a new, freely available corpus of eye movement and electrical brain activity recordings during natural reading as well as during annotation. This is the first dataset that allows for the comparison between these two reading paradigms. We described the materials and experiment design in detail and conducted an extensive validation to ensure the quality of the recorded data. Since this corpus is tailored to cognitively-inspired NLP, the applications and re-use potentials of this data are extensive. The provided word-level and sentence-level eye-tracking and EEG features can be used to improve and evaluate NLP and machine learning methods, for instance, to evaluate linguistic phenomena in neural models via psycholinguistic data. In addition, because the sentences contains semantic relation labels and the annotations of all participants, it can also be widely used for relation extraction and classification. Finally, the two carefully constructed reading paradigms allow for the comparison between normal reading and reading during annotation, which can be relevant to improve the manual labelling process as well as the quality of the annotations for supervised machine learning.",6 "char_len": 22971,7 "document_hash": "38a634c68d4fa25e4896acf1e85a9f0f8870d65f95e5784cc4892f58bc920977",8 "questions": [9 {10 "question_id": "q00",11 "qtype": "qasper",12 "question": "Did they experiment with this new dataset?"13 }14 ]15}