spb/wp30_uqo

1{2  "document_id": "lb_qasper_c3dc6e575eeee19145547b339ae1c7af914dfde59a652000",3  "family": "public_qasper",4  "split": "test",5  "text": "Introduction\nAttempts toward constructing human-like dialogue agents have met significant difficulties, such as maintaining conversation consistency BIBREF0. This is largely due to inabilities of dialogue agents to engage the user emotionally because of an inconsistent personality BIBREF1. Many agents use personality models that attempt to map personality attributes into lower dimensional spaces (e.g. the Big Five BIBREF2). However, these represent human personality at a very high-level and lack depth. They prohibit the ability to link specific and detailed personality traits to characters, and to construct large datasets where dialogue is traceable back to these traits.\nFor this reason, we propose Human Level Attributes (HLAs), which we define as characteristics of fictional characters representative of their profile and identity. We base HLAs on tropes collected from TV Tropes BIBREF3, which are determined by viewers' impressions of the characters. See Figure FIGREF1 for an example. Based on the hypothesis that profile and identity contribute effectively to language style BIBREF4, we propose that modeling conversation with HLAs is a means for constructing a dialogue agent with stable human-like characteristics. By collecting dialogues from a variety of characters along with this HLA information, we present a novel labelling of this dialogue data where it is traceable back to both its context and associated human-like qualities.\nWe also propose a system called ALOHA (Artificial Learning On Human Attributes) as a novel method of incorporating HLAs into dialogue agents. ALOHA maps characters to a latent space based on their HLAs, determines which are most similar in profile and identity, and recovers language styles of specific characters. We test the performance of ALOHA in character language style recovery against four baselines, demonstrating outperformance and system stability. We also run a human evaluation supporting our results. Our major contributions are: (1) We propose HLAs as personality aspects of fictional characters from the audience's perspective based on tropes; (2) We provide a large dialogue dataset traceable back to both its context and associated human-like attributes; (3) We propose a system called ALOHA that is able to recommend responses linked to specific characters. We demonstrate that ALOHA, combined with the proposed dataset, outperforms baselines. ALOHA also shows stable performance regardless of the character's identity, genre of the show, and context of the dialogue. We plan to release all of ALOHA's data and code.\nRelated Work\nTask completion chatbots (TCC), or task-oriented chatbots, are dialogue agents used to fulfill specific purposes, such as helping customers book airline tickets, or a government inquiry system. Examples include the AIML based chatbot BIBREF5 and DIVA Framework BIBREF6. While TCC are low cost, easily configurable, and readily available, they are restricted to working well for particular domains and tasks.\nOpen-domain chatbots are more generic dialogue systems. An example is the Poly-encoder from BIBREF7 humeau2019real. It outperforms the Bi-encoder BIBREF8, BIBREF9 and matches the performance of the Cross-encoder BIBREF10, BIBREF11 while maintaining reasonable computation time. It performs strongly on downstream language understanding tasks involving pairwise comparisons, and demonstrates state-of-the-art results on the ConvAI2 challenge BIBREF12. Feed Yourself BIBREF13 is an open-domain dialogue agent with a self-feeding model. When the conversation goes well, the dialogue becomes part of the training data, and when the conversation does not, the agent asks for feedback. Lastly, Kvmemnn BIBREF14 is a key-value memory network with a knowledge base that uses a key-value retrieval mechanism to train over multiple domains simultaneously. We use all three of these models as baselines for comparison. While these can handle a greater variety of tasks, they do not respond with text that aligns with particular human-like characteristics.\nBIBREF15 li2016persona defines persona (composite of elements of identity) as a possible solution at the word level, using backpropagation to align responses via word embeddings. BIBREF16 bartl2017retrieval uses sentence embeddings and a retrieval model to achieve higher accuracy on dialogue context. BIBREF17 liu2019emotion applies emotion states of sentences as encodings to select appropriate responses. BIBREF18 pichl2018alquist uses knowledge aggregation and hierarchy of sub-dialogues for high user engagement. However, these agents represent personality at a high-level and lack detailed human qualities. LIGHT BIBREF19 models adventure game characters' dialogues, actions, and emotions. It focuses on the agent identities (e.g. thief, king, servant) which includes limited information on realistic human behaviours. BIBREF20 pasunuru2018game models online soccer games as dynamic visual context. BIBREF21 wang2016learning models user dialogue to complete tasks involving certain configurations of blocks. BIBREF22 antol2015vqa models open-ended questions, but is limited to visual contexts. BIBREF23 bordes2016learning tracks user dialogues but is goal-oriented. BIBREF24 ilinykh2019meetup tracks players' dialogues and movements in a visual environment, and is grounded on navigation tasks. All of these perform well in their respective fictional environments, but are not a strong representation of human dialogue in reality.\nMethodology ::: Human Level Attributes (HLA)\nWe collect HLA data from TV Tropes BIBREF3, a knowledge-based website dedicated to pop culture, containing information on a plethora of characters from a variety of sources. Similar to Wikipedia, its content is provided and edited collaboratively by a massive user-base. These attributes are determined by human viewers and their impressions of the characters, and are correlated with human-like characteristics. We believe that TV Tropes is better for our purpose of fictional character modeling than data sources used in works such as BIBREF25 shuster2019engaging because TV Tropes' content providers are rewarded for correctly providing content through community acknowledgement.\nTV Tropes defines tropes as attributes of storytelling that the audience recognizes and understands. We use tropes as HLAs to calculate correlations with specific target characters. We collect data from numerous characters from a variety of TV shows, movies, and anime. We filter and keep characters with at least five HLA, as those with fewer are not complex enough to be correctly modeled due to reasons such as lack of data. We end up eliminating 5.86% of total characters, and end up with 45,821 characters and 12,815 unique HLA, resulting in 945,519 total character-HLA pairs. Each collected character has 20.64 HLAs on average. See Figure FIGREF1 for an example character and their HLAs.\nMethodology ::: Overall Task\nOur task is the following, where $t$ denotes “target\":\nGiven a target character $c_t$ with HLA set $H_t$, recover the language style of $c_t$ without any dialogue of $c_t$ provided.\nFor example, if Sheldon Cooper from The Big Bang Theory is $c_t$, then $H_t$ is the set of HLA on the left side of Figure FIGREF1.\nWe define the language style of a character as its diction, tone, and speech patterns. It is a character specific language model refined from a general language model. We must learn to recover the language style of $c_t$ without its dialogue as our objective is to imitate human-like qualities, and hence the model must understand the language styles of characters based on their traits. If we feed $c_t$'s dialogue during training, the model will likely not effectively learn to imitate language styles based on HLAs, but based on the correlation between text in the training and testing dialogues BIBREF26.\nWe define character space as the character representations within the HLA latent space (see Figure FIGREF4), and the set $C = \\lbrace c_1,c_2,...,c_n\\rbrace $ as the set of all characters. We define Observation (OBS) as the input that is fed into any dialogue model. This can be a single or multiple lines of dialogue along with other information. The goal of the dialogue model is to find the best response to this OBS.\nMethodology ::: ALOHA\nWe propose a three-component system called ALOHA to solve the task (see Figure FIGREF6). The first component, Character Space Module (CSM), generates the character space and calculates confidence levels using singular value decomposition BIBREF27 between characters $c_j$ (for $j = 1$ to $n$ where $j \\ne t$) and $c_t$ in the HLA-oriented neighborhood.\nThe second component, Character Community Module (CCM), ranks the similarity between our target character $c_t$ with any other character $c_j$ by the relative distance between them in the character space.\nThe third component, Language Style Recovery Module (LSRM), recovers the language style of $c_t$ without its dialogue by training the BERT bi-ranker model BIBREF28 to rank responses from similar characters. Our results demonstrate higher accuracy at retrieving the ground truth response from $c_t$. Our system is also able to pick responses which are correct both in context as well as character space.\nHence, the overall process for ALOHA works as follows. First, given a set of characters, determine the character space using the CSM. Next, given a specific target character, determine the positive community and negative set of associated characters using the CCM. Lastly, using the positive community and negative set determined above along with a dialogue dataset, recover the language style of the target.\nMethodology ::: Character Space Module (CSM)\nCSM learns how to rank characters. We can measure the interdependencies between the HLA variables BIBREF29 and rank the similarity between the TV show characters. We use implicit feedback instead of neighborhood models (e.g. cosine similarity) because it can compute latent factors to transform both characters and HLAs into the same latent space, making them directly comparable.\nWe define a matrix $P$ that contains binary values, with $P_{u,i} = 1$ if character $u$ has HLA $i$ in our dataset, and $P_{u,i} = 0$ otherwise. We define a constant $\\alpha $ that measures our confidence in observing various character-HLA pairs as positive. $\\alpha $ controls how much the model penalizes the error if the ground truth is $P_{u,i} = 1$. If $P_{u,i} = 1$ and the model guesses incorrectly, we penalize by $\\alpha $ times the loss. But if $P_{u,i} = 0$ and the model guesses a value greater than 0, we do not penalize as $\\alpha $ has no impact. This is because $P_{u,i} = 0$ can either represent a true negative or be due to a lack of data, and hence is less reliable for penalization. See Equation DISPLAY_FORM8. We find that using $\\alpha =20$ provides decent results.\nWe further define two dense vectors $X_u$ and $Y_i$. We call $X_u$ the “latent factors for character $u$\", and $Y_i$ the “latent factors for HLA $i$\". The dot product of these two vectors produces a value ($X_u^TY_i$) that approximates $P_{u,i}$ (see Figure FIGREF9). This is analogous to factoring the matrix $P$ into two separate matrices, where one contains the latent factors for characters, and the other contains the latent factors for HLAs. We find that $X_u$ and $Y_i$ being 36-dimensional produces decent results. To bring $X_u^TY_i$ as close as possible to $P_{u,i}$, we minimize the following loss function using the Conjugate Gradient Method BIBREF30:\nThe first term penalizes differences between the model's prediction ($X_u^TY_i$) and the actual value ($P_{u,i}$). The second term is an L2 regularizer to reduce overfitting. We find $\\lambda = 100$ provides decent results for 500 iterations (see Section SECREF26).\nMethodology ::: Character Community Module (CCM)\nCCM aims to divide characters (other than $c_t$) into a positive community and a negative set. We define this positive community as characters that are densely connected internally to $c_t$ within the character space, and the negative set as the remaining characters. We can then sample dialogue from characters in the negative set to act as the distractors (essentially negative samples) during LSRM training.\nAs community finding is an ill-defined problem BIBREF31, we choose to treat CCM as a simple undirected, unweighted graph. We use the values learned in the CSM for $X_u$ and $Y_i$ for various values of $u$ and $i$, which approximate the matrix $P$. Similar to BIBREF29 hu2008collaborative, we can calculate the correlation between two rows (and hence two characters).\nWe then employ a two-level connection representation by ranking all characters against each other in terms of their correlation with $c_t$. For the first level, the set $S^{FL}$ is the top 10% (4582) most highly correlated characters with $c_t$ out of the 45,820 total other characters that we have HLA data for. For the second level, for each character $s_i$ in $S^{FL}$, we determine the 30 most heavily correlated characters with $s_i$ as set $S^{SL}_i$. The positive set $S^{pos}$ are the characters which are present in at least 10 $S^{SL}_i$ sets. We call this value 10 the minimum frequency. All other characters in our dialogue dataset make up the negative set $S^{neg}$. These act as our positive community and negative set, respectively. See Algorithm 1 in Appendix A for details, and Figure FIGREF11 for an example.\nMethodology ::: Language Style Recovery Module (LSRM)\nLSRM creates a dialogue agent that aligns with observed characteristics of human characters by using the positive character community and negative set determined in the CCM, along with a dialogue dataset, to recover the language style of $c_t$ without its dialogue. We use the BERT bi-ranker model from the Facebook ParlAI framework BIBREF32, where the model has the ability to retrieve the best response out of 20 candidate responses. BIBREF12, BIBREF19, BIBREF0 choose 20 candidate responses, and for comparison purposes, we do the same.\nMethodology ::: Language Style Recovery Module (LSRM) ::: BERT\nBIBREF28 is first trained on massive amounts of unlabeled text data. It jointly conditions on text on both the left and right, which provides a deep bi-directional representation of sentence inference. BERT is proven to perform well on a wide range of tasks by simply fine-tuning on one additional layer. We are interested in its ability to predict the next sentence, called Next Sentence Prediction. We perform further fine-tuning on BERT for our target character language style retrieval task to produce our LSRM model by optimizing both the encoding layers and the additional layer. We use BERT to create vector representations for the OBS and for each candidate response. By passing the first output of BERT's 12 layers through an additional linear layer, these representations can be obtained as 768-dimensional sentence-level embeddings. It uses the dot product between these embeddings to score candidate responses and is trained using the ranking loss.\nMethodology ::: Language Style Recovery Module (LSRM) ::: Candidate response selection\nis similar to the procedure from previous work done on grounded dialogue agents BIBREF0, BIBREF19. Along with the ground truth response, we randomly sample 19 distractor responses from other characters from a uniform distribution of characters, and call this process uniform character sampling. Based on our observations, this random sampling provides multiple context correct responses. Hence, the BERT bi-ranker model is trained by learning to choose context correct responses, and the model learns to recover a domain-general language model that includes training on every character. This results in a Uniform Model that can select context correct responses, but not responses corresponding to a target character with specific HLAs.\nWe then fine-tune on the above model to produce our LSRM model with a modification: we randomly sample the 19 distractor responses from only the negative character set instead. We choose the responses that have similar grammatical structures and semantics to the ground truth response, and call this process negative character sampling. This guides the model away from the language style of these negative characters to improve performance at retrieving responses for target characters with specific HLAs. Our results demonstrate higher accuracy at retrieving the correct response from character $c_t$, which is the ground truth.\nExperiment ::: Dialogue Dataset\nTo train the Uniform Model and LSRM, we collect dialogues from 327 major characters (a subset of the 45,821 characters we have HLA data for) in 38 TV shows from various existing sources of clean data on the internet, resulting in a total of 1,042,647 dialogue lines. We use a setup similar to the Persona-Chat dataset BIBREF0 and Cornell Movie-Dialogs Corpus BIBREF33, as our collected dialogues are also paired in terms of valid conversations. See Figure FIGREF1 for an example of these dialogue lines.\nExperiment ::: HLA Observation Guidance (HLA-OG)\nWe define HLA Observation Guidance (HLA-OG) as explicitly passing a small subset of the most important HLAs of a given character as part of the OBS rather than just an initial line of dialogue. This is adapted from the process used in BIBREF0 zhang2018personalizing and BIBREF10 wolf2019transfertransfo which we call Persona Profiling. Specifically, we pass four HLAs that are randomly drawn from the top 40 most important HLAs of the character. We use HLA-OG during training of the LSRM and testing of all models. This is because the baselines (see Section SECREF31) already follow a similar process (Persona Profiling) for training. For the Uniform Model, we train using Next Sentence Prediction (see Section SECREF12). For testing, HLA-OG is necessary as it provides information about which HLAs the models should attempt to imitate in their response selection. Just passing an initial line of dialogue replicates a typical dialogue response task without HLAs. See Table TABREF19. Further, we also test our LSRM by explicitly passing four HLAs of `none' along with the initial line of dialogue as the OBS (No HLA-OG in Table TABREF19).\nExperiment ::: Training Details ::: BERT bi-ranker\nis trained by us on the Persona-Chat dataset for the ConvAI2 challenge. Similar to BIBREF0 zhang2018personalizing, we cap the length of the OBS at 360 tokens and the length of each candidate response at 72 tokens. We use a batch size of 64, learning rate of 5e-5, and perform warm-up updates for 100 iterations. The learning rate scheduler uses SGD optimizer with Nesterov's accelerated gradient descent BIBREF34 and is set to have a decay of 0.4 and to reduce on plateau.\nExperiment ::: Training Details ::: Uniform Model\nis produced by finetuning the BERT bi-ranker on the dialogue data discussed in Section SECREF15 using uniform character sampling. We use the same hyperparameters as the BERT bi-ranker along with half-precision operations (i.e. float16 operations) to increase batch size as recommended BIBREF7.\nExperiment ::: Training Details ::: LSRM\nis produced by finetuning on the Uniform Model discussed above using negative character sampling. We use the same hyperparameters as the BERT bi-ranker along with half-precision operations (i.e. float16 operations) to increase batch size as recommended.\nEvaluation ::: CSM Evaluation\nWe begin by evaluating the ability of the CSM component of our system to correctly generate the character space. To do so, during training, 30% of the character-HLA pairs (which are either 0 or 1) are masked, and this is used as a validation set (see Figure FIGREF9). For each character $c$, the model generates a list of the 12,815 unique HLAs ranked similarly to BIBREF29 hu2008collaborative for $c$. We look at the recall of our CSM model, which measures the percentage of total ground truth HLAs (over all characters $c$) present within the top N ranked HLAs for all $c$ by our model. That is:\nwhere $HLA_{c}^{gt}$ are the ground truth HLAs for $c$, and $HLA_{c}^{tN}$ are the top N ranked HLAs by the model for $c$. We use $N = 100$, and our model achieves 25.08% recall.\nTo inspect the CSM performance, we use the T-distributed Stochastic Neighbor Embedding (t-SNE) BIBREF35 to reduce each high-dimensionality data point to two-dimensions via Kullback-Leibler Divergence BIBREF36. This allows us to map our character space into two-dimensions, where similar characters from our embedding space have higher probability of being mapped close by. We sampled characters from four different groups or regions. As seen in Figure FIGREF4, our learned character space effectively groups these characters, as similar characters are adjacent to one another in four regions.\nEvaluation ::: Automatic Evaluation Setup ::: Five-Fold Cross Validation\nis used for training and testing of the Uniform Model and LSRM. The folds are divided randomly by the TV shows in our dialogue data. We use the dialogue data for 80% of these shows as the four-folds for training, and the dialogue data for the remaining 20% as the fifth-fold for validation/testing. The dialogue data used is discussed in Section SECREF15. This ensures no matter how our data is distributed, each part of it is tested, allowing our evaluation to be more robust to different characters. See Appendix C for five-fold cross validation details and statistics.\nEvaluation ::: Automatic Evaluation Setup ::: Five Evaluation Characters\nare chosen, one from each of the five testing sets above. Each is a well-known character from a separate TV show, and acts as a target character $c_t$ for evaluation of every model. We choose Sheldon Cooper from The Big Bang Theory, Jean-Luc Picard from Star Trek, Monica Geller from Friends, Gil Grissom from CSI, and Marge Simpson from The Simpsons. We choose characters of significantly different identities and profiles (intelligent scientist, ship captain, outgoing friend, police leader, and responsible mother, respectively) from shows of a variety of genres to ensure that we can successfully recover the language styles of various types of characters. We choose well-known characters because humans require knowledge on the characters they are evaluating (see Section SECREF40).\nFor each of these five evaluation characters, all the dialogue lines from the character act as the ground truth responses. The initial dialogue lines are the corresponding dialogue lines to which these ground truth responses are responding. For each initial dialogue line, we randomly sample 19 other candidate responses from the associated testing set using uniform character sampling. Note that this is for evaluation, and hence we use the same uniform character sampling method for all models including ALOHA. The use of negative character sampling is only in ALOHA's training.\nEvaluation ::: Baselines\nWe compare against four dialogue system baselines: Kvmemnn, Feed Yourself, Poly-encoder, and a BERT bi-ranker baseline trained on the Persona-Chat dataset using the same training hyperparameters (including learning rate scheduler and length capping settings) described in Section SECREF20. For the first three models, we use the provided pretrained (on Persona-Chat) models. We evaluate all four on our five evaluation characters discussed in Section SECREF28.\nEvaluation ::: Key Evaluation Metrics ::: Hits@n/N\nis the accuracy of the correct ground truth response being within the top $n$ ranked candidate responses out of $N$ total candidates. We measure Hits@1/20, Hits@5/20, and Hits@10/20.\nEvaluation ::: Key Evaluation Metrics ::: Mean Rank\nis the average rank that a model assigns the ground truth response among the 20 total candidates.\nEvaluation ::: Key Evaluation Metrics ::: Mean Reciprocal Rank (MRR)\nBIBREF37 looks at the mean of the multiplicative inverses of the rank of each correct answer out of a sample of queries $Q$:\nwhere $rank_i$ refers to the rank position of the correct response for the $i$-th query, and $|Q|$ refers to the total number of queries in $Q$.\nEvaluation ::: Key Evaluation Metrics ::: @!START@$F_1$@!END@-score\nequals $2 * \\frac{precision*recall}{precision+recall}$. For dialogue, precision is the fraction of words in the chosen response contained in the ground truth response, and recall is the fraction of words in the ground truth response contained in the chosen response.\nEvaluation ::: Key Evaluation Metrics ::: BLEU\nBIBREF38 generally indicates how close two pieces of text are in content and structure, with higher values indicating greater similarity. We report our final BLEU scores as the average scores of 1 to 4-grams.\nEvaluation ::: Human Evaluation Setup\nWe conduct a human evaluation with 12 participants, 8 male and 4 female, who are affiliated project researchers aged 20-39 at the University of [ANON]. We choose the same five evaluation characters as in Section SECREF28. To control bias, each participant evaluates one or two characters. For each character, we randomly select 10 testing samples (each includes an initial line of dialogue along with 20 candidate responses, one of which is the ground truth) from the same testing data for the automatic evaluation discussed in Section SECREF28.\nThese ten samples make up a single questionnaire presented in full to each participant evaluating the corresponding character, and the participant is asked to select the single top response they think the character would most likely respond with for each of the ten initial dialogue lines. See Figure FIGREF41 for an example. We mask any character names within the candidate responses to prevent human participants from using names to identify which show the response is from.\nEach candidate is prescreened to ensure they have sufficient knowledge of the character to be a participant. We ask three prescreening questions where the participant has to identify an image, relationship, and occupation of the character. All 12 of our participants passed the the prescreening.\nResults and Analysis ::: Evaluation Results\nTable TABREF44 shows average results of our automatic and human evaluations. Table TABREF45 shows average Hits@1/20 scores by evaluation character. See Appendix F for detailed evaluation results. ALOHA is the model with HLA-OG during training and testing, and ALOHA (No HLA-OG) is the model with HLA-OG during training but tested with the four HLAs in the OBS marked as `none' (see Section SECREF17). See Appendix G for demo interactions between a human, BERT bi-ranker baseline, and ALOHA for all five evaluation characters.\nResults and Analysis ::: Evaluation Challenges\nThe evaluation of our task (retrieving the language style of a specific character) is challenging and hence the five-fold cross validation is necessary for the following reasons:\nThe ability to choose a context correct response without attributes of specific characters may be hard to separate from our target metric, which is the ability to retrieve the correct response of a target character by its HLAs. However, from manual observation, we noticed that in the 20 chosen candidate responses, there are typically numerous context correct responses, but only one ground truth for the target character (for an example, see Figure FIGREF41). Hence, a model that only chooses dialogue based on context is distinguishable from one that learns HLAs.\nRetrieving responses for the target character depends on the other candidate responses. For example, dialogue retrieval performance for Grissom from CSI, which is a crime/police context, is higher than other evaluation characters (see Table TABREF45), potentially due to other candidate responses not falling within the same crime/police context.\nResults and Analysis ::: Performance: ALOHA vs. Humans\nAs observed from Table TABREF44, ALOHA has a performance relatively close to humans. Human Hits@1/20 scores have a mean of 40.67% and a median over characters of 40%. The limited human evaluation sample size limits what can be inferred, but it indicates that the problem is solved to the extent that ALOHA is able perform relatively close to humans on average. Notice that even humans do not perform extremely well, demonstrating that this task of character based dialogue retrieval is more difficult than typical dialogue retrieval tasks BIBREF19, BIBREF12.\nLooking more closely at each character from Table TABREF45, we can see that human evaluation scores are higher for Sheldon and Grissom. This may be due to these characters having more distinct personalities, making them more memorable.\nWe also look at Pearson correlation values of the Hits@1/20 scores across the five evaluation characters. For human versus Uniform Model, this is -0.4694, demonstrating that the Uniform Model, without knowledge of HLAs, fails to imitate human impressions. For human versus ALOHA, this is 0.4250, demonstrating that our system is able to retrieve character responses somewhat similarly to human impressions. Lastly, for human versus the difference in scores between ALOHA and Uniform Model, this is 0.7815. The difference between ALOHA and the Uniform Model, which is based on the additional knowledge of the HLAs, is hence shown to improve upon the Uniform Model similarly to human impressions. This demonstrates that HLAs are indeed an accurate method of modeling human impressions of character attributes, and also demonstrates that our system, ALOHA, is able to effectively use these HLAs to improve upon dialogue retrieval performance.\nResults and Analysis ::: Performance: ALOHA vs. Baselines\nALOHA, combined with the HLAs and dialogue dataset, achieves a significant improvement on the target character language style retrieval task compared to the baseline open-domain chatbot models. As observed from Table TABREF44, ALOHA achieves a significant boost in Hits@n/N accuracy and other metrics for retrieving the correct response of five diverse characters with different identities (see Section SECREF28).\nResults and Analysis ::: Performance: ALOHA vs. Uniform Model\nWe observe a noticeable improvement in performance between ALOHA and the Uniform Model in recovering the language styles of specific characters that is consistent across all five folds (see Tables TABREF44 and TABREF45), indicating that lack of knowledge of HLAs limits the ability of the model to successfully recover the language style of specific characters. We claim that, to the best of our knowledge, we have made the first step in using HLA-based character dialogue clustering to improve upon personality learning for chatbots.\nALOHA demonstrates an accuracy boost for all five evaluation characters, showing that the system is robust and stable and has the ability to recover the dialogue styles of fictional characters regardless of the character's profile and identity, genre of the show, and context of the dialogue.\nResults and Analysis ::: Performance: HLA-OG\nAs observed from Table TABREF44, ALOHA performs slightly better overall compared to ALOHA (No HLA-OG). Table TABREF45 shows that this slight performance increase is consistent across four of the five evaluation characters. In the case of Sheldon, the HLA-OG model performs a bit worse. This is possibly due to the large number of Sheldon's HLAs (217) compared to the other four evaluation characters (average of 93.75), along with the limited amount of HLAs we are using for guidance due to the models' limited memory. In general, HLA Observation Guidance during testing appears to improve upon the performance of ALOHA, but this improvement is minimal.\nConclusion and Future Work\nWe proposed Human Level Attributes (HLAs) as a novel approach to model human-like attributes of characters, and collected a large volume of dialogue data for various characters with complete and robust profiles. We also proposed and evaluated a system, ALOHA, that uses HLAs to recommend tailored responses traceable to specific characters, and demonstrated its outperformance of the baselines and ability to effectively recover language styles of various characters, showing promise for learning character or personality styles. ALOHA was also shown to be stable regardless of the character's identity, genre of show, and context of dialogue.\nPotential directions for future work include training ALOHA with a multi-turn response approach BIBREF0 that tracks dialogue over multiple responses, as we could not acquire multi-turn dialogue data for TV shows. Another potential is the modeling of the dialog counterpart (e.g. the dialogue of other characters speaking to the target character). Further, performing semantic text exchange on the chosen response with a model such as SMERTI BIBREF39 may improve the ability of ALOHA to converse with humans. This is because the response may be context and HLA correct, but incorrect semantically (e.g. the response may say the weather is sunny when it is actually rainy). HLA-aligned generative models is another area of exploration. Typically, generative models produce text that is less fluent, but further work in this area may lead to better results. Lastly, a more diverse and larger participant pool is required due to the limited size of our human evaluation.",6  "char_len": 33277,7  "document_hash": "7b872ba2d0d0fc9cb33bb399cbc48cea11a23cd9f306254f576c66e876d1ed8e",8  "questions": [9    {10      "question_id": "q00",11      "qtype": "qasper",12      "question": "How big is the difference in performance between proposed model and baselines?"13    }14  ]15}