300

British Journal of Educational Psychology (2015), 85, 300–315 © 2015 The British Psychological Society www.wileyonlinelibrary.com

Forming a belief: The contribution of comprehension to the evaluation and persuasive impact of argumentative text Irene-Anna N. Diakidoy1*, Stelios A. Christodoulou1, Georgios Floros2, Kalypso Iordanou3 and Philip V. Kargopoulos4 1

Department of Psychology, University of Cyprus, Nicosia, Cyprus Department of English Studies, University of Cyprus, Nicosia, Cyprus 3 School of Sciences, University of Central Lancashire, Pila, Cyprus 4 Department of Psychology, Aristotle University of Thessaloniki, Greece 2

Background. Research has shown substantial belief change as a result of reading text and the pervasive influence of prior belief in the evaluation of short arguments. Both outcomes have been attributed to the depth to which the text or the argument has been processed. This study brings together critical thinking and text comprehension research by employing an extended argumentative text and varying the quality of its arguments. Aim. The study examines the contribution of comprehension outcomes to the critical evaluation and persuasive impact of argumentative text. Sample. One hundred and sixteen first-year graduate and third- and fourth-year undergraduate university students. Method. Measures of initial topic-related beliefs, perceived topic knowledge, and need for cognition were obtained. Students read one of two versions of a two-sided, implicitly persuasive argumentative text (677 words) varying in argument quality. Post-reading tasks included main claim recall, overall recall, inference generation, claim agreement, and text evaluation. Results. The text was positively evaluated and highly persuasive regardless of argument quality, but half of the students either failed to identify the main claim promoted or confused it with individual arguments. Despite a modest but positive association between inference generation and text evaluation, no comprehension measure had a significant main or interactive effect. Need for cognition contributed to positive evaluations in the absence of prior topic knowledge regardless of argument quality. Conclusions. The findings suggest a dissociation between the elaboration associated with deep comprehension and the elaboration associated with critical evaluation with implications for belief formation and the teaching of thinking.

Argumentative text aims to justify one or more claims and to persuade its readers in favour of a particular belief or course of action. As a result, it is a primary medium for belief formation and change. Argumentative texts vary in argument structure, that is the extent to which they include counterarguments (one-sided vs. two-sided), and in the degree to

*Correspondence should be addressed to Irene-Anna N. Diakidoy, Department of Psychology, University of Cyprus, Kallipoleos 75, PO Box 20537, Nicosia 1678, Cyprus (email: [email protected]). DOI:10.1111/bjep.12074

Forming a belief

301

which they explicitly seek to change prior beliefs (refutational vs. non-refutational) (e.g., Alexander, Buehl, & Sperl, 2001). Nevertheless, their persuasive impact, as indicated by changes in initial beliefs, can also be hypothesized to relate to text comprehension and argument evaluation outcomes. A substantial amount of work has examined individuals’ ability to understand and engage in argumentation, generate arguments in writing, and to change their prior beliefs as a result of reading persuasive argumentative text (Felton & Kuhn, 2001; Glassner, Weinstock, & Neuman, 2005; Iordanou, 2010; Murphy, Holleran, Long, & Zeruth, 2005; Reznitskaya et al., 2001). In contrast, relatively little research has focused on the comprehension of argumentative text exclusively (Larson, Britt, & Larson, 2004) or in conjunction with the evaluation of its claims and arguments and persuasive impact. Moreover, research with argumentative text has remained relatively separate from work focusing on the critical evaluation of arguments of varying quality (Klaczynski, Gordon, & Fauth, 1997; Macpherson & Stanovich, 2007; Neuman & Weizman, 2003). Nevertheless, bridging these two lines of work is important given the prevalence of argumentative text, its potential to shape public opinion and beliefs, and to guide decision-making. More importantly, the task of understanding and critically evaluating claims and arguments presented in text represents an educational challenge and opportunity as it requires instructional effort to be directed at the development of both comprehension and critical thinking skills. This study borrows from and contributes to both argumentative text and critical thinking research by examining the connection between the comprehension of an extended argumentative text with its evaluation and persuasive impact.

Persuasive impact and critical thinking Evidence for the persuasive impact of argumentative text has been provided by several studies showing readers with initial beliefs inconsistent with those promoted by the text to manifest greater post-reading belief change in the direction suggested by the text than readers whose initial beliefs were somewhat more consistent (Kardash & Scholes, 1996; Murphy & Alexander, 2004; Murphy et al., 2005). These findings seem impressive in the light of evidence from research on critical thinking highlighting the pervasive influence of prior belief and opinion in evaluating and accepting arguments, evidence, and conclusions (Klaczynski et al., 1997; Macpherson & Stanovich, 2007; Stanovich & West, 1997). Klaczynski et al. (1997) presented college students with short argument scenarios that were either consistent or inconsistent with students’ occupational goals and beliefs. Specifically, the scenarios were individualized, presenting arguments that portrayed each student’s previously indicated occupational goals favourably or unfavourably in terms of quality of life (finances, health, relationships) and personal characteristics (intelligence, ethics). In some cases, arguments could be contested on the basis of representing an informal reasoning fallacy, such as hasty generalization from personal experience and violating, thus, the law of large numbers (LLN) (conclusions based on small sample). Students evaluated belief-consistent, goal-enhancing arguments and evidence as more persuasive, regardless of the fallacy embedded. In contrast, they evaluated beliefinconsistent, goal-threatening arguments and evidence as less persuasive, correctly identifying the LLN fallacy as the basis for discrediting claims and conclusions (Klaczynski et al., 1997). Similar belief bias effects have been manifested with a variety of informal reasoning tasks and regardless of cognitive ability (Macpherson & Stanovich, 2007; Stanovich & West, 2007). In this study, we controlled for prior belief by selecting a persuasive argumentative text on a relatively neutral topic and proceeded to examine its

308

Irene-Anna N. Diakidoy et al.

to read the text once, but carefully, with the goal of understanding it. Then, they completed a filler task that was similar to the Prior Knowledge and Belief Questionnaire but included different statements. Finally, the post-reading tasks were administered. Session 2 lasted about 60 min (Table 4).

Results Preliminary results Preliminary analyses indicated that the text topic was relatively neutral in terms of initial Belief bias either for or against the text’s main claim (M = 3.97, SD = 0.71, min/max = 3/5). Half of the students (50%) indicated that they had no opinion for or against, while the rest indicated that they somewhat agreed (27%) or somewhat disagreed (23%) with the claim that corresponded to the view promoted by the text. However, no students indicated definite or strong beliefs either for or against the claim. All individual difference, comprehension, and text evaluation variables were normally distributed (skewness .05). The data in connection with the Main Claim Identification task reflected its difficulty with 29.3% of the students missing the main claim (score = 0) and another 19% confusing it with an argument (score = 1). Overall, only half of the students (50.8%, score = 2) were able to accurately identify and recall the text’s main claim. It can be seen from Table 5 that students who had a higher Need for Cognition were also more likely to rate their Prior Knowledge high. Overall, however, students rated the Table 4. Study overview and tasks Session 1 (45 min) Need for Cognition Scale Prior Knowledge and Belief Questionnaire Session 2 (after 2–3 weeks, 60 min) Reading of the Text Filler Task Post-Reading Tasks Main Claim Identification Claim Agreement Text Evaluation Text Recall Table 5. Descriptive statistics and correlations between individual difference, comprehension, and text evaluation measures (N = 116) Measures 1. NC 2. PK (perc.) 3. Recall 4. Inferences 5. Evaluation M SD

1

2

3

4

5

– .19* .09 .07 .17 61.70 9.11

– .04 .04 .02 2.95 1.51

– .25** .02 0.09 0.06

– .23* 2.61 1.61

– 5.32 1.18

Note. *p < .05; **p < .01.

Forming a belief

303

inferences that support its integration with background knowledge in working memory (Cote, Goldman, & Saul, 1998; Graesser, Millis, & Zwaan, 1997; McNamara & Magliano, 2009). The quality of the resulting representation depends on the complex interplay of person-related factors, such as inference-making skills, working memory capacity, comprehension monitoring, and the availability of relevant prior knowledge (e.g., Cain, Oakhill, Barnes, & Bryant, 2001; Carretti, Caldarola, Tencatti, & Cornoldi, 2014; Elbro & Buch-Iversen, 2013; Perez, Paolieri, Macizo, & Bajo, 2014). Moreover, the text may support inferencing and integration through cohesion markers and additional elaborations that can compensate to some extent for lack of prior knowledge and/or refutations that by highlighting prior knowledge and belief as inaccurate or unproductive can facilitate their revision (e.g., Braasch, Goldman, & Wiley, 2013; Diakidoy, Mouskounti, & Ioannides, 2011; Kamalski, Sanders, & Lentz, 2008; McNamara, Kintsch, Songer, & Kintsch, 1996). An extended argumentative text, however, poses additional cognitive demands as readers need to represent an entire line of argument – often in relation to a parallel line of counterargument – and in connection to one or more claims. The difficulty of this task was demonstrated indirectly by Larson et al. (2004), who found college students to achieve only 30% accuracy in identifying the main claim promoted by a text. Nevertheless, Neuman (2003) found text comprehension to strongly predict high school students’ ability to identify fallacies in short argument scenarios. In this study, we sought to extend the basic critical thinking paradigm to the more common and naturally occurring task of understanding and evaluating an extended argumentative text. We employed a two-sided persuasive text on a topic that was relatively neutral for our sample and measured readers’ ability to recall the main claim promoted, to recall overall text information, and to generate inferences. However, in line with reading comprehension theory and research, we conceptualized only inference generation and main claim recall as indicative of the extent to which the text was processed deeply. Moreover, our primary interest was in the extent to which comprehension supports the critical evaluation of complex argument lines embedded in text, as opposed to evaluations reflecting general liking or interest in the topic. Therefore, we varied overall argument quality on the basis of LLN – a fallacy we could manipulate without altering the content of individual arguments and creating confounding differences between text versions. However, individual arguments embedded in text cannot be modified separately from each other without altering the cohesion that contributes to understanding the text as whole. Therefore, the argument quality manipulation was introduced indirectly and at the level of the text in a way that accounted for clusters of arguments.

General hypotheses On the basis of previous findings (e.g., Murphy et al., 2005), we expected to observe changes in initial beliefs as a result of reading. Moreover, we expected persuasive impact to be positively associated with text evaluation. However, we also reasoned that, in the case of an extended text, comprehension must be a necessary prerequisite for any evaluation and belief change. Nevertheless, for comprehension to support the evaluation of a complex argument line, the resulting representation must accurately specify claims in relation to arguments along with their implications. Otherwise, the basis for subsequent evaluation and belief change becomes questionable, if not arbitrary. Therefore, we predicted measures of deep comprehension, such as the ability to represent the main

304

Irene-Anna N. Diakidoy et al.

claim promoted by the text and to generate inferences, to be more associated with evaluation than simple memory as indicated by the amount of text information recalled. Although inferential elaboration is a characteristic of deep comprehension, it is unclear the extent to which, in the case of argumentative text, this can be assumed to also involve critical elaboration. We hypothesized that if deep comprehension entails analytical processing, then its influence on evaluation would be a function of argument quality. Therefore, we would expect readers who manifest a deeper level of comprehension – as indicated by the accurate representation of the claim promoted and the generation of inferences – to give positive evaluations and be persuaded by a high-quality argument line only. In contrast, we would expect readers who manifest a more shallow comprehension level to remain uninfluenced by argument quality in their subsequent text evaluations and beliefs. Finally, there is some evidence concerning the influence of person-related factors, such as prior knowledge and need for cognition, on argument evaluation and persuasive impact. Perceived topic knowledge has been found to be negatively related to persuasive impact (Murphy & Alexander, 2004), whereas need for cognition (i.e., the tendency to engage in reflective, effortful, and complex thinking) has been hypothesized to support elaboration and belief change (Cacioppo, Petty, Feinstein, & Jarvis, 1996; Dole & Sinatra, 1998; Murphy et al., 2005) and shown to be modestly associated with critical evaluation (Klaczynski et al., 1997; Macpherson & Stanovich, 2007). To account for these potentially moderating influences, we also included measures of need for cognition and perceived topic knowledge.

Method Participants To maximize the sample’s representativeness of the educated young adult population, a combination of purposive and convenience sampling was employed to recruit participants from different universities at different levels of their study and majoring in different fields. All recruited participants were enrolled in psychology courses that were either required or electives for their field of study and obtained extra credit for their research participation. The result of this process was a sample that included 116 graduate and undergraduate university students. Although the majority of the participants were women (n = 100), this male/female ratio is representative of the student body majoring in the fields of education, social sciences, and humanities at both universities. The undergraduate students were in the third (n = 48) and fourth (n = 30) years of study at a mediumsized public university. The majority of the undergraduate students (n = 61) came from different departments in the School of Education and Social Sciences, while the rest were from the Schools of Economics, Natural Sciences, and Humanities. All graduate students (n = 38) had a first degree in Education and were in their first year towards a Master’s degree in Educational Psychology at a small private university.

Text The text was selected from a pool of authentic argumentative articles compiled from newspapers, magazines, and the Internet. A previous pilot study with a different yet comparable sample of advanced undergraduate students attending the public university (N = 57) had indicated that the majority of the students showed no

Forming a belief

305

extreme bias in favour of either the claim promoted by the text or the counterclaim. The title of the article was ‘Moral resistance’ and promoted the claim that managers and supervisors are justified in supporting hard-working but low-paid employees with time-off and other favours beyond those specified by company regulations (Aggelopoulos, 2010). The article was implicitly persuasive promoting the main claim indirectly through the inclusion of a higher number of arguments in favour (65%) than counterarguments (35%). The original article (421 words long) was further elaborated with additional information preserving, however, the original proportion of arguments to counterarguments. This modification resulted in a longer text (an average of 677 words across text versions) that included a greater variety of arguments and counterarguments than the original article (Table 1). Two independent raters parsed the text into clauses (90.5% agreement, Cohen’s K = .81, p < .01) and identified each clause as either an argument or a counterargument (89.2% agreement, Cohen’s K = .78, p < .01), and all differences were resolved in conference. Subsequently, two different text versions were created on the basis of LLN. The argument quality manipulation involved the strategic insertion of quantitative modifiers (many/a few or some) indicating indirectly the extent to which a claim was based on knowledge derived from a large sample or a limited number of cases. Therefore, in the high argument quality version, the claim was shown to derive from the fact that a large number of people were adversely affected by specific company policies, whereas in the low argument quality version the claim was shown to derive from a limited number of cases and, thus, more likely to represent a hasty generalization on the part of the author (Table 1).

Prior Knowledge and Belief Questionnaire A Prior Knowledge and Belief Questionnaire was constructed to measure Initial Belief (and possible bias) and Prior Knowledge as demonstrated by self-ratings (Perceived Prior Knowledge). The questionnaire included eight statements representing claims related to several topics (technology, environment, politics, etc.), but only one was directly related Table 1. Text excerpts representing arguments and counterarguments Claim related ..... and he [supervisor] talked about how bad he feels to have to supervise hard-working people whose wages cannot even cover basic needs like food and shelter. In the course of her [author] research, she discovered that (many/a few) managers and supervisors felt similarly stressed by this situation ..... She also discovered that (many/a few) [managers] found ways to undermine the system by secretly handing out goods or time off to help them [workers] cover basic needs and care for sick children ..... What is the greater injustice here? Violating a company regulation or helping people who, despite working hard, cannot support their families? ..... Treating workers with fairness and understanding not only has no adverse consequences for the company but may also increase productivity in the long run Counterclaim related ..... There is nothing moral in these acts as they [supervisors] are stealing from their employer ..... There is always the possibility that handouts are given on the basis of personal relationship and not on the basis of true need Note. Excerpts were translated from Greek. Argument quality manipulations are underlined and shown in parentheses.

306

Irene-Anna N. Diakidoy et al.

to the topic of the selected text (work relations/economy). Each statement was followed by a 7-point agreement scale and a 7-point prior knowledge rating scale. Participants were asked to read each statement, to indicate their degree of agreement (1 = strongly disagree, 7 = strongly agree), and to indicate how much they knew about the topic reflected in the statement (1 = nothing, 7 = very much). Only the scores obtained for the statement that corresponded to the selected text’s main claim were of interest and included in the analyses. Two versions of the questionnaire with statements appearing in different order were created and counterbalanced between participants (see Table 2 for example items).

Need for Cognition The short version of the need for cognition scale (Cacioppo et al., 1996) was used in this study. The scale was translated into Greek and then back into English to verify that the translation reflected accurately the meaning of the original statements. The scale included 18 statements, nine indicating a high need for cognition and nine statements indicating a low need for cognition which were scored in reverse. Each statement was followed by a 5-point scale on which participants were to rate how much the statement characterized them (1 = not at all, 5 = very much). The reliability of the scale was acceptable (Cronbach’s a = .79) given the scope and goals of this study (Nunnally, 1978).

Post-reading tasks The purpose of the post-reading tasks was to provide measures of text comprehension, text evaluation, and persuasive impact. The first task required students to recall what they thought was the main claim that the text promoted. Responses that represented the general topic, the counterclaim, or an irrelevant detail were given a score of 0, whereas responses that represented an argument – indicating a confusion of the claim with arguments – were given a score of 1. Responses that indicated that the student had correctly identified the main claim were given a score of 2 (92.5% agreement between two independent raters, Cohen’s K = .85, p < .01, differences resolved in conference). Sample responses in each scoring category are shown in Table 3. The score received by each student in this task represented his/her Main Claim Identification score. Table 2. Sample items of the Prior Knowledge and Belief Questionnaire Statements Managers should always adhere to company rules and policies as a matter of company loyaltya The marketing and use of genetically modified products should be strictly limited by law The death penalty is warranted in cases of very serious and serial crimes For the sake of the environment, a proportion of industrial and farm land should be converted into wild parks Questions following each statement How much do you agree with the statement? How much do you know about the issue reflected in the statement? Note. Excerpts were translated from Greek. Statement corresponding to the counterclaim of selected text (agreement rating was scored in reverse).

a

Forming a belief

307

Table 3. Sample responses to the Main Claim Identification task in each scoring category Responses scored 0 This text is about workers’ rights in a global economy (student 67) The claim that this passage promotes is that sometimes actions that seem good are in fact stealing (student 52) The passage promotes equality between employers and employees (student 8) Responses scored 1 This text claims that if you help workers they will be more productive (student 53) Some managers feel guilty supervising people who work very hard but make very little money (student 85) The article argues that it is not fair to work hard but earn enough to have a decent life (student 18) Responses scored 2 At a time of financial hardship and economic injustice, it is important for people to do what they can to support their fellow humans (student 6) The claim is that human rights are more important than any company rules and profit that may violate them (student 104) The passage promotes the claim that managers also have a responsibility towards their employees (not just their employer) and their well-being in and out of the workplace (student 21) Note. Responses were translated from Greek.

Subsequently, students were asked to indicate on a 7-point scale the degree to which they agreed with the main claim that the text promoted (‘How much do you agree with the claim the text promotes?’, 1 = strongly disagree, 7 = strongly agree). This Claim Agreement task corresponded to the Belief Rating task included in the Prior Knowledge and Belief Questionnaire, therefore, allowing for direct comparisons between before- and after-reading beliefs. The Claim Agreement scale was followed by a Text Evaluation scale asking participants to indicate the degree to which they found the text and its arguments convincing (‘How convincing do you think are the arguments the text presents to promote the claim?’, 1 = not at all convincing, 7 = very convincing). The final task was a free recall task that asked students to write all they could remember from the text they had read. The recall protocols were parsed into clauses by two independent raters (96% agreement, Cohen’s K = .92, p < .01), and each clause was identified as either representing an explicit text idea in terms of gist (verbatim or paraphrase) or a valid inference that exemplified a connection that was left implicit by the text (bridging inference) or a reader-provided elaboration (elaborative inference) (86% agreement, Cohen’s K = .71, p < .01, differences resolved in conference). The scoring of the recall protocols yielded two measures: An Overall Recall Score representing the proportion of text ideas recalled (total number of text ideas in recall/total number of ideas in text) – an index of memory – and the total Number of Valid Inferences generated in recall – representing an index of elaboration and deep comprehension.

Procedure Data collection took place in two group sessions scheduled 2–3 weeks apart. Session 1 involved the administration of the Need for Cognition scale and the Prior Knowledge and Belief Questionnaire. In Session 2, the students were randomly assigned to read one of the two text versions (high argument quality or low argument quality). They were instructed

308

Irene-Anna N. Diakidoy et al.

to read the text once, but carefully, with the goal of understanding it. Then, they completed a filler task that was similar to the Prior Knowledge and Belief Questionnaire but included different statements. Finally, the post-reading tasks were administered. Session 2 lasted about 60 min (Table 4).

Results Preliminary results Preliminary analyses indicated that the text topic was relatively neutral in terms of initial Belief bias either for or against the text’s main claim (M = 3.97, SD = 0.71, min/max = 3/5). Half of the students (50%) indicated that they had no opinion for or against, while the rest indicated that they somewhat agreed (27%) or somewhat disagreed (23%) with the claim that corresponded to the view promoted by the text. However, no students indicated definite or strong beliefs either for or against the claim. All individual difference, comprehension, and text evaluation variables were normally distributed (skewness .05). The data in connection with the Main Claim Identification task reflected its difficulty with 29.3% of the students missing the main claim (score = 0) and another 19% confusing it with an argument (score = 1). Overall, only half of the students (50.8%, score = 2) were able to accurately identify and recall the text’s main claim. It can be seen from Table 5 that students who had a higher Need for Cognition were also more likely to rate their Prior Knowledge high. Overall, however, students rated the Table 4. Study overview and tasks Session 1 (45 min) Need for Cognition Scale Prior Knowledge and Belief Questionnaire Session 2 (after 2–3 weeks, 60 min) Reading of the Text Filler Task Post-Reading Tasks Main Claim Identification Claim Agreement Text Evaluation Text Recall Table 5. Descriptive statistics and correlations between individual difference, comprehension, and text evaluation measures (N = 116) Measures 1. NC 2. PK (perc.) 3. Recall 4. Inferences 5. Evaluation M SD

1

2

3

4

5

– .19* .09 .07 .17 61.70 9.11

– .04 .04 .02 2.95 1.51

– .25** .02 0.09 0.06

– .23* 2.61 1.61

– 5.32 1.18

Note. *p < .05; **p < .01.

Forming a belief

309

topic as relatively unfamiliar, with only 16% of the students rating their topic knowledge high. Need for Cognition and Perceived Prior Knowledge did not correlate significantly with comprehension outcomes, such as Recall and Number of Valid Inferences. Nonparametric correlations indicated that Main Claim Identification was similarly unrelated to Need for Cognition (rs = .02, p = .832) and Perceived Prior Knowledge (rs = .06, p = .508). Finally, the correlations between individual difference variables and Text Evaluation were also non-significant (Table 5).

Comprehension and evaluation Overall, the text was evaluated positively (Table 5) with the majority of the students (79.3%) judging it and its arguments to be somewhat to very convincing. Our first hypothesis concerned the association between deep comprehension and Text Evaluation. As expected, Number of Inferences, but not Recall, correlated positively with Text Evaluation. Students who generated more inferences in their recalls were also more likely to evaluate the text positively. However, no similar significant relationship was found between our second deep comprehension measure, Main Claim Identification, and Evaluation (rs = .07, p = .451). It is noteworthy that Main Claim Identification was also unrelated to inferencing (rs = .02, p = .800). To examine this problematic result further, we performed logistic regression for Main Claim Identification with Need for Cognition, Perceived Prior Knowledge, Recall, and Number of Inferences as the predictors. Although the final model was not significant, v2 (8) = 12.98, p = .112, R2 = .11 (Cox & Snell), Number of Inferences contributed significantly to confusions, B = 0.48, Exp(B) = 0.62, Wald v2 = 5.24, p = .022. Students who generated more inferences were also more likely to confuse the main claim with individual arguments. Our second hypothesis concerned the association between deep comprehension of argumentative text and its critical evaluation. To test this hypothesis, we performed ANCOVA with Need for Cognition, Perceived Prior Knowledge and Number of Inferences as the covariates, and Text Main Claim (missed, confused, or accurately identified) and Argument Quality (high vs. low) as the independent variables. The results indicated significant main effects of Need for Cognition, F (1, 114) = 7.20, p = .008, g2 = .06, and Perceived Prior Knowledge, F (1, 114) = 5.29, p = .023, g2 = .05, and a significant interaction between these two covariates, F (1, 114) = 5.56, p = .020, g2 = .05. It can be seen from Table 6 that a higher Need for Cognition was associated with more positive evaluations in the absence of Prior Knowledge than a lower Need for Cognition. Contrary Table 6. Means of text evaluation ratings as a function of Need for Cognition and Perceived Prior Knowledge (N = 109) Perceived Prior Knowledge

Need for Cognition Lower Higher Cohen’s d

Lower

Higher

d

5.07 (1.28) 5.74 (0.87) .61

5.36 (1.18) 5.31 (1.18) .04

.24 .41

Note. Standard deviations are shown in parentheses. Need for Cognition and Perceived Prior Knowledge were split on the median for presentation purposes only.

310

Irene-Anna N. Diakidoy et al.

to our expectations, however, neither Argument Quality nor any of the deep comprehension measures had any significant independent or interactive effects on Text Evaluation (p > .05). Students who read the low argument quality version of the text gave similar evaluations as those who read the high argument quality version, M = 5.22, SD = 1.26 and M = 5.44, SD = 1.09 respectively, t (112) = 1.02, p = .308.

Evaluation and persuasion To examine the connection between Text Evaluation and persuasive impact, we had asked students to indicate the degree to which they agreed with the main claim the text promoted – a measure that corresponded to the Belief Bias measure in the Prior Knowledge and Belief Questionnaire. However, as 34 students had missed the text’s main claim (rendering it impossible to determine with what they agreed or disagreed), we decided to exclude their data for this part of the analyses. In the restricted sample (n = 81), Claim Agreement ratings were negatively skewed (M = 5.77, SD = 1.15, min/ max = 1/7, skewness = 1.39) and, therefore, only nonparametric analyses were performed. As expected, Claim Agreement correlated positively with Text Evaluation (rs = .59, p = .000). However, this measure did not correlate with any other variables, and it is noteworthy that its correlation with the corresponding initial Belief measure was also not significant (rs = .15, p = .168). In fact, a Wilcoxon signed rank test for related samples indicated that the differences between initial Belief (M = 3.94, SD = 0.71, Mdn = 4) and post-reading Claim Agreement (M = 5.77, SD = 1.15, Mdn = 6) were significant (p < .01). To further explore influences on belief change, a difference score was created by subtracting the initial Belief score from the Claim Agreement score. The resulting Belief Change score was also negatively skewed (skewness = 1.24) with M = 1.83, SD = 1.29, Mdn = 2, min/max = 4/4. Belief Change correlated significantly with Text Evaluation only (rs = .53, p = .000). Finally, a Mann–Whitney U-test indicated that Argument Quality did not influence Belief Change (p > .05).

Discussion The study examined the contribution of comprehension to the evaluation and persuasive impact of an argumentative text in the absence of strong prior bias. The quality of the arguments was varied between text versions, and measures of perceived prior knowledge and need for cognition were also obtained. The findings highlight (1) the difficulty of comprehending extended argumentative text, and (2) its high persuasive impact in the absence of a critical evaluation of its argument line. Consistent with previous findings (e.g., Murphy et al., 2005), the text employed in this study was also successful in altering initial beliefs. In fact, 89% of the students who were able to identify the text’s main claim modified their initial beliefs to make them more congruent with the view promoted by the text. As expected, students who evaluated the text positively were also more likely to be persuaded by its arguments, manifesting greater belief change than students who gave lower text evaluations. Positive evaluations, in turn, were associated with increased number of inferences generated in recall and higher need for cognition in the absence of relevant prior knowledge. However, half of the students either missed the main claim of the text or confused it with an argument. This finding is consistent with those of Larson et al. (2004) and

Forming a belief

311

underscores the difficulty of mentally representing complex argument lines embedded in text. We had reasoned that the ability to identify the main claim, as an index of deep comprehension, would support the critical evaluation of arguments as opposed to simple memory for text information. However, this was not the case, as neither main claim recall nor overall recall contributed to evaluation. Moreover, although inference generation contributed to positive evaluations, it did not interact with argument quality. This finding in conjunction with the fact that inferences also contributed to main claim confusions suggests that the inferences generated represented more local connections and/or argument-irrelevant elaborations (Cote et al., 1998) than elaborations that would support the critical evaluation of arguments in relation to each other and to claims. Our findings concerning the influence of need for cognition on evaluation are only partly consistent with previous claims and findings (Cacioppo et al., 1996; Klaczynski et al., 1997; Murphy et al., 2005). Contrary to our expectations, however, this factor interacted with perceived topic knowledge instead of argument quality. Students with a higher need for cognition but lower relevant knowledge found the text more convincing regardless of the overall quality of its argument line. In a large meta-analytic study, Cacioppo et al. (1996) reviewed evidence indicating small but reliable interactions of need for cognition with argument quality in evaluation and persuasion. Individuals with a high need for cognition were more influenced by the quality of the arguments to which they were exposed than individuals with a low need for cognition. Argument quality, however, was operationalized as argument strength on the basis of prior subjective ratings (Cacioppo et al., 1996). Instead, in this study we attempted to formalize argument quality on the basis of an established informal reasoning fallacy, that is the violation of the law of large numbers. Nevertheless, the need to control for content across text versions prevented a direct and explicit manipulation of argument quality masking, thereby, any effects of this thinking disposition variable as well as those of comprehension. Nevertheless, we consider an overarching manipulation of argument quality more ecologically valid and viable in the case of the complex and interrelated lines of argument normally found in extended argumentative text. Although the present findings, taken together, seem to be consistent with a dualprocess framework, in general, and the motivated reasoning hypothesis (Klaczynski et al., 1997) that specifically predicts shallow processing of an argument line in the absence of strong prior bias, we take issue with the extent to which at least half of our sample engaged in shallow processing. Specifically, we would consider students who identified and represented the main claim and generated inferences to have processed the text deeply and to have constructed a coherent and elaborated representation of its content despite their initial neutral beliefs. Nevertheless, our findings suggest a dissociation between the elaboration characterizing deep comprehension and the elaboration involved in the critical examination of arguments (Evans, 2012; Gl€ ockner & Witteman, 2010; Stanovich et al., 2008). Critical thinking research has shown automatic, intuitive processing to predominate in informal reasoning and decision-making tasks (e.g., Kahneman & Frederick, 2002; Tversky & Kahneman, 1974), and the present findings can be taken as further evidence. However, the extent to which the integrative and elaborative processes involved in the comprehension of an unfamiliar and demanding text can be solidly classified as automatic and intuitive is debatable (van den Broek, Lorch, Linderholm, & Gustafson, 2001) and unsupported by the comprehension-related findings of the present study. Nevertheless, deliberate comprehension and analytic processes may be employed for different purposes.

312

Irene-Anna N. Diakidoy et al.

Comprehension concerns the construction of representations (Gl€ ockner & Witteman, 2010), whereas analytic processes concern the manipulation of representations (Evans, 2012). Therefore, manipulations involving task instructions and/or reader goals for reading may provide a better context for studying the extent to which and how comprehension may contribute to critical evaluation. Further research should also examine the extent to which current results generalize across texts and topics. In this study, a single text was pre-selected and used to control for strong bias in favour or against the claim it promoted. Nevertheless, that leaves open the possibility of text characteristics influencing the results obtained, especially those related to comprehension and critical evaluation (e.g., Kamalski et al., 2008; McNamara et al., 1996). Moreover, the text presented a moral dilemma faced by a group of people and its main argument line capitalized on universal principles of fairness and justice. Considering that our sample of young educated readers are likely to display a higher level of moral reasoning (Nucci & Turiel, 2009), it also remains possible that the observed persuasive impact may reflect a close match between the text’s argument line and the sample’s increased concern for human rights and their protection. Similarly, the positive evaluations obtained may in part reflect our readers’ increased ability to co-ordinate different social and moral goals (Turiel, 2003). Therefore, research employing different texts on different kinds of controversial topics or issues should help untangle any influence of argument quality on critical evaluation from that of text and topic. The underlying assumption that motivated this study was that comprehension must be a prerequisite for critical thinking. Understanding and critically evaluating an extended argumentative text presents a greater challenge than understanding and critically evaluating individual arguments or short argument scenarios. Especially in the absence of well-established topic-related knowledge, processing resources need to be first allocated to mentally representing the content of the text, possibly in competition with the more analytic processing required for the critical evaluation of arguments (Handley, Newstead, & Trippas, 2011). Nevertheless, argumentative text remains a primary medium for belief formation, belief change, and learning in general. As such, it can provide the basis for decision-making in different contexts with consequences and implications ranging from trivial to critical. Therefore, the ease with which adult readers agree with claims and accept arguments, even when alternatives are presented, is a matter of social and educational concern. Our findings suggest that addressing this concern would necessitate simultaneous attention to the development of both reading comprehension and critical thinking skills. There is substantial evidence that collaborative reasoning in the context of dialogic enquiry can advance younger and older students’ critical thinking and argument skills (e.g., Felton & Kuhn, 2001; Iordanou, 2010; Iordanou & Constantinou, 2014; Kuhn, Zillmer, Crowell, & Zavala, 2013; Reznitskaya & Gregory, 2013; Reznitskaya et al., 2001). Moreover, explicitly targeting specific thinking and argument evaluation skills across different subjects in the context of an infusion approach appears promising with multiple potential benefits on critical thinking, critical reading, and metacognitive monitoring skills (e.g., Bensley & Spero, 2014; Dewey & Bento, 2009). Specifically, the acquisition of knowledge and skill regarding the parts of an argument, the connection between evidence and claims, the criteria for evaluating this connection and the contribution of counterarguments can form additional objectives of direct instruction in relevant subject matter enriching, thereby, both the existing curriculum and the learning outcome.

Forming a belief

313

Acknowledgements This research was funded in part by a University of Cyprus Internal Research Grant (8037P16019) to the first author. We thank Melina Ioannou, Constantina Alkiviadous, Andri Anastasiou, Eleni Papachristou, and Irene Andreou for their valuable help with data collection and scoring.

References Aggelopoulos, G. (2010, February 26). Moral resistance. Newspaper The News. Retrieved from http://democracy-rethymno.blogspot.com/2010/03/blog-post_03.html Alexander, P. A., Buehl, M. M., & Sperl, C. T. (2001). The persuasiveness of persuasive discourse. International Journal of Educational Research, 35, 651–674. doi:10.1016/S0883-0355(02) 00008-3 Bensley, D. A., & Spero, R. A. (2014). Improving critical thinking skills and metacognitive monitoring through direct infusion. Thinking Skills and Creativity, 12, 55–68. Braasch, J. L. G., Goldman, S. R., & Wiley, J. (2013). The influences of text and reader characteristics on learning from refutations in science texts. Journal of Educational Psychology, 105, 561– 578. doi:10.1037/a0032627 Cacioppo, J. T., Petty, R. E., Feinstein, J. A., & Jarvis, B. G. (1996). Dispositional differences in cognitive motivation: The life and times of individuals varying in need for cognition. Psychological Bulletin, 119, 197–253. Cain, K., Oakhill, J. V., Barnes, M. A., & Bryant, P. E. (2001). Comprehension skill, inference-making ability, and their relation to knowledge. Memory and Cognition, 29, 850–859. Carretti, B., Caldarola, N., Tencatti, C., & Cornoldi, C. (2014). Improving reading comprehension in reading and listening settings: The effect of two training programmes focusing on metacognition and working memory. British Journal of Educational Psychology, 84, 194–210. doi:10.1111/ bjep.12022 Cote, N., Goldman, S. R., & Saul, E. U. (1998). Students making sense of informational text: Relations between processing and representation. Discourse Processes, 25, 1–53. Dewey, J., & Bento, J. (2009). Activating children’s thinking skills (ACTS): The effects of an infusion approach to teaching thinking in primary schools. British Journal of Educational Psychology, 79, 329–351. doi:10.1348/000709908X344754 Diakidoy, I. N., Mouskounti, T., & Ioannides, C. (2011). Comprehension and learning from refutation and expository texts. Reading Research Quarterly, 46, 22–38. doi:10.1598/RRQ.46.1.2 Dole, J. A., & Sinatra, G. M. (1998). Reconceptualising change in the cognitive construction of knowledge. Educational Psychologist, 33, 109–128. doi:10.1207/s15326985ep3302&3_5 Elbro, C., & Buch-Iversen, I. (2013). Activation of background knowledge for inference making: Effects on reading comprehension. Scientific Studies of Reading, 17, 435–452. doi:10.1080/ 10888438.2013.774005 Evans, J. S. B. T. (2012). Questions and challenges for the new psychology of reasoning. Thinking & Reasoning, 18, 5–31. doi:10.1080/13546783.2011.637674 Felton, M., & Kuhn, D. (2001). The development of argumentative discourse skill. Discourse Processes, 32, 135–153. Glassner, A., Weinstock, M., & Neuman, Y. (2005). Pupils’ evaluation and generation of evidence and explanation in argumentation. British Journal of Educational Psychology, 75, 105–118. doi:10.1348/000709904X22278 Gl€ ockner, A., & Witteman, C. (2010). Beyond dual-process models: A categorisation of processes underlying intuitive judgement and decision making. Thinking & Reasoning, 16, 1–25. doi:10.1080/13546780903395748 Graesser, A. C., Millis, K. K., & Zwaan, R. A. (1997). Discourse comprehension. Annual Review of Psychology, 48, 163–189.

314

Irene-Anna N. Diakidoy et al.

Handley, S. J., Newstead, S. E., & Trippas, D. (2011). Logic, beliefs, and instruction: A test of the default interventionist account of belief bias. Journal of Experimental Psychology: Learning, Memory, & Cognition, 37 (1), 28–43. doi:10.1037/a0021098 Iordanou, K. (2010). Developing argument skills across scientific and social domains. Journal of Cognition and Development, 11, 293–327. doi:10.1080/15248372.2010.485335 Iordanou, K., & Constantinou, C. P. (2014). Developing pre-service teachers’ evidence-based argumentation skills on socio-scientific issues. Learning and Instruction, 34, 42–57. doi:10.1016/j.learninstruc.2014.07.004 Kahneman, D., & Frederick, S. (2002). Representativeness revisited: Attribute substitution in intuitive judgment. In T. Gilovich, D. Griffin & D. Kahneman (Eds.), Heuristics and biases: The psychology of intuitive judgment (pp. 49–81). New York, NY: Cambridge University Press. Kamalski, J., Sanders, T., & Lentz, L. (2008). Coherence marking, prior knowledge, and comprehension of informative and persuasive texts: Sorting things out. Discourse Processes, 45, 323–345. doi:10.1080/01638530802145486 Kardash, C. M., & Scholes, R. J. (1996). Effects of pre-existing beliefs and repeated readings on belief change, comprehension and recall of persuasive text. Contemporary Educational Psychology, 20, 201–221. Klaczynski, P. A., Gordon, H. D., & Fauth, J. (1997). Goal-oriented critical reasoning and individual differences in critical reasoning biases. Journal of Educational Psychology, 89, 470–485. Kuhn, D., Zillmer, N., Crowell, A., & Zavala, J. (2013). Developing norms of argumentation: Metacognitive, epistemological, and social dimensions of developing argumentative competence. Cognition and Instruction, 31, 456–496. doi:10.1080/07370008.2013.830618 Larson, M., Britt, A. M., & Larson, A. A. (2004). Disfluencies in comprehending argumentative texts. Reading Psychology, 25, 204–224. doi:10.1080/02702710490489908 Macpherson, R., & Stanovich, K. E. (2007). Cognitive ability, thinking dispositions, and instructional set as predictors of critical thinking. Learning and Individual Differences, 17, 115–127. doi:10.1016/j.lindif.2007.05.003 McNamara, D. S., Kintsch, E., Songer, N. B., & Kintsch, W. (1996). Are good texts always better? Interactions of text coherence, background knowledge, and levels of understanding in learning from text. Cognition and Instruction, 14, 1–43. McNamara, D. S., & Magliano, J. (2009). Toward a comprehensive model of comprehension. The Psychology of Learning and Motivation, 51, 297–384. doi:10.1016/S0079-7421(09)51009-2 Murphy, K. P., & Alexander, A. P. (2004). Persuasion as a dynamic, multidimensional process. American Educational Research Journal, 41, 337–363. doi:10.3102/00028312041002337 Murphy, K. P., Holleran, A. T., Long, F. J., & Zeruth, A. J. (2005). Examining the complex roles of motivation and text medium in the persuasion process. Contemporary Educational Psychology, 30, 418–438. doi:10.1016/j.cedpsych.2005.05.001 Neuman, Y. (2003). Go ahead, prove that God does not exist! On high school students’ ability to deal with fallacious arguments. Learning and Instruction, 13, 367–380. doi:10.1016/S0959-4752 (02)00011-7 Neuman, Y., & Weizman, E. (2003). The role of text representation in students’ ability to identify fallacious arguments. The Quarterly Journal of Experimental Psychology, 56, 849–864. doi:10.1080/02724980244000666 Nucci, L., & Turiel, E. (2009). Capturing the complexity of moral development and education. Mind, Brain, and Education, 3, 151–159. Nunnally, J. C. (1978). Psychometric theory. New York, NY: McGraw-Hill. Perez, A. I., Paolieri, D., Macizo, P., & Bajo, T. (2014). The role of working memory in inferential sentence comprehension. Cognitive Processes, 15, 405–413. doi:10.1007/s10339-014-0611-7 Petty, R. E., & Cacioppo, J. T. (1986). Communication and persuasion: Central and peripheral routes to attitude change. New York, NY: Springer-Verlag. Reznitskaya, A., Anderson, R. C., McNurlen, B., Nguyen-Janiel, K., Archodidou, A., & Kim, S. (2001). Influence of oral discussion on written argument. Discourse Processes, 32, 155–175.

Forming a belief

315

Reznitskaya, A., & Gregory, M. (2013). Student thought and classroom language: Examining the mechanisms of change in dialogic teaching. Educational Psychologist, 48, 114–133. doi:10.1080/00461520.2013.775898 Stanovich, K. E., Toplak, M. E., & West, R. F. (2008). The development of rational thought: A taxonomy of heuristics and biases. Advances in Child Development and Behavior, 36, 251– 285. Stanovich, K. E., & West, R. F. (1997). Reasoning independently of prior belief and individual differences in actively open-minded thinking. Journal of Educational Psychology, 89, 342–357. Stanovich, K. E., & West, R. F. (2007). Natural myside bias is independent of cognitive ability. Thinking & Reasoning, 13, 225–247. doi:10.1080/13546780600780796 Turiel, E. (2003). Resistance and subversion in everyday life. Journal of Moral Education, 32, 115– 130. Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases. Science, 185, 1124–1131. van den Broek, P., Lorch, R. F., Linderholm, T., & Gustafson, M. (2001). The effects of readers’ goals on inference generation and memory for texts. Memory & Cognition, 29, 1081–1087. Received 24 December 2013; revised version received 9 March 2015

Forming a belief: The contribution of comprehension to the evaluation and persuasive impact of argumentative text.

Research has shown substantial belief change as a result of reading text and the pervasive influence of prior belief in the evaluation of short argume...
172KB Sizes 0 Downloads 7 Views