Journal of Linguistics and Language Teaching
Volume 13 (2022) Issue 1
Washback of a High-stakes Language Exam on Spanish Teachers' Practice in Brazil
Yasser Abdullah Al Tamimi (Riyadh, Saudi Arabia), Gladys Quevedo-Camargo (Brasília, Brazil) & Rafael Sena Raposo de Melo (Brasília, Brazil)
Abstract (English)
The present study explores the impact of a three-phased high-stakes admission exam on the practice of Spanish language teachers as they prepare their high-school students for meeting the entry requirements at the University of Brasilia. The Serial Assessment Program (PAS) is an annual reading test in Spanish, French or English functioning as a diagnostic, summative, and formative assessment tool and offering an entry opportunity for low-income students. Using a case study, an online questionnaire of twenty-nine Spanish public school teachers and three interviews, the study identifies potential washback of the Spanish exam on the teaching practice of public school Spanish language teachers when they teach Spanish reading skills. Despite the teachers' consensus that the PAS exam is essential in its socio-economic context, data analysis points to an impoverished Spanish language content integrated into the Spanish curriculum. Evidence of washback was identified as being restricted to past exam papers used to familiarize learners with the exam item format, topics, themes, and genres utilized in the PAS Spanish language component. Bottom-up approaches to reading seem to predominate, and a more consistent alignment of reading competencies and abilities to the exam framework was not identified.
Keywords: Washback, Programa de Avaliação Seriada exam-PAS, Spanish language teachers; the University of Brasilia, top-down and bottom-up approaches; Watanabe's dimensions
Abstracto (Español)
Este estudio investiga el impacto de un examen de admisión y alto riesgo hecho en tres fases en la práctica de los profesores de español mientras preparan a sus estudiantes de secundaria para cumplirlo como oportunidad de ingreso en la Universidad de Brasilia. El programa de evaluación seriada (PAS) representa una prueba anual de lectura en español, francés o inglés que funciona como herramienta de diagnóstico y también tiene función sumativa y formativa. Utilizando el estudio de caso, un cuestionario en línea hecho por veintinueve profesores de lengua española de escuelas públicas y tres entrevistas, el estudio identifica el potencial impacto del examen en la práctica de los docentes cuando enseñan la habilidad lectora en español. A pesar del consenso de los profesores de que el examen es esencial en su contexto socioeconómico, el análisis de los datos apunta a un contenido empobrecido del español integrado en el currículo escolar. La evidencia del efecto retroactivo se identificó con prácticas y usos de documentos y exámenes anteriores utilizados para familiarizar a los estudiantes con el formato del examen, con los temas y los géneros utilizados. Los enfoques ascendentes de la lectura parecen predominar y no se identificó una alineación más consistente de las competencias y capacidades de lectura con el marco del examen.
Palabras clave: Efecto retroactivo, examen Programa de Evaluación Seriada (PAS), profesores de español, Universidad de Brasilia, enfoques descendentes y ascendentes; dimensiones de Watanabe.
1 Introduction
The University of Brasília (UnB), founded in 1960, is one of the largest Brazilian federal universities. It is situated in the capital of the country, Brasília – Federal District, right in the middle of the national territory, and hence, it holds primary social, economic, political and intellectual status. Traditionally, admission to the UnB undergraduate programs required a single annual entrance exam – the High-school National Exam (Exame Nacional do Ensino Médio – ENEM). However, in 1995, and in response to the community needs, the UnB instituted in parallel the Serial Assessment Program (Programa de Avaliação Seriada – PAS) as a reliable and valid admission test. The PAS is divided into three principal stages (PAS 1, PAS 2, and PAS 3) that correspond to completing the first, second and third high school grades. The learning outcomes of the high school core reading content to be considered in the PAS assessment were developed by specialized representative teachers across the Federal District. Over the years, a test model was created in which students' knowledge and skills could be analyzed in terms of school performance, thus promoting a more coherent dialogue between schools and the University.
Due to its importance as an UnB admission opportunity, the PAS has significantly influenced higher education in Brasília and the surrounding region. Thus, to help students pass the exam, high schools ordinarily offer supplementary exam preparation courses. For further support, some Federal District schools have integrated into their curricula the skills, competencies, methods and concepts regularly evaluated by the PAS.
Even though the PAS has been used as a principal gateway for high-school graduates to join the UnB for over a quarter of a century now, being a valid instrument for diagnostic, formative and summative assessment with a recognized social relevance, it has barely received research attention. In fact, prior to conducting the present paper, we expected to find much literature on its impact on the teaching and learning process in different areas, but surprisingly, we only found one study, viz. Pessoa's (2004) M.A. dissertation that exclusively focuses on the washback of the English track of the PAS exam. The lack of studies on the washback of the other two tracks – Spanish and French – constitutes a gap in the literature that the present study attempts to partially bridge by investigating the washback of the PAS Spanish track on the teaching situation at Spanish high schools. More specifically, the study aims to identify evidence of this washback in the Spanish high school teachers' practice as they teach Spanish reading skills, the main focus of the PAS exam, in regular public schools in Brasília, Federal District Brazil.
2 Background
2.1 PAS: Overview and Significance
The PAS is divided into three stages to provide gradual knowledge assessment during the three high-school grades. After each stage, a raw score is generated and completed in the third grade of high school (PAS 3), when the students choose the course they wish to take among the UnB ninety-nine undergraduate programs. Each stage has a weight corresponding to the grade, so stage 1 has weight 1, stage 2 has weight 2, and stage 3 has weight 3. This results in a classification score in descending order proportional to the number of vacancies offered by the administration of the courses. The selection of students takes place in descending order until all vacancies are filled.
Although taking the PAS is not compulsory, it is still beneficial for its takers as it offers a formative, diagnostic and summative assessment of their learning outcomes by enabling them to access their performance data through the UnB Undergraduate Course Selection Information System, a website provided by the University. Thus, the Program generates subsidies that guide students based on their results since they receive a performance report.
The PAS is an annual exam, usually held on Sundays, for more than four thousand students who, upon passing the exam, can competitively secure admission at the UnB. According to the exam organizing committee, the number of exam takers in 2019 was 12,260. Unfortunately, the exam was not administered in 2020 due to the pandemic, but in 2021 there were 17,346 test-takers. Luckily, the number of candidates is increasing gradually with the continuing return to normal. In addition to the vacancies available for wide competition, the exam organization committee incorporates affirmative actions in all courses for black and indigenous people, as well as for public school students whose family income is less than or about one and a half minimum Brazilian wage (approximately 330 USD nowadays).
The PAS is not the only gateway to the University of Brasília. Students can also take the traditional university entrance exam – the Vestibular – or the High-school National Exam (Exame Nacional do Ensino Médio – ENEM). The difference is that apart from being a gradual approval process, the PAS exam breaks down barriers across disciplines.
2.2. PAS Structure
Interdisciplinarity is an aspect that aims at developing students' critical ability to articulate different contents, such as math and science, in a pragmatic, applicable and genuine way (e.g. Graff 2015 and Allwood et al. 2020). The PAS exam fulfills this purpose by creating cross-cutting themes in its items that mix knowledge from different areas and eliminate the formal boundaries of the disciplines. This idea of interdisciplinary construction refers to the conception of an exam being free from memorization and conditioning, thus different from traditional entrance exams. The exam is intended to address current issues concerning young people's daily lives, and therefore, it involves many areas of knowledge, not restricted to assessing only the content proposed by a single area. This is systematized through a framework that consists of five competencies or domains (Portuguese and foreign languages, natural and social phenomena, decision making, critical thinking, and social agency) and four skills: interpretation, planning, performing and criticizing. Only part of the first competence – foreign languages and interpretation and performing skills are operationalized in the PAS exam to provide evidence of foreign language knowledge.
The exam consists of 100 multiple-choice items divided into four types: A, B, C, and D. Type A items contain a text, a command and an assertion that should be judged as right or wrong. Type B items are restricted to science and mathematics and require students to solve them utilizing equations, for instance. Type C items have a four-answer multiple-choice format and focus primarily on social themes in comics or cartoons since they require the ability to interpret structural knowledge and correlate it with sociocultural knowledge to arrive at the correct answer. Finally, the Type D item is a written task, in which students have to write a short essay on the content of any area of knowledge selected by the exam board.
Ten items of the exam focus on content related to foreign languages, and candidates can choose one of the three languages offered: Spanish, French or English. The Spanish language component consists of nine Type A items and one Type C item, as shown in Figures 1 and 2 taken from the exam website.
Figure 1: Example of type A item in the PAS Spanish component
(https://www.professorjailton.com.br/novo/provas-vestibulares/PAS-UnB/
Subprograma%202019-2021/1Etapa/461_PAS_1_2019_Matriz.pdf.; 19-11-2021)
Figure 2: Example of type C item in the PAS Spanish language component
(https://www.professorjailton.com.br/novo/provas-vestibulares/
PAS-UnB/Subprograma%202019-2021/1Etapa/461_PAS_1_
2019_Matriz.pdf; 19-11- 2021)
Despite its relevance and innovative proposal in terms of Interdisciplinarity, gradual knowledge assessment and access to performance feedback, little is known about whether or not the exam exerts any washback on instructors' teaching practices or students' learning processes.
3 Literature Review
Washback in language learning is commonly defined as:
the impact or influence that external exams, particularly high-stakes exams, as well as achievement tests may potentially have on language teaching and learning processes, the curriculum, material design and stakeholders' attitudes" (Quevedo-Camargo & Scaramucci, 2016: 287).
Although the academic community has always acknowledged its existence, washback started being studied more deeply and thoroughly after Alderson & Wall (1993) emphasized its complexity and discussed methodological alternatives to investigate it. Since then, numerous studies have been carried out in different parts of the world, focusing on different exams and educational contexts (Tsagari & Cheng (2017) for a historical perspective and studies on washback).
Theoretical models have also been proposed to explain the washback mechanism. Bailey (1996) suggested three main elements that may be influenced by the nature of a test:
the participants (learners, teachers, administrators, coursebook writers and publishers),
the processes (any action related to learning), and
the products (what was taught or learned).
Burrows (2004) presented a curriculum-innovation model in which he highlighted the importance of taking into account teachers' knowledge, beliefs, assumptions, and participants' reactions to the changes caused by tests.
Watanabe (2004) proposed a washback model consisting of five dimensions:
The first dimension is value, which can be positive, negative or neutral: the changes caused by an exam can be beneficial or not, or even irrelevant to exam takers and the teaching and learning process. It has to be noted that the mere existence of an exam is not a sufficient condition for an impact to be generated by it (Shohamy et al. 1996). In addition, changes can be perceived differently by different stakeholders.
The second dimension is intensity. Intensity can be vital if the exam significantly influences classroom practices and participants, or weak if its influence is only partial. Based on Shohamy et al. (1996), Watanabe (2004: 130) points out that intensity can vary
over time with the status of the test (low or high stakes), the status of the language being tested, the purpose of the test, the format of the test, and skills tested.
The third dimension is specificity, which can be general or specific. General specificity refers to the "type of effect that may be produced by any test" (Watanabe 2004: 20) regardless of its characteristics or content. This dimension is specific when it is related "to only one specific aspect of a test or one specific test type" (ibid.: 20).
The fourth dimension is length. It refers to how long the test impacts the stakeholders' lives. It can be short when the impact disappears as soon as the test is taken or long when such impact remains present after the test was administered.
The fifth and last dimension is intentionality. It can be intentional or unintentional, i.e., there are conducts and tests intended to cause impact and those that do not have such an objective. Watanabe's five-dimension model has been considered in this study because it rules out the possibility of dissociating washback from the environment and the social context in which the exam is held.
Due to its contextual relevance and meaning, Washback is typically associated with large-scale and high-stakes examinations. Thus, it is more likely that people are attracted to exams that have a greater expression for the social, economic or cultural impact they may cause. According to Kunnan, an exam is considered high-stakes when
decisions based on the interpretation of test results or assessments (of a single test) bring consequences for those who perform it, and when these consequences radically alter the professional path or life event." (Kunnan (2013: 1).
Nichols et al. (2012: 3) add that "those whose results [...] can produce essential decisions for students, teachers and administration.
Due to its broad scope in education, washback has been studied in various contexts in different parts of the world. Booth (2018), for instance, studied TOEIC washback in South Korea. Sato (2019) carried out an investigation in Japan focused on Japanese students' English learning behavior while preparing for tests. Maksud Ali & Obaidul Hamid (2020) studied factors that potentially contribute to the negative washback effect on English language teaching in secondary schools in Bangladesh, and Tsang & Isaacs (2022) investigated washback on students’ learning in a high-stakes school exit examination in Hong Kong.
The potential power of such exams can be favorable or unfavorable to their target groups. Having one's fate decided by an exam is a double-edged sword: while it seems frightening, it may be the best opportunity ever in someone's life. No doubt, the exam analyzed in this study, the PAS, is an example of a large-scale, high-stakes external exam, as it is designed and applied by agents outside the school (Wiebusch 2012), periodically applied to a high number of candidates (Tofolli et al. 2016), and as its results are used for decision-making (Nichols Glass & Berliner 2012).
In the foreign-language components of the PAS exam, the primary construct is reading ability, which represents, above all, a social act (Grabe & Stoller 2013) which requires a purpose and the development of macro- and micro-skills as well as strategies for comprehension (Grabe & Stoller 2013, Brown & Lee 2015). The PAS Spanish-language component is consistent with the two classical reading approaches: bottom-up and top-down. In the bottom-up approach, readers adopt a more data-driven strategy, as they
must first recognize a multiplicity of linguistic signals (letters, morphemes, syllables, words, phrases, grammar, discourse markers) and use their linguistic data-processing mechanisms to impose some sort of order on these signals. (Brown & Lee 2015: 390).
The top-down approache, on the other hand, requires a conceptual-driven strategy, as "reading is primarily directed by reader goals and expectations'' (Grabe & Stoller 2013: 25). Both approaches are essential as they are used for different purposes. The bottom-up approach seems to be the most used one to answer the PAS Spanish language items. The top-down approach should theoretically be valued at school, as it provides opportunities for collaboration and the exchange of social experiences. This leads to the reflection of whether or not schools focus more on bottom-up rather than on top-down reading, as this is required by the PAS Spanish-language component. Evidence of such a change would indicate the existence of washback.
4 Methodology
This research represents an interpretive multiple-case study (Faltis 1997, Bhattacharya 2008, Creswell & Poth 2018) whose objective is to verify any evidence of washback caused by the Spanish-language component of the PAS exam in the way public high-school Spanish language instructors in the city of Brasília teach Spanish reading skills.
4.1 Data Collection
Data collection was carried out in two methodologically acceptable phases. In the first phase, a questionnaire consisting of one multiple-choice and six Likert-scale items, as shown in Table 1 below, was sent out to 100 Spanish language teachers, using their official emails provided by the local Secretariat of Education and teachers groups' social media platforms. The questionnaire link was open for two months (May and June 2020), and twenty-nine Spanish language teachers responded. Those respondents statistically formed a random sample as all recipients, who were assigned numbers from 1 to 100, had an equal chance of being selected:
Table 1: Content of the Online Questionnaire
The second phase consisted of a semi-structured interview with three out of the twenty-nine respondents selected according to the following criteria:
holding an academic degree in Spanish or Spanish-Portuguese;
teaching in regular public schools in the Federal District; and
teaching at least one of the three high-school levels.
Each of the three interviewees met all three criteria. However, for confidentiality, the selected teachers were assigned pseudonyms: Carlos – a 30 year-old teacher with a bachelor's degree in Spanish-Portuguese and three years of experience with first, second and third-grade high-school students; Luis – a 35 year-old teacher with the same degree and seven years of experience with the same teaching levels; and Rafaela – a 38-year-old teacher with a Spanish bachelor’s degree and10 years of experience with the same levels. In addition, the disparity in their responses to the online questionnaire was considered for selecting them.
The questions shown in Table 2 below are aimed at expanding the interviewees' comments about the fifth question in Table 1 above:
Table 2: Semi-structured Interview Follow-up Questions
The original design of this study included classroom observation. However, due to the Covid-19 pandemic, the interviews were conducted online in July and August 2021, with an average length of twenty-five minutes per respondent. Thematic analysis (Creswell & Poth, 2018) categorized the transcriptions. At the end of the interviews, the interviewees were asked to provide the researchers with a reading activity they had designed and used in class to develop their students' reading skills. This request was meant to deepen our understanding of the respective responses and provide subsidies for the analysis of potential washback.
5 Results and Discussion
5.1 General Aspects
The data analysis results are presented below according to the successive sequence of the online questionnaire items, followed by the results of the interview analysis.
Our first question referred to the high-school grade the respondents taught most frequently:
Figure 3: Respondents' distribution according to high-school level most frequently taught
Figure 3 above represents the distribution of the 29 Spanish teachers according to the high-school grade they teach. Teachers working in the first high-school grade form the majority (n = 13; 45%), then come teachers working in the third grade (n=11; 38%), which is followed by teachers working in the second grade (n = 5; 17%).
Table 3 below, which reflects the dominance of the first and third-year levels, provides demographic information about the three interviewees who met the selection criteria as stated in Section 4.1 above. In addition, the table offers further details about their teacher education course, experience, and high school teaching level:
Table 3: Interviewee's Teacher Course, Experience, and High-school Grade Most Frequently Taught
The second interview question referred to teachers’ familiarity with the PAS assessment framework:
Figure 4. Respondents' knowledge of the PAS assessment framework
The PAS assessment framework is not very popular among the teachers, as 59% reported knowing a little, and 14% knew nothing about it. This result seems contradictory given the importance usually attributed to the PAS exam in schools and the local society. Further insight into this knowledge is provided by the interviewees, as shown in Table 4 below:
Table 4: Interviewees' Knowledge of the PAS Assessment Framework
Surprisingly, Rafaela, the most senior and experienced third-grade high school teacher of Spanish, reported that she only knew a little about PAS. However, both Carlos and Luís, the first-grade high school teachers, know it quite well, and this may point to their interest in finding ways to best prepare their students for the test.
The third question referred to the teachers’ frequency of adjusting their planning so as to favour alignment with the PAS framework:
Figure 5: Respondents' Frequency of Planning Adjustment for Alignment with the PAS Framework
As shown in Figure 5 above, 59% (38% + 21%) of the respondents “many times” or “always” align their lesson planning with the PAS framework. This finding seems at odds with the previous finding that 73% of the respondents indicated that they knew nothing or only a little about the exam framework ( Figure 4 above). From this result, it can possibly be concluded that such an alignment occurs unknowingly when the teachers merely reproduce the exam item formats in classroom reading activities, without realizing that the PAS framework focuses on competencies to be worked on with students. A similar contradiction is noticed in Rafaela's answer to the same question, as shown in Table 5 below:
Table 5: Interviewees' Frequency of Planning Adjustment for Alignment with the PAS Framework
As displayed in Table 5, the heterogeneity in the interviewees' responses seems to corroborate Alderson & Wall (1993)' conclusion that the same exam can impact respondents in different ways.
The question of whether in the teacher’s classroom tests, there was any reference to content or items found in the PAS Spanish language component was answered as follows:
Figure 6: Reference to the PAS Spanish Component in Respondents' Classroom Tests
As shown in Figure 6, more than half of the respondents (58%) reported they referred "a lot" or "all the time" to such content and items This result may substantiate the idea that the exam item format, the content approached, and the textual genre used are probably the main elements that stand out in the process, regardless of the actual exam framework. The interviewees' responses to the point in question were again heterogeneous, as shown in Table 6 below:
Table 6: Interviewees' Reference to Content or Items from the PAS Spanish Component
By comparing Tables 5 and 6, it seems clear that curriculum adjustment and alignment to the PAS framework does not necessarily mean referring to contents or exam items in classroom tests, as seen, for instance, in Carlos's and Luís's responses.
In the next question, the respondents were asked whether they had ever designed materials for students using or referring to PAS Spanish language component items:
Figure 7: Respondents' Materials Design according to the PAS Spanish Language Component
66% of the respondents stated having designed materials based on the PAS Spanish language component (Figure 7), even though 73% of them affirmed knowing a little or nothing about the exam framework (Figure 4). Again, it could be concluded that such a design was solely based on item formatting, topic selection or the exam genre. Rafaela is a clear case in point: although she knew only “a little” about the PAS assessment framework (Table 4), she indicated that she permanently designed her exam materials to conform with it (Table 7 below):
Table 7: Interviewees' Material Design per the PAS Spanish Component
However, Carlos and Luís represent a different case, as both stated knowing the PAS framework 'quite well' (Table 4 above). Hence, their conformity is highly anticipated, given their prior knowledge of the significance of the exam.
The sixth question was about whether the respondents thought it was essential to work with themes or topics approached in the PAS component in their classes:
Figure 8: Importance of Working with Themes or Topics approached in the PAS Component in Class
As shown in Figure 7, the respondents placed great importance on working with themes or topics approached in the PAS Spanish components. More specifically, 55% of the teachers judged this in-class practice as “important”, while 28% judged it to be “very important”, which totals 83%. The relatively high value assigned to the importance of this practice sounds impressive in the circumstance. Given some uncertainty about the respondents' understanding of themes or topics ranging from social inequality to love or verb forms, a further investigation of the subject matter is recommended. The interviewees' responses to this question echo those of the respondents, as shown in Table 8 below:
Table 8: The importance of Working with the PAS Themes or Topics from the Interviewees’ Perspective
That all the three interviewees stated that it is important or very important to work with themes brought about by the foreign language component may indicate washback in terms of content selection for their classes when teaching Spanish reading skills.
The seventh question was about whether the respondents’ and interviewers believe that the PAS Spanish language component contributes to more satisfactory teaching and learning:
Figure 9: Respondents: Contribution of the PAS Spanish Component to Teaching and Learning
Figure 9 shows that 41% of the respondents believe the PAS Spanish component contributes a little to more satisfactory teaching and learning, and 28% do not believe it does at all, equaling 69% disfavouring this contribution. This result sounds contradictory, especially when the replies to this particular point are contrasted with those provided in response to questions three (about planning adjustment and alignment), four (about referring to content and items in the classroom), five (about material design), and six (about the importance of working with exam themes and topics), as shown above in figures 5-8, respectively. In the group of interviewees, Carlos is no exception, as shown in Table 9 below:
Table 9: Interviewees: contributions of the PAS Spanish component to teaching and learning
Carlos' words "I do not believe it" are not consonant with his responses to the previous four questions. A possible explanation for this is that the exam is only seen as a gateway to the University and not a pedagogical tool to boost better teaching and learning.
Based on the previous analysis of the respondents' and interviewees' replies to the online questionnaire items adopted as an instrument in this study that only focuses on the teacher's perspective, and considering Watanabe's (2004) theoretical dimensions discussed in section 3 above, washback could be understood as explained in Table 10 below:
Table 10: Washback Analysis Based on the Responses to the Questionnaire Items
In addition to the previous analysis of the interviewees' responses to the survey questions, their replies to the semi-structured interview follow-up questions, as introduced in Table 2 above, were also analyzed. These questions aimed to provide more information about the fifth survey question: “Have you ever designed materials for your students using or referring to the Spanish PAS exam items?” Selected responses are discussed below for each case: Carlos, Luís, and Rafaela.
5.2 Case 1: Carlos
The school where Carlos works is located in a low-to-middle income area. When asked about the form and the structure of the materials developed for his classes, Carlos replied:
Carlos: I make photocopies, I use digital material (...), PDFs, videos, listening comprehension, that Nuevo Español en Marcha manual has a lot of material online, right?! They have a YouTube channel with a lot of material (...) and something that we can take advantage of from the textbook provided by the secretariat, which for me is quite poor…. but you can use something. (...)
Researcher: In these materials you said you work with infographics, the texts themselves (…). Do you think they have any relationship with the PAS Spanish language component?
Carlos: There is, because we … we have this demand from the coordination, right?! (...). So we have this profile of always giving priority to the use of the PAS. In the face-to-face model, our tests, all the evaluation methods, are designed based on PAS items. (...) I always bring items from previous exams.
(Interview with Carlos; 11-06-2021)
Carlos' expresses a “demand from the coordination” to prioritize the PAS exam, which explicitly connects with washback. This is likely attributed to the socio-economic context in which the school is situated, as the PAS is more valued just because it is the exam with the highest probability of access to public higher education. Although the ENEM, the High-school National Exam, may also be relevant, it turns out not to be attractive for many local students who would not have any possibility to move away from where they live. This impact explains Carlos's material selection and corroborates Hughes' (2003) words about the connection between impact and social factors.
Carlos also revealed that teachers have the opportunity to offer the courses they design themselves at his school. He said:
The school I work at is a pilot school for the new high school model, you see? So we started to adopt this high school's new framework last year. And then, each teacher had to have a so-called formative itinerary, right?! which was an… elective course, like an elective one (...) and then I worked with focus on the PAS.
(Interview with Carlos; 11-06-2021)
Carlos states that he worked with past exam papers in these classes and simulated the PAS exam organization. This reveals the influence of the exam on his teaching practice, especially in the teaching of Spanish reading skills. As the entire course is focused on interpreting texts, the intensity of the washback effect is vigorous (Watanabe 2005).
Concerning the content of his teaching, Carlos stated:
(…) more reading comprehension (...) for the learner to know how to read and interpret a text, he needs to have subsidies, right?! He needs to know what a verb is, he needs to know what an adjective is, a noun, so I like to work with this grammatical part first so that when he comes across the text that is precisely the PAS model, he knows how to recognize the structures there (...).
(Interview with Carlos; 11-06-2021)
Carlos' input above illustrates the existence of washback in his teaching of reading. He gives priority to the bottom-up approach in which the main focus is the morphosyntax of the language. His goal is to give learners subsidies that enable the final product- interpretation. His belief is reinforced by the PAS Spanish items that focus on structural knowledge.
The in-class activities offered by Carlos are past exam paper based. Thus, as per Watanabe's (2004) model, the impact in Carlos's case can be analyzed as follows:
Table 10: Analysis of Carlos' Case
5.3 Case 2: Luís
Luís' school is located in a town considered to be of upper-middle-income. When asked about the structure of the materials he used in his classes, he replied:
Well, the structure of these materials aims to enable the student to develop a skill... a reading-for-specific-purposes skill, right?!, considering that the PAS only demands this skill, unfortunately. So these are materials that are designed, basically, for this purpose. These are materials, texts that are in line with… with the PAS reference framework, right?! (...) that the student can read in a foreign language, right?! interpret to answer the items.
(Interview with Luís; 31-05-2021)
It is possible to identify some evidence of washback in Luís' approach to the teaching of reading, as there is an impact when an exam affects the direction of language teaching (Shohamy 1992). This is exactly what happens when Luís teaches. He uses materials that can provide learners with practice in reading for specific purposes aiming at successfully answering the exam items.
Unlike Carlos, the socio-economic context in which Luís teaches does not seem to affect his work. However, the learners in the school in which Luís works have the PAS exam as an option for access to higher education, and thus, his job is to provide them with relevant knowledge and information should they decide to take it.
When asked about the goals he set when preparing these materials, Luis stated:
The specific objective is to understand the text style, usually an informative text, a science communication text, so there are some elements that are typical of the nature of this type of text... some linguistic elements belong to the nature of this textual type (...) for example, knowing that 'pero' is an adversative word, that 'sin' does not mean 'yes', but 'without' (...).
(Interview with Luís; 31-05-2021)
The ability to exchange conjunctions without semantic change and find meaning mirrors some of the items found in the PAS Spanish language component, which require this type of knowledge. Hence, the practice of teaching to read is limited to what the test tests, which is another sign of the existence of impact and is aligned with the bottom-up reading approach. The materials made available by Luís also rely heavily on the PAS past exam papers. In light of Watanabe (2004), Luis' input can be analyzed as shown in Table 11 below:
Table 11: Analysis of Luís Case
When comparing Tables 10 and 11, we can conclude that Carlos and Luis' cases coincide in terms of specificity, intensity, length and intentionality but differ in value due to their respective beliefs about washback on teaching and learning.
5.4 Case 3: Rafaela
Rafaela teaches in a high-income area in the city of Brasília. When asked about the materials she designs, she replied:
the PAS exam is based on the ability to read and interpret, and eventually, they add grammar, it's not much, but they do. The demand at school is not even the PAS anymore; it is ENEM. In the last three years, it has completely changed its focus. I work in the sense of focusing on processes, on reading and interpreting inference, deduction, not interpretation. I take their own tests (PAS), I take the exam items, and I work with them… I do this a lot!
(Interview with Rafaela, 02-06-2021)
As stated above, the ENEM is a national large-scale high-stakes exam that enables access to public and private universities all over Brazil. Although the PAS exam offers an excellent opportunity for admission at the University of Brasília, where competition is more restricted to the city of Brasília and surrounding areas, it is not a matter of focus for Rafaela's students. This less concern is probably attributed to the socio-economic context of the area the school is located in, where families are typically affluent and usually prefer the ENEM as it also enables access to universities in other cities or states in the country. This seems to corroborate Hughes' (2003) observation of a strong correlation between washback and the social context.
Rafaela uses PAS past exam papers to teach reading skills in her classroom, suggesting an inclination to the top-down approach, emphasizing text interpretation. Therefore, there is some evidence of washback in her teaching practice as she includes reading skills in her teaching. Thus, considering Watanabe's (2004) model, Rafaela's case can be summarized as follows:
Table 12: Analysis of Rafaela's Case
The analyses of the three cases seem to confirm a close relationship between washback on the one hand and the socio-economic and cultural context in which the exam is inserted, on the other (Hughes 2003). In addition, washback on teachers' practice is intertwined with their respective conceptions and culture. Finally, regarding the use of materials for reading skill development, there was consensus among the interviewees about using past exam papers as a primary tool to work with reading in their classes.
Overall, this analysis corroborates Hughes' (2003) finding of a close relationship between washback and the socio-economic and cultural context in which the exam is administered. The analysis also reveals that washback on teachers' practice is intertwined with their respective conceptions and cultures. Finally, the interviewees had a consensus on the relevance of past exam papers to developing students' reading skills.
5 Conclusion
The PAS is a three-phased local exam that serves as a legitimate entrance option at the University of Brasília. As an external large-scale high-stakes examination, it consists of 100 items, and the Spanish language component has ten items that focus on reading. However, despite the consensus on its significance in the socio-economic context in which it is administered, data collected by means of an online questionnaire as completed by twenty-nine Spanish-language public-school teachers and three interviews indicate an impoverished contribution of the Spanish language component to the development of learners' reading skills in the respective language.
Evidence of washback was identified as being restricted to the use of past exam papers, most probably to familiarize students with item format, topics, themes, and genres integrated into the Spanish language component. Bottom-up approaches to reading are apparently predominant, and no evidence of work with the competencies and skills from the exam framework was identified.
Though not conducted on a large scale, the present study raises some essential pedagogical issues that merit further consideration by both exam developers and users. As far as exam developers are concerned, there should be more variety in the Spanish-language-component items in terms of genres and text selection to stimulate different reading skills and approaches, mainly the top-down approach which is often overlooked by teachers in favour of the bottom-up approach.
On top of that, more work on publicizing and discussing the exam framework would help increase teachers' familiarity with the exam, which is expected to positively impact the teaching of Spanish reading skills. A booklet in which the respective items are presented and explained would be a means to enable teachers and students to understand the principles underlying each item as well as the whole foreign language component. As for PAS Spanish-language-component users, who mainly are teachers, more consistent knowledge of the framework would be beneficial and focus on teaching reading in Spanish rather than teaching to the exam.
Finally, it needs to be noted that COVID-19 has imposed some limitations on the study, including the inability to conduct class observations and collect more comprehensive and representative data, which would otherwise not have been created..
References
Alderson, Charles & Dianne Wall (1993) Does Washback Exist? In: Applied Linguistics 14/2, 115-129 (https://doi.org/10.1093/applin/14.2.115.; 15-06-2022)
Allwood, Jens, Olga Pombo, Clara Renna & Giovanni Scarafile (eds.) (2020): Controversies and Interdisciplinarity: beyond Disciplinary Fragmentation for a New Knowledge Model. Amsterdam: Benjamins.
Bailey, Kathleen (1996): Working for Washback: A Review of the Washback Concept in Language Testing. In: Language Testing 13/3, 257-279 (https://doi.org/10.1177/026553229601300303; 15-06-2022)
Bhattacharya, Himika (2008): Interpretive research. In: Given, Lisa M. (Ed.). The SAGE Encyclopedia of Qualitative Research Methods. Los Angeles et al.: Sage Publishing, 465-468
Booth, Dawn Karen (2018): The Sociocultural Activity of High Stakes Standardized Language Testing: TOEIC Washback in a South Korean Context. Cham: Springer International.
Brown, H. Douglas & V. Lee (2015): Teaching by Principles: an Interactive Approach to Language Pedagogy. 4th edition. London: Pearson.
Burrows, Catherine (2004): Washback in Classroom-Based Assessment: a Study of the Washback Effect in the Australian Adult Migrant English Program. In: Cheng, L., Y. Watanabe & A. Curtis (Eds.): Washback in Language Testing: Research Contexts and Methods. Mahwah, New Jersey & London: Lawrence Erlbaum Associates, 113-128
Creswell, John & Cheryle Poth (2018): Qualitative Inquiry & Research Design: Choosing among Five Approaches. 4th edition. Los Angeles et al.: Sage Publishing.
Faltis, Christian (1997): Case Study Methods in Researching Language and Education. In: Hornberger, N. H. & D. Corson (Eds). Encyclopedia of Language and Education. Volume 8: Research Methods in Language and Education. Dordrecht, Boston & London: Kluwer Academic Publishers, 145-152
Grabe, William, & Fredricka Stoller (2011): Teaching and Researching Reading. Harlow (UK): Longman/Pearson.
Graff, Harvey (2015): Undisciplining Knowledge: Interdisciplinarity in the Twentieth Century. Baltimore (MD): John Hopkins University Press.
Hughes, Arthur (2003): Testing for Language Teachers. 2nd edition. Cambridge (UK): Cambridge University Press.
Kunnan, Antony (2013): High-stakes Language Testing. In: Chapelle, C. A. (Ed.): The Encyclopedia of Applied Linguistics. Hoboken (NJ): Blackwell, 1-6.
Maksud, Ali & Obaidul Hamid (2020): Teaching English to the Test: Why Does Negative Washback Exist within Secondary Education in Bangladesh? In: Language Assessment Quarterly 17/2, 129-146 (https://doi.org/10.1080/15434303.2020.1717495; 15-06-2022).
Nichols, Sharon, Gene Glass, & David Berliner (2012): High-stakes Testing and Student Achievement: Updated Analyses with NAEP Data. In: Education Policy Analysis Archives 20/20 (https://doi.org/10.14507/epaa.v20n20.2012; 15-06-2022).
Pessoa, Aline (2004): Efeito retroativo de um exame de seleção no ensino de inglês. In: Trab. linguist. apl. 43/2, 287-305 (https://doi.org/10.1590/S0103-18132004000200006; 15-06-2022).
Quevedo-Camargo, Gladys & Ricardi Scaramucci (2016): Methodology in Washback Studies. In: Gitsaki, C. & C. Coombe (Eds.): Current Issues in Language Evaluation, Assessment and Testing. Newcastle upon Tyne (UK): Cambridge Scholars Publishing, 187-304
Sato, Takanori (2019): An Investigation of Factors Involved in Japanese Students' English Learning Behavior during Test Preparation. In: Papers in Language Testing and Assessment 8/1. 69-95 (https://arts.unimelb.edu.au/__data/assets/pdf_file/0003/3060417/8_1_S4_Sato.pdf; 15-06-2022).
Shohamy, Elena (1992): Beyond Proficiency Testing: a Diagnostic Feedback Testing Model for Assessing Foreign Language Learning. In: Modern Language Journal 76/4, 513–521 (https://doi.org/10.2307/330053; 15-06-2022).
Shohamy, Eleana, Samadar Donitsa-Schmidt & Irit Ferman (1996): Test Impact Revisited: Washback Effect Over Time. In: Language Testing 13/3, 298-317. (https://doi.org/10.1177/026553229601300305; 15-06-2022).
Tsagari, Dina & Liying Cheng (2017): Washback, Impact, and Consequences Revisited. In: Shohamy, E., I. Or, S. May (eds). Language Testing and Assessment. Encyclopedia of Language and Education. 3rd ed. Cham: Springer, 359-372. https://doi.org/10.1007/978-3-319-02261-1_24; 15-06-2022).
Tsang, Chi Lai & Talia Isaacs (2022): Hong Kong Secondary Students' Perspectives on Selecting Test Difficulty Level and Learner Washback: Effects of a Graded Approach to Assessment. In: Language Testing 39/2, 212-238 (https://doi.org/10.1177/02655322211050600; 15-06-2022).
Watanabe, Yoshinori (2004): Methodology in Washback Studies. In: Cheng, L., Y. Watanabe & A. Curtis (Eds.): Washback in Language Testing: Research Contexts and Methods. Mahwah, New Jersey & London: Lawrence Erlbaum Associates, 19-36
Wiebusch, Elosia (2012). Avaliação em larga escala: uma possibilidade para a melhoria da aprendizagem. Anped Sul, IX, 1-13. (https://1library.org/article/avalia%C3%A7%C3%B5es-em-larga-escala-como-orienta%C3%A7%C3%A3o-de-pol%C3%ADticas.z3odx07z; 03-06-2022).
Authors:
Yasser Abdullah Al Tamimi
Associate Professor of Linguistics
Alfaisal University
Saudi Arabia
Email: yaltamimi@alfaisal.edu
Gladys Quevedo-Camargo
Associate Professor of Applied Linguistics
University of Brasília
Brazil
Email: gladys@unb.br
Rafael Sena Raposo de Melo
Public school Spanish teacher
Secretariat of Education
Brazil
Email: raposorafaelsena@gmail.com