From the Design of Assessments to Language Assessment Literacy

Theoretical Framework

Characteristics of Useful Language Assessments

An assessment is an instrument in which students can show whether they have learned or progressed in their language skills. Examples of assessment instruments include traditional ones, such as tests with multiple-choice, true-false, and matching items; and rubrics for assessing speaking or writing performance. Alternative instruments include self-assessment or peer-assessment checklists and protocols for managing portfolios. Whatever the type, the educational purpose of language assessment is to collect clear information about either skills, contents, or learning objectives in a particular language curriculum (Brown, 2011; Green, 2014). Then, the information collected through these means is used to document or further foster language learning and teaching (Green, 2014; Bachman & Damböck, 2018) within a particular language education context.

To further contribute to an assessment's usefulness, teachers designing or using assessment instruments should consider their theoretical qualities. Table 1 is a synthesis of five central qualities of assessments (Bachman & Palmer, 2010; Giraldo, 2019; Green, 2014):

Table 1: Qualities of Language Assessments

Source: Author's own elaboration (2021).

All the qualities above are relative; hence, the use of the phrase the degree to which: an assessment is not totally authentic or inauthentic, but it largely depends on the purpose and context where it is used, as it is commented above. One final feature of a useful assessment is that it is professionally designed -following technical design guidelines - to achieve its purpose. Below we review core guidelines for creating language assessments.

Technical Guidelines for Designing Language Assessments

For quality control, assessment designers should plan their products by writing a document with specifications (Davidson & Lynch, 2001; Fulcher, 2010). This document includes the purpose, the specific skills, the assessment method, number of items or tasks and their nature, as well as any other information that can help to ensure the assessment is planned and designed to meet its purpose. Among these specifications, one needs to be underscored: the specific skills to be assessed need to be clearly delineated; this is commonly called construct definition (Carr, 2011; Fulcher, 2010). Lack of clarity in this regard may lead to an assessment that is not useful. In Table 2, we synthesize major specific guidelines for listening/reading and speaking/writing assessments.

Table 2: General Guidelines for Designing instruments to Assess Language Skills

Source: Author's own elaboration (2021).

As stated, the previous table synthesizes fundamental guidelines for test construction; nevertheless, for design specifics, other specialized works can be consulted: Buck (2001) for listening; Alderson (2010) for reading; Luoma (2004) for speaking; and Cushing (2010) for writing. For test construction in general, useful resources are Alderson et al. (1995) and Carr (2011).

Related Research

As we mention earlier, limited research has been done regarding analyses of language assessment instruments. The first three studies below aimed to describe language teachers' assessment instruments used in their assessment practices. The second set of studies described the assessment instruments teachers designed as they were engaged in language assessment courses.

Frodden et al. (2004) studied the assessment instruments used by foreign language teachers (English and French) working in two Colombian universities. In their findings, the authors report that teachers placed emphasis on assessing vocabulary and grammar, but not so much on authentic language use. Additionally, the instruments tended to have problems with construct validity, i.e., it was not clear what they really aimed to assess as, for example, no scoring procedures were stated. Finally, as the authors report, traditional instruments were used more so than were alternative assessments.

Giraldo (2018b) conducted a study delving into English teachers' beliefs and practices involving the design and use of final achievement tests. The instrument analyses indicated that teachers in this study assessed linguistic and pragmatic aspects of language, with minor attention to sociolinguistic issues. Additionally, while tests tended to assess language in context, there were problematic areas in rubric design that led to reliability issues, i.e., lack of clarity regarding how to score language skills.

In a recent study, Villa-Larenas and Brunfaut (2022) critically examined the LAL of twenty language teacher educators in Chile. As part of their analysis, the researchers studied a set of assessment instruments used by these stakeholders. The authors indicate that the teacher educators used a variety of assessment techniques, most notably "fill-in-the-gaps, [...] constructed response, matching, sequencing, and sentence completion" (p. 11).

The existing literature indicates the positive impact of language assessment training on teachers' design of assessment instruments. Overall, authors report that teachers design authentic language use tasks connected to their classrooms (Koh et al., 2018; Montee et al., 2013); clearly operationalize the language skills to be assessed (Arias et al., 2012; Koh et al, 2018). Additionally, these authors report that the programs were based on contextual needs analysis of language teachers' LAL, which included the study of assessment instruments. Thus, the research has suggested that analyzing assessment instruments may be used as one source of feedback for planning and implementing successful LAL training for teachers.

Why does the Design of Assessments Matter?

From the language performance that students show in an assessment, teachers are supposed to determine whether students have developed language skills; also, the instrument is designed to meet a purpose. Thus, properly designed assessments aid in doing the aforementioned tasks; poorly designed assessments may give erroneous results about students' language skills and, therefore, not be fit for a given purpose.

Another reason why design in assessment is crucial is the LAL needs teachers have reported in this regard. Although teachers design or resort to already designed instruments, studies have shown that they want and expect training in how to professionally construct instruments (Fulcher, 2012; Hasselgreen et al., 2004; Vogt & Tsagari, 2014). In response to this need, LAL programs have indicated that a design-based course impacts teachers' LAL: primarily knowledge and skills, with secondary attention to principles (Giraldo, 2021).

Against this background, in this paper we report an analysis of language assessment instruments in order to elucidate and interpret the skills dimension (particularly instrument design) of a group of LAL teachers. This analysis helped us to generate implications and recommendations for planning the language assessment course for the teachers in our study. We provide details from our research in the methodology and findings sections below.

Participants

Thirty English language teachers consented to share two assessment instruments, which formed the corpus in our study. These teachers agreed to participate in an online language assessment course. To design the course, a major source to draw these participants' LAL were the instruments we studied. However, we also asked them about their LAL through a questionnaire and an individual interview.

In terms of the participants, fifteen teachers had an MA degree, fourteen a BA, and one a specialization, as their highest educational level. Their teaching experience is presented in Table 3, while some of their assessment practices can be seen in Table 4, both with rounded percentages.

Table 3: Experience Teaching English

Source: Author's own elaboration (2021).

Table 4: Selected Teachers' Assessment Practices

Source: Author's own elaboration (2021).

Data Collection and Analysis

To collect the data (i.e., the assessment instruments), we contacted a group of 30 English language teachers from state high schools in the coffee region of Colombia. They were invited to participate in a professional development course in language assessment. To accomplish this, they agreed to share two assessment instruments with our research group: one assessment for receptive skills and one for productive skills. The teachers shared their instruments by email; then, we deleted all personal and institutional information written in the documents. Finally, we stored the instruments in a digital Google Drive folder, only accessible to us as researchers.

To organize and analyze the content in the instruments, we used two Google Forms (one for listening/reading assessments and one for speaking/ writing assessments) which included a description section and an analysis section. For the former, we described the nature of the instruments and their items or tasks: methods, instructions, texts used (for listening and reading), scoring criteria (for speaking and writing), number of tasks or sections, and point allocation.

Regarding the latter, we analyzed qualities of assessments such as validity, authenticity, and reliability. Each one of us scrutinized between thirteen and sixteen instruments. Then, the principal investigator reviewed all the analyses and noted down areas that needed clarification. Comments that needed elaboration were discussed and resolved in the research team. After our analyses, we derived interpretations regarding what was done well and what needed to improve in design.

This content analysis, as commented, was done conceptually and ecologically, i.e., based on emerging trends in the data. We finalized data analysis by grouping content codings, which led us to the major findings we present and discuss below. For example, a major trend in the assessments for listening/reading skills was that many items could be answered without actually listening or reading; speaking and writing assessments were mostly grammar based. These two codes were grouped under one major trend: construct-related issues. As for instruments seeking to assess speaking or writing, we noticed that they did not include clear scoring criteria, or they included construct-irrelevant factors. We labelled these open codings under another major trend: reliability issues.

Technical and Theoretical Issues in Construct Representation in the Instruments

In this section, we provide a closer, critical analysis of the nature of the 51 instruments we studied. We start by presenting recurring problems that we identified in the design of items for assessing listening and reading skills. Then, we explain the issues we elucidated in the assessments of speaking and writing skills.

Listening and Reading Assessments

Besides, other technical problems we identified in the assessments for listening and reading skills (in addition to questions that can be answered based on prior knowledge) were items that could not be answered and items whose answer was not totally clear. Item 4 below, meant to assess reading, does not have an answer in the text. Item 13 has options that overlap.

Ins. 16, ReadingT-F4

True (T) or False (F)

Item 4: The students write the English answers on the board. T_F_

The corresponding segment from the source text reads as follows:

For developing the listening skills, the teacher asks us to listen audios to answer questions like this "How old are you? We pay attention to the answers and write them on a piece of paper.

Thus, there is no evidence in the reading to state whether Item 4 is true or false.

Ins. 25, ListeningMCQ13

Item 13: Bludworth is_when Pepys finds him.

A. Angry and dirty

B. Angry and hot

C. Hot and dirty

D. Tired and dirty

Supposedly, the answer was C, but options A and D also have the word dirty; option B has the word hot, so there is overlap among the options: there is no one single, undoubtedly correct answer.

The technical difficulties in the items above, lead to problems with two qualities of language assessment. Validity can be negatively affected because, if a student answers an item without listening or reading, then the resulting mark or score cannot be interpreted as a demonstration of these skills. In addition, reliability is reduced in items that do not have one clearly correct answer: some students may fail because they choose an answer which is partially true (or false), but that is keyed as the opposite -the performance may be providing reliable information about the student's skill but the scoring itself is not. Also, this problem may exacerbate if students, theoretically, chose a different answer if they took the same test again. In synthesis, all these design glitches in the listening or reading assessments may not allow teachers to make accurate inferences about constructs. Based on their design, the assessments did not accurately and wholly address the constructs of listening or reading.

Speaking and Writing Assessments

We found two major issues with the instruments to assess these skills. On the one hand, the rubrics that the teachers used did not fully state what about these skills was to be assessed; a related problem was the lack of rubrics to assess these skills. According to the rubric in Table 5 below, the teacher was meant to assess knowledge of the content, language grammar and vocabulary, and voice. However, these aspects are not clearly defined, which is a key condition for validity in assessment: it is not clear what specific aspects of grammar or vocabulary are assessed.

Table 6: Sample Rubric for a Speaking Assessment 3

Source: Author's own elaboration (2021).

The next instrument is meant to assess writing skills. In this, the only information we could retrieve were the instructions as there was no rubric to assess performance.

Ins. 53, Writing

Make an infographic to adolescents inviting them to carry a healthy life.

Use images, tips, an interesting title or question.

It can be in pairs or individual.

On the other hand, the other issue with the speaking and writing assessments in the corpus was the lack of authenticity. Since there was a tendency to address grammar in the instruments, the teachers proposed tasks that bore a limited resemblance to real-life situations and use of the English language. For example, the writing assessment below does not target writing as it is done in real-life situations.

• Ins. 2510, Writing

• 1. You /finish/your lunch/yet/?

• ______________________________________

• 2. Your teacher/plan/this class activities/already

• ______________________________________

• 3. Claudia/tell/me /a secret/just/

• ______________________________________

The writing assessment above presents authenticity issues because in real life situations we do not order preset sentences; instead, we write the sentences from scratch according to our goal (e.g., writing an email, an essay, etc.); in other words, a context is needed for the writing task to be more authentic.

The findings in our study reiterate those in other studies (Frodden et al., 2004; Koh et al., 2018; Villa-Larenas & Brunfaut, 2022): Teachers with limited training in language assessment design instruments that present issues in reliability, validity, and authenticity. This trend may be problematic in practice because language instruments with design and theoretical problems -like the ones we identified in the present study- may not be useful to document or contribute to language learning.

Based on the areas for improvement, we suggested that a language assessment program for these teachers should focus on two major areas. First of all, the teachers may benefit from a design-based course in which they can carefully evaluate and create instruments based on rigorous design guidelines (see Table 1 for examples). This pedagogical implication for teaching language assessment to in-service teachers is supported by studies that have shown how teachers foster their LAL through assessment design (see Arias et al., 2012 and Kremmel et al., 2018 for examples). To complement this technical aspect, the teachers can study qualities of language assessments (e.g., validity, reliability, authenticity) and how they can be used to improve assessments. These two dimensions -the technical and the theoretical- can be used as arguments to explain to teachers how properly designed instruments are useful to achieve educational purposes.