QI Linyi, DI Aiying, LIU Xingang, ZHU Jiangqin, LUO Maoju, ZHOU Nan

University of Electronic Science and Technology of China

　　Abstract：

　　Since the issue of China's Standards of English Language Ability(CSE), considerable research has been conducted, yet scant attention has been paid to its application in speaking assessments. This study addresses this gap by developing a CSE-based speaking test and its corresponding marking scheme to assess the entry speaking levels of non-English majors at a first-class university. The empirical evidence for its validity is investigated using Weir's social-cognitive validity framework, exploring CSE’s role in guiding the development of speaking tests and thus offering an empirical case study in this regard.

　　The study unfolded in two stages: the development of the speaking test and its marking scheme followed by a validation process. In stageⅠ, the test was developed based on the can-do descriptors of levels 3-6 of CSE subscales for oral expression. The constructs were defined as description, narration, argumentation, exposition and interaction, covering topics ranging from personal experience, interests, daily life scenarios, study, life choices to hot social events. The test form was designed as an Oral Proficiency Interview, including conversations with one interlocutor, individual mini-presentation and paired discussion. An analytic marking scheme was developed by drawing on criteria features of spoken language from CSE-level descriptors and adding in more discriminative adjectives or adverbs and descriptions of negative language performances. The scheme covers communication and interaction, fluency and coherence, language, and pronunciation. Subsequently, 112 freshmen out of 530 were selected using stratified sampling to sit the final test, which was video recorded and double-marked after rigorous training on standardization and pilot marking.

　　In stage Ⅱ, the study applied Weir’s social-cognitive validity framework to assess the contextual, theoretic, scoring and predictive(criterion-related) validity of the test. Contextual and theoretic validity were explored through a post-test questionnaire, to elicit test takers’ reflections on contextual factors, understanding of the test, and the necessary cognitive process for completing the test. Scoring validity was investigated by examining rater reliability, test reliability, difficulty and discrimination. Finally, predictive validity was evaluated by correlating Year 1 students’ mid-term speaking test scores and teachers’ ratings based on classroom observation with students’ CSE levels on our speaking test, thereby probing into the extent to which the latter could predict the former.

　　The results show that this CSE-based speaking test has relatively high contextual and theoretic validity, with most of the test takers believing the test design, task requirements, test context and topics were effective in eliciting their language abilities and strategies specified in test constructs. Moreover, the test demonstrates high reliability, suitable difficulty levels and good discrimination, and the marking scheme was generally effective in informing raters’ marking. The majority of the test takers scored level 4 or 5, consistent with the CSE's prediction of speaking proficiency levels of first-year university students. The test takers’ CSE levels on our speaking test significantly correlated with their later speaking performance on their English course, underscoring the test’s predictive validity.

　　Overall, the CSE’s can-do statements for oral expression effectively informed the design of a speaking test, facilitating the evaluation of test takers’ spoken communication levels and the prediction of their performance in later speaking tests of English courses.

　　Key words：

　　China’s Standards of English Language Ability(CSE); speaking test; marking scheme; Weir’s social-cognitive validity framework

　　Source:

　　Journal of China Examinations

　　No. 5, 2024