Quantifying impact on MSc International Business

Co-authored by Andreas Georgiadis of Leeds University Business School (LUBS)

Introduction: The English for Academic Purposes (EAP) in- sessional within Leeds University Business School (LUBS) was a newly developed component of the MSc International Business course which ran throughout the 22/23 academic year. As arrangements for the 22/23 academic year were being organised, this project was coordinated to measure whether the implementation of EAP in-sessional classes would have a positive impact on student experience, outcomes, or attainment and explore if and how the impact of the EAP sessions could be measured from a quantitative perspective.

Measuring impact of in-sessional EAP provision has predominantly been facilitated through qualitative data gathering processes, however the quantitative data analysis approach relevant to this study is representative of an innovative approach to evaluating impact within EAP in-sessional contexts. The accompanying poster presentation provides contextual information relevant to the profile of the students on the MSc International Business course and reports on a large-scale comparative study (n131) which aimed to measure improvements in the writing and academic language of students who either did or did not attend the EAP in-sessional provision. It also details the methodology, quantitative data analysis of student writing samples and results.

Methods: Relevant data for this study was collected through collaboration with departmental colleagues, formative assessment practices, and the gathering of student writing samples at three key points across the academic year, complimented by a short questionnaire at the conclusion of each semester. The collected student writing samples were analysed using an adapted Language Centre assessment criteria for content, grammar, and use of academic vocabulary. The structure, cohesiveness, and coherence of the samples were also evaluated. These samples include student’s personal statements collected pre-semester 1, a case report timed examination at the conclusion of semester 1 and a research proposal at the conclusion of semester 2.

The personal statements were analysed by tutors and the EAP specialist pre-course using the adapted LC criteria to provide a baseline and identify any students who may benefit from attending the EAP sessions (although sessions would be available to all students). This led to identification of 51 students as a target group, of which 42 regularly attended the EAP sessions. In addition, 43 students from the non-identified group attended the sessions, meaning that 85 attended in total. In turn, 9 of the identified students chose not to attend alongside the remaining 37 students (46 in total). This process allowed for the identification of comparative groups whose formative grades could be tracked and analysed to demonstrate whether or not student attendance of the EAP classes impacted grades across the three writing tasks during the academic year.

Analysis: The quantitative data analysis process included the use of a statistical impact evaluation, used systematically in educational economics to monitor added value, also known as the change in performance (grades), from before to the end of the programme and to identify any change over time. This would provide insight into the impact of the intervention, on this occasion comparing average outcomes of different groups of students attending the sessions with those not attending the sessions. One analysis compared changes in the average score of content, grammar, and vocabulary between before the baseline (pre-support stage) and final assessment of attendants and non-attendants. Measuring changes over time allows to adjust for systematic differences in performance between the two groups that may reflect other factors than EAP support, such as educational background, including country of origin and disciplinary background, and demographic characteristics, such as gender.  Moreover, the two groups were quite similar in terms of their assessment scores in the baseline period, which suggests that those not attending form a valid comparison group relative to which the performance of those attending the sessions can be gauged. A second analysis examined used data from all periods, not only the initial and final assessment, by comparing differences in changes in scores between baseline and end of semester 1 and between the end of semester 1 and semester 2.

Results: We discover that EAP attendance raises scores generally and in terms of content, structure, and GVS by an average of 7–14 percentage points.

According to Table 1's results, individuals who participated in the EAP sessions saw an increase in their assessment scores of approximately 7 percentage points between the start and end of the academic year in all categories, including overall, content, structure, and GVS scores. In addition, those who did not participate in the sessions demonstrated a significant decline in their assessment scores, which was approximately 4 points in the overall and structure scores, and 5 and 2 points in content and GVS scores, respectively.

According to Table 2 results, students who attended the EAP sessions improved on assessments in every category, including overall, structure, content, and GVS, by around 7 points per semester, or 14 points more than they would have during the entire academic year. Conversely, students who did not attend the sessions saw a decline in scores in every category, by around 3 points per semester.

Discussion: The findings of this impact study demonstrate that attending EAP in-sessional provision can support MA level students develop their understanding of academic language, style, and conventions within a specific disciplinary context, as well as exemplifying high-quality, inter-departmental collaborative practice. The findings of this study adds to the outcomes of similar studies conducted from an EAP perspective (Benson and Andersen, 2019; Pearce, Sloan and Stamps, 2021).

Poster presentation

Please feel free to contact Clark or Andreas (a.georgiadis1@leeds.ac.uk) if you have any questions.