You are here

Assessment Overview

Overview: 

UAWP Assessment Executive Summary

In our assessment report for AY 15-16, the UAWP assessed its pilot curriculum for ENGL 101/107 and determined that we needed to emphasis rhetorical purpose more. Drs. Aimee Mapes and Rochelle Rodrigo revised the curriculum during Summer 2016 for use with our new Graduate Teaching Assistants (GTAs) during AY 16-17. The revision of the curriculum also included more systematic reflection throughout the semester on students’ achievement of the SLOs to prepare them for a final reflection at the end of the course. This metacognitive reflection serves as the foundation for a more sustainable portfolio-based assessment of UAWP courses that we are piloting in AY 17-18.

 

In summary, the University of Arizona Writing Program (UAWP) reported in our AY 15-16 Assessment report (1) that we needed to focus on:

  • emphasizing rhetorical purpose more in ENGL 101/101A/107

  • conducting a self-study and APR of the UAWP in October 2016 (through the Council of Writing Program Administrators Consultant-Evaluator Service)

  • designing a new program assessment that will measure the achievement of SLOs in the full composition sequence

​We also implemented a new course in Fall 2016, ENGL 106, designed to meet the increasing language needs of international students.

During the AY 16-17 academic year, we

  • implemented the revised curriculum for ENGL 101/101A/107
  • assessed students' perceptions of learning in ENGL 101/101A/107 (results reported below)
  • assessed students' and instructors' perceptions of the effectiveness of the new ENGL 106 curriculum (results reported below)
  • conducted a self-study and APR of the UAWP in October 2016
  • conducted a program-wide dynamic criteria mapping (Broad, 2009) exercise to develop a new program assessment plan

(1) Prior UAWP Assessment Reports from 14-15 and 15-16 are included in file attachments section.

 

Expected Learning Outcomes: 

Our assessment report for AY 16-17 includes assessments of both our ENGL 101/101A/107 courses and ENGL 106. The courses have separate, but related student learning outcomes (SLOs):

 

Student Learning Outcomes for ENGL 101/101A/107 (also ENGL 102/108)

At the end of the First-Year Composition (FYC) sequence, students will be able to

 

Goal 1: Rhetorical Awareness. Learn strategies for analyzing texts’ audiences, purposes, and contexts as a means of developing facility in reading and writing.

1.1. identify the purposes of, intended audiences for, and arguments in a text, as situated within particular cultural, economic, and political contexts.

1.2. analyze the ways a text’s purposes, audiences, and contexts influence rhetorical options.

1.3. analyze how genres shape reading and composing practices.

1.4. read in ways that contribute to their rhetorical knowledge as writers.

1.5. develop facility in responding to a variety of writing contexts calling for purposeful shifts in structure, medium, design, level of formality, tone, and/or voice.

Goal 2: Critical Thinking and Composing. Use reading and writing for purposes of critical thinking, research, problem solving, action, and participation in conversations within and across different communities.

2.1. employ a variety of research methods, including primary and/or secondary research, for purposes of inquiry.

2.2. make informed judgments about the quality, appropriateness, and credibility of sources.

2.3. demonstrate facility in incorporating evidence, such as through summaries, paraphrases, quotations, and visuals.

2.4. synthesize research findings in development of an argument

2.5. support ideas or positions with compelling discussion of evidence from multiple sources.

2.6. compose persuasive researched arguments for various audiences and purposes, and in multiple modalities.

Goal 3: Reflection and Revision. Understand composing processes as flexible and collaborative, drawing upon multiple strategies and informed by reflection.

3.1. adapt composing and revision processes for a variety of technologies and modalities.

3.2. produce multiple revisions on global and local levels.

3.3. suggest useful global and local revisions to other writers.

3.4. identify the collaborative and social aspects of writing processes.

3.5. evaluate and act on peer and instructor feedback to revise their texts.

3.6. reflect on their progress as academic writers.

Goal 4: Conventions. Understand conventions as related to purpose, audience, and genre, including such areas as mechanics, usage, citation practices, as well as structure, style, graphics, and design.

4.1. demonstrate knowledge of linguistic structures, including grammar, punctuation, and spelling, through practice in composing and revising.

4.2. reflect on why genre conventions for structure, paragraphing, tone, and mechanics vary.

4.3. identify and effectively use variations in genre conventions, including formats and/or design features.

4.4. demonstrate familiarity with the concepts of intellectual property (such as fair use and copyright) that motivate documentation conventions.

4.5. apply citation conventions systematically in their own work.

 

Student Learning Outcomes for ENGL 106

Goal 1: Genre Awareness. Learn strategies for analyzing texts’ audiences, purposes, and forms as a means of developing awareness of academic genres.
Student Learning Outcomes: At the end of English 106, students will be able to

  • Recognize and explain shifts in register (e.g., academic, conversational) in terms of choices in vocabulary and grammar.
  • Use examples of a genre to identify conventional choices for content and its organization.

Goal 2: Academic Literacy Practices. Learn and use strategies for developing academic literacy practices typical to US universities.
Student Learning Outcomes: At the end of English 106, students will be able to

  • Identify key components of writing assignment prompts, including the purpose, genre, and formatting requirements.
  • Write a text in a familiar genre that displays appropriate development and coherence.
  • Develop an idea using explanations, examples, and/or details.
  • Summarize a familiar text using some conventional summary language (e.g., third person, reporting verbs).
  • Use multiple strategies for reading academic texts (e.g., skimming, scanning, identifying organizational markers like headings, previewing, and using visual cues).
  • Distinguish main ideas and supporting details in lengthy or difficult readings.
  • Understand concepts of intellectual copyright and plagiarism as defined in US academic environments and at the University of Arizona.

Goal 3: Reflection and Revision. Understand composing processes as flexible and collaborative, drawing upon multiple strategies and informed by reflection.
Student Learning Outcomes: At the end of English 106, students will be able to

  • Participate in collaborative writing practices, including peer review, as a means of improving their own writing.
  • Revise their writing on global and local levels.
  • Make use of peer and instructor feedback when revising their texts.
  • Identify common patterns of error in their own writing.
  • Use a range of self-editing strategies.
  • Reflect on their progress as academic writers.

Goal 4: Academic Language and Conventions. Develop written academic language and conventions in English.
Student Learning Outcomes: At the end of English 106, students will be able to

  • Identify and use common academic phrases for functions such as organizing ideas, expanding on ideas, providing examples, expressing personal views, attributing ideas, and making claims or sharing a thesis.
  • Apply conventional formatting features to their writing, such as use of capitalization, titles, sentence boundaries, paragraphs, and document layout.
  • Produce simple written text that conveys the intended meaning to readers.
  • Incorporate some syntactic and lexical variety into their writing.
Assessment Activities: 

The UAWP conducts ongoing assessment. We rotate which student learning outcomes (SLOs) we assess each year. We conducted direct assessment of student learning in ENGL 101/107 during AY 15-16 related to Goal 3: Reflection and Revision. This year we conducted indirect assessment in ENGL 101, 101A, 106, and 107 of students' (and instructors', in ENGL 106) perceptions of learning for all outcomes.

The AY 16-17 report provides an overview of two year-long assessment projects. We assessed students' perceptions of learning in ENGL 101/101A/107 (specifically responding to the revision of the curriculum in AY 15-16 and focusing on all outcomes), and we also assessed students' and teachers' perceptions of the effectiveness of our new course, ENGL 106, designed to support international students.

ENGL 101/101A/107

Seeking to measure students’ perceptions of their own learning in relationship to the program student learning outcomes (SLOs), the University of Arizona Writing Program (UAWP) designed and implemented an assessment project involving a large-scale student survey and statistical analysis. This project measured students’ perceptions of their own learning regarding five (5) writing program outcomes that were deemed most appropriate for students in the first course of the two-course general education requirement [English 101, 101A (Basic Writing), or 107 (International)]. Each outcome is followed by its corresponding goal category. At the end of the Foundations Writing sequence, students will be able to:

  • Identify the purposes of, intended audiences for, and arguments in a text, as situated within particular cultural, economic, and political contexts (Rhetorical Awareness)
  • Analyze the ways a text’s purposes, audiences, and contexts influence rhetorical options (Rhetorical Awareness)
  • Incorporate evidence, such as thorough summaries, paraphrases, quotations, and visuals (Critical Thinking and Composing)
  • Follow appropriate conventions for grammar, punctuation, and spelling, through practice in composing and revising (Conventions), and
  • Apply citation conventions systematically in their own work (Conventions).

Research Question: Which, if any, differences are significant in students’ self-perceptions between English 101 and 101A, 101 and 107, and 107 and 101A?

Methods

The UAWP Director asked all instructors teaching ENGL 101, 101A, and 107 to distribute a survey via Qualtrics at the end of the Fall 2016 semester. The survey consisted of eight (8) Likert scale “can-do” statements designed to assess learning for all four SLO major areas. Graduate Assistant Director Maria Conti conducted a usability test of the survey with students from each of the three courses and used student feedback to modify the survey with the assistance of the UAWP Director and Assistant Director Aimee Mapes. Of the 4,787 students in these courses, the Writing Program received 1,474 responses (~ 31% response rate) across ENGL 101 (n=1,144), 101A (n=256), and 107 (n=74). Maria Conti analyzed the quantitative data using a 2-Way Repeated Measure ANOVA and Post Hoc Tests to determine statistical significance (p<.01).

ENGL 106

During the 2016-17 academic year, a study was conducted by the UAWP to determine students’ and instructors’ perceptions of the newly implemented ENGL 106 course, which is meant to support international students as they enter a U.S. academic environment. Through surveys of both students and instructors, observations of ENGL 106 instructor collaborative meetings, and interviews with a focus group of ENGL 106 students, the study (led by Rachel LaMance) investigated these questions:

  • Are students in English 106 achieving the proposed student learning outcomes (SLOs)?
  • What are the strengths and weaknesses of English 106, and how can they inform future iterations of the curriculum?
Assessment Findings: 

ENGL 101/101A/107

Lack of Significant Differences—101A and 107

One of the project’s most interesting findings was that there were no statistically significant differences between 101A and 107. This may be due to UA’s placement mechanism. Currently, only international (visa) students are placed into 107, while other second language writers such as resident immigrants and Generation 1.5 learners are often placed into 101A. The data suggest that the UAWP should consider designing a revised placement process for domestic students that more thoroughly takes into account their previous educational and language experiences.

Audience: 107 Rated Themselves Lower than 101

Another interesting result is the response for the question about audience [“I can identify a text’s intended audience (person or group the writer wishes to address)”]. There was a significant difference in the perceptions of 107 and 101 students in this area, with 107 students rating themselves lower. This may be due to the variation in their background knowledge as compared to U.S.-educated students. The concept of “general audience” concept may be confusing for writers who have experienced the variety of people and cultural contexts that could be suggested by this term. The data suggests that L2 writers often have a more complex understanding of audience than their L1 counterparts, although more research is needed to explore this further.

ENGL 106

Quantitative and qualitative data collected from both students and instructors show primarily positive reactions to the first iteration of the ENGL 106 course. The data also helped identify sites for future course improvement, which will be addressed in semesters to come. A brief overview of study findings, organized by research question, is below:

Are students in English 106 achieving the proposed student learning outcomes?

  • Students were asked to rate their agreement with can-do statements regarding course SLOs at the beginning of the semester and again at the end of the semester. In the second survey, an average of 26% more students responded “agree” or “strongly agree” (i.e. the highest confidence ratings).
  • The 10 most frequent responses to the open-ended survey question “Which skills, strategies, terms, concepts, etc. did you learn in English 106 this semester?” related directly or indirectly to course SLOs: citation and formatting, developing and supporting ideas, register/formality, writing skills, grammar, reading skills, organization, plagiarism, thesis statements, academic English.

What are the strengths and weaknesses of English 106, and how can they inform future iterations of the curriculum?

  • 93% of students (72 total respondents) indicated that the topics of World Englishes and language variation were interesting to them.
  • Survey responses and collaborative meeting observations also show that the topics were interesting to instructors.
  • Despite interesting topics, certain readings and assignments were too challenging for some students. Some instructors eliminated these readings and altered assignments for the second semester of implementation.
  • Multiple instructors appreciated having fewer main assignments (three rather than four), which allowed more time to focus deeply on content as well as the writing process.

Focus group interview data indicate that ENGL 106 fosters learning about U.S. (academic) culture while learning about writing. 

Change in Response to Findings: 

The UAWP will continue to monitor the new curricula in ENGL 101/101A/107 and ENGL 106. Based on the recommendations of our APR (conducted in October 2016), we will be

  • desigining and implementing a new placement procedure for all domestic students, modeled after the success of our pilot placement process for international students

  • piloting a new outcomes-based portfolio assessment for all 100-level writing classes

In addition, we will continue the process of refining our curriculum so that ENGL 101/101A/107 and ENGL 102/108 work together to prepare students for the work they do after they finish the Foundations Writing sequence. Our new program assessment, piloted in AY 17-18, will help us assess the achievement of learning outcomes as we revise our curriculum.

AttachmentSize
PDF icon AY 14-15 UAWP Assessment Report311.46 KB
PDF icon AY15-16_Assessment.pdf290.13 KB
Updated date: Wed, 11/29/2017 - 20:16