The AME Department has established several specific and regularly scheduled assessment cycles for measuring outcomes, reviewing its educational objectives and outcomes, and identifying the needs of its constituencies. The assessment has “internal” and “external” components.
The Undergraduate Studies Committee is charged as the focal faculty committee to oversee the assessment process within the department. A member of the committee is designated as the coordinator of assessment activities. The committee is asked to conduct its business to explicitly optimize the impact of its activities on the Educational Objectives and Learning Outcomes of the department at all times. All data obtained from the range of assessment vehicles are directed to the Undergraduate Studies Committee. The committee is asked to review the data, interpret them and bring recommendations to the faculty for consideration at a faculty meeting at least once each year.
The Methods Used to Assess Intended Student Outcomes
The principal instruments (both quantitative and qualitative) used in this regard are:
- Undergraduate Advisory Committee: continuous.
- Senior Exit Survey: conducted once each year.
- Alumni Survey: conducted by the College of Engineering: every year.
- Evaluation of Senior Design Projects by Judges from Industry: every year.
- Student Course/Instructor Evaluations: conducted every semester in every course according to University policy.
- Fundamentals of Engineering Examination: periodic tabulation of the results.
- Review and Assessment by Industrial Advisory Council (and new AME Advisory Board): annually as needed, especially in response to their needs.
- Performance Assessment from Industry: every three years; to learn about the performance of our graduates.
- Faculty (Undergraduate Studies Committee) Assessment of Curriculum: conducted periodically as needed, but parts of the curriculum are reviewed every year.
- Academic Program Review: once every seven years, mandated by the University. The report is transmitted to the Arizona Board of Regents.
- ABET Accreditation Review: once every six years. The report is transmitted to the Arizona Board of Regents.
- Job Placement Data (Academic Services, College of Engineering): Data are collected from pre-commencement programs, departments, and faculty, from employers who advertise through Academic Services, and from the CareersEng listserv survey conducted by Academic Services. Periodic examination and interpretation of data.
Senior Exit Survey: Through Spring 2004, this survey was administered every semester in the capstone design class for ME students. Starting Fall 2004, it has been administered every semester in the Senior Colloquium (AME 495s). It is our belief that this survey is a very valuable assessment tool. Student experiences with faculty, other personnel, and the infrastructure are especially important for the closure of the assessment feedback loop. These data are made available to the Undergraduate Studies Committee and the Department Head for their actions to improve the program.
Alumni Survey: Conducted by the College of Engineering every year (individual students are contacted every other year). The surveys are directed at alumni with three and five years of experience following graduation. The survey not only provides a perspective on the students’ educational experiences, but the first page of the survey also yields important information on their professionalaccomplishmentsandcareer developmentactivities. The alumni survey asks how well the department has prepared them for industrial settings: multidisciplinary teamwork, communication skills, lifelong learning, ethical responsibilities, and problem and design formulation.
Evaluation of Senior Design Projects by Judges from Industry: In their final year, undergraduate students are required to take a sequence of two interdisciplinary capstone design courses in which AME students interact directly with companies and their peers from other engineering disciplines to design and deliver a real-life product by the end of the academic year. Representatives from industry act as judges as part of the final evaluation process. Since Spring 1999, their input has been garnered through an evaluation form addressing difficulty of design, creativity, quality of design and hardware, level of analysis, and effectiveness of presentation.
Student Course/Instructor Evaluations: This is done via standard questionnaires available from the office of Assessment and Enrollment Research of the University. The mandatory survey is filled out by students near the end of each semester for each course, and the information is processed by the University. Options are available for modifying the survey to meet the specific educational objectives of a course. Most conventionally, however, the so-called short form is used.
Fundamentals of Engineering Examination (FE): Some AME graduates take this examination in anticipation of PE certification in later years. The FE examination is not a graduation requirement, but it is a nationally-normed examination that provides useful information, both on the quality of the program as well as on the students who take them.
Review and Assessment by the AME Advisory Board: The AME Advisory Board has in the past provided a vehicle for industry input to the periodic evaluation of program educational objectives and the efforts to continuously improve the program on an occasional basis. The AME Advisory Board is currently comprised of representatives from local industry, alumni, and academics from state institutions of higher education with close ties to the department. This newer, more local structure was adopted (in Fall 2009) in the aftermath of recent very steep cuts in the state budget, which mandate better coordination with state institutions and more involvement and support by local industry.
Performance Assessment from Industry: Another measure of the success of the Mechanical Engineering Program is the on-the-job performance of the graduates working in industry. Representative employers of our graduates are requested to fill out a performance survey every three years. The survey specifically requests information on our graduates in the areas of technical ability, communication and professional growth, and eagerness to engage in life-long learning. Both the Department Head and the Undergraduate Studies Committee review the feedback.
Faculty (Undergraduate Studies Committee) Assessment of Curriculum: Curriculum (i.e., program) issues are discussed by the Undergraduate Studies Committee and recommendations are brought to the faculty for discussion at faculty meetings. Recent examples of such discussions are the introduction of new technical electives on biomechanics and nanoengineering into the curriculum and modifications in the structure of some courses.
With respect to assessment at the course-level, it is the faculty, via their day-to-day contact with students, who have the best understanding of how well the learning outcomes are met. The assessment is done in conventionally accepted ways using prerequisite quizzes, examinations, projects and homework. Information on the first is provided to the Undergraduate Studies Committee and the Department Head. The use of the prerequisite quiz is now (since Spring 2010) mandatory for a list of required and elective courses. The University also provides grade statistics for each course at semester’s end for review by the Department Head, who may request input/action from the Undergraduate Studies Committee if he/she feels that the program outcomes are not met.
A future objective is to construct a short survey that instructors can use in each course to provide an overall assessment on how well the students met the applicable program outcomes through standardized questions (e.g. selected FE exams) to remove any inconsistencies resulting from different levels of course difficulty and instructor standards
Academic Program Review: The review consists of a self-study report and the report/ recommendations of the review committee. The last review was carried out during the Spring of 2008. The committee consisted of three external (one NAE member), an alumni, and two faculty from the UA College of Engineering (6 total). A community representative was to have been part of the committee but did not participate due to illness. The final report of the committee is available upon request; the self-study report is available from the Department. This is a comprehensive review of the department, including the graduate program and research activities. The report was largely favorable and included various suggestions for improvements. The report, together with departmental input on recommendations, was submitted to the provost in October 2008. No action has thus far taken on any of these recommendations because of severe budget problems the university encountered in fiscal years 2008, 2009, and 2010.
: The review consists of a self-study questionnaire and an on campus visit/evaluation by a ABET evaluators. The self-study documents whether students, curriculum, faculty, administration, facilities, and institutional support meet the established criteria
. During the on-campus visit, the evaluation team reviews course materials, student projects, and sample assignments and interviews students, faculty, and administrators. The team investigates whether the criteria are met and addresses any questions raised by the self-study.
Job Placement Data: Data are collected from pre-commencement programs, departments, and faculty, from employers who advertise through Academic Services, and from the CareersEng listserv survey conducted by Academic Services..
Summary of Assessment Results
The Senior Exit Surveys have told us that the students in general are satisfied with the educational experiences and agree that the program is teaching them the necessary skills. The two areas that warrant the most attention for improvement are the physical laboratories and the number and quality of TAs. An Undergraduate Laboratory Committee has been formed to investigate the first deficiency and implement improvement strategies. However, the financial constraints have limited progress. In addition, current TA assessment process will be improved by implementing a student survey of the TAs in each course to which a TA has been assigned. The TA surveys will be designed to identify what the problems are and solicit suggestions regarding how to address the problems.
The results from the Alumni Survey are in agreement with those of the Senior Exit Survey. The AME program is doing well in educating its students in all the hard sciences areas, e.g., mathematical skills, design, etc. However, the alumni survey indicates that improvement should be made to "understand contemporary issues”, and to "understand the impact of engineering solutions in the global context." The Department will be investigating different ways to address this shortcoming.
Evaluation of Senior Design Projects by Judges from Industry is between Fair and Good. This is an assessment tool that we don't fully understand. The Undergraduate Studies Committee is investigating the reasons. As the committee identifies the reasons, they will propose corrective measures to improve the capstone design projects.
Student Course/Instructor Evaluation results are in agreement with the Exit Survey and Alumni Survey. AME students are satisfied with the course work. They feel that they are learning from our curriculum.
The performance of AME graduates in the NCEES Fundamentals of Engineering (FE) Examination is very good, surpassing the national average passing rate. Specifically, 96.5 percent of the 33 AME graduates who took this national examination in between April 2005 and April 2009 passed it, compared to the national average of 83 percent.
The previous APR Report provides a comprehensive review of the department. Faculty, staff, new building, and the undergraduate programs are identified as the strengths of the department. The Alumni Survey shows us that more, compared to 1998-2004 Alumni Survey results, of our undergraduate students feel that AME encourages and/or permits them to participate in research, independent studies, internships, and extra activities. Currently (2009-10 academic year), the undergraduate student-to-faculty ratio is 28.2, which is high compared to our national peer institutions. The graduate student-to-faculty ratio is 4.4, which is comparable to our national peer institutions.
The ABET accreditation review will occur in AY 2010/2011. Currently, an ABET committee is drafting the self-study questionnaire. The on campus visit will be arranged during Fall 2010.
AME graduates receive very positive evaluations from their employers. However, given the small sample size, these results serve as a supplement to other assessment results. The fact that many big companies recruit, hire, and retain our graduates is a strong indication of the success of the AME program in training our students.