You are here

Economics: Graduate Programs

Overview: 

 

The doctoral program in Economics leads to a Ph.D. in Economics that trains students in state-of-the-art methods in economics, allowing them to obtain excellence in teaching and develop frontier research. In our program, students take courses during their first three years, with emphasis on economic theory, economic dynamics, econometrics, and specialized field courses. Students start on research from their first year, and by their third year, the majority of their effort is devoted to their dissertation research. Graduates from our program have obtained faculty positions and postdoctoral fellowships at prestigious research universities, liberal arts colleges, think-tanks and consulting firms.

Expected Learning Outcomes: 

Consistent with our program goals, we measure both teaching and research performance.

A) Teaching outcomes and measurement

We seek to measure the following teaching outcomes:

  1. Designing an economics course structure that effectively conveys relevant material.
  2. Giving spoken presentations of economic knowledge, clearly and at a level appropriate for the audience.

Our specific measurements – starting in December 2012 – are:

Specific Measurement

Outcome Measured

Date occurs

Teaching audition

2

1st or 2nd year

Review of syllabus for courses taught

1

2nd year & beyond

Lecture observation for courses taught

2

2nd year & beyond

TCE evaluations

1, 2

2nd year & beyond

 

The teaching evaluation process begins with Ph.D. students delivering a lecture during the middle of the spring semester of their 1st or 2nd year. In a simulated undergraduate classroom setting, each graduate student delivers a 20-minute lecture on a pre-assigned topic to an audience of undergraduate students and a faculty observer. To pass the audition, a graduate student has to be approved by a majority vote of the undergraduate students, and separately, by the faculty observer(s). All of the lectures are on principles-level topics and the student audience is comprised of upper-division economics students who are instructed to evaluate the lectures as if they were taking Economics 200 for the first time.

Faculty and student observers at the audition fill out the Ph.D. Presentation of Knowledge Rubric. Ph.D. students who are assigned to teach a class as the primary instructor must submit their course syllabus for approval. Faculty members evaluate syllabi using the Ph.D. Course Structure Design Rubric. Once they have started to teach the class, a faculty observer attends a class lecture (multiple times if necessary), evaluating teaching again using the Ph.D. Presentation of Knowledge Rubric. Assessment information from the rubrics and the TCE evaluations, is compiled and updated yearly.

 

B) Research outcomes and measurement

We seek to measure the following research outcomes:

  1. Demonstrating mastery of core economic methods, specifically economic theory and econometrics
  2. Demonstrating mastery of two sub-fields of economics, from among the offered fields of study
  3. Creating high quality and novel research that is publishable in a top academic journal.

Our specific measurements – starting in December 2012 – are:

Specific Measurement

Outcome Measured

Date occurs

Written comprehensive exams

1

1st year

Second year focus group

1

2nd year

Third year focus group

2, 3

3rd year

Second year paper

2, 3

2nd year

Third year paper

2, 3

3rd year

Oral comprehensive exam

3

4th year

Practice job market seminar

3

5th year

Oral thesis defense

3

5th year

Initial graduate placement

1,2,3

5th year

 

Written comprehensive exams take place at the end of the first year of study, and the results are tabulated and maintained yearly for analysis. Attending faculty fill out the PhD Research Rubric at the second year paper, third year paper, oral comprehensive exam, practice job market seminar, and oral thesis defense for each student. Initial graduate placements are tabulated and updated yearly by the graduate coordinator. 

Each spring, the Department conducts separate focus group sessions with 2nd and 3rd year graduate students to collect feedback on various aspects of the doctoral program. These sessions use GroupSytem’s ThinkTank electronic collaboration system, which allows for anonymous but collaborative commentary. The Director of Graduate Studies prepares a summary of the focus groups for the Department to review. 

Assessment Activities: 

A) Teaching auditions

Eighteen doctoral students participated in the 2012 teaching auditions process. Two-thirds, or 12 of the 18 students, passed the audition. In 2013, thirteen doctoral students participated in the auditions process.  Six of the thirteen students passed the audition. While the pass rate is lower for 2013 than for the year before, the results across these two years are otherwise fairly similar and we describe overall results below. 

Undergraduate student evaluators graded several aspects of each lecture, including organization, delivery, and an overall evaluation score, using the standard A = 4, B = 3, C = 2, D =1, E = 0 grading scale. Average scores are reported in the table below, broken out by passing students vs. failing students.

Table: Average Scores in Teaching Auditions
 

Students Passing the Audition (18 of 31)

Students Failing the Audition (13 of 31)

Overall Evaluation  3.32 2.04
Lecturer Organization  3.57 2.27
Lecture Delivery 3.22 2.25

There was significant variation in “overall evaluation” scores on the lectures, with scores ranging from a high of 4.0 to a low of 1.38.  Note that the mean score for lecture “organization” for passing students was higher than their mean score for the “delivery” of their lecture. This is not surprising. A good part of organizing an effective lecture requires, beyond thorough knowledge of the subject matter, sound judgment and hard work. Providing a polished and engaging delivery typically requires a lot of experience, which these graduate students do not have at this point in their careers. We believe that the specific findings are very encouraging results since the majority of these students will, in the future, acquire considerable experience in the classroom. Interestingly, for those who failed the audition, the two respective means were virtually identical. This suggests that to become effective teachers these graduate students need to enhance all relevant skills. 

Besides the above questions, an additional question that the audience was asked to answer was: did you understand the lecture? The response was a unanimous “yes” for all eighteen graduate students that passed the audition. In fact, three of the students who failed the audition were also in this category and five of the students who failed the audition had a majority of the audience indicate that they understood the lecture. In the remaining five cases one-half or more of the undergraduates indicated that they did not understand the lecture. This is a very positive result given the importance of understanding what is being taught. Students who fail the teaching audition in one year are given the opportunity to audition again the following year.

More details on the teaching audition scores are available here. The results have been updated to include teaching audition results through 2016.

 

B) TCE evaluations

All graduate student teachers in Economics are evaluated on their teaching using the university TCE system. Over the period 2006–2013, we were able to obtain ratings from 126 courses. We report on those ratings.

Economics graduate students who teach courses generally perform very well on teaching evaluations. Our teaching evaluations are done on a 5-point scale. The key important question is “Overall instructor effectiveness,” where 3 represents “sometimes effective,” 4 represents “usually effective,” and 5 represents “almost always effective.” On this question, the mean of the means of teaching ratings is 3.81.

Another important question is “Overall course rating,” where 3 represents “about the same as usual,” 4 represents “better than usual,” and 5 represents “one of the best.” On this question, the mean of the means of teaching rates is 4.19. Thus, we believe that our students who have passed the department teaching audition process provide high quality instruction that is better than average.

Overall, these teaching rates suggest that students are effective at delivering high quality instruction that is well received by the undergraduate students.

More details on the TCE evaluations are available here. The results have been updated to include TCE results through the Fall 2015 semester.

 

C) Paper presentations and oral comprehensive exams

We recently implemented a scoring rubric for 2nd year papers, 3rd year papers and oral comprehensive exams. Since the establishment of the rubric, 16 students have completed the oral comprehensive exams. The mean score on the preliminary oral exams for these five students was 6.7, where 6 indicates that they meet expectations for progress and 9 indicates that they exceed expectations. On the research question and design section, 14 of 16 Ph.D. students have met the expectations, which demonstrates that they can formulate an appropriate and reasonable research question. Two students were judged to be slightly below the meeting expectations benchmark. Several students approach the level of exceeding expectations, indicating that their research question may be superior, in the sense of being novel and/or proposing or using new cutting-edge methods. In terms of economic methods, all students except one met the expectations, demonstrating that they have adequate economic theory and/or quantitative skills necessary to complete the project. Some students demonstrate a strong knowledge of economic methods, which demonstrates innovative modeling and/or data analysis. In terms of the contribution to economic knowledge, all students except one again met the expectations, demonstrating that they raise ideas that may make meaningful contributions to the literature. Comparatively fewer of them exceeded the expectations, defined as raising an idea that will be significant or even transformative.

Overall, the assessment of our preliminary oral exams demonstrates that students at this stage in their Ph.D. career are asking appropriate economic questions and have the tools with which to answer those questions. Some of the students exceed these expectations and it will be useful to find ways to get more students to exceed them, through enhanced training in graduate classes and more interaction with faculty members. Because these measurements take place early on in their research careers, relatively few students have made a significant or transformative contribution to the economics literature. We expect that this measure will increase throughout their career, as students continue to hone their methodological and research design skills.

We began using the 9-point scoring rubric last year for the 2nd year and 3rd year papers. Six students presented a 2nd year paper. Their average score was 6.3, where 6 indicates met expectations and 9 indicates exceeds expectations. All students except one met expectations for research design, economic methods and contribution to economic knowledge. One student did not pass the 2nd year paper evaluation, but this student re-wrote his paper and was passed based on the revised paper. One student received a unanimous exceeds expectations rating from faculty reviewers for the economic methods component of her paper.

Fifteen students presented a 3rd year paper. Their average score was 6.64. Thirteen out of fifteen students met expectations for their 3rd year paper.   One student earned an overall score of 8.7, where 9 indicates exceeds expectations. Two students did not pass the evaluation, but both students revised their papers during the subsequent summer and passed the exam by the next semester.

 

D) Focus groups

We established a focus group for the 2nd year students in 2011, and expanded this to separate focus groups for 2nd and 3rd year students in 2012. Focus group questions were revised for 2013 to include questions about doctoral program learning outcomes. The focus groups were performed with software developed by Jay Nunamaker of the MIS Department at the University of Arizona (GroupSystem’s ThinkTank), and allowed students to view each other’s comments in real time on a common screen in our seminar room, all the while preserving the anonymity of individual comments.

Overall, the focus groups show that students are happy with the program. Students generally valued their courses except for ones taught by instructors who were only temporarily at the UA or likely planned to leave UA. Moreover, the concerns about these courses reflect concerns about the quality of teaching of these courses, not the overall programmatic content. Students appreciate the high-quality faculty and camaraderie among their peers.

A central concern of students is the lack of faculty in particular fields and the limited numbers of faculty overall. This concern has eased somewhat since 2011 as we have hired a number of new research-active faculty (though lost others). Some concerns remain in theory, behavioral/experimental, economic history, and labor. Another concern of students is the relatively small stipends given to teaching assistants and research assistants.

Students generally valued the paper presentations. However, some fine-tuning regarding faculty input would be useful. It is worth noting that some of the comments from 2011, regarding the inability to find an advisor, were addressed by an improved process of advisor assignment and grading, established in 2012. Students would also like more guidance about expectations for the various program milestones, notably prelims and paper presentations.

Several questions from 2013 focus groups address learning outcomes. Students believe they have mastered economic theory. Students feel they have learned fundamental econometric concepts, but in some cases would like additional opportunities to gain experience with computational tools for these concepts. Students generally agree that they are on track to produce novel, high quality research. One student expressed the view that students should be “apprenticed into” research earlier. Third year students feel they can design and teach an economics course that effectively conveys relevant material. Second year students are less sure about this, but we note that these students have not taught their own courses yet. Some second year students expressed a desire to have opportunities to practice teaching, for example by leading recitation sections of undergraduate courses. Students generally feel that the program has enabled them to give effective, clear presentations.

More details on the focus group outcomes are available here.

 

E) Initial placement

We have data on all 48 students who completed our program during the period 2006–13. Of the 48 students, 19 or 40% of the total, obtained initial placements as tenure-track assistant professors at research academic institutions. The institutions represented in this list are Rochester Institute of Technology, University of Mississippi, University of Mannheim, HEC Montreal, University of Michigan, University of Colorado, Louisiana State University, University of Konstanz, University of Alabama, Washington University in St. Louis, University of Florida (2), University of Calgary, and UC Riverside, UC Merced, ITAM, and University of Canterbury. Eight students obtained initial placements as post-doctoral fellows at research institutions. The institutions represented in this list are Heidelberg University, LaTrobe University, University of Technology Sydney, UC Merced, California Institute of Technology, Harvard University, Yale University, and Columbia University.

A number of other students obtained positions as tenure-track assistant professors at liberal-arts colleges, including Hobart and William Smith Colleges, Rowan University, Baylor University, San Diego State University, and Franklin & Marshall University. Finally, a number of students obtained initial positions in research think-tanks and antitrust consulting firms, including the Max Planck Institute (2), CEPS/INSTEAD, and Bates White Economic Consulting.

Overall, we believe that our Ph.D. placement record is strong. All except one of our graduating students obtained high caliber employment that effectively utilized their advanced training in economics: our students’ career paths in research academia, liberal arts colleges, and antitrust consulting all require the skills of an economics Ph.D. Importantly, the majority of our students place as tenure-track professors or post-doctoral fellows at research academic institutions. This implies that a majority of our students proceed to careers whose goal is to create and disseminate economic research. Finally, a number of our students have obtained initial positions at institutions that are ranked comparably or higher than our own department, such as the University of Michigan and Washington University in St. Louis, implying that our program is successful in developing researchers who are at the frontier of their fields.

More details on the initial placement are available here. The placement details have been updated through 2015.

 

 

Assessment Findings: 

Teaching Outcome 1: Designing an economics course structure that effectively conveys relevant material.

Eighteen of thirty-one graduate students passed teaching auditions during 2012-2013. For these students there was a unanimous audience response of “yes” to the question: Did you understand the lecture? Even for many of the graduate students who failed the audition, a majority of the undergraduates in the audience responded “yes” to this question. Overall, this is a positive result given the importance of understanding what is being taught. Graduate students who passed the audition achieved high scores on the “organization” of their lectures; higher scores on “organization” as compared to “delivery”. While organizing a lecture is a smaller task than designing a course structure, achieving proficiency in organizing a lecture is a necessary step toward this teaching outcome. Graduate students who teach their own courses receive evaluations from undergraduate students via the university TCE system. TCE results indicate that our graduate students are effective at delivering high quality instruction that is well received by the undergraduate students, as measured by responses to the “Overall course rating” question. Finally, responses from the focus groups noted that graduate students have ample opportunities to teach their own classes, and that 3rd year students are confident in their ability to be effective instructors.

 

Teaching Outcome 2: Giving spoken presentations of economic knowledge, clearly and at a level appropriate for the audience.

Responses from graduate students at the focus groups are generally positive about achieving this teaching outcome. The evidence from teaching auditions regarding student understanding of lectures is positive for this teaching outcome, just as it is for teaching outcome 1. Delivering spoken presentations that are clear and at an appropriate level is key for achieving good understanding of students. Teaching audition results indicate that graduate students scored lower on “delivery” of lectures than on “organization” of lectures. Providing a polished and engaging delivery typically requires a lot of experience, which these graduate students do not have at the time of their teaching auditions. TCE results indicate that our graduate students are receiving overall course ratings above 4 (“better than usual”). This suggests that our graduate students are achieving this teaching outcome by the time they are teaching their own courses.

 

Research Outcome 1: Demonstrating mastery of core economic methods, specifically economic theory and econometrics.

A key assessment for this outcome is the written comprehensive exam. This is a rigorous examination of core economic methods that students must master in order to be successful economic researchers. During the last 3 years, 29 of our graduate students have passed and 8 have failed this exam. Given the rigorous nature of the exam, we are not surprised to have a non-negligible failure rate. Moreover, the failure rate for our written comprehensive exam is similar to that of other economics Ph.D. programs. Students must pass this exam before they can move on to take field courses and work on their dissertation. Focus group responses from 2nd and 3rd year cohorts (all of whom have passed the written comprehensive exam) indicate high confidence in mastery of economic theory and of econometrics concepts. Both cohorts exhibited some concerns about mastering application of econometric methods.

 

Research Outcome 2: Demonstrating mastery of two sub-fields of economics, from among the offered fields of study.

We do not include sub-fields in our written comprehensive exams, so assessment of this research outcome comes mainly from results of the oral comprehensive exams and 2nd and 3rd year papers. Most students are meeting or exceeding expectations in the orals and these two papers, and this indicates that students are generally able to apply material from their sub-fields. Focus group responses from 2nd and 3rd year students raised two issues about paper requirements. One is that expectations for 2nd and 3rd year papers were not clear to all students. A second issue is that some students did not receive adequate advising, or sought advising only at a late date, and were therefore less successful with their paper than they might have been. This issue has also been a concern expressed by several faculty members.

Placement at the end of the program provides another measure of the extent to which students are mastering sub-fields that are viewed as important by prospective employers. We have achieved a very high placement rate, and all placements are in positions that utilize skills learned in our program.

 

Research Outcome 3: Creating high quality and novel research that is publishable in a top academic journal.

Assessment results for the oral comprehensive exams and the 2nd and 3rd year papers indicate that students are meeting or exceeding expectations for formulating appropriate research questions, utilizing economic research methods, and making meaningful contributions to economics knowledge. These results provide evidence that students are on track to achieve research outcome 3. Focus group responses are consistent with this evidence. Placement also provides a measure of the extent to which this outcome is being achieved. In particular, 40% of our placements in recent years are academic positions and such hiring for such positions is based on a judgement that the candidate has strong potential for creating high quality and novel research.

Change in Response to Findings: 

Two changes in graduate program practices were made early in 2013, based on findings from prior assessment activities. One change was to institute regular meetings with each cohort of graduate students about program milestones and expectations. This change was in response to focus group comments from graduate students and also aimed at improving performance on 2nd and 3rd year papers. A second change was adoption of an admitted student day during spring semester, in which students who have been admitted (or, are likely to be admitted) are invited to campus for a day of meetings with faculty and graduate students and a social event at the end of the day. This change was aimed at increasing the yield on our admissions and improving the quality of enrolling students.

As noted earlier in this document, several changes were made in measurements during the last year which were aimed at collecting information that better addresses learning outcomes. At a department faculty meeting in May, 2014, the faculty voted to make changes in the process and requirements for 2nd and 3rd year papers, in response to research outcome findings. The main changes are as follows:

1) The dates for submission of the 2nd year paper and student presentation of the 2nd year paper are delayed from April of the 2nd year to the beginning of the fall semester of the 3rd year. This change will give students more time to receive and respond to advising from faculty members.

2) In order to ensure that students receive guidance throughout the process of writing their 2nd and 3rd year papers, they will be required to have a faculty advisor for their 2nd and 3rd year papers. For the 2nd year paper a student must find a faculty member willing to serve as advisor by February 15 of the 2nd year. For the 3rd year paper a student must find a faculty member willing to serve as advisor by October 15 of the 3rd year.

3) A detailed explanation of expectations and requirements for 2nd and 3rd year papers is now available for students. This explanation clarifies what a student must accomplish in order to receive a passing grade on their paper. These changes have been incorporated into a revised Graduate Student Handbook for economics doctoral students. The revised handbook may be accessed at: http://econ.arizona.edu/docs/doctoral/Grad%20handbook_revised%205_10_201....

Updated date: Wed, 06/29/2016 - 15:58