Getting Graduate Program Assessment off to a Quick and Easy Start

A great way to start to assess your program is to examine the culmination of students’ work in a program, their theses and dissertations. These works represent many of the deep skills and much of the knowledge your students have acquired over their studies and thus the work reflects much of their education.

Below, we’ve provided a brief description of how to use your students’ theses and dissertations for program assessment. If you would like a broader and more thorough discussion of using dissertations for assessment, you might like Barbara Lovitts’ book Making the Implicit Explicit (2007). Copies have been ordered for each department and program and will be dispersed to department or program chairs. Copies have already been made available to members of the GSAS Assessment Committee, and others can be ordered at your request.

Starting at the End: Using Theses and Dissertations for Program Assessment

  • A “program-level student learning goal” is simply something you intend your students to attain from completing a graduate degree in your program. Your goals can include knowledge, skills, behaviors, attitudes, habits, etc. The best way to identify your program’s goals is to complete the statement “Upon graduating from our program, a student should ….” Identify outcomes that are essential to your discipline and that every student should attain. For example, most disciplines require that students develop critical thinking skills. 

    An objective is simply a concrete manifestation of a more abstract goal. For example, Psychology faculty might expect a Psychology graduate student to exhibit critical thinking by analyzing a research design. A Philosophy graduate student might demonstrate achievement of the same goal by identifying the assumptions in an argument. A single learning goal might be reflected in many objectives. To extend the previous example, you might expect a student to demonstrate critical thinking in the way she selects texts or articles for the literature review of her thesis or how he discusses the strengths and limitation of his own work.

  • To proceed, you need the following:

    (a) A set of goals that program faculty have for the education of your students (“program-level student learning goals”): You can’t determine whether students have accomplished the program goals unless you know what the goals are. For this assessment cycle you need to identify at least one goal.

    (b) A set of objectives that describe more concretely what students should be able to do if they’ve accomplished the program goals.

    (c) Criteria for deciding what a student’s work would have to have in order to demonstrate that the student had met an objective.

  • With this information decided by program faculty (or some subset of faculty to whom that responsibility and authority has been delegated), you need only apply it to the students’ theses and dissertations. You could draw up a “rubric” that expresses both the objectives and the criteria. Alternatively, one could draw up a checklist of qualities (e.g., uses language effectively with attention to connotation and denotation, maintains professional tone) or content (e.g., presents a clear thesis, uses primary sources to support thesis) expected in the work. Faculty can use it to indicate whether or not a thesis or dissertation possesses each quality or contains the specified content.

    Next, you (and your colleagues) collect your rubrics or checklists and summarize the evidence: How many students performed at each level? If you used a rubric, you can count the number of students who performed at each of your levels (say, outstanding, acceptable, unacceptable). If you used a checklist, you can count the number of students who demonstrated each item.

  • Finally, you and your colleagues review the findings and discuss what, if anything, needs to be done in response. Suppose, for example, that your assessment shows the following:

    (a) Almost every dissertation contained a clear, well-articulated thesis statement.

    (b) About half of the students provided minimal or inadequate support from primary sources.

    From these results, it would appear that students need more support or instruction on finding and using primary sources. N.B.: Before settling on that conclusion, the faculty who worked with the students as they produced these papers should weigh in. If the faculty had to exert Herculean efforts to get the students to articulate thesis statements, or if there were extenuating circumstances that prevented students from getting primary sources, attend to that information. Once you’ve determined the strengths and weaknesses of the students, consider what might be done to improve your students’ education. You may need to review your courses, for example, to determine whether students get sufficient opportunity to learn and practice the skills required in their dissertation research. Is there some way to improve students’ preparation for the dissertation (and hence education throughout the program)? Together, the program faculty should decide on how to proceed, either by further investigation to determine how to improve the program or by implementing changes to the program.

  • You need not include every students’ work in the assessment, a sample may suffice. Decide the size of your sample based on the size of your graduating class. See “Sampling for Assessment” for ideas. Some of this information may also apply to Master’s programs with capstone papers, internship papers, or field experience papers, while capstone projects for other types of Master’s and Certificate programs may need to be evaluated using a different approach.