Getting Program Assessment off to a Quick and Easy Start

If your program requires students to complete a capstone course, thesis or other culminating work, a great way to start to assess your program is to examine that student work. These works represent many of the deep skills and much of the knowledge your students have acquired over their studies and thus the work reflects more of their education than just whatever happens in the capstone course itself.

Below, I've provided a brief description of how to use your students' capstone work for program assessment. If you would like a broader and more thorough discussion of using student course work for assessment, you might like Chapter 11 from Barbara Walvoord and Virginia Anderson's book Effective Grading: A Tool for Learning and Assessment in College (2nd ed., 2010). A limited number of copies of this book are available through the Office of Institutional Research and Assessment. Please contact us at [email protected] if you would like a copy.

Starting at the End: Using Capstone Seminars and Theses for Program Assessment

  • A “program-level student learning goal” is simply something you intend your students to attain from majoring in your program. Your goals can include knowledge, skills, behaviors, attitudes, habits, etc. The best way to identify your program’s goals is to complete the statement “Upon graduating from our program, a student should ….” Identify outcomes that are essential to your discipline and that every student should attain. For example, most disciplines require that students develop critical thinking skills.

    An objective is simply a concrete manifestation of a more abstract goal. For example, Psychology faculty might expect a Psychology major to exhibit critical thinking by analyzing a research design. A Philosophy major might demonstrate achievement of the same goal by identifying the assumptions in an argument. A single learning goal might be reflected in many objectives. Similarly, when you examine evidence that students have met an objective, the evidence will be shaped by the assignment and thus by the course.

  • To proceed, you need the following:

    (a) One or more capstone experiences (e.g., seminar, portfolio, thesis, performance) that all (or almost all) students are required to take and that is typically taken in the senior year (or after the student has completed most of the major requirements).
    (b) A set of goals that program faculty have for the education of your students ("program-level student learning goals"): You can't determine whether students have accomplished the program goals unless you know what the goals are.
    (c) A set of objectives that describe more concretely what students should be able to do if they've accomplished the program goals.
    (d) Criteria for deciding what a student's work would have to have in order to demonstrate that the student had met an objective.

  • With this information decided by program faculty (or some subset of faculty to whom that responsibility and authority has been delegated), you need only apply it to the students' papers. You could draw up a "rubric" that expresses both the objectives and the criteria. Faculty can use the rubric while reading the papers for grading purposes or at a second reading for assessment purposes alone. Alternatively, one could draw up a checklist of qualities (e.g., uses language effectively with attention to connotation and denotation, maintains professional tone) or content (e.g., presents a clear thesis, uses primary sources to support thesis) expected in the work. Faculty can use it to indicate whether or not a paper (or other piece of work) possesses each quality or contains the specified content.

    Next, you (and your colleagues) collect your rubrics or checklists and summarize the evidence: How many students performed at each level? If you used a rubric, you can count the number of students who performed at each of your levels (say, outstanding, acceptable, unacceptable). If you used a checklist, you can count the number of students who demonstrated each item.

  • Finally, you and your colleagues review the findings and discuss what, if anything, needs to be done in response. Suppose, for example, that your assessment shows the following:

    (a) Almost every paper contained a clear, well-articulated thesis statement.
    (b) About half of the students provided minimal or inadequate support from primary sources.

    It would appear that students need more support or instruction on finding and using primary sources. N.B.: Before settling on that conclusion, the faculty who worked with the students as they produced these papers should weigh in. If the faculty had to exert Herculean efforts to get the students to articulate thesis statements, or if there were extenuating circumstances that prevented students from getting primary sources, attend to that information. Once you've determined the strengths and weaknesses of the students, consider what might be done to improve your students' education. You may need to review your intermediate and introductory level courses, for example, to determine whether students get sufficient opportunity to learn and practice the skills required in their capstone course. Is there some way to improve students' preparation for the capstone (and hence education throughout the program)? Together, the program faculty should decide on how to proceed, either by further investigation to determine how to improve the program or by implementing changes to the program.

  • This approach is most useful when the capstone is a program requirement, that is, when all of your students must participate. When capstone courses are optional, only some students (often the better, more motivated students) tend to take the course so you may miss important information that would be available only from a broader sample.

    This approach will not shed light on your program if the work required in the course is not a continuation of work students have done in prior courses. The course should therefore involve application and extension of skills students learned in previous courses. Of course, the capstone course content may be entirely new, a more in-depth examination of a familiar topic, or a new integration of several familiar topics and so forth. Novel aspects of the course are fair game for course assessment, but they won't tell you much about your program.

  • You need not include every students' work in the assessment, a sample may suffice. Decide the size of your sample based on the size of your graduating class. If you have more than 50 majors in a graduating class, consider sampling 20-25% of their work. Pick a random sample or organize your sampling to ensure that you have a range of student performance. If you have a smaller graduating class, start by sampling a random set of 5-10. Summarize your findings and then read 5 more randomly selected papers. If the 5 new papers are commensurate with the first 10, you're done. If they tell you something more, incorporate your new findings and repeat the process, starting with 5 additional randomly-selected papers. Stop when the process stops yielding new information. (Note: there's nothing magical about 5-10 papers. Select numbers that make sense in the context of your program. Your primary goal should be to get useful (and hence reasonably accurate) information about your program.)