A Tale of Two Assessments: Part Deux

In my last blog, I presented an example of assessment that came from the world of Kenyon sports. Today I would like to look on the academic side for an example of assessment research that grew out of our own need to know something about our students. Last year, RAAS (for those off campus readers, that's our Resource Allocation and Assessment Subcommittee) reported in its annual report on Gen Ed Assessment that our students were perceived as somewhat deficient in their ability to locate and assess information. More specifically, common areas of concern included the inability of students to assess the quality of web-based sources, the inability or unwillingness of students to use primary sources. RAAS sent this information from the departmental GEARS on to LBIS. LBIS has devised a fairly simple but I think effective means of determining what our students either know or don't know about doing research.

As the LBIS section of the facpac reported, with support from NITLE and HEDS, LBIS is currently administering the Research Practices Survey to our first-year students. The survey is a fifteen-minute questionnaire, developed by and for liberal arts institutions, that assesses undergraduate students‘ experiences, dispositions, and proficiencies in conducting academic research. If the response rate is high enough, this survey should give us a fairly detailed snapshot of the first year class' research abilities and inclinations (based largely on their high school experience). Not only will LBIS and your liaisons be able to tailor their instruction to students more effectively, it might also give the faculty a better sense of what our entering students know and don't know about doing research. The survey will be repeated in the spring with first year students to build a picture of their learning over their first year. Eventually, if this project is successful, the survey could be given (perhaps not even every year) to entering students during orientation (or the summer before) and then spring semester junior year (just before beginning comps).

So far 79 students in the first year class have taken this survey, representing about 17% of the class. The survey does not close until October 24, so there is still time. Students: go take the survey, now; it's a good form of procrastination. Faculty, tell your students to go take this survey. Let's try to get a better sampling.

What sorts of questions does the Research Practices Survey ask? Always such good questions. The survey seeks information about students' experience with research (how often have you done it and how challenging is it for you?), quizzes them on familiarity with research terms and strategies (Boolian searching), and quizzes them on evaluation.

An example of questions about experience: How many assignments, papers, or research projects have you completed in the past academic year that required you to include at least three sources in the Bibliography? 43% (34) of the respondents said 5 or more over the last year. Only 17% (14) said one or two.

When they were given research assignments in the past, 74.68% reported that they almost always were required to use a specific footnoting format.

Here's a surprising one: When asked whether they had consulted a librarian during the past year to help with research, 33% responded "sometimes," but a whopping 56.41% responded "rarely" and "never." 65.82% said they never consulted a writing lab.

Only about 22% of the respondents divide their work across the time given for an assignment; 65.82% do most of it at the end (time management).

The survey also tests students on their knowledge of search terminology and strategy, so the survey will give us some direct assessment of students' abilities and not just their attitudes and experiences. For instance, 59. 49% did not know how to truncate a word in an electronic search (to be fair I forget that one all the time too). The students who took the survey overwhelmingly knew how to read a journal article in a bibliography accurately. However, 56% could not distinguish in a bibliographic entry between a journal article and a chapter in a book.

Here's a really important finding. 89.87% could distinguish accurately between a primary and a secondary source. 62% could accurately identify what a peer reviewed journal was. That's good news. The best news of all? Fully 82.28% of the respondents correctly identified when a citation is NOT required (You are describing your own findings or analysis), suggesting that our students know the difference between their own work and plagiarism.

Now granted the sample is still small; who knows if the findings will hold up if we can get all or most of the students to do it. But remember there is still time. The survey runs for another week.

Is this information you can use? You betcha.