BIOL 5174 - Intro to Grad Studies - Biological Sciences: Evaluating Sources - Criteria

Once you find information on your research topic, it's useful to get in the habit of critically evaluating the sources you found based on criteria such as relevance, accuracy, currency, and more.

For general criteria, you can always use the following as a guideline to evaluate information sources:

1. Authority – Is the person, organization, or institution responsible for the intellectual content of the information knowledgeable in that subject?
2. Accuracy – How free from error is this piece of information?
3. Objectivity – How objective is this piece of information?
4. Coverage – How well does this piece of information cover the topic?
5. Currency – When was the item of information published or produced?
6. Audience – Who is this information written for?

For scientific research articles, try the 11-Step framework outlined in this blog post and described below:

“How to read and understand a scientific paper,” by Jennifer Raff, Assistant Professor, Department of Anthropology, University of Kansas at Lawrence. This post was published August 25, 2013 on her blog titled Violent Metaphors: Thoughts from the Intersection of Science, Pseudoscience, and Conflict."

Pre-reading: consider the journal where the scientific research article was published. What can you find out about this journal? Who is on its editorial board? Does a professional association oversee its content and editorial policies? What other articles has it published?

Scientific Terminology Note: Try out these sources, or even Wikipedia to get a quick understanding of unfamiliar vocabulary.

• Talking Glossary of Genetic Terms from the National Human Genome Research Institute 
• NCBI Bookshelf from the National Center for Biotechnology Information (the folks who bring you PubMed!)

Steps from "How to Read and Understand a Scientific Paper ...":

1. Begin by reading the article's Introduction or Background section, not the abstract.
   • The abstract often reflects the authors' interpretation of the study - wait to read this so you can form your own impression.
2. Identify the BIG QUESTION.
   • Why is the research discussed in this article being done? Why is it important to the field, to all of us?
3. Summarize the background in five sentences or less.
   • What work came before this study (as described by the authors in their introduction)? What are limitations of previous work? What needs to be done (according to the authors)?
4. Identify the Research Questions / Specific Questions for this scientific research article.
   • What specific questions are the authors trying to answer by conducting this research study? What do they want to find out? What outcomes are they looking to measure?
5. Identify the approach
   • What do the authors say they will do/research to answer their specific questions?
6. Read the Methods Section. Draw a diagram for each experiment, showing exactly what the authors did.
   • Drawing a diagram for each experiment described will help you visualize the steps and consider each one so that you can better look up more information about the steps or pieces that you need to learn more about to understand fully.
7. Read the Results Section. Write one or more paragraphs to summarize the results for each experiment, each figure, each table.
8. Do the results answer the Research Questions / Specific Questions from #4?
9. Read the Discussion/Conclusion/Interpretation Section
   • ​​Do you agree with the authors' interpretation of the results?
10. Now, go back and read the Abstract
    • ​​Does it summarize what the authors said in the article? Does it match your interpretation of the article?
11. What do other researchers say about this paper, or the topic?
    • ​​How does this article fit in the conversation between researchers on this topic? (Find and read other articles, blog posts by other researchers; look for comments on the article you read.)
12. (optional) Go through the References / Bibliography / LIterature Cited at the end of the article - what articles did the authors cite to support their background, methods, and analysis of results? Do any of these seem important to read to understand the topic or methods more fully?

 

Further details to consider for each of the General Evaluation Criteria above:

Authority

Determining the knowledge and expertise of the author of information is an important aspect of evaluating the reliability of information. Anyone can make an assertion or a statement about some thing, event, or idea, but only someone who knows or understands what that thing, event, or idea is can make a reasonably reliable statement or assertion about it.
Some external indications of knowledge of or expertise are

--a formal academic degree in a subject area
--professional or work-related experience- businessmen, government agency personnel, sports figures, etc. have expertise on their area of work
--active involvement in a subject or organization by serious amateurs who spend substantial amounts of personal time researching and --studying that subject area
--organizations, agencies, institutions, corporations with active involvement or work in a particular subject area.
HINT: Be careful of opinions stated by professionals outside of their area of work expertise.

Accuracy

Establishing the accuracy, or relative accuracy, of information is an important part of evaluating the reliability of information. It is easier to establish the accuracy of facts than it is opinions, interpretations, or ideas. The more an idea, opinion, or other piece of information varies from the accepted point of view on a particular topic the harder it is to establish its accuracy. It may be completely accurate but corroborating it is both more necessary and more difficult.
An important aspect of accuracy is the intellectual integrity of the item.

--Are the sources appropriately cited in the text and listed in the references?
--Are quotations cited correctly and in context? Out of context quotations can be misleading and sometimes completely erroneous.
--Are there exaggerations, omissions, or errors?

These are difficulty to identify if you use only one source of information. Always use several different sources of information on your topic. Analyzing what different sources say about a topic is one way to understand that topic.
In addition to errors of fact and integrity, you need to watch for errors of logic. Errors of logic occur primarily in the presentation of conclusions, opinions, interpretations, editorials, ideas, etc. A number of these occur so frequently that they have been described and named. 

Objectivity

Establishing the objectivity, or relative objectivity, of information is the third aspect of evaluating information. While it is unlikely that anything humans do is ever absolutely objective, it is important to establish that the information you intend to use is reasonably objective, or if it is not, to establish exactly what the point of view or bias is. There are times when information expressing a particular point of view or bias is useful, but you must use it consciously. You must know what the point of view is and why that point of view is important to your project.
Another part of objectivity / point of view / bias worth considering is the question of your own personal point of view or bias. Having a point of view or bias on a topic is not bad - you should have developed some sort of opinion about your project. The question revolves around what you do with it. You can use your own point of view to shape your interpretation of the information available on a topic, as long as you:

--honestly acknowledge all points of view on an issue
--logically and fairly address those which disagree with your own
--do not ignore or dismiss information because it does not support with your point of view
--document your point of view to the best of your ability

Coverage

Determining exactly what the information covers and to what extent it fully covers the topic is another aspect of evaluating information. It is important to determine the extent of coverage. Information sources with broad, shallow coverage mean that you need to find other sources of information to obtain adequate details about your topic. Information sources with a very narrow focus or a distinct bias mean that you need to find additional sources to obtain the information on other aspects of your topic. The extent and depth of coverage can tell you a lot about the usefulness of the information you have found.
Some indications that coverage is adequate:

--clearly an overview of a topic or an aspect of a topic
--clearly an in-depth treatise or analysis of a topic
--clearly an in-depth analysis of a particular aspect of a topic
--presents new information about a topic

Currency

Determining when an item of information was published or produced is an aspect of evaluating information. The date information was published or produced tells you how current it is or how contemporaneous it is with the topic you are researching. There are two facets to the issue of currency.

--Is the information the most recent version
--Is the information the original research, description, or account
The question of most recent version of information versus an original or primary version can be a critical one.

For example: If you were doing a project on the survival of passengers in car crashes, you would need the most recent information on automobile crash tests, structural strength of materials, car wreck mortality statistics, etc

If, on the other hand, you were doing a project on the feelings of college students about the Viet Nam War during the 1960s, you would need information written in the 1960s by college students (primary sources) as well as materials written since then about college students in the 1960s (secondary sources).
A careful definition of your topic will indicate what level of currency you need. You can always discuss this with your instructor, another knowledgeable person, or ASK A LIBRARIAN for advice.

Audience

Identifying the intended audience of the information or product is another aspect of evaluating information. The intended audience of an item generally determines the style of presentation, the level of technical detail, and the depth of coverage.
For example, books on food sanitation written for children, for restaurant workers, or for research microbiologists will be very different even though they all cover the same topic.
Determining the intended audience of a particular piece of information will help you decide whether or not the information will be too basic, too technical, too general, or just right for your needs. The intended audience can also indicate the potential reliability of the item because some audiences require more documentation than others.

For example, items produced for scholarly or professional audiences are generally produced by experts and go through a peer evaluation process. Items produced for the mass market frequently are not produced by experts and generally do not go through an evaluation process.
 
Some indications of the intended audience are:

--highly technical language, complex analysis, very sophisticated/technical tools can indicate a technical, professional, or scholarly audience
--how-to information or current practices in "X" are frequently written by experts for practitioners in that field
--substantive and serious presentations of a topic with not too much technical language are generally written for the educated lay audience
--popular language, fairly simple presentations of a topic, little or no analysis, inexpensive tools can indicate a general or popular audience
--bibliographies, especially long bibliographies, are generally compiled by and for those doing research on that topic