Types of Evaluation Designs and 2 Types of Methodologies

¶ … criminal procedure and the idiosyncrasies of criminal practice vary widely from jurisdiction to jurisdiction" (Jaros, 2010, p. 445). If what Jaros states is true, then it is probably true as well that evaluating the different circumstances surrounding the commission of crime is also widely diverse in its practice. There are a number of methodologies that are used in various research including studies relating to the study of criminal justice and different aspects of that arena; two of the more commonly used methodologies employ quantitative and qualitative methods of research.

The quantitative methodology is used by researchers who are seeking to quantify certain areas of study or the results of such studies. Quantifying involves numbers, percentages and numerically evaluated data. One of the benefits that can be derived when using quantitative evaluation is that such a method provides numerical data for comparative studies. Comparative studies show specific numbers calculated from participant responses.

Although many experts can argue concerning discussions and conclusions of quantitative results, very few of these experts will argue with the numerical aspects of those conclusions However, not all experts agree that studies that use quantifying methodologies are effective in the area of criminal justice. A 2005 article by Max Travers suggested that a "provisional critique" of the criminal justice evaluation should include a footnote on the fact that evaluation research is "predicated on a misguided notion of objectivity" (Travers, 2005, p. 48).

Quantitative evaluations are supposedly objective since numbers are not normally considered to be capable of lying. Travers suggests that "most evaluation research is methodologically poor…when assessed by the standards employed in the academic peer-reviewed disciplines of criminology and sociology" (p. 48). Travers then continues by asserting that the lower standards required in quantitative methodology "does not offer the same intellectual interest or fulfillment as a traditional academic work" (p. 40).

Travers would much rather use qualitative methods in criminology studies and seeks to overcome researcher bias by gathering data on the why's of crime rather than the numbers. Qualitative studies are conducted in almost the exact opposite manner as quantitative studies. Qualitative studies are not looking to gather numerical data, instead, the focus of a qualitative study is on how the participants feel, think or perceive certain situations.

One of the benefits of a qualitative study is that it provides researchers with a deeper understanding of the subject's motivational thinking regarding the subject of the study. Oftentimes a qualitative study will provide the researcher with the opportunity to examine the depth of feelings towards the subject. A qualitative study is the type of study that many researchers use to study behavior or phenomena in natural environments. Many experts, however, disagree with Travers premise concerning the strength of qualitative vs. quantitative studies.

Don Weatherbum answers Traver's assertions by stating that while "quite a few criminologists appear to share his (Traver's) skeptism about the possibility of disinterested and objective enquiry" (Weatherbum, 2005, p. 416) but that in reality experts "routinely reject large numbers of studies on the grounds of poor design, poor statistical analysis, low sample sizes, weak measures of key variables, inadequate controls and so on" (p. 416).

Weatherbum contends that methodologies are not the problem since most academic researchers understand that the "purpose of the evaluation is to see whether a program is achieving its stated aims and that it is not always necessary to interview anyone to find out whether it is" (p. 418). Weatherbum is talking about a summative evaluation of specific programs regarding criminal justice.

According to Weatherbum, governments engage academia to answer a specific set of research questions when conducting criminal justice studies, and to accomplish that requires quantitative methodologies not the qualitative measures that assist a researcher to press his or her own agenda. There are numerous examples of using both research methodologies in the same study to complement each other.

One expert tells how a recent study using both was conducted by stating "the questionnaire was conducted through a face-to-face interview with the day-shift officer permanently assigned to the housing unit of each prisoner in the random sample" (Williams, Lindquist, Hill, Baillargeon, Mellow, Greifinger, Walter, 2009, p. 1287). These researchers used face-to-face interviews usually considered qualitative in nature, complemented by a questionnaire that is seen as quantitative in methodology and employed a random sample without a comparative group with is also seen as qualitative in style.

Oftentimes, the use of a certain methodology depends on what information is being sought by the researcher and whether the evaluation is considered formative or summative in nature. It also depends on what the results of the study are going to be used for. One of the most common uses of evaluations in the criminal justice arena can concern the criminals themselves. One expert writes "evaluations of competence to stand trial (CST) are the most common type of criminal forensic evaluation conducted" (Behavioral Sciences, 2006).

The CST offers an evaluators the opportunity to examine the circumstances around the crime and the individual who committed the deed. One expert wrote "it is clear that competence evaluations can considerably impact criminal proceedings and a defendant's liberty" (Warren, Murrie, Steiskal, Colwell, Morris, Chauhan, Dietz, 2002). Oftentimes evaluations can determine if the individual who committed the crime can be admitted to programs that are designed to help them. Programs such as the MERIT program help to "break the drug -- crime cycle by involving defendants in treatment and rehabilitation" (Merit, 2006).

Of course, to get admitted to such programs, the individual must submit to an intense evaluation by a medical professional. An irony not lost on this author is that the MERIT program was evaluated as to its effectiveness using a summative evaluation. The evaluation discovered that "of the 238 participants referred to MERIT, 72% were accepted, with half of those entering the program successfully completing it" (NSW, 2003). Other evaluative measures taken in the criminal justice system assists the system in determining whether the defendant or prisoner is being rehabilitated.

A recent study determined that "analyzing consequences of interventions, the impact of labels and the outcomes of treatment systems are essential" (Clancey, Howard, 2006, p. 380). Some experts worry that there are too many evaluations taking place and that the courts may be abdicating their responsibilities when it comes to deciding whether individuals who have committed crimes are either being imprisoned or sent to a rehabilitation center on the advice of the evaluator.

One study reports that "in all but one of 328 Alabama cases the court accepted the mental health professional's opinion -- a 99.6% agreement rate between the court and evaluator" (Zapf, Hubbard, Galloway, Cox, Ronan, 2004, p.30). It may make more sense to accept the evaluator's findings, and it might be the easy route for overcrowded court systems, but that does bring into account the evaluator's veracity, and the reliability and validity of the methodologies the evaluator employs.

One expert states "decision making…is effectively delegated to clinical evaluators making low-visibility and essentially unreviewed decisions pursuant to a vague, open-textured standard (Winick, 1995, p. 572). Another area of stringent evaluation in the criminal justice industry is the pre-trial chamber. One expert recently espoused that "the pre-trial chamber has a main role in preparing the confirmation of charges. It has to ensure that a confirmation hearing takes place within a reasonable period and that the evidence is disclosed between the parties to the proceedings" (de Beco, 2007, p. 470).

Other studies can help determine exactly what type of individual commits crimes and how to deter them from doing so. As an example, one study showed that "drug users, especially heavy users, commit a disproportionate amount of crime" (Chaiken & Chaiken, 1982) and another study shows that "studies of restorative justice demonstrate the complexity of implementing and evaluating an alternative approach to traditional criminal justice processing" (Bonta, Wallace-Capretta, Rooney, McAnoy, 2002, p. 320).

This particular evaluation seeks to assist criminal justice officials in deciding whether those convicted of crimes should face jail time or be remanded to an alternative treatment center. This study sought to determine whether "restorative justice can have a positive influence on the views of those who have been harmed; and regarding the impact on the recidivism of those who caused the harm" (p. 321). What is really interesting is that this particular study said that researchers must remain "mindful of the methodological limitations of many studies" (p. 321).

Each methodology does display certain limitations as to how to determine the effective use of criminal justice. Comparing the two methodologies is a straightforward exam. The qualitative researcher is armed with a method to explore by using open-ended questions, thoughts and conversations to determine the participant's thoughts and perceptions. The evaluator often uses in-depth interviews, focus groups and one-on-one scenarios to assist in determining how the participant's thought process is working.

The researcher can be armed with general questions and ideas based on the focus of the study, but the majority of time the hypothesis of the study will change based on the participant's beliefs, thoughts, observations and perceptions. The evaluator attempts to determine the positive or negative nature of the responses but quantifying such qualitative measures is sometimes quite difficult. An additional benefit to qualitative studies is that it may not require groupings with the participants.

Qualitative studies can also include how the participants feel or think about certain concepts and these feelings, thought and perceptions can often times change. This can lead to skewed results, especially if a pre-test and post-test is administered within the study. However, a quantitative study is often conducted using a pre-test and a post-test in order to compare results. The researcher may also use close-ended questions to elicit responses instead of the open-ended that is favored by the qualitative researcher.

Whatever methodologies are employed by the evaluator, care must be taken to ensure the validity and reliability of the study or evaluation. Whether a formative or summative evaluation is needed to ensure the evaluator accomplishes the goals and objectives of the court should also be considered. Formative evaluation is a "method of judging the worth of a program while the program activities are forming or happening, formative evaluation focuses on the process" (Bhola, 1990).

The use of a formative evaluation in the criminal justice system can assist both the court and the defendant in addressing problems as they are happening. Interventions (such as the MERIT program) can be applied to certain situations as they are discovered. If a summative evaluation was to be used instead (in these circumstances) then those interventions surely would not take place. Summative evaluations, on the other hand, are normally applied 'after the fact'. Using a summative evaluation to determine whether a program is working.