Comparing the research experience of traditional and professional doctoral programs in Australia

¶ … Australian Doctoral Students

Educational institutions around the world have undertaken various benchmarking initiatives in recent years in which they compare themselves to other universities on various metrics in an effort to document their current levels of performance and to identify opportunities for continuous self-improvement. This study on the research experience of doctoral students will compare students enrolled in traditional PhD and taught/professional doctorate programs such as DBA, EdD, using the "Postgraduate research experience questionnaire"; in addition, the study uses a qualitative method for interviewing respondents to compare the interview data with the publications "Research training in doctoral programs: What can be learned from professional doctorates" and "The doctoral education experience: Diversity and complexity." An introduction to the issues involved is provided in the introductory chapter, followed by a critical review of the relevant peer-reviewed, scholarly and organizational literature in chapter two. Chapter three describes more fully the study's methodology and chapter four presents an analysis of the statistical data and the concluding chapter provides a summary of the research, salient conclusions and recommendations for policymakers and doctoral students alike.

Chapter 1: Introduction

Statement of the Problem

Purpose of Study

Importance of Study

Scope of Study

Rationale of Study

Overview of Study

Chapter 2: Review of Related Literature

Chapter 3: Methodology

Description of the Study Approach

Data-gathering Method and Database of Study

Chapter 4: Data Analysis

Chapter 5: Summary, Conclusions and Recommendations

Table 1.

List of Figures

An Analysis of the Research Experience of Recent Australian Doctoral Students

Chapter 1: Introduction

Universities throughout the world are undertaking benchmarking exercises in which they compare themselves to other universities on various metrics in an effort to document their current levels of performance and to identify opportunities for continuous self-improvement (Marsh, Rowe & Martin, 2002). To this end, graduates of Australian higher education institutions who have completed a doctoral or research masters degree are invited by their institution and Graduate Careers Australia (GCA) on an annual basis to respond to the Postgraduate Research Experience Questionnaire (PREQ), typically around four months following their graduation, and there have been six PREQ surveys conducted since 1999 (PREQ, 2007). The PREQ instrument was developed in 1999 by the Graduate Careers Council of Australia (GCCA, now GCA) and the Australian Council for Educational Research (ACER) (ACER 1999). The instrument focuses on dimensions central to the postgraduate research experience in most fields of education. The items were based on a review of the literature and discussions within focus groups of research higher degree students, and they were revised on the basis of item performance during two trials. The PREQ queries doctoral graduates concerning 28 statements about seven aspects of their research degree (PREQ, 2007). The responses from these graduates are used to identify potential areas for improvement in supervision, and to provide information concerning graduate students' perceptions of the resources and the climate for research in an academic work unit (Ramsden, 1998).

The first 27 statements of the PREQ instrument are designed to gauge the quality of supervision, the intellectual climate of the department in which the respondent studied, the development of generic skills, the quality of the infrastructure provided by the university, the thesis examination process, and the clarity of goals and expectations; a final 28th item asks that graduates indicate their overall level of satisfaction with their research experience. For this purpose, a five-point Likert-scaled response is provided for all items that ranges from 'strongly disagree' to 'strongly agree'; in addition, a neutral 'does not apply' option is also provided. The analyses of the results to date indicate that the PREQ scales provide valid and reliable measurement of the targeted constructs (the PREQ report also notes that, by convention, item and scale statistics are reported in a -100, -50, 0, 50 and 100 PREQ reporting metric that corresponds with the five-point 'strongly disagree', 'disagree', 'neither disagree nor agree', 'agree' and 'strongly agree' response scale) (PREQ, 2007).

Results pertaining to graduates' perceptions of research supervision are captured by six PREQ items. While only 60.0 per cent of respondents agreed that they received good guidance in their literature search, 83.6 per cent agreed that supervision was available when they needed it. A scale score was computed for the 2,998 graduates who responded to at least four of the six items about supervision. The mean percentage agreement for the scale was 73.8, up from 72.8 per cent in the 2004 survey and 69.1 per cent in 1999. These figures suggest that the quality of supervision has increased slowly but steadily over the last seven years.

The five PREQ items about 'intellectual climate' measure whether graduates felt that their department had made efforts to integrate them into the academic community. At the national level, only 50.8 per cent of respondents agreed that the research ambience in the department or faculty stimulated their work. The highest level of agreement in this scale was to the statement that 'The department provided opportunities for social contact with other postgraduate students'. Since 1999 there have been very slight increases in graduates' perceptions of the intellectual climate surrounding their research, but it remains the area in which they express the lowest levels of agreement. The mean percentage agreement score for the 2,808 graduates who responded to at least four of the five intellectual climate items was only 58.2 per cent. It appears that Australian universities have a long way to go in terms of enhancing this important aspect of the research degree experience.

The PREQ includes five items designed to assess graduates' perceptions of the extent to which their course helped develop their generic skills. Such skills include the ability to transfer knowledge, apply analytical techniques to new situations, solve problems, plan work, and communicate effectively in writing. Over the years, and again in the 2005 survey, skill development has been the area for which responding graduates have expressed the highest levels of agreement. Nationally, 85.8 per cent agreed that 'As a result of my research, I feel confident about tackling unfamiliar problems' and 94.2 per cent agreed that 'My research sharpened my analytic skills'. The scale mean for this quality was 90.7 per cent, reflecting the perceptions of 2,959 respondents; the national scale means have varied very little over time, ranging from a low of 89.5 per cent in 2000, to 91.0 per cent in the 2004 survey (PREQ, 2007).

Statement of the Problem

In order to use benchmarking exercises effectively, there is a need for a comprehensive set of benchmark indicators that focus on appropriate outcomes; measure functional effectiveness rather than superficial criteria (i.e., are selected because they are easily "countable"); are systematically developed to have good content (and "face") validity; and differentiate between universities to provide appropriate standards as a basis of measuring excellence and continuous improvement (Marsh et al., 2002). Within such a benchmarking framework, it is particularly difficult to establish appropriate outcomes to measure the effectiveness of programs for PhD and postgraduate research students. Even at the undergraduate teaching level where there is widespread use of students' evaluations of teaching effectiveness, there is a limited basis for making comparisons across universities or across similar academic departments from different universities. At the PhD level, though, there remains a paucity of research into the systematic use of student surveys to evaluate the quality of PhD research supervision and, apparently, no research that attempts to compare effectiveness across large numbers of different universities (Marsh et al., 2002). Within the broader context of a benchmarking exercise there is a need for the examination of substantive issues relevant to the evaluation of research student supervision, the development and evaluation of an appropriate survey instrument for collecting the data, and methodological issues associated with the appropriate analysis of such data (Marsh et al., 2002).

Purpose of Study

The purpose of this study was to compare students enrolled in traditional PhD and taught/professional doctorate programs such as DBA, EdD, using the "Postgraduate research experience questionnaire."

Importance of Study

While research degrees often almost by definition involve learners identifying, analysing and reporting patterns in knowledge and practice, it remains critical that supervisors and others frame learning with appropriate pedagogical structures and expectations (Marsh et al., 2002). Likewise, it is an academic leadership responsibility to ensure that support and supervision arrangements available to graduate students are clearly articulated, and to be certain that supervisors are fully competent to fulfill their duties (Ramsden, 1998). Moreover, as McWilliam and her colleagues (2002) emphasize, "The professional doctorate has been, and continues to be, a paradoxical area of activity in doctoral education. On the one hand, the current increase in number of professional doctorate programs in Australian universities would seem to indicate a burgeoning interest in, and knowledge of, this relatively new form of doctoral education. On the other hand, there are many signs that these programs sit somewhat awkwardly within the postgraduate offerings of Australian universities, an outsider to all but those who manage, teach and study within a particular professional doctorate program" (p. 1).

Scope of Study

The literature review includes American, British and Australian sources, and the study will be targeted at one Australian university.

Rationale of Study

According to De Valero (2001), "Given the high costs associated with graduate education, the current national climate of diminishing resources for higher education, and an increased competition for these resources between undergraduate and graduate programs, understanding and examining the factors that affect students' ability to complete their degree requirements in a timely manner and considering the implications of these factors becomes crucial" (p. 341). In this regard, Neumann (2005) reports that, "The reasons for length of time to completion and non-completion rates are important considerations for universities and governments," and notes that: "Concerns about slow time to completion relate to the commitment of staff and other resources for extended periods, the desirability of concentrating graduate studies in a reasonable time period, the possibility that information obtained in the research may become obsolete before completion of the thesis, and delays in researchers moving on to other projects" (Neumann, 2005, p. 16). The growing body of evidence to date suggests that the reasons for degree noncompletions fall within three broad areas: (a) personal or family factors, (b) changes in career aspirations and development, - issues of quality of supervision and to these a fourth should probably be added: "Growing pressure for the PhD to be everything to everybody" (Neumann, 2005, p. 16). Furthermore, almost all data in higher education are inherently multilevel, even though this feature of the data is typically not taken into account; depending on the application, the different levels of analyses could include countries, geographic regions or states, different universities, faculties or departments within universities, and individual students or academic staff. As illustrated in the instant study, research, policy questions, data, and statistical analyses that are appropriate at one level of analysis may be inappropriate or even misleading when evaluated at another level of analysis.

Overview of Study

This study used a five-chapter format to accomplish the above-stated research purpose. The first chapter introduced the topics under consideration and provided the purpose, importance, scope and rationale of the study. Chapter two below provides a critical review of the relevant peer-reviewed, scholarly and organizational literature and chapter three describes more fully the study's methodology. Chapter four presents an analysis of the statistical data and the concluding chapter provides a summary of the research, salient conclusions and recommendations for policymakers and doctoral students alike.

Chapter 2: Review of Related Literature

Background and Overview.

The relationship between doctoral students and the supervisory faculty at graduate universities is inextricably related to how the research experience will likely unfold for the students involved. According to Delamont, Atkinson and Parry (2004), "Academic disciplines and their subject-matter are mutually constitutive. A discipline furnishes its members with definitions-often tacit-of what is 'thinkable', with appropriate assumptions as to what 'counts' as research problems, suitable research methods, definitions of research programmes and the approved modes of graduate student research" (p. 173). Given the importance of this relationship, it is reasonable to posit that the better the supervision provided, the more favorable the doctoral students will perceive their university experience and the support they received during their research experiences. According to Walfish and Hess (2001), "Graduate school course work emphasizes the preparation of graduate students for research" (p. 167). In this regard, Greene, Hardy and Smith (1995) report that, "Research training plays the central role in the present definition of doctoral education. But as new graduates are discovering, research inside the ivory tower often has little in common with what happens outside. In the past, university research has been a two-for-one bargain: cutting-edge research paid for with the same dollars that educated the next generation of researchers" (p. 59).

Doctoral Students' Perception of Their University and Research Experiences.

While there is a growing body of research concerning undergraduate students' evaluations of classroom teaching effectiveness and some research on the quality of supervision of research and PhD students, there remains a paucity of current research concerning the reliability and validity of PhD students' evaluations (Marsh et al., 2002). According to the most recent the Report of the Postgraduate Research Experience Questionnaire, "Simultaneous analysis of data from the 2003, 2004 and 2005 surveys showed that there is quite a substantial amount of variation among the annual PREQ scores for many if not most institutions. While such variation was most closely associated with those institutions that provided only very few responses, this was not always the case and it is apparent that most institutions' scores vary over the years. The composite institution-level estimates show that there are generally only statistically significant differences between institutions at the top and bottom of each distribution, and that there are only a few institutions which are routinely located in the top five for all facets of the experience measured by the PREQ" (PREQ, 2006, p. viii).

One intended purpose of PhD students' evaluations is to provide informative feedback that will lead to the improvement of research supervision. There is clear evidence that feedback from students' evaluations of teaching, coupled with appropriate consultation, can lead to improved teaching effectiveness. For example, in a study by Marsh and Roche (1993), randomly assigned intervention- and control-group teachers completed self-evaluations and were evaluated by students before and after the intervention. An essential component of the intervention was a set of teaching strategy booklets -- one for each factor on the student evaluation instrument. Teachers selected the factor to be targeted in their individually structured intervention and then selected the most appropriate strategies from a book of strategies relevant to that factor. The intervention teachers improved significantly more than control group teachers. Furthermore, for the intervention group (compared to control group), targeted dimensions improved substantially more than nontargeted dimensions. The study demonstrated that feedback from students' evaluations of teaching and consultation are an effective means of improving teaching effectiveness.

These authors also note that it is important to keep in mind that this intervention can only be conducted with a well-designed, multidimensional instrument and that the specificity of the intervention effects to the targeted dimensions further supports the construct validity of multidimensional students' evaluations of teachings. The lessons from this research that may be useful for improving the quality of postgraduate supervision based on PhD students' evaluations are that the feedback needs to be specific to each supervisor; supervisors may need concrete strategies about how to improve their supervision; and this feedback may need to be complemented by a trained consultant. Even when supervisors are motivated to improve their supervision and have feedback about their strengths and weaknesses, they still require professional assistance on how actually to improve their supervision (Marsh et al., 2002).

The PREQ represents a contentious policy issue in higher education in Australia for three reasons:

It is used to develop performance indicators with comparisons between universities;

There is the threat of funding being granted or reduced as a result of performance indicators; and,

There is some disagreement about the validity of the PREQ, especially since many supervisors have an insufficient number of postgraduates to satisfy a reliability criterion (and they cannot be identified for ethical reasons) and the intended unit of analysis is broad 'fields of study' within a university (rather than supervisors) (Waugh, 2001).

A concomitant analysis of data from the 2003, 2004 and 2005 surveys demonstrated a significant amount of variation among the annual PREQ scores for many, if not the majority of Australian institutions; although this level of variation was most closely associated with those institutions that provided only very few responses, this was not always the case and it is apparent that most institutions' scores vary over the years (PREQ, 2007). The composite institution-level estimates show that there are generally only statistically significant differences between institutions at the top and bottom of each distribution, and that there are only a few institutions which are routinely located in the top five for all facets of the experience measured by the PREQ (PREQ, 2007).

Review of Recent Australian Government Publications Concerning Doctoral Programmes and the Research Experience of Doctoral Students.

In their publication, "Researching training doctoral programmes: what can be learned from professional doctorates?," McWilliam and her associates discuss the relatively recent experience of offering doctoral education through professional doctorate programmes as a contribution to the improvement of doctoral education in Australian universities. The evaluation focused on the extent to which such programmes had developed practices for sustaining closer collaboration between universities and industry. According to this authors, "Doctoral education in Australia is currently under pressure to become more industry focused. When set against the 800-year history of the PhD, the professional doctorate is a young doctorate, the first being set up in Australia within the last two decades. The nature and status of professional doctorates remains unclear to many, including a number of university administrators of research training, as well as government and industry personnel. The fact that 61 per cent of professional doctorate programs fall under the classification of 'research' higher degrees is not widely understood. Moreover, the 131 programs we found to exist in 35 of the 38 Australian public universities, exhibit a wide range of structures and features.

While there is strong evidence of an increase in the number of professional doctorates being offered in Australian universities, and there is some evidence of innovation in a number of professional doctorate programs, it appears that industry-focused doctoral education is still in its infancy. With a few exceptions, neither industry nor universities were engaging in any significant way to develop sustainable partnerships to serve and support the work of doctoral education. While the government White Paper Knowledge and Innovation (Kemp, 1999a) is clearly having an impact on universities in terms of active improvement of the quality and accountability of research training, industry remains to be engaged in any systematic or sustained way.

The majority of operational professional doctorates programs may currently be characterised as having 'surface' level links, in that they exhibit the following features:

particular industry or group of industries is the source from which most clients come and to which they return;

There is some attempt made to involve non-academic individuals from industry and/or a professional group in course delivery, supervision or assessment (this is likely to be limited and ad-hoc);

Research and research activities are workplace-based; and,

Marketing materials stress the value of the program to targeted professions (McWilliam et al., 2002).

In her study, "The doctoral education experience: Diversity and complexity," Neumann (2005) provides the following description of the study's methodology: "Doctoral students' educational experiences across four discipline groups and six universities were examined. Over 130 interviews were conducted, two-thirds with students, the remainder with experienced supervisors, departmental and faculty coordinators and deans, postgraduate association presidents and senior managers. Consideration was given to type of doctorate (PhD or professional doctorate), mode of enrolment (full or part time), and stage of research (early, middle or late). The intention was to give doctoral students a 'voice' and to enhance the understanding of the complexity and diversity of their educational experiences" (p. xiii).

The study by Neumann found that doctoral experiences are shaped by a range of contextual considerations, including:

Mode of enrolment makes a difference to the intensity of the experience while different stages of enrolment highlight different aspects of the research process;

The type of doctorate does not appear significantly to affect the nature of the doctoral experience;

Disciplines shape the doctoral experience and institutional contexts can influence disciplinary perspectives;

Financial and resource issues are strongly differentiating factors, both between full and part time students and between students in the hard and soft disciplines;

The opportunities for student feedback to their institution on their experiences and satisfaction are limited and inconsistent; and,

The changing and flexible nature of the doctorate is evident from the respondent interviews (Neumann, 2005).

This author also cites some fundamental structural differentiation between PhDs and professional doctorates, including the following:

The major difference between the PhD and the professional doctorate lies in the mode of entry;

Students in professional doctorates maintain that their research could have been undertaken within a PhD;

Both academics and students believe that the PhD is sufficiently flexible to include coursework where needed and to accommodate non-traditional topics;

Both types of doctorate involve either coursework provision or research only. Around half of all doctorates in the study specify coursework and coursework is found across all discipline groups, although its purpose varies;

There is a divergence of view on the appropriateness of the U.S. PhD model for Australian universities, but the need is argued for more flexible entry and exit points within the doctorate;

Very few students enter a doctorate with a specific career in mind and while an academic career is never far from many of their minds, it is seen as an undesirable or unrealistic aspiration for the majority;

Students following professional doctorate programs do not see the qualification as particularly relevant to advancing their careers;

There is seen to be a largely unmet need for universities to offer career development opportunities for doctoral candidates, but the only career development opportunity commonly available is undergraduate teaching (Neumann, 2005).

In addition, Neumann identified disciplinary variations within the doctoral experience, including the following key findings:

Subsidisation of doctoral study from external funds accounts for the major resource differences between the hard and the soft disciplines;

up" scholarships are seen as necessary to attract students into the hard disciplines, especially the hard applied ones;

Hard disciplines are more likely to avoid recruiting part time students and certain categories of overseas students;

There can be tensions in industry funded doctoral research but in the hard applied fields they are manifest in the industry partners; expectations that students will work on non-thesis related activities.

In soft applied fields topic definition emerges as a major issue;

The absence of a departmental role in acculturation opportunities most noticeably affects students in soft disciplines and part time students across the disciplinary range (Neumann, 2005).

In some of the cases studied by Neumann, there were also institutional factors identified that can affect the doctoral experience, including the following:

Dissatisfaction with resource allocation and support is stronger in some universities than in others;

Satisfaction with acculturation opportunities is stronger in research intensive universities;

All universities in the study have in place policies relating to doctoral candidature. However they are differentially positioned for quality assurance; and,

The readability and accessibility of university policy statements and handbooks for doctoral students is highly variable (Neumann, 2005).

Finally, other important issues identified in the study by Neumann include:

There is strong emphasis on initial topic selection and reducing the time taken for topic refinement across all discipline groups;

There is evidence of prestructured "tick the boxes" topics in some hard fields;

The institutional allocation of funds for doctoral students is considered to be opaque;

Universities are monitoring progress more closely, with the intention of providing students with more meaningful and regular feedback on progress, such as through academic transcripts;

The introduction of an annual report form is designed to provide students an opportunity to raise concerns and panel reviews without the supervisor are particularly welcomed by students.

Universities are generally positive about government moves to improve completions within time, but are concerned that the reduced timeframe may raise potential problems of inflexibility (Neumann, 2005).

The author reports that the greatest differences identified in "The Doctoral Education Experience" were found between the hard and soft disciplines: "The hard disciplines, such as physics, biology and engineering, presented much tighter knowledge structures and a more gregarious social organisation than those of the soft fields, history, politics, cultural studies, management, law and education. Supervisors and departments in hard disciplines had little difficulty in specifying available research topics for students. In the soft disciplines it was far more a case of "make your own niche" within the loose parameters of a department's research emphases. It was not unusual, moreover, for an academic specialising in, say, American history to supervise students undertaking projects in Australian history, whereas a much closer matching of research was expected in hard fields" (p. 27).

Differing philosophical approaches and attitudes could also be identified. Within the soft pure fields, philosophical differences appeared in the form of competing disciplinary paradigms. The pressure on students to adopt a postmodernist perspective was felt acutely in one university and to some degree in a second but not at all in the others. In this case, one of the newer universities, students were attracted to enroll in doctoral study with a new professor having a reputation for open mindedness and a tolerance of theoretical diversity. The student profile in all fields of four of the universities found greater differentiation in mode of enrolment than in the remaining two; the latter universities are described in the report as being ery research intensive and one of the very large institutions had the most traditional student profile in the study (a larger proportion of males, full time students and students enrolled straight from honours) (Neumann, 2005). This trend was most discernable in the hard fields, and their student recruitment strategies were specifically developed to maintain this profile and position them well for meeting time requirements for completions as shown in Figure ____ below (Neumann, 2005, p. 29).

Figure ____. Mode of enrolment by university.

Source: Neumann, 2005 at p. 155.

According to Biegel, Hokenstad, Singer and Guo (2006), criticisms of part-time doctoral education include the following:

Greater length of time for students to complete their degree;

Lack of interaction of part-time students with other doctoral students thus limiting the opportunity for peer learning (which is believed to be a critical element of doctoral education);

Lack of interaction with faculty outside the classroom;

Lack of opportunity to attend research seminars and presentations outside of the classroom;

Lack of development of students into research scholars; and,

Lack of socialization into scholarship; and lack of socialization into the profession (p. 231).

Chapter 3: Methodology

Data-gathering Method and Database of Study

The study used a variety of sources to develop the relevant background and overview of the issues involved, but focused on those reports issued by the Australian government as part of its PREQ and like-minded initiatives. These resources included those developed during the course of the literature review research as well as interviews with.

Chapter 4: Data Analysis

The most recent PREQ survey was distributed to a total of 6,007 research degree graduates as part of the 2005 survey, and a total of 3,019 valid PREQ responses were received in time for inclusion in the report, providing a response rate of 50.3 per cent, a minor increase from the 49.3 per cent rate reported in 2004 (PERQ, 2007). The response rates for the 39 higher education institutions participating in the survey ranged from 3.4 to 100.0 per cent. Comparison against population data indicated that the sample of secured responses was representative in terms of broad field of education, respondent sex and age, home language background and disability, but not in terms of attendance type, mode of study or permanent residence status (PERQ, 2007).

The qualities of research experience measured by the PREQ are listed in Table 1 below; this table lists the title of the scale used to measure each of these qualities, provides a brief description of the focus of each scale, and shows the number of items in each scale.

Table ____.

Qualities of the postgraduate research experience measured by the PREQ.

Focus

Items

Supervision the accessibility and quality of research degree supervision

Intellectual Climate the learning community and conditions provided by the institution

Skill Development the extent of generic analytical and communication skill development

Infrastructure the quality of learning infrastructures such as space, equipment and finance

Thesis Examination whether the examination process was timely, fair and satisfactory

Goals and Expectations the clarity of learning structures, requirements and standards

Overall Satisfaction overall satisfaction with the recently completed degree

Source: PREQ, 2007, p. 1.

The PERQ surveys doctoral students' perceptions across a range of major issues as reflected in the figures below; the mean for the overall category is reflected in the first bar to the left.

Figure ____. PREQ Supervision results.

Figure ____. PREQ Intellectual climate results.

Figure ____. PREQ Skills development results.

Figure ____. PREQ Infrastructure results.

Figure ____. PREQ Thesis examination results.

Figure ____. PREQ Goals and expectations results.

Figure ____. PREQ Overall satisfaction results.

Source: Based on chart data in PREQ, 2007 (see complete graph at Appendix ____).

The highest levels of agreement in the 2005 PREQ were those for PREQ01 ('Supervision was available when I needed it') and PREQ21 ('My supervisor/s provided helpful feedback on my progress'); moreover, the relative orderings of the items has not changed since the inception of the PERQ (PREQ, 2007). The results of the most recent PREQ also showed, among other things, that most research graduates tended to agree that their research degree developed their skills, that goals and expectations were clear, and that they were generally satisfied with their degree and thesis examination. Of the seven qualities measured, the lowest levels of agreement were expressed in terms of infrastructure and intellectual climate. These two qualities were associated with the most varied patterns of national response (PREQ, 2007). According to the most recent report, "Graduates' perceptions of the intellectual climate in which they learned are important, as they reflect their socialisation into a community of scholarship and inquiry. The idea of intellectual climate is complex, and incorporates social, formal academic and more personal aspects of experience. In terms of the 2005 PREQ, around two-thirds of responding graduates agreed with PREQ05 ('the department provided opportunities for social contact with other postgraduate students'). Of the five dimensions of intellectual climate measured, the least agreement has tended to be expressed to PREQ23 ('the research ambience in the department or faculty stimulated my work')" (PREQ, 2007, p. 26).

The development of certain generic skills is also important for the majority of research degrees. Concerning skill development, PREQ respondents agreed most with PREQ14 ('My research sharpened my analytic skills'). At the national level, the lowest levels of agreement were associated with PREQ20 ('Doing my research helped me to develop my ability to plan my own work') and PREQ26 ('as a result of my research, I feel confident about tackling unfamiliar problems'). Indeed, over the years these two ends of the spectrum appear to have mapped out two distinct components of skill development within research degrees (PERQ, 2007, p. 26).

While research study involves many formative kinds of ongoing assessment and review, the PREQ thesis examination scale focuses squarely on the final component of postgraduate research study. It may perhaps be of little surprise that survey respondents were generally very happy with the thesis examination process given that surveyed individuals have by definition successfully completed their course. Almost 90 per cent of respondents agreed with PREQ02 ('the thesis examination process was fair'); however, the report also shows that approximately two-thirds of graduates agreed with PREQ25 ('the examination of my thesis was completed in a reasonable time'). This figure has been trending upwards over the last three years, suggesting that institutions may be streamlining examination processes (PREQ, 2007).

There most recent PREQ report also notes that there was very little variation between the three PREQ items administered to probe graduates' perceptions of the clarity of learning goals and expectations. There has also been relatively little movement in the levels of agreement to each of these items over the last six survey administrations. In recent years, there have been slightly more respondents agreeing with PREQ04 ('I developed an understanding of the standard of work expected') than with PREQ11 ('I understood the required standard for the thesis') or PREQ19 ('I understood the requirements of thesis examination') (PREQ, 2007).

While results for PREQ28, the single item used to measure graduates' overall satisfaction with course quality ('Overall, I was satisfied with the quality of my higher degree research experience'), trended upwards between 2002 and 2004, there has been a very slight downturn in the 2005 survey. The figures are fairly high and stable, however, with the average over the six surveys being 83.0 per cent and the figure for 2005 being 83.4 per cent. In sum, more than 75 per cent of responding graduates reported being satisfied with the quality of their higher degree research experience (PREQ, 2007).

The foregoing analysis of the item-level scores in the most recent PREQ showed that research graduates tended to be least satisfied in terms of the support offered to help them integrate into university research communities. They expressed the highest levels of agreement to items which address graduates' perceptions of personal generic skill development. The lowest levels of missing data were provided for items concerning supervision and learning expectations, while between 7.6 and 9.3 per cent of respondents either omitted or used the "not applicable" responses concerning the provision of learning supports and infrastructures (PREQ, 2007).

To identify differences between research disciplines, Table __ and Figures ____ through ____ provide PREQ scale mean percentage agreement scores for the 15 largest narrow fields of education for research doctorate graduates

Table ____.

PREQ scale mean percentage agreement scores for the 15 largest narrow fields of education for research doctorate graduates.

NFOE n Super-vision Intellec-tual Climate Skill Development Infra-structure Thesis Exam. Goals & Expectations Overall Satis. Psychology 131 75.7-52.1-91.6-68.7-77.1-92.6-84.0 Gen. Med. 86-75.5-65.7-95.4-79.8-82.8-92.6-94.1 History 73-73.2-53.7-86.6-48.6-78.8-88.6-79.5 Educ. Studies 54-85.3-52.4-93.2-70.9-79.6-94.8-87.0 Biochem. & Cell Bio. 53-69.2-64.2-91.2-75.7-81.8-89.3-83.0 Med. Sci. 51-77.3-65.7-94.1-77.6-85.0-94.1-82.4 Studies in Human Soc. 47-74.6-40.2-91.1-50.3-80.5-91.5-80.4 Agricult. Science 43-84.5-62.8-92.9-78.2-85.7-88.4-93.0 Business Manag. 43-69.2-57.8-75.9-58.7-79.1-84.9-73.2 Political Science 38-71.9-48.4-85.4-69.1-79.3-88.6-88.9 Physics 37-64.1-63.8-91.7-75.6-87.4-93.7-91.9 Chemical Sciences 36-76.6-68.3-91.1-77.8-81.5-92.6-88.6 Biological Sciences 34-70.1-52.9-94.1-63.3-71.7-88.2-93.9 Botany 34-77.5-67.9-95.8-81.3-82.4-91.2-90.9 Natural & Phys. Sciences 33-76.2-74.4-95.8-81.8-80.8-98.0-90.9

Source: Based on tabular data in PREQ, 2007 at p. 40.

Figure ____. PREQ scale mean percentage agreement scores for Psychology.

Figure ____. PREQ scale mean percentage agreement scores for General Medicine.

Figure ____. PREQ scale mean percentage agreement scores for History.

Figure ____. PREQ scale mean percentage agreement scores for Educational Studies.

Figure ____. PREQ scale mean percentage agreement scores for Biochemical and Cell Biology.

Figure ____. PREQ scale mean percentage agreement scores for Medical Science.

Figure ____. PREQ scale mean percentage agreement scores for Studies in Human Sociology.

Figure ____. PREQ scale mean percentage agreement scores for Studies in Agricultural Science.

Figure ____. PREQ scale mean percentage agreement scores for Studies in Business Management.

Figure ____. PREQ scale mean percentage agreement scores for Studies in Political Science.

Figure ____. PREQ scale mean percentage agreement scores for Studies in Physics.

Figure ____. PREQ scale mean percentage agreement scores for Studies in Chemical Sciences.

Figure ____. PREQ scale mean percentage agreement scores for Studies in Biological Sciences.

Figure ____. PREQ scale mean percentage agreement scores for Studies in Botany.

Figure ____. PREQ scale mean percentage agreement scores for Natural and Physical Sciences.

Source: Based on tabular data in PREQ, 2007 at p. 40.

Interview Summaries

Chapter 5: Summary, Conclusions and Recommendations

Summary

Conclusions

Recommendations