Data acquisition

Bartel, Lee (2006). Trends in Data Acquisition and Knowledge Development. in Colwell, R. (Ed). MENC Handbook of Research Methodologies. Oxford University Press.



TRENDS IN DATA ACQUISITION AND KNOWLEDGE DEVELOPMENT

Lee R. Bartel

INTRODUCTION

Research is "a systematic process by which investigators gather information, organize it in a meaningful way, and analyze and interpret it" (Asmus & Radocy, 1992, p. 141). While this definition does not indicate why investigators gather and work with information, a clue is in "a meaningful way"— the research process should be meaningful. Organizing, analyzing, and interpreting is a process of developing knowledge. Another way to express this is to say that research is a systematic process of data acquisition and knowledge development. The terms "data acquisition" and "knowledge development" resonate clearly with the current computer lexicon (Knowledge Discovery in Databases or KDD). This is intentional because technology is one of the most influential forces affecting research in the past 30 years. I have selected these terms to serve as "lenses" with which to examine trends in research.

A trend is direction of movement, a course, an inclination. To identify a trend means to note change over time and then to project the "direction" and "movement" of the change into the future. I am not a futurist. Nevertheless, I identify seven trends in research: (1) Construct complexity—researchers are taking into account more dimensions, facets, or connections as they construct research problems and build conceptual frameworks for studies. (2) Ethical complexity—recognizing and honoring the complexity of constructs such as teaching and learning takes the researcher into dimensions of students' lives that have strong ethical implications. This trend intersects with a growing attention to individual and human rights and requires compromise with traditional design principles. (3) Methodological complexity — the pursuit of satisfying answers to questions consisting of complex constructs requires greater diversity and multiplicity of method. (4) Data complexity—multi-variate and multi-method studies result in complexity in data as a whole and within specific types of data such as video recordings. (5) Analytical complexity—data and construct complexity dictates analytical complexity in the process of fitting most appropriate analyses with specific question and data, and developing knowledge through patterns, relationships, similarities and differences, etc. (6) Representation complexity—technological developments are providing new and easier ways of presenting data, data reductions, and knowledge claims while epistemological argument is redefining what counts as “knowledge” with the result that knowledge representation is becoming increasingly complex. (7) Dissemination complexity—web site postings, e-journals, teleconferences, video journals, CDROMs are current indicators of dissemination opportunity, choice, and complexity. Perhaps there are two mega trends: (1) the use of technology and (2) increasing complexity in all facets of the process. Due to constraints on length our focus is principally on three of the areas in which complexity is increasing.

I draw attention to complexity because the natural function of research is to simplify. Complexity, a property of an object, idea, phenomenon, organism, or system, stems from "compoundness"—multiple parts, layers, or dimensions. In addition, complexity lies not only in an entity as multiple components, but also in the interconnectedness or interwoveness of the parts, each of which may depend on or influence the other, neither of which is in a fixed relationship or quantity, nor is related to a fixed behavior. For example, learning is complex. Learning depends on the many attributes of the learner, the home, the teacher, the curriculum, the school, and the environment. Each of these in turn is complex, with related and dependent dimensions. Considering learning without considering these interconnected dimensions would be to deny the essential nature of learning. Complexity, as a quality of being differentiated yet integrated, is commonly regarded as "the direction in which evolution proceeds" (Csikszentmihalyi, 1993, p. 157). Since I am looking at trends or changes in research, my perspective is a view over time.

Explanation of changes in complexity over time, or simply explanation of complexity itself is the focus of Complexity Theory. Although I am not making a direct application of Complexity Theory (or Chaos Theory), the implication of my observations and explanations may have some commonalities with it. An important assumption of Complexity Theory is its inherent dialectic of simplicity and complexity: “what looks incredibly complicated may have a simple origin, while surface simplicity may conceal something stunningly complex” (Briggs & Peat, 1999, p. 79). In its application in research the most important premise of Complexity Theory is that complexity and simplicity are not so much “inherent in objects themselves, but in the way things interact with each other, and we, in turn, interact with them” (Briggs & Peat, 1999, p 89).

To understand the seven trends more fully, we need to reconceptualize research in terms of "knowledge development" and "data acquisition." This may transcend the qualitative - quantitative division by focusing on essential processes of research.

KNOWLEDGE DEVELOPMENT

Constructs as Knowledge

Graduate student Pierre plays with questions of creativity for a research study. He wonders whether it is an inherent potential or learned ability and then realizes he needs to clarify what he means by "creativity." Pierre reflects on instances when he was told he was especially creative. He remembers the feelings of anguish when he had to improvise in music class and the sense of euphoria when he crafted an exceptional poem. He describes the characteristics of some people he considers creative. He realizes the meaning "creativity" has in his mind is an accumulation of at least (1) personal experiences that were designated "creative," (2) demonstrations of others engaging in "creativity," (3) stories involving "creativity," and (4) the meanings of other words associated with creativity like spontaneous, artistic, novel, unique, and special. Pierre has constructed this set of meaning connections with creativity over many years. A complex construct relates to and depends on a multitude of other constructs; it constantly and continuously develops.

In research the term "construct" generally means a defined concept, a formalized description of an informal notion, a distillation of an idea so that it can be operationalized with a "test" or checklist or categorical assignments. I use the term "construct" to reflect individuals’ active mental involvement in building a definition, the engagement in a constructive process to give clear meaning to an idea. The mind selects some things, rejects some things, connects some things. For example, "creativity" as a commonly used term, clearly has a shared meaning that communicates something to many people in our culture. But, to study it in a formal research context, a researcher might define creativity in a way related to manipulation of materials, to certain procedures, or to observable products with defined characteristics; in other words, as an operational definition.

An infant’s first recognitions of a violin and its associated sound, Pierre's association with "creativity," or the researcher’s theoretical construct all share the feature of being a locus of meaning constructed by a mental process of selection and association from a host of "perceptual data." "Cronbach has concluded that all human action is constructed, not caused, and that to expect Newton-like generalizations describing human action, as Thorndike did, is to engage in a process akin to 'waiting for Godot'" (Donmoyer, 1990, p. 178, referring to Cronbach, 1982).

Basic constructs exist in descriptive, denotative "noun" form to which connotations are added through active experience. Qualitative descriptive constructs—adjectives and adverbs—emerge from attributions or characterizations of basic constructs and tend to take on values on a dichotomous continuum (e.g., hot—cold, excellent—mediocre, loud—soft). Explanatory, or “theory-making,” constructs develop as mental structures that encompass the links and relationships among constructs. Kelly (1955) refers to these in his Psychology of Personal Constructs; they are also the constructs described by Schema Theory. The relational, explanatory construct can be seen as an hypothesis or, with development, a theory. It acts as a means of making predictions about the world. As the person experiences a stream of sensory data from any or all the bodily senses, these hypothesis constructs offer "explanations" of the data by allocating them to existing constructs and relationships among constructs previously experienced, or introduce "modifications" to existing constructs.

In most cases in this chapter I refer to descriptive and explanatory level constructs simultaneously. Where necessary I will differentiate by referring to schema-type constructs as "explanatory constructs." For example, knowledge is the sum of consciously and intentionally accessible constructs (in this case both descriptive and explanatory types). In addition, I assume that constructs exist not only linguistically as word meanings, but, in relation to all perceptual modes (Phenomenal, Linguistic, Kinaesthetic, Affective, and Performance (Perlmutter & Perkins, 1981)) and to all forms of intelligence (Gardner, 1999).

Knowledge development and understanding is concerned with: (1) increasing the complexity of constructs (adding pages with more on them to the web site), (2) increasing the associations among constructs (creating more links between sites), (3) increasing the complexity of explanatory constructs, (4) increasing the extent of construct consciousness or clarity, (5) making associations more readily accessible, and (6) increasing facility at accessing and using the links, thereby (7) increasing the accuracy of explanatory constructs to anticipate and predict the future. The purpose of research is knowledge development.

Constructs in research design

Fine (1998) states, "What science makes is knowledge, which includes concepts and theories, along with things and even facts" (p. 4). Research is about constructing knowledge. Especially in the social sciences where most music education research is methodologically situated, research works with constructs. We rarely work with objects or matter, as physicists do, where only the most radical constructivists might argue properties do not exist apart from representations of them. The things that concern music educators are more likely theoretical constructs such as musicianship, creativity, aptitude, preferences, attitudes, abilities, effectiveness, competence, artistry, achievement, excellence, learning, response, or understanding, for which we normally use "indicators" or representations from which we infer what we can know. What our research efforts are aimed at, then, is to refine, elaborate, or clarify our constructs.

Research design is generally a data acquisition and analysis plan for the purpose of developing knowledge. It is possible to make a plan to acquire data without a clear view of what knowledge exactly will be developed, but in most cases, even in research trying to develop "grounded theory," the researcher has a prediction of the type of knowledge, the category of constructs, that will emerge. The way to acknowledge this prediction is in the form of a question. A real question, one without an already formulated answer, is the means to examining the nature and adequacy of constructs. A question demands a method of answering. The plan to secure an answer is the research design.
The constructs a researcher will think to question, the kinds of questions, and the kinds of methods admissable as legitimate for answering those questions are culturally influenced— whether that is a religious, societal, or research culture.

A researcher may decide to further knowledge in a defined area, on one facet of a problem, or a simple construct, but must recognize that resulting "incompleteness" and thus the possible irrelevance of the knowledge to the holistic and complex essence of education and life. Simplifying a construct in research may clarify method and design but risks distortion. Imposing a favorite method or design on possible questions thereby eliminates potentially useful and important questions. In addition, exploring phenomena one or two unidimensional constructs at a time is a slow process within a changing context. Cronbach (1975) stresses this problem of social science research: "The trouble, as I see it, is that we cannot store up generalizations and constructs for ultimate assembly into a network. It is as if we needed a gross of dry cells to power an engine and could only make one a month. The energy would leak out of the first cells before we had half the battery completed. So it is with the potency of our generalizations" (p.123).

Construct contemplation and analysis is of greatest importance as a step toward research design. The next step, deciding what data will provide the sources of information from which knowledge can be advanced is just as crucial. What cannot be over-emphasized is that all that can be obtained is a "representation" of the construct under study. The number on an ability test is a representation (or estimate in statistical terms) of ability-related tasks as defined by a theory-driven testmaker. If a student is observed demonstrating behaviors that are physical, verbal, or musical representations one may infer something about that student’s ability. An interview with a person will obtain a linguistic representation, as well as some gestural representation, of an internal state.

Researchers are concerned about construct validity, i.e., the extent to which the data being obtained adequately represent the construct theorized. If we are studying creativity, data might be observations of the manipulation of materials, of certain procedures, or of observable products with defined characteristics. Such data, selected to relate to a theoretical construct, may well ignore many aspects of the "creative experience" like the creator's feelings, some of the thought process occurring during the process of creation, the kinesthetic abilities limiting realization of an idea, or the ideas rejected during the process. A serious concern for researchers of creativity is one of construct validity, i.e., do the dimensions included in the "observables" or the "test" really match the richness of meaning we give the word creativity. The problem may be that a complex construct is simplified into a "theoretical construct" that is too one-dimensional, too reliant on the immediately evident, or too poorly understood by the researcher.

Theoretical construct validity depends both on how well one knows the "common" construct and on how faithful the selected representations are to the "reality" they are taken to represent. If the constructs are thoroughly known, questions can be anticipated and the research design improved.
If the complexity entailed in a situation is not understood, or if the researcher believes there may be unanticipated questions from incomplete construct-awareness, the research may begin exploring the environment and attend to the constructs being "tweaked" by the incoming data. The researcher then is becoming aware of the constructs employed in classifying and filing incoming data. Whether through formal “instruments” or mentally analyzed informal observations, this is a process of data acquisition and knowledge development

Constructs and analysis

Given that the data in a large database are of various types, a program to make sense of them must be flexible enough to accommodate the diversity. One way is to apply to software the concepts of Kelly's (1955) "Psychology of Personal Constructs." The fundamental principle holds that a person's psychological processes are directed by explanatory constructs. Kelly sees the main function of explanatory constructs (or schema) as creating personal predictions about the world. The person then tests the predictions, and the construct is reinforced or changed depending on how well the experiential reality matches the prediction. Data mining software begins with constructs of postulated relationships among data. These anticipatory constructs are then confirmed, altered or mutated in an iterative process. Knowledge constructs are "discovered" in the database by the computer.

In a similar manner, the researcher conducting a case study of a classroom acquires a constant stream of "data." These data are processed with existing explanatory constructs—the result of all previous "learning" (knowledge development). As the researcher's explanatory constructs encounter data that cannot be accommodated by existing structures, new ones are created or existing ones modified. This process of the mind is essentially the same as that of normal living that results in our "informal" knowledge or common sense. However, when the researcher makes the process conscious by making the construct structure conscious, it is clearly recognizable as an analysis process. Researchers gathering verbal, gestural, artistic, or emotive representations must be especially aware of the "data mining" nature of their analysis.

The quantitatively descriptive researcher engages in virtually the same process but in a slower and more linear manner. Data are representations, believed to have validity related to the constructs under study. The representations are analyzed with an existing construct and the degree to which they coincide with the construct is made conscious. The researcher then decides whether the basic construct or the anticipatory/explanatory construct is confirmed or requires modification.

Implication of Constructs for Method

If Cronbach (1982) is right in that all human action is constructed, not caused, and the explanations I have presented are plausible, then research method begins with and accounts for the nature of the constructs to be studied. Further, method and analysis must acknowledge that in social science, the phenomena of interest are usually enmeshed with the construct structure of each individual in the study. That emphasizes the urgent need for attention to construct validity in any study and that validity probably rests to a great extent on the complexity of the representations as well as what Keeves (1997a) calls "band width" (range of manifestations) and "fidelity" (observational unidimensionality). To honor the complexity inherent in most constructs, researchers must plan for a program of research, multiple data types, multi-methods, or close identification with a "family" of studies contributing to a shared knowledge.


DATA ACQUISITION

A basic research process

Without data there is no research. The data employed in knowledge development may not be recent, but they nonetheless are data and should be recognized as such. The philosopher engaged in criticism of a praxial approach to music education may employ constructs acquired from Kant or Sparshott ten years ago. The historian may have strong personal memories of participation in an organization she is now studying. The experimenter gathers his computer file of data. The anthropologist writes copious field notes, creates audio and video recordings, and gathers representative objects. Data acquisition is crucial in every case.

Types of Data

All data are representations. An achievement test score is a representation of the student's achievement (itself a theoretical construct). Numbers are not quantities, they are mental surrogates for quantities. The numeral "7" in and of itself is an indication of "sevenness" in whatever property to which it is applied. The words in the interview transcription are not the meanings and thoughts of the interviewee, they are surrogates for those meanings. Therefore, to understand the types of data, we must explore types of representations. (Just to clarify the obvious, I am assuming there is a reality apart from representations of it.)

Representations of knowledge are both internal (the way we represent images and concepts internally) and external (the way we represent our knowledge to communicate it to others). For example, through internal representation we create a construct we label "up" by about kindergarten. It develops with every lift off the floor or climbing of the stair (kinesthetically), with tossing a toy or pointing to the sky (spatial), with making a vocal glissando (musical), with counting while building the block tower (mathematical), and with adding words to all these experiences (linguistic). We can communicate something about "up" to others through external representations––moving our hand upward (gestural), pointing upward (symbolic), drawing an arrow pointing to the "top" of the page (iconic), saying the word "up" (linguistic), saying a series of numbers representing increasing quantities (numeric), or making a sound increasing rapidly in pitch (musical). Representations fall at least into the areas of
Gardner's (1999) basic intelligences: linguistic, logical-mathematical, musical, bodily-kinesthetic, spatial, and, probably, intra– and inter-personal, and possibly even naturalist and spiritual. Within each there are different expressions and forms. For example, linguistic representations may be a single word, a logical proposition, a poem, or a story. Bodily-kinesthetic representations may include symbolic gestures like sign language, kinesthetic analogues, dance, or pantomime. The important conclusion here is that all forms of external representation can be regarded as "data" and therefore recorded and analyzed.

Roles of Data

Data, as knowledge representation, have distinct roles. Davis, Shrobe, & Szolovits (1993) identify five roles of a knowledge representation that are pertinent to artificial intelligence applications on computers. Three are particularly relevant here. The first and most obvious is the basic role of a knowledge representation as a surrogate—something that substitutes for the thing itself. The second role of data is as "set of ontological commitments." External reality can be represented in various ways, and no representation is perfect. Each attempted representation captures some things but necessarily omits others. Therefore, every time researchers decide on what they will focus in the world and how they will represent it, they are "making a set of ontological commitments. The commitments are in effect a strong pair of glasses that determine what we can see, bringing some part of the world into sharp focus, at the expense of blurring other parts" (Davis, Shrobe, & Szolovits, 1993, p.20).

A third role a knowledge representation fulfils is as a "medium of human expression." This is most important in research in terms of communicating knowledge developed in the research process. An important point here is that knowledge as a construct is generally complex, and the selection of a means of representing it again focuses on some facets and ignores others. Expression and communication are different. A person may choose a way to express knowledge and it may satisfy that person, but it may not communicate what was intended or meant. It is one thing to record video data of a conductor's gesture, analyze the gestures as knowledge representations, and reshape existing knowledge about musical problem-solving, and quite another to appear in front of a research conference and demonstrate that knowledge only with gestures. Gesture may be needed to communicate aspects of knowledge that are not easily represented in words, but words are pragmatically useful. Anyone traveling in a country where his or her language is not understood knows how difficult gestural communication alone is. At the same time, in many contexts, seeing another's gestures enhances communication accuracy.

Data Acquisition Concerns

The primary requirement of data, of a representation as a surrogate, is that it is something out of which we can make meaning that is warranted and that the meaning it contributes is supported or at least not contradicted through multiple perspectives (i.e., it is valid). Although validity often appears to hinge on methodology (e.g., randomization and control in experiments, population and sampling in descriptive, triangulation in interpretivist), validity is primarily about credible, defensible meaning-making from the selected representation. Cronbach (1971) states, "One validates, not a test, but an interpretation of data arising from a specified procedure" (p. 447). Zeller (1997) explains, "it is not the indicant itself that is being validated, but rather, it is the purpose for which the indicant is being used that is submitted to validation procedures" (p. 824). If a professor uses a highly “valid” music history examination as a measure of achievement in a musical acoustics class, there certainly would be indefensible meaning-making.

Because the constructs we deal with in social science are complex, selecting any one representation from which we can credibly infer meaning is nearly impossible. Consequently, researchers concerned about validity may need to draw on several types of representations (complex data) and multiple method. Zeller (1997) argues, "A valid inference occurs when there is no conflict between messages received as a result of the use of a variety of different methodological procedures" (p. 829).



SEVEN TRENDS

Trend 1: Construct complexity

The most fundamental trend affecting the social and behavioral science dimensions of music education research is the growing recognition by researchers of complexity as a central characteristic of all constructs and phenomena and the increasing complexity of the constructs themselves. Constructs do not exist in external reality—only the sources for our constructs do. Constructs are mental creations and develop toward complexity. Complexity is characterized by "compoundness"—multiple parts, layers, or dimensions—and interconnectedness of the parts, each of which may depend on or influence the other, none of which are in a fixed relationship or quantity, nor are related to a fixed behavior. For example, the researcher today may acknowledge that "self-concept" as a research construct is more complex than researchers thought 15 years ago, but the self-concept construct in the typical research subject's mind may be more complex today than it was 25 years ago due to an increase in comparative images from the media, peer comparisons through internet chat lines, or societal standards resulting from cultural diversity.

The complexity problem was first described in relation to the problem of generalizability of specific research findings. The hope enunciated by E. L. Thorndike (1910) was to "tell every fact about everyone's intellect and character and behavior,... the cause of every change in human nature... the result which every educational force... would have" (p. 6). In 1957 Cronbach reiterated this hope, but argued that to accomplish this researchers would have to focus on the effects of interactions rather than the effects of treatments. Frustration from inconsistent findings from similar studies led Cronbach (1975) to conclude that, "Once we attend to interactions, we enter a hall of mirrors that extends to infinity. However far we carry our analysis—to third order or fifth order or any other—untested interactions of still higher order can be envisioned" (p.119).

A major part of the problem seemed related to changing cultural context and the speed with which studies can be done. Cronbach concluded: "The trouble, as I see it, is that we cannot store up generalizations and constructs for ultimate assembly into a network. It is as if we needed a gross [12 dozen] of dry cells to power an engine and could only make one a month. The energy would leak out of the first cells before we had half the battery completed. So it is with the potency of our generalizations" (1975, p.123).

A problem for research is that we are dealing with complex constructs within a complex context that is evolving rapidly toward greater complexity. One social science reaction to this problem may be the rise of qualitative research. "Qualitative" researchers frequently point to the simplifying, linear approach of quantitative researchers as focusing on phenomena or constructs that have no relevance in the complicated real world. Quantitative researchers have been slow to respond with analytic techniques that can accommodate the complexity inherent in education or even to acknowledge the complexity. The arguments between the two groups have been so drenched in philosophic fundamentals of Cartesian reductionism, hermeneutics, epistemology, Realism, Relativism, Constructivism, and so on, that the more practical issues of construct complexity have been lost behind the smokescreen of the paradigm wars in educational research. Cziko (1989) observes that "the debate, centered on issues related to quantitative versus qualitative approaches to research, has at the very least raised serious questions.... There appears, however, to have been little discussion among educational researchers of what may be an even more basic issue, that is, the possibility that the phenomena studied in the social and behavioral sciences are essentially unpredictable and indeterminate [complex]" (p. 17). Evolution toward increasing complexity has not only affected the social sciences. Perhaps not yet openly perceived as a paradigm war in "hard science," limitations of the Newtonian Paradigm are showing and the "a major fault line has developed in the episteme" (Hayles, 1991, p. 16). The wedge now driving open this fault is Complexity Theory.

Qualitative researchers reacted to the simplicity and linearity problem of traditional scientific research by abandoning formal measurement and calculation. Naturalistic contexts are inherently complex and make credible analysis and understanding difficult. Complexity-informed researchers in social science are responding now in several ways. One is to imitate the physical scientists in searching for "chaotic order in dynamic data sets measured across very large numbers of time points" (Byrne, 2000, p. 2) with statistically rigorous analysis of the dynamics of chaos in social science data. Another approach is the development through computer-based simulations "of very elegant graphical representations of the behavior of uninterpreted deterministic chaotic expressions" (Byrne, 2000, p. 3). This allows for the examination, for example, "of whole system properties corresponding exactly to Durkheim's conception of social facts, and allows for examination of their emergence and transformation over time" (Byrne, 2000, p. 3).

A third way complexity-informed social science researchers are changing is to look in a new way at both quantitative and qualitative approaches. In quantitative research we either have or can envision large and complex data sets—of the sort acquired through national assessments, polls, census, market surveys, government revenue data, school evaluations, and so on. These data sets are complex because they contain varied data about individuals and groups in a time-series manner. Perhaps logistic regression and loglinear procedures are appropriate analysis techniques because of the possibility for handling interaction. However, Bryne (2000) states and Cronbach (1975) might agree, "It has been unfortunate that the emergent complexity generated by interaction has so often been reduced to terms in linear equations. A way of thinking informed by complexity leads us to treat procedures which enable the identification of significant interaction as qualitative exploratory devices rather than as ways of establishing law-like models...Instead of generating models which seek to describe determinants of outcomes, and writing interactions as terms in those models, we can see interactions as signs of the presence of complexity in general" (p. 4). In qualitative research we are seeing a trend toward computer-assisted analysis. The influence of Complexity researchers will probably result in an emphasis on time-ordered data and analysis that looks for development and change.

Trend 2: Ethical Complexity

At least three factors have contributed to a trend of increasing ethical complexity in conducting music education research. The first is the increasing complexity of the constructs under study. No longer are rigorous researchers satisfied to administer a short paper-pencil test and claim, for example, it is a valid measure of self-concept. The acknowledgement of construct complexity means researchers are probing deeper into sensitive areas—psychological issues, home and private matters, relationships, roles and perceptions—all of which potentially can destabilize the participant’s mental state or raise questions that provoke transformative reflection. Also, some constructs studied in music education are diversified and include topics that may touch on sensitive or confidential data, e.g., mentoring (Lamb, 1997b, 1998, 1999), peak experience (Sundin, 1989; Gabrielsson, (1991), negative feedback (Cameron & Bartel, 2000; Jacques, 2000), or brainwave manipulating sounds (Bartel, 2000).

A second factor is the increasing complexity of the social environment within which music education takes place. In cosmopolitan cities, cultural diversity continues to increase. Not long ago a researcher in Canada or the United States could send a letter requesting permission written in English to parents with considerable confidence that it could be read and understood. Today in Toronto, for example, most schools have upwards of 30 language groups represented among the students and many of the families are recent immigrants. Although students are learning English, parental permission for research necessitates extensive translation of letters and further raises fears among many people of privacy invasion through student observation or questioning. Subject diversity of language, culture, or religion becomes a prominent issue in the interpretation of data representing most constructs. This is less of a problem with simple constructs but a serious problem in complex constructs. An ethical issue resides in the integrity of our interpretations—do we really understand and correctly represent what people know, believe, experience, perceive, or achieve.

A third factor increasing ethical complexity is "the rights revolution" (Ignatieff, 2000). Beginning with the 1948 Universal Declaration of Human Rights, the past 50+ years have seen a constant struggle for freedom, civil rights, equal rights, self-government, or human rights. Often the battle has been between individual rights versus special group rights. The development of an increasingly litigious culture has exacerbated the situation. One consequence of these struggles has been a climate where both subjects and researchers are much more aware, and ethical watch-dog agencies are much more cautious. As a result researchers must anticipate and clarify procedures and effects, communicate clearly and accurately to all touched by the research, and obtain legally defensible permissions for all research.

Trend 3: Methodological Complexity

Research methodology is the plan employed to acquire data and make meaning (develop knowledge) out of those data. It is an intentional process to reshape or confirm existing constructs (knowledge). In education this process has always been relatively complex because educational research is inherently multidisciplinary—the concerns of education draw on many disciplines. Historical and philosophical research have methodological traditions pursued in music education research. Psychology as a discipline developed in the past 120 years and has two, at times competing, orientations—experimental and correlational (Cronbach, 1957; Shulman, 1988). Both psychological research orientations have been prominent in music education research. Musicology, in addition to its historical dimension, is an analytic approach, one that is applied in music education research. Sociological research in the past relied heavily on surveys—questionnaires and interviews. This is one of the most common approaches in music education research. Anthropology has had less application in music education research until recently, but ethnomethodology has been rising in popularity. Some disciplines like linguistics, economics, demography, have had little effect on music education research.

Educational research is inherently multilevel—concerned about both individuals and groups. Educational research deals with problems that are inherently complex—"it is rare that a problem in educational research can be reduced in such a way that it can be viewed in terms of only two constructs or variables" (Keeves, 1997b, p. 278). Educational research often examines learning over time making multiple or repeated measures necessary.

As a result of being multidisciplinary, multilevel, and inherently complex, educational research is multimethod. However, the existence of many methods, if these were clearly differentiated but not integrated, would demonstrate diversity but only limited complexity. The growing acceptance of multiple epistomologies, however, adds another layer of real complexity.

The often heated paradigm wars in educational research that attempted to demonstrate the "rightness" of one paradigm over another, assumed that distinct paradigms existed and that, if so, only one could be "true." There are serious flaws in the concept of distinct paradigms (Walker & Evers, 1997), but as foundational theories for a specific set of research methodologies they do have utility. Walker and Evers (1997) explain:
In offering a broader, three-way taxonomy of research to account for diversity in inquiry, Popkewitz (1984, p.35) says: "the concept of paradigm provides a way to consider this divergence in vision, custom and tradition. It enables us to consider science as procedures and theories of social affairs." He assumes that "in educational sciences, three paradigms have emerged to give definition and structure to the practice of research." After the fashion of "critical theory" (Habermas, 1972), he identifies the paradigms as "empirical-analytic" (roughly equivalent to quantitative science). "symbolic" (qualitative and interpretive or hermeneutical inquiry), and "critical" (where political criteria relating to human betterment are applied in research). (Walker & Evers, 1997, p. 23)

The coexistence of these epistomological perspectives is important. They do not supercede each other as paradigms in the natural science are assumed to do (although that is now in question as well), or as Lincoln and Guba (1984) seem to assume with their "paradigm eras."

Another facet of research methodology has greatly affected the overall complexity of knowledge development processes in the past 15 years—theoretical orientation. These orientations are differentiated in specific ways from each other, with common features between some, and selective discipline and paradigm acceptance among them. Babbie (1995) argues that these orientations co-exist and co-contribute to a complete view of social and behavioral phenomena because each “offers insights that others lack—but ignores aspects of social life that other[s] … reveal" (p. 41).

Babbie (1995) and Rothe (1993) identify the following as distinct theoretical orientations affecting methodological decisions: (1) Social Darwinism— research finding factors predicting or facilitating survival of the fittest (e.g., finding characteristics of band students most likely to succeed). (2) Conflict theory—questions related to class or group (e.g., how “ethnic minorities” are excluded from mainstream musical culture through the maintenance of “racial” stereo-typing). (3) Symbolic interactionism—concerns about how the individual constructs an image of self through interactions with groups and individuals, is able to take on the perspective of the other, and then tailors communication in anticipation of the reaction of the "generalized other" For example, orchestra members may have a view that conductors are volatile, passionate, dictators. Consequently they act cautiously, await instructions, and tolerate abusive behavior. The conductor sees them as musicians to be instructed, controlled, and manipulated, possibly by dramatized anger in order that they will play with adrenalin and passionate vigor. The interaction between the group and the individual is symbolic.
(4) Role theory—examinations of how taking on a role restricts or permits certain behaviors, e.g., the role of passionate conductor. Dolloff (1999) focuses on how music teachers have an image of “teacher” that sets behavioral expectations for themselves when they assume the role of teacher. (5) Ethnomethodology—examines a culture in context to make sense of how people live and what meaning they give to their reality, assuming that people's realities are constructed in cultural context. For example, a community choir can be distinct culture and the researcher could “live” awhile in this culture to discover the realities experienced there. (6) Structural functionalism—organizations or societies are systemically structured like organisms—consisting of components that each contribute to the function of the whole (e.g., what distinguishes tenured professors from untenured professors from graduate students from undergraduate students by examining values, norms, community types, or individual roles). (7) Feminist theory—an orientation that has common features usually including a rejection of positivism, a ubiquitous concern with gender, stress on the value-ladeness of all research, the adoption of liberatory methodology, and the pursuit of nonheirarchical research relationships (Haig, 1997, pp 180-181). (8) Exchange theory—examination of the rationality applied in weighing costs and benefits of all choices (Homans, 1974). A researcher might examine what constitutes music students’ cost and benefit ledger related to time-intensive practice and the benefits of musical achievement. (9) Phenomenology—a focus on how people internalize the objective world into consciousness and negotiate its reality in order to make it livable and shareable, e.g., what is it like being a musical child prodigy? (10) Conversational analysis—assumes conversation is central in interpersonal behavior. Particular interest is in status and power revealed in conversational structure, for example, through interruptions and overlaps. The “talk” between trumpet teacher and trumpet student might be the means of studying student/teacher relationship. (11) Social Ecology—assumes "physical properties of territory act as reference points for people's interactions with one another" (Rothe, 1993, p. 56). A field experiment manipulating music room environment would address its social ecology. (12) Action research—not so much a theory as an attitude with focus on improvement. A teacher might enlist the cooperation of a choir to record, in a reflective journal, responses to some innovative strategies to build a sense of community. Analysis of the students’ and teacher’s journals would serve as data to determine whether the innovations result in improvement.

Other discipline-related orientations are surely possible, but these serve to illustrate the complexity that develops as paradigm, discipline, and orientation interact in the creation of research method. Music education research does not reveal all of these orientations but a basic trend is to a greater emphasis on sociology of music, constructivism, and the social aspects of music making and learning, and with that will come specific social research orientations (McCarthy, 1997, 2000; Paul, 2000).

The relevance of discipline, paradigm, and orientations to research method is evident if one sees research method not simply as the data acquisition method but rather as an interaction among the question posed, the analysis required to answer the question, and the data appropriate for the analysis. An important role of the orientations is in influencing what questions will be asked, but they also influence who will be asked, how answers will be obtained, what will count as valued representations and as knowledge, and what analyses will be conducted.

I must make an important observation: The primary form of data (representation) in empirical-analytic research is the numeral, hence the common designation "quantitative." The primary form of data (representation) in symbolic and critical research is the word. Words allow for use of adjectives and adverbs, the description of qualities, and hence the designation "qualitative." However, we differentiate among quantitative methods on the basis of: (1) how the data are acquired (by asking questions requiring ratings it could be survey, by asking questions on a "test" it could be an experiment or a correlational study, and by counting checkmarks on a list representing behaviors it could be an observational study), (2) from whom the data are acquired (random sample assigned to comparative groups could be an experiment, or a single group volunteer sample might be a survey), (3) what happens between data acquisition efforts (if a planned treatment is administered it may be an experiment), (4) and what analysis is planned for the acquired data (if two sets of scores from a group of individuals are compared for relationship it is correlational, if the averages are compared it could be an experiment). The question(s) that motivate the research are directly answered by the analysis and so have an important defining role for methodology.

These four aspects of research: (1) how the data are acquired, (2) from whom the data are acquired, (3) what happens between data acquisition efforts, and (4) and what analysis is planned for the acquired data, along with the defining role of the research questions and their explanatory intention, determine method. They do so in "quantitative" research and they do so in "qualitative" research. Consequently, the general category of qualitative research contains many specific methods. However, the definition of these methods may not be settled since the field is still evolving.

The most direct way to methodological relatedness is in multi-method studies. Exploring complex constructs raises multiple questions which frequently demand multiple methods. In the past these were most likely methods within a single epistomological paradigm, but researchers are beginning to combine "quantitative" and "qualitative" methods in a single study. Zeller (1997) states that "one method used in isolation does not provide compelling answers to many research problems. The reason for this is clear. Different techniques differ dramatically in their purposes, goals, objectives, tactics, strategies, strengths, and weaknesses. The original questions that prompt the research and the new questions that emerge from its results require a blending of methods" (p. 828). In the discussion of validity in research, Zeller argues further that, "inferences about validity do not lend themselves to solely quantitative or statistical solutions. On the contrary, an interactive research strategy combining quantitative and qualitative is advocated. A valid inference occurs when there is no conflict between messages received as a result of the use of a variety of different methodological procedures" (Zeller, 1997, p. 829).

Trend 4: Data Complexity

All data are representations, intentionally acquired or captured for study. An achievement test score sets a fixed point, a freeze-frame of sorts, for the individual's ever developing achievement. A video-recording of a piano lesson can later be viewed and re-viewed. The image on the video screen is not the actual lesson—it is a representation of the lesson. It is selected for study and is fixed (selected, made stable, not repaired) to the extent that it can be repeatedly viewed.

The researcher’s selection of the phenomenon or representation for study is one of the most important and urgent decisions in any research effort. In traditional empirical-analytic research this is done early in the process, and study design is created accordingly. In interpretivist approaches, the researcher may decide that some aspect of teaching may be studied, commence video-recording many episodes of teaching, and delay the decision on what specific phenomena or representations to describe or interpret. But, regardless of delay, or multiple viewings of a tape, eventually the researcher selects the words, actions, or gestures to include in the analytic and interpretive process.

Ways to fix representations for study include: (1) audio recorded language (often transcribed into written form); (2) written language of various types; (3) audio recorded sound; (4) video recordings of the phenomena; (5) test scores; (6) verbal or numeric categorical assignments; (7) numeric rating or rank values; (8) graphic displays; (9) photographic or pictorial prints; (10) artifacts; (11) symbols and symbolic notations. Obviously categories and subcategories are possible. For example, audio recorded sound could have subcategories like music, ambient room noise, subject's non-verbal sounds etc.

There is a simple to complex specificity range in fixed data. Consider the following: a checkmark in a category, the number on a rating scale, or the test score; an audio recording of an interview or a written narrative account; a video recording of an interpretive dance, or an artist’s self-portrait. All hold meaning and can be interpreted; however, the layers, strands, connections, and inter-relationships of meaning in the later are considerably greater.

An important characteristic to note about different types of data is data depth. Certain types of data, like categorical assignments or test scores, have a simple relationship with the construct they represent. Other data types, however, are more complex. The data fixed by a video recording of a band rehearsal can be subjected to various theoretical orientation "lenses" or to various research questions. Such data has "depth" that can be mined and refined repeatedly before its store is depleted. The more complex the data type is, the more depth it offers for data mining.

Trend 5: Analytical Complexity

Despite the increasing complexity of constructs, methodology, and data, analysis is essentially a process of simplification, a process of creating order within the represented reality that allows for meaning making (interpretation). Analysis is a process of clarification, a process of reducing unknowns. In a computer systems context, complexity is defined as “the existence of unknowns” (Kafre, 1998). The goal of research is to eliminate unknowns through the development of knowing, and it may do so by imposing a simplification system. It is in analysis that the tension between complexity and simplification exists most obviously. In statistical analysis, the researcher lets a number represent a phenomenon, manipulates the collected numbers to reduce them to a number representing something new, and postulates an implication of the number for the original observed phenomenon. In qualitative analysis, the researcher frequently transcribes or notates the representations, categorizes and classifies these for further comparative examination, and postulates an explanation or interpretation about the observed phenomena. However, the phenomena and constructs of interest to music educators are complex and analyses leading to unidimensionality, simple linearity, or “stories that dissolve all complexity” (Shenk, 1997, p. 157), are no longer useful if progress in research is to be made.

General Responses to Complexity

In educational research, attendance to complexity can be seen as giving greater place to individual uniqueness and context, to multiple perspectives, and different representations. Several developments in research design, analysis, and measurement illustrate how these requirements can be accommodated.

Rasch Analysis. Rasch analysis recognizes the inevitable interaction of a measurement technique with the person being measured (Keeves, 1997a). The technique links qualitative analysis to quantitative methods by converting dichotomous and rating scale observations into linear measures. It is often misclassified under item response theory or logit-linear models. These describe data. “Rasch specifies how persons, probes, prompts, raters, test items, tasks, etc. must interact statistically for linear measures to be constructed from ordinal observations. Rasch implements stochastic Guttman ordering, conjoint additivity, Campbell concatenation, sufficiency and infinite divisibility” (Linacre, 1995).

Single-Case Design and Change Measurement. Recognizing complexity raises serious doubt about the generalizability of results from large samples. When researchers average results from groups of subjects they omit considerable richness in the data (Sharpley, 1997, p. 451). Practitioners’ concerns with individuals rather than aggregates mean that the scientific theory generalized from research actually has limited application in practice. “Even statistically significant findings from studies with huge, randomly selected samples cannot be applied directly to individuals in particular situations; skilled clinicians will always be required to determine whether a research generalization applies to a particular individual, whether the generalization needs to be adjusted to accommodate individual idiosyncrasy, or whether it needs to be abandoned entirely with certain individuals in certain situations” (Donmoyer, 1990, p. 181). As a result, research aimed at establishing the utility of specific interventions rather than at establishing scientific principles is finding single-case design particularly useful (Jutai, Knox, Rumney, Gates, Wit, & Bartel, 1997).

Single-case studies can have a macro-focus (Stake, Bresler, & Mabry, 1991) or a micro-focus (Jutai, et al 1997) and can be designed to yield various types of data. For those generating numeric data, the measurement of change in the interrupted time series is the primary analytic task. In the past there was debate between advocates of simple visual analysis of a graphic representation of research participants’ change and advocates of statistical analysis. The problem was that graphic representation was easily distorted and that traditional statistical procedures were not properly applicable because simple pre- and post comparison were not adequate. One of the concerns influencing analysis is both the within person change and the between person change. Least-squares regression analysis has been used for within person change and weighted least-squares regression analysis for between person change. However, recent developments have made change measurement more powerful, assuming the collection of considerable longitudinal data. High quality analysis of within and between person models can be conducted with hierarchical linear modeling (Bryk & Raudenbush, 1992) or with covariance structure analysis (Willett & Sayer, 1994, 1995).

Meta Analysis. Glass (1976) coined the term “meta-analysis” for an approach that allows the quantitative, analytic integration of the results from many studies on a particular phenomenon. Although the statistical procedure treats the studies selected for integration as being alike, in actual fact the meta-analysis merges multiple perspectives and varying definitions of constructs. It is then a form of analysis that simplifies the complexity inherent in research in a particular field. If “reality” is complex and any one study cannot adequately encompass that complexity to form an explanation of it, then integrating all attempts at such explanation in a meta-analysis may more adequately honor the complexity.

Integration of analyses from multiple independent studies had been done in some form before Glass (1976). Cotes (1722) describes how astronomers’ observations were combined by using weighted averages. Pearson (1904) integrated five separate samples by averaging estimates of correlation. Birge (1932) attempted to integrate results in physics by weighting combinations of estimates; Cochran (1937) and Yates and Cochran (1938) did so with agricultural experiments. Tippett (1931) and Fisher (1932) proposed a method of combining probability values.

The use of meta-analysis has been increasing, especially in the past 15 years. Barrowman (1998) surveying of the use of meta-analysis in medical research, found two or three studies per year in the early 1980’s, increasing to about 10 per year by the mid 1980’s, and to 80 by the end of the 80’s. Egger and Smith (1997, p. 1) found this trend continued with close to 800 meta-analysis studies reported in Medline by 1996. In music education few meta-analysis studies have been done. A search of the major journals revealed only one, Standley (1996). However, “meta-analysis seems well suited . . . to make a larger and larger contribution to the social and behavioral sciences, and to a broad range of policy-relevant research” (Wachter & Straf, 1990, p. 28).

The most common approach to meta-analysis is to calculate the effect size on each study selected for inclusion. Effect size is basically the mean of the treatment group minus the mean of the control group divided by the standard deviation of the control group (Light & Pillemer, 1984). The pattern of effect sizes can then be examined visually by plotting them on frequency distributions or funnel displays that plot effect size, for example, against sample size. Statistical procedures are used to calculate cumulative effects, e.g., overall mean effect size (Glass, McGraw, & Smith, 1981; Hunter, Schmidt, & Jackson, 1982; Rosenthal, 1991).

Criticisms of meta-analysis focus primarily on three issues: (1) independence, (2) the “apples” and “oranges” problem, and (3) the “file drawer” problem. When a single study consists of multiple “experiments,” several effect-size estimates result. These cannot be used separately in a meta-analysis since they may be drawn from the same sample and, consequently, are not independent. Glass, et al. (1981) acknowledge the problem and suggest calculating an average effect for the study as a whole. Since these individual study effects are mostly related to distinct variables, critics argue that averaging them is not an adequate solution (Chou, 1992). This is a criticism that focuses on the tension between the complexity of reality and the analytic drive to simplify for understanding. The “apples” and “oranges” problem is somewhat related. Critics point out that meta-analysis integrates effects from variables based on different theoretical constructs. Glass, et al. (1981) argue that researchers constantly aggregate data from different individuals who are like “apples” and “oranges.” The important consideration is that the meta-analysis concern must be about “fruit” and then “apples” and “oranges” can be included. The “file drawer” problem is based on the premise that researchers selecting studies for inclusion in a meta-analysis will favor published studies because of availability and that there is a publication bias: the selection process for publication tends to favor studies that show significance (the studies that do not reject the null hypothesis remain in the file drawer) (McGaw, 1997). Rosenthal (1991) argues this problem does not exist but Glass, et al. (1981) show that effects from meta-analyses of theses and dissertations are weaker than effects from published studies.

Recent developments with Bayesian statistics have been applied to meta-analysis with considerable promise. DuMouchel (1990), Efron (1996), and Smith, Spiegelhalter, and Thomas, (1995) demonstrate that Bayesian methods provide a framework for handling meta-analytic issues such as fixed-effects vs. random-effects models, appropriate treatment of small studies, possible shapes of the population distribution, and how to include covariates (Barrowman, 1998)

Concept Mapping. The concept map is familiar to most teachers as an efficient teaching strategy; it has more recently been used as a technique in interpretivist research. Specifically, it can be used to probe and visualize the complexity of constructs residing in a research participant’s mind. Novak (1990) employed the concept map in research with elementary students as a means of facilitating analysis of interview data by applying this structural device for representing a student’s understanding of a complex phenomenon. The visual, often hierarchical, display of ideas does clarify participants’ understanding, but the common two-dimensionality of visualization may distort or at best inadequately represent the participants’ knowledge. “The limitation [of concept mapping in a research context] is inherent in a hierarchical representation of knowledge in which complex concepts, or complexes of concepts, are established in a superordinate/subordinate relationship. The concept complexes situated in the subordinate positions are, however, multilevel entities whose constituent parts often relate in a complex manner to the focus concept” (Lawson, 1997, p. 294).

An alternative to the “hand-drawn,” hierarchical, qualitative concept maps has been emerging. Particularly when a group of participants is involved, a method described by Trochim (1989a, 1989b) that draws on multivariate analysis can be used. The concept or domain to be mapped is focused for the group, statements are generated through brainstorming, the statements are placed on cards, each participant sorts the cards into piles that make sense to the person (not a forced distribution like Q-sort), statements are rated (e.g., on priority or preference), the cards in sorted piles are then analyzed through cluster analysis and non-metric multi-dimensional scaling, and finally displayed in various maps for interpretation. Although this approach is described by Trochim (1989b) as “soft science or hard art,” it is an indication of a trend in analysis of complex constructs—the use of statistical analysis
with “qualitative” data and attention to complexity by statistical analysts.

Dual Scaling. One of most common analysis techniques of interpretivist research, is the identification of “themes” in data—essentially a process dependent on the categorization of observations (verbal statements or descriptions of behavior). Computer programs now assist in categorization, or at least in tracking and retrieving, of text data for purposes of thematic analysis. Using the categorization for further analysis by doing numeric, statistical analysis on the categorical data, however, is not so common. A recent method designed to extract quantitative information from non-numerical (qualitative) data is dual scaling (Nishisato, 1980, 1984, 1994). It addresses simultaneously in analysis a number of the characteristics of complex data — it is descriptive, optimal, multidimensional, and multivariate. For example, it will derive the most reliable scores for respondents from multiple-choice data, as well as provide all the necessary coordinates for multidimensional representation of data. It handles both incidence data and dominance data. Its potential for music education research is still to be explored.