Instrumentation and Innovation in Design

Instrumentation and Innovation in Design Experiments Dan Schwartz     

Design research can contribute by developing interventions that strike the right balance of innovation and efficiency experiences and that place students on a trajectory towards adaptive expertise. This research should also create ways to determine whether students are on that trajectory, and this requires more than borrowing standard off-the- shelf measures of efficiency.

As the Design-Based Research Collective (2003) noted, “the use of randomized trials may hinder innovation studies by prematurely judging the efficacy of an intervention” (p. 6)

One peril is that because the work is about innovation, it often needs to let go of current theories. This can create a tower of innovation babble with little short-term hope of cumulative knowledge. A second peril is that if innovations must stand on their own, with limited support from prior theory, the research is difficult and runs a high risk of failure. diSessa and Cobb (2004), for example, argue that a preeminent goal of design experiments is to create new theories. This may be a fine goal for brilliant researchers at the top of their game, but for the rest of us, it is a recipe for heart-felt platitudes.

Some researchers reject the idea of using measures because they worry the measures will foreclose the possibility of detecting the unanticipated. Consequently, many rely on narratives rather than discrete measures to create inter-subjectivity…We do not know of any evidence one way or another that determines whether narrative yields precise agreement between the researcher and the audience.

Unlike ethnographers, design researchers are orchestrating “what could be” rather than observing what already exists, and therefore, they have must have some goal in mind. Ideally, this goal would be specific enough that it is possible to begin precisely measuring its attainment.

Notably, none of the research we describe is about proving causes. Instead, it is about demonstrating the discriminant and ecological validity of our instructional apparatus and measures, which we believe is one place where design research can excel in contributing to scientific knowledge.

—> Would this paper be of use in explaining transfer from blocks to text?

The experiment arose from a concern that most current assessments of knowledge use sequestered problem solving (Bransford & Schwartz, 1999). Like members of a jury, students are shielded from contaminating sources of information that might help them learn during the test. It appears that this assessment paradigm has created a self-reinforcing loop where educators use efficiency-driven methods of procedural and mnemonic instruction that improve student efficiency on sequestered tests. However, we suppose that a goal of secondary instruction is to put students on a trajectory towards adaptive expertise so they can continue to learn and make decisions on their own. Like many others, we fear that measures of the wrong outcomes drive instruction the wrong way.

The candidate made a mistake and started to defend causal claims about the teachers’ success. This single case could never crisply defend causal claims. The work came off as speculation and craft knowledge.

diSessa and Cobb (2004), for example, propose that a significant goal of design research is “ontological innovation”—the invention of new scientific “categories that do useful work in generating, selecting among, and assessing design alternatives.” (p. 78). The goal is to uncover categories of phenomena that support explanation. We suppose these authors actually meant “epistemic innovation,” because ontology refers to what exists and epistemology refers to knowledge of what exists. These authors are not proposing that the value of design experiments is to create new existences, but rather to create new knowledge.      —> Challenge this?

it is important to appreciate that the goal of educational design research is not technological innovation per se, but rather innovation in learning practices. Thus, pointing to an innovative technology is less compelling than pointing to an innovative learning practice it creates.

This is the notion of praxis, where the proof of a theory is in the change it creates (Cook, 1994).

We proposed that it might be profitable to position design experiments in a larger space of innovation and efficiency. The question is, what features might be added to design research to ensure it maximizes the chances for innovation while also setting the stage for more standard tests of scientific value.

We argued that a logical warrant for innovation is the resolution of incommensurables.

TO READ:  diSessa and Cobb (2004) Design-Based Research Collective (2003) (Cook, 1994)