NSF DR-K12 Reviewing

Criteria

  • What is the potential for the proposed activity to
    • advance knowledge and understanding within its own field or across different fields (Intellectual Merit); and
    • benefit society or advance desired societal outcomes (Broader Impacts)?
  • To what extent do the proposed activities suggest and explore creative, original, or potentially transformative concepts?
  • Is the plan for carrying out the proposed activities well-reasoned, well-organized, and based on a sound rationale? Does the plan incorporate a mechanism to assess success?
  • How well qualified is the individual, team, or institution to conduct the proposed activities?
  • Are there adequate resources available to the PI (either at the home institution or through collaborations) to carry out the proposed activities?

Review template

Rating

Intellectual Merit

Strengths

Weaknesses

Broader Impact

Strengths

Weaknesses

Summary

2200765 (Primary Panelist)

Exploring & Research at the STEM-C Academy (E&R-STEM-CA)

Rating: Fair

Intellectual Merit

Goals:

  • Produce a STEM pipeline from middle school through college for minoritized students.
  • Integrating CS, engineering, and research into secondary curricula
  • Research experiences in STEM+C for minoritized secondary students
  • PD for math/science teachers
  • Mentorship for minoritized secondary students
  • Promote digital equity

Strengths

  • The goals listed are very important challenges in the areas of computing education and engineering education.

Weaknesses

  • The goals listed are important, but what specific challenges to these goals would this project address?
  • This proposal is not organized around research questions, nor is it framed in terms of a synthesis of existing research.
  • The research cited in support of integrating CS, engineering, and research into curricula pertains to 1-to-1 laptop programs, not CS.
  • The measures proposed (Table 1) are very general, consisting mostly of

Broader Impact

Strengths

  • Participant institutions are minority-serving institutions.
  • The team has a strong background in mentoring and an excellent network connecting
  • After completion of project, institutions will have an institutionalization plan.

Weaknesses

  • It is not clear how the activities all relate to each other. It would be helpful to have a logic model to align activities with expected outcomes. For example, research internships, saturday academies, research symposia, and classroom interventions are all worthwhile ways to support youth in developing interest in STEM careers, but how do they all fit together? Why were these specific activites chosen over others?

Summary

This Learning Strand implementation and improvement-type proposal from an alliance of educational organizations in Puerto Rico and the USVA would be led by Scientific Caribbean Foundation (SCF) and would implement a research mentoring program serving minorized secondary students and science/math teachers.

The Broader Impacts of this study could be profound: the proposal includes a wide array of stakeholders serving minoritized populations, as well as large reserach institutions with experience administering projects such as that proposed.

However, the intellectual merit of this proposal is not as strong. The proopsal does not draw on the substantial existing body of literature on integrating CS, engineering, and research (three distinct topics to be sure), nor does it frame specific research questions. Unaddressed are which specific parts of integrating CS, engineering, and research into curricula and pedagogical practice have been difficult in the past and how this proposal would address them.

2201421 (Primary Panelist)

Reprogramming High School Science: Investigating a Professional Learning Model Integrating STEM/CS through Teacher-Student Co-Learning

Rating: Excellent (Later changed to Very Good)

Intellectual Merit

Strengths

  • Clear goals: develop the field’s understanding of co-learning, 2) describe how teachers integrate CT and engineering practices into domains of expertise, 3) study the impact on students’ knowledge, interests and identity. Noteworthy is that this study will investigate how teachers learn, not just show whether they do.
  • Research builds on a workshop taught between 2017 and 2021 with evidence of effectiveness.
  • Combining non-hierarchical teacher learning and student out-of-school learning is an innovative and promising model.
  • Aligned with NSF Big Idea “Growing Convergence Research”
  • Articulates key design features: co-learning, supporting teachers in changing their practice, and including teachers as co-designers.
  • Participatory DBR model (with teacher alumni of the program)

Weaknesses

Broader Impact

Strengths

  • Clear, research-based strategy for supporting marginalized students in becoming interested in STEM.
  • Aligned with NGSS.
  • Well-articulated, differentiated workshop curriculum with specific research-backed strategies for equitable inclusion.
  • Advisory board and external assessor

Weaknesses

Summary

This Teaching Strand Early Stage Design and Development Study would have high school students and teachers learn learn computer science (CS), engineering design, and computational thinking (CT) in the context of biology, specifically by modeling biological phenomena in robots. These workshops have already run for several iterations with preliminary evidence of effectiveness. The current study would expand the workshops and engage in more systematic design-based research improving the model and research on how teachers and students learn in a non-hierarchical co-learning context.

This proposal is very strong, both in terms of intellectual merit and broader impacts. Its theory of action is clear and thorough and extensively supported by a coherently-synthesized research base.

2200917/2200918/2200919 (Secondary Panelist)

Collaborative Research: Designing Computational Modeling Curricula across Science Subjects to Study How Repeated Engagement Impacts Student Learning throughout High School (DC-Models)

Rating: Very Good

Intellectual Merit

Strengths

  • Builds on strong research foundation showing the value of computational modeling for science learning, particularly across multiple subjects. The proposed research and theory of action also builds on a strong research base for teacher learning, developing effective professional development in the context of NGSS learning goals.
  • Clearly-framed and specific research methodology.

Weaknesses

  • No theoretical rationale is given for the proposed extensions to StarLogo Nova.
  • Takes a fairly narrow view of learning: little engagement with sociocultural or critical factors shaping students’ classroom experiences.

Broader Impact

Strengths

  • Focused on problems of practice using design-based implementation research methodology.
  • Master’s level and undergraduate research is incorporated in specific ways.
  • Will serve marginalized populations. There is a clear plan for building the research-practice partnership and for the outcomes of the partnership.
  • Prior outreach to teachers and evidence of teacher support for the project based on pilot activities.

Weaknesses

Summary

This Learning Strand Early Stage Design and Development proposal would be a research-practice partnership between researchers and the Washington, DC school district, addressing two problems of practice: (1) the district wants all students to access computational modeling in science classes, and (2) district teachers need professional development around NGSS.

The study would focus on three design-based implementation research (DBIR) questions: How do students’ work with computational modeling affect students’ learning core science ideas? (in one course and across multiple courses) And what kinds of design supports do students and teachers need to use science units integrating computational modeling?

2201061 / 2201062 (Secondary Panelist)

Collaborative Research: Advancing Middle School Computer Science Learning with Intelligent Game-Based Pedagogical Agents

Rating: Good.

Intellectual Merit

Strengths

  • Builds on prior work developing game-based CS learning environments.
  • Research questions are well-framed with supporting literature.
  • The meaasures proposed are specific and well-suited to the research questions. In particular, the use of continuous assessment of students’ strategies and their concept understanding seems promising for rapid iteration of the pedagogical agents.

Weaknesses

  • The goals of an inclusive computing culture are very important, as illustrated by the anecdotal scenario. But these goals seem somewhat orthogonal to the designs proposed for pedagogical agents. How are the software designs connected to the pedagogy? Little background is given on what pedagogies are effective in teaching with game-based learning environments and how teachers can learn the relevant content knowledge and pedagogical content knowledge to use these games effectively in a classroom.
  • I was not persuaded that combining evidence for game-based learning with evidence for intelligent tutoring systems leads to the conclusion that pedagogical agents are promising for game-based learning systems. The methods of analyzing code structure seem sound, but these will not necessarily lead to a modeling of students’ misconceptions or affective states. This is particularly a question for youth who sometimes have affective reasons for asking for help in addition to getting information. So while I am personally somewhat skeptical of a successful outcome, the research questions are definitely sound and toward the unknown.

Broader Impact

Strengths

  • Once the software is developed, it will be easily scalable to many more students.

Weaknesses

  • Little detail is given on how the teachers and students would be included as co-designers, nor on the professional development practices which will be used to support teacher adoption. Given the long history of educational software which goes unused in practice, this feels important to address in ensuring broader impact of the project.

Summary

This Learning Strand Design and Development study would build on prior NSF support developing a game-based middle-school computer science learning environment to study how an automated pedagogical agent might scaffold problem-solving, with particular emphasis on how agents agents co-designed by marginalized students might support their learning. Using a variety of built-in and separately-administered assessments, the researchers propose to iteratively develop pedagogical agents which support students’ concept-learning and dispositions toward CS.

2201139 (Secondary Panelist)

integrating STEM and LIteracy with Computation in Education (iSLICE)

Rating: Fair

Intellectual Merit

Strengths

  • Many of the activities seem conducive to powerful learning experiences. For example, the interdisciplinary lessons illustrated in Table 5 seem highly-valuable.

Weaknesses

  • The parallels drawn between elements of CT (decomposition, patterns, abstraction, and algorithmic solutions) and ELA practices (Table 2) are superficial. As a former high school English teacher and middle-school CS teacher, and as a researcher who has studied how textual literacy practices and CT might support one another, I feel confident in saying that the way novices and experts use these practices is so different across disciplines that there is little value in suggesting a linkage.
  • The research qustions focus on program evaluation, not on developing basic understandings of how CT can be taught and learned in an interdisciplinary context. There has been a robust literature on this topic in the last decade which is not cited.

Without such a framing, I am concerned about using

Broader Impact

Strengths

  • Well-thought-out, coherent, and feasible proposed activities.
  • Would serve a rural, high-need population in Tennessee. Not much computing education research has been done in this context.
  • Would build on an existing network of partnerships.
  • Emphasis on feasibility and scalability.
  • High school summer camp is likely to provide youth with familiarity with college environments; may support more STEM-interested youth in pursuing college after high school.

Weaknesses

  • Critical needs are undertheorized. This project would (admirably) serve rural, low-SES, first-generation students. But the nature of their distinctive needs is not framed theoretically, nor are they linked to specific features of the proposed activities. Furthermore, there is no discussion of how these dimensions of marginalization intersect more-frequently-discussed dimensions of race and gender.

Summary

This Teaching Strand early stage Design and Development project would integrate computing and literacy to support students’ computational thinking skills,with emphasis on future workforce skils, interdisciplinary STEM, equity, and the contxt of COVID-19 schooling. The project would consist of ongoing teacher professional development and a summer camp at the university for high school students. While the project is well-conceived in terms of broader impact for a population (rural, first-generation, low-SES, Appalachian) which has receiveld little funding or research attention, the project’s intellectual merits are inadequate. The research questions address program effectiveness but not a contribution to the body of literature on interdisciplinary computational thinking (particularly connecting to literacy), which is not cited here.

2201258 (Secondary Panelist)

Improving Computer Science Instruction Through Automated Feedback to Teachers

Rating: Fair

Intellectual Merit

Strengths

  • Research on teacher uptake in CS is promising. Like noticing and asking follow-up questions, substantial research shows the importance of uptake in other STEM subjects. Research developing pedagogical content knowledge in CS is much-needed.
  • The proopsal makes a strong case for providing teachers formative feedback as a means of developing their pedagogical skill.

Weaknesses

  • While the proposal correctly notes that high-quality coaching is expensive, the proposal does not justify its assumption that automated formative feedback might be an effective substitute. Even if the system worked perfectly, the aspects of coaching which situate a teacher’s work in a community of professional practice would be missing, and they would likely be replaced with the kind of mistrust associated with black-box AI and workplace surveillance.

  • As a CS education researcher with a recent graduate degree in NLP, I feel this proposal oversells the capabilities of NPL as giving “computers the ability to understand text and spoken words in much the same way human beings can.” Aside from the acoustic challenge of collecting high-quality classroom audio, NLP automated transcription is just reaching acceptable quality levels. We are far from NLP being able to characterize the situated, multimodal meaning of speech in context.

  • The emphasis on objective measures of uptake, and lack of a participatory design process, seems likely to exacerbate existing inequities rooted in cultural and linguistic practices.

  • RCT: What is the control group? Should be some kind of surveillance.

Broader Impact

Strengths

Weaknesses

  • The proposal is largely focused on basic research; it is not clear to what extend the proposed project would have direct broader impact.

Summary

This Assessment Strand Design and Development project would develop a system for providing real-time feedback to computer science teachers related to uptake of student ideas via recording classroom discourse and analyzing it using natural language processing methods. The proposal is focused on an important research question and an urgent problem of practice, but the feasibility of the proposed line of inquiry is doubtful. Broader impacts are not sufficiently addressed.

2201271 (Secondary Panelist)

SELECT: Tools for Selecting Curricular Resources that integrate Computationally-Focused Science Practices

Rating: Very Good

Intellectual Merit

Strengths

  • Well-framed focus on translating epistemic practices into learning experiences for students, recognizing the distance between this goal and current teaching practices, and the limitations of existing curricular offerings intended to support science-as-practice.
  • Design-based implementation methodology likely to build on strengths of researchers and practitioners, and to produce results with external validity. Design conjectures to be explored are clearly-articulated.
  • Builds on pilot study.

Weaknesses

  • Data collection and analysis could use more detail. It would be helpful to include quantitative measures to show the the tool’s impact across contexts, in addition to the interpretive case study approach.

Broader Impact

Strengths

  • Partnership with Pennsylvania Department of Education to disseminate results statewide.
  • Partnerships with two school districts which had existing interest in transforming STEM practices; distinct characteristics. Both are low-SES; one has a high percentage of racially-marginalized students; the other is rural.
  • Development of a website to support teachers in accessing tools developed.

Weaknesses

Summary

This Teaching Strand Design and Development project would study how a tool for high-school biology teachers to select curricular resources could support the science and engineering practices of using and developing models, using math and computational thinking, and analyzing and interpreting data. The project would use a design-based implementation research methodology, working with teachers in two demographically-different school districts to develop tools for selecting curricular materials. Data collection and analysis would focus on qualitative case studies of professional development, teachers’ classrooms, and individual participants’ perspectives. While the project’s aims are incremental, it is very soundly-designed and seems likely to yield results having intellectual merit and broader impact.

2201289 (Secondary Panelist)

Curated Pathways to Innovation: Engaging Learners to Develop Computing Career Interest

Rating: Very good.

Intellectual Merit

Strengths

  • Makes a clear case for the need for programs such as this one to support marginalized youth developing interest in computing. The mechanism of curating is promising.
  • Builds on prior research with clear evidence of effectiveness.
  • Robust and feasible methodology.

Weaknesses

  • Existing and proposed affordances of the software are not backed by design rationales. This may limit analysis of why or how the tool was effective.
  • The theory of change (four-phase model of interest development; expectancy-value theory) does not address dimensions of marginalization (as presented in Table 1, for example) What is different about these groups or their experiences in school or with STEM? Without this framing, it will be difficult to interpret the data collected and act on it.

Broader Impact

Strengths

  • Existing track record of reaching marginalized populations and being valued by them.
  • Focuses on important questions about which demographics are able to benefit from the tool.

Weaknesses

  • While the project appears to already have strong and ongoing broader impacts (e.g. the tool is in use in schools), it is unclear how this specific proposal would increase the overall project’s broader impact.

Summary

This Learning Strand Level II Late-Stage Design and Development project would explore how an established remote/hybrid middle-school STEM+C program is experienced differently for marginalized students, and then improve the program based on what is learned and measure the results. With participation from education researchers, content experts, and teachers, this project would conduct a user experience study, engage in iterative design and development, and conduct a pilot study of the new design before disseminating results. This project addresses an important issue and has a high likelihood of yielding results with intellectual merit and broader impacts.

Summaries

2201421 (Nielson)

This Teaching Strand Early Stage Design and Development Study would have high school students and teachers learn computer science (CS), engineering design, and computational thinking (CT) in the context of biology, specifically by modeling biological phenomena in robots. The research would focus on non-hierarchical co-learning between teachers and high school students. The work would encompass studying two instantiations of the workshop with 15 students each and 5 teachers each combined with an additional week for teachers to develop their curriculum materials. These workshops have already run for several iterations with preliminary evidence of effectiveness. The current study would expand the workshops and engage in more systematic design-based research improving the model and research on how teachers and students learn in a non-hierarchical co-learning context. The proposal’s data management plan and postdoctoral mentoring plan are present and adequate.

The panel agreed that the emphasis on non-hierarchical student/teacher co-learning would be an interesting and important contribution, particularly in the context of teacher professional development and with an emphasis on teacher expertise. There was strong coherence between the project’s aims, the framing of the research questions, and the proposed methods.

One important limitation of the propsal was its oversight of the long history of research on students and teachers learning together (e.g. from the fields of Constructionism and Learning Sciences) as well as the more recent literature on computing and computational thinking in science. This absence weakened the proposal’s case for the novelty of its research focus. In order to bolster the research importance of studying co-learning in a professional development context, it would have been helpful to frame the research with existing literature on teacher learning in professional development and its translation into practice. Similarly, if the non-hierarchical quality of interactions is important, it would be helpful to theorize its importance (particularly as the proposal notes that it does not aim to challenge hierarchies which exist in classrooms).

The panel also felt that the proposal’s research design was missing important details in a few areas. The proposal lacked detail on what data would be collected when, as well as on the co-design structure and how iterative design-based research would progress. Using conjecture mapping (Sandoval, 2014) to articulate specific design conjectures and theroetical conjectures would be one way of making the design-based research process more specific while retaining necessary flexibility.

The proposal has substantial potential for broader impacts. For participants in the workshops, the opportunity to interact with practicing scientists, and to learn across the teacher/student divide would likely prove valuable. However, the panel had some concerns about how the learning in workshops will be translated into practice. As noted above, mechanisms by which this would take place were undertheorized and data collection around how the workshops translated into practice was limited (it appears that the teaching logs are intended to document how teachers put their learning into practice, but what these would include and how they would be studied is unclear.) It was promising that teacher alumni would be included as co-designers and that teacher and student alumni would serve on the advisory board, but the panel felt the absence of district representation on the team, and the absence of an expert on curriculum design or on professional development.

Finally, the panel felt that the likely broader impacts were incommensurate with the project’s requested budget. It felt like a reach to claim that the project would reach 3600 students by counting all of the students of the teacher participants when translating learning from the workshops into classroom practice was largely beyond the scope of this project.

2200917 (Klopfer)

This Learning Strand Early Stage Design and Development proposal would be a research-practice partnership between researchers and the Washington, DC school district, addressing two problems of practice: (1) the district wants all students to access computational modeling in science classes, and (2) district teachers need professional development around NGSS. The study would focus on three design-based implementation research (DBIR) questions: How do students’ work with computational modeling affect students’ learning core science ideas? (in one course and across multiple courses) And what kinds of design supports do students and teachers need to use science units integrating computational modeling?

One major strength of this proposal is that its research questions are important from the perspective of intellectual merit as well as broader impacts: research on how students learn through computational modeling in one course and across courses will contribute to fundamental questions about learning while likely having a substantial impact on practice. The proposal builds on strong research foundation showing the value of computational modeling for science learning, particularly across multiple subjects. The proposed research and theory of action also builds on a strong research base for teacher learning, developing effective professional development in the context of NGSS learning goals.

There were a few areas where more details on the proposed research would have been helpful. No theoretical rationale is given for the proposed extensions to StarLogo Nova, nor is a design process articulated for these features. Additionally, the proposal’s framing of science learning does not engage substantially with sociocultural or critical factors shaping students’ classroom experiences, potentially limiting its ability to engage with questions of equity and marginalization which may emerge within the proposal’s central research focus. It would also be helpful to have more details on the proposed assessments, and how the research would disentangle contextual differences across classrooms and schools from effects of the proposed intervention.

This proposal has very strong prospects for broader impacts. The work intends to engage with 45 teachers and 2,680 students as part of the curriculum development and implementation, and to offer research opportunities to master’s and undergraduate-level students. The depth of commitment to this project from both the research team and the school district is a distinct strength of the proposal, which makes it more likely to be impactful. The project budget and data-management plan are also clear and well-developed. The project would leave DC Public Schools with substantial curricular resources and enhanced capacity for future research-practice partnerships. More broadly, the findings would likely be relevant to policymakers and administrators making decisions about whether and how to invest in computational resources, tools, curricula and professional development for their science teachers.