NSF CAREER Reviews

Panel Summary

Description of project

This project proposes to develop assessments that operationalize cognitive, situated, and critical framings of computational thinking, deploy them into classrooms to validate assessments, and then understand their role in a pre-service CS teacher preparation context. The work is done in the context of computational storytelling. The panel interpreted this proposal as a detailing of a collection of assessment techniques and understanding situated and critical computing literacies.

Intellectual Merit

Strengths

Aim to develop a model to articulate the mechanisms that this will be applied to neutral outcomes. It’s important to assess curriculum and see whether it’s making its impact to assess CS curriculum or culturally relevant.
The panel expressed appreciation for the terms and figures that were explained well (like ground truth value).
The panel discussed how computational thinking as a construct in the field is still poorly defined. The panel indicated that this proposal could help deconstruct three levels of assessment with cognitive, situated, and critical, specifically on how to operationalize these. One panelist pointed out that while many studies have looked at cognitive level, this proposal’s merit was focused on the assessment in the critical spaces, which was intriguing.

Weaknesses

The panel discussed about the target population and the size, and indicated that while the community was mentioned as diverse, the proposal didn’t break down the demographics. What does this mean to be exceptionally diverse? What about the teachers? Are these teachers diverse? What does this mean in terms of the kinds of learning that ensues?
One panelist recommended that the PI should consider adding Kim Scott for culturally relevant computing expertise.
One panelist mentioned that although the proposal mentioned it would be building an assessment with measures of validity, the proposal ignored literature that discusses how to create assessment with measures of validity.
One panelist pointed out that while they were expecting strong examples of computational thinking, it was presented as measures of storytelling, which they felt was problematic. There was a mixed discussion about what constitutes computational thinking and whether the storytelling examples were salient enough to represent CT.
Some panelists were concerned about the text-based storytelling. It was unclear how the assessments might apply to different domains. Panelists indicated that this work was very discrete and statistical modeling was being used to design the assessment and described tensions between situated and qualitative terms.

Broader Impacts

Strengths and Weaknesses

The panel discussed at length about the use of San Diego schools. Although it was impressive to use a large, diverse district for such an intervention, it also felt unethical since the interactions with students and their data was not well thought out. One panelist mentioned that it felt cavalier to use all this student data with such little planning. The panel believes that more consideration and planning would yield a stronger proposal. A deeper discussion of how to ensure those students are best served by this approach as well as a broader discussion of how to gain adoption beyond SDUSD (and teacher prep at University of Buffalo) would be welcome additions to the proposal.

Solicitation-specific Review Criteria

Data management plan

The data management plan was detailed and concrete.
The letter of support from the department chair was supportive.
The integration of research and education is well-integrated, but needs a deeper discussion of how to ensure those students are best served by this approach.

Constructive suggestions for improvement

The panel suggested that this proposal lays foundations for instrumentation and investment for future research. The panel suggests that perhaps consider removing the implementation with the San Diego school district (or develop a more thoughtful plan) would be advantageous. One panelist mentioned that just examining the three different levels of assessment would be impactful and could be the focus of future proposals.

Additional Comments

Summary Statement

Overall, the panel appreciated the focus on the three different levels of assessment and working with the term computational literacies. However, the use of 26,000 students’ data from the SDSUD district lacked a clear plan and rationale and lacked detail.

The summary was read by/to the panel and the panel concurred that the summary accurately reflects the panel discussion.

PANEL RECOMMENDATION: Low Competitive

Review 1

Rating: Very Good

Summary

The premise of this proposal is that the PI wants to operationalize computational literacies into cognitive, situated, and critical framings of computational thinking via assessments that will be developed and then integrated into his Unfold software platform that allows students to develop programming skills through the use of narrative storytelling. The PI also aims to leverage those stories to understand and assess how making computer science learning visible through these assessments helps pre-service teachers conceptualize and implement just and equitable CS learning in their classrooms.

This is an incredibly well-organized proposal. It’s very clear. The research design is very strong.

A strength of the proposal is that it aims to develop a model that’s going to help articulate the mechanisms by which what appears to be “neutral CS curricula” can actually lead to disparate outcomes across race and gender, and even at those intersections. The literature on Black girls, as an example, reveals that this phenomenon happens quite a bit. As such, having some type of a model that can help to really articulate and refine and make more salient how that happens, even in well-meaning CS curricula, is really important.

A strength is that this proposal is incredibly explanatory. Most of the terms are explained, and the figures are explained. Terms like ground truth value are explained. The PI’s thought process and the rationale for why particular design decisions were made is very visible, which is a real strength of this proposal.

The proposal states that the context of the targeted population will be middle school computer science students across San Diego Unified School District, which enrolls 121,000 students (K-12), and 26,000 of which are in grades six through eight. That information is important, but the PI needs to go further. For example: What’s the demographic data for those 26,000 students in grades six through eight? What backgrounds do they come from? The PI states that San Diego is “an exceptionally diverse community”, but does not discuss how that translates in terms of targeted student populations, which would definitely add strength to the proposal.

Just like the demographics of the San Diego Unified School District are needed, so are the demographics of the teachers. This is important because there may be teachers who find themselves in classrooms, especially computing classrooms, teaching students who are demographically very different from them. It makes a difference in terms of the type of learning experience that ensues and the ways in which teachers and students interact and engage with each other. The demographic data is important in trying to think through and unpack how those differences may show up in the classroom and how they may manifest themselves in the classroom. Additionally, that data is important in thinking through and discussing how the demographics of teachers and students contribute to the creation and sustaining of, or the breaking down and detriment of equitable and just learning.

I actually would recommend that one person be added to the advisory board, Kimberly Scott, specifically because her work deals with culturally relevant and culturally responsive curricula within computing, which is a component that is missing here, especially given that equitable and just have not been adequately defined.

Broader Impacts

A strength of the proposal from the perspective of broader impacts is that the proposal aims to address issues of equity and justice in computing. This is an understudied aspect of CS education that needs more attention and study.

The biggest weakness is that just and equitable CS learning is not defined. It is not clear what the PI believes comprises a just or an equitable CS learning environment. This is important because just and equitable can mean different things for different people depending upon their backgrounds and where and how they’re situated with respect to the learning environment itself. The PI states this positionality makes a difference in the proposal, but the proposal doesn’t describe what a just and equitable CS learning environment actually is or what it looks like.

In the very beginning, the PI makes a statement that says, “learning experiences whose outcomes are measured in exclusively cognitive terms are likely to be particularly alienating for minoritized students that perpetuate longstanding inequities in CS education”. However, the proposal doesn’t explain why that’s the case. Making the argument regarding why measuring CT exclusively in cognitive terms is likely to be particularly alienating for minoritized students and why that positionality perpetuates longstanding inequities and CS education is needed.

Solicitation-specific criteria

The education plan is well integrated into the research plan.

Summary Statement

This is a strong proposal with a strong research design. Additional expertise is needed around culturally-relevant and responsive curricula and how these inform the design of equitable and just learning environments. Further, equitable and just learning environments needs to be defined. However, the research design itself is strong enough that I rated the proposal very good.

Review 2

Rating: Poor

Summary

The project intends to develop three proposed computational literacies related to computational thinking: cognitive, situated, and critical. The project then intends to develop assessments of these literacies and gather evidence of validity for those assessments. As part of the data collected to validate the assessments, he plans to develop a system to help predict which students are struggling with different dimensions of CT and aid instructors by providing those predictions.

Intellectual Merit

Strengths

A better understanding of computational thinking would be of value to the broad computing education community. Computational thinking is poorly defined, often meaning everything and nothing at the same time. This project has the potential to better refine what computational thinking means, particularly in a K-12 context.

Weaknesses

This reviewer is deeply concerned about the plan to develop validated assessments in this proposal. Given existing work on these literacies, the next step is to better define these three dimensions for the community by developing materials, soliciting feedback from instructors and researchers, and gaining a deeper understanding of how students view these issues. Moreover, the examples given in the proposal for situated and critical competencies are arguably not computational. They examine student’s story telling ability which is indeed interesting, but the connection to computing is weak at best. Given that there isn’t agreement yet in the community about whether these three literacies should be the focus of instruction to K-12 students, trying to create a validated assessment of these literacies is simply premature.

In addition, there are existing methods for gaining evidence of validity for assessments and those are mostly ignored in this proposal. The PI has no prior support from the NSF nor does the PI have any background in assessment design or validation. For example, a key step in designing an instrument is gaining consensus from potential adopters on what should be measured and, given these dimensions of CT are new, this is extremely difficult to accomplish at this point and the proposal notably does not address this challenge. In addition, there are established methods to gather evidence of validity of the assessment when piloted (Classical Test Theory, Item Response Theory) that are not mentioned here. It could be that the PI has good reason to use other approaches than those established in the community, but if so, I would expect to see a justification for why he is not using those techniques. Overall, given the validation process as proposed, I do not believe the assessments created in this project will be of value to the community.

Data management plan

Acceptable

Broader impacts

Strengths

A key strength of this proposal is in the broader view of computational thinking that may better capture the value of computational thinking to a diverse population of students.

Weaknesses

Dissemination and adoption are key components of the broader impact of an assessment. As mentioned under IM, the lack of involvement in stakeholders in the design of the instrument vastly limits potential adoption.

This reviewer is also deeply concerned that this pilot of computational thinking will be run with 26,000 students and that the data gathered will be used to create predictive tools of student success. Given the lack of preliminary research findings on the benefits of this approach to teaching CT to students, having this large a number in that initial research study, particularly one that aims to predict student outcomes, poses significant ethical concerns that are unaddressed in the proposal.

Solicitation-specific criteria

Letter of Departmental Support: Department is supportive of his effort.

Integrating research and education: The use of Unfold Studio in schools means that this will likely be used by students in the SDUSD. A deeper discussion of how to ensure those students are best served by this approach as well as a broader discussion of how to gain adoption beyond SDUSD (and teacher prep at University of Buffalo) would be welcome additions to the proposal.

Collaboration plan (if any); letters of collaboration: His collaboration with SDUSD is supported by ongoing efforts.

Summary Statement

Constructive suggestions for improvement: Please see my comments under weaknesses for both IM and BI.

Overall: There are truly creative ideas present in this proposal with a major potential contribution of better scoping/defining computational thinking for a diverse population of students. There are significant weaknesses, however. The plans to design and validate the assessment are ill-defined, perhaps reflecting the PI’s lack of experience in assessment design. Moreover, the lack of preliminary findings demonstrating the value of these lenses on computational thinking make this proposal premature both in the desire to create assessments with evidence of validity and question whether it is ethical to expose an entire school district to this approach. If the proposal is funded, I would expect to see these major concerns addressed by the PI before starting the work. If the proposal is not funded, I hope the PI addresses this feedback and submits again, as there is value in the ideas of the proposal.

Review 3

Rating: Very Good

Summary

Intellectual Merit

The core intellectual merit of the project is in addressing the notable gap in research and instruments for understanding situated and critical computing literacies.

Strengths

The research is clearly novel, given the substantial gaps in prior work in this space.
The research would enable other work, answering questions about the interactions and tradeoffs between different types of literacy in teaching, or reveal interactions between particular types of literacies and student identity.
The work advances discourse on what constitutes computational literacy, an area that is still developing, but is critical to shaping both research and practice.
The research framing and methods are strongly grounded in well-established design traditions
The cognitive operationalizations are creative and novel, focusing on both process and product
The critical operationalizations are speculative, but potentially revealing, and show some promise.
The use of the preservice course as a context for summative evaluation seems risky, but the use of two contexts with and without the dashboard may help reveal helpful contrasts.
The advisor board offers strong complementary expertise.

Weaknesses

Some of the operationalizations are necessarily situated on the storytelling domain, which may limit their application in other domains of computational literacy. This is an unavoidable weakness, however; it’s likely many operationalizations would be needed for different domains. Storying is a reasonable domain in which to start.
There is some risk that the highly discrete, quantitative, and modeling-focused nature of the project will end up clashing with the many social nuances of situated and critical literacy. The long history of qualitative methods in this space was likely no accident, and so there are reasons to be skeptical that these approaches will have merit. This weakness is tempered by the PI’s history of success in this space, and so if anyone were to make progress with this approach, it would likely be the PI.

Data management plan: The plan was detailed, concrete, and specific about what would be gathered, how it would be archived and protected, demonstrating a clear sense of the proposed work plans and the data to be produced.

Broader impacts

Strengths

The direct interaction with SDUSD promises some degree of direct impacts on students and classrooms.
There are clear synergies between the proposed work and the PI’s role in pre-service teacher preparation.
The grant offers new kinds of research opportunities at the PI’s institution

Weaknesses

It was hard to see how the work would have any direct impacts on equitable CS learning or teaching, as it largely focuses on research infrastructure (e.g,. instrument development), and so the potential impact on broadened participation in CS is highly speculative (but promising).

Solicitation-specific criteria

Letter of Departmental Support: The letter offers compelling indicators of growing support and mentorship at the institution and nuanced awareness of the synergies between research, teaching, and service.

Integrating research and education: The integration plans were high level, and largely focus on emergent, opportunistic integrations through a community of practice spanning different contributors to the project. The students in teacher preparation courses are a compelling part of this work, however, adding some bridge between the assessment design and teacher education classrooms. It would have been nice to see more concrete ideas about how the activities interact: for example, was it experiences with pre-service teachers that led to a demonstrated need of situated and critical assessments and dashboard? How might both pre and in-service teachers been engaged in shaping the work? There was a sense that the work would be done for teachers rather than with them, which is a missed opportunity for integration.

Summary Statement

If the proposed work develops meaningful and reusable instruments, it would empower other research and practice that is not currently possible. That is potentially transformative, but contingent upon how reusable the work is in domains and platforms chosen for the project. This is potentially transformative.

Due to the many strengths, only modest weaknesses, and high potential of the work’s impact on future research and practice, I recommend the proposal for funding.