2021 NSF AISL Reviews

Agency Name: National Science Foundation
Agency Tracking Number: 2115969
NSF Program: AISL
PI/PD: Proctor, Christopher
Application Title: Weaving Computer Science into the Fabric of Cultural Worlds Through Interactive Storytelling

Review 1

Rating: Very Good

Intellectual Merit

The project aims to refine and disseminate Unfold Studio as a tool to improve underrepresented students’ computational thinking skills and broaden participation in CS through interactive storytelling workshops.

Strengths

The researcher brings a wealth of experience to the project, including the initial development of Unfold Studio—an interactive storytelling program which allows students to create their own adventures.
Unfold Studio will be utilized at the Foundry, a community-based makerspace, in which middle and high school students will participate in integrated CS and literacy workshops.
The researcher also outlines in great detail how past theories have continued the need to broaden participation in CS education. If this project is successful, it will contribute key information to the field about how to support underrepresented populations in learning CS.

Weaknesses

More details would have been helpful about how the participants will be recruited for this project. While mention is made about the Foundry’s connection to the community, it is still unclear about where the students will come from and how they will be selected to participate.
Further details about the length and structure of the afterschool workshops would have also been helpful.

Broader Impacts

Strengths

The project aims to produce a number of deliverables, which can be broadly used by a variety of stakeholders.
Peer-reviewed papers and articles will help the research community better understand how to tap into marginalized students’ identities to help them learn CS and explore CS-related careers.
Unfold Studio can be used in formal and informal educational settings, along with the pedagogical and curricular resources. Altogether, these deliverables can help to replicate and scale the findings from the proposed project.

Weaknesses

More detail would have been helpful to more fully understand plans for the afterschool workshop.
Particularly given the researcher’s past implementation of similar workshops, an outline of activities or a typical daily workshop agenda would have offered up key information.

To what extent does the proposal identify the characteristics and needs of the targeted underrepresented groups (public or professional) to be served?

The researcher outlined characteristics of the Buffalo community and how they could benefit from this project.
In particular, an explanation was provided for dividing the participating students into 7th-8th and 9th-10th grade bands based upon their developmental needs and typical interests.

To what extent does the proposal include specific plans or strategies for addressing or accommodating the specific interests, community or cultural perspectives, and/or educational needs of participants of the identified underrepresented groups?

The theoretical underpinning of this project is that past attempts to broaden participation in CS have relied upon marginalized communities being recruited to participate in the dominant-controlled culture of CS classrooms.
Instead, this project proposes turning the tables so that the culture identities of “others” are recognized as legitimate within CS.

Summary Statement

Overall, this project offers a solid model for broadening participation of underrepresented youth in learning computational thinking through literacy-based workshops which emphasize identify and belonging. The theoretical framework was quite strong; however, more details about the program implementation itself would have been helpful in providing a clearer picture of the proposed efforts.

Review 2

Rating: Very Good

Intellectual Merit

The goal of this Innovations and Development proposal is to develop overlapping literacies: computational, textual, and geographic to support Computer Science learning between two groups of K-12 students: (1) 7th and 8th grades, and (2) 9th and 10th grades. The collaboration will be between the PI at the Univ of Buffalo and the Foundry, a community Makerspace in Buffalo, NY. This proposal is grounded in the hypothesis that the persistently low participation in CS by marginalized youth occurs because the focus is on skills- and knowledge-based training, but with little attention being paid to ones identities and cultures.

Strengths

A reframing of computational thinking more generally along three dimensions: (1) cognitive, situated, and critical, and the PI’s deriving a meta-framework that he has termed as computational literacies to create structural relations between these three dimensions. This proposal adopts this framework for informal CS learning involving theory dialogue in K-12 CS education. This is translated into literacies that involve interpersonal and contextual meaning-making practices.

Developing this framework in terms of interactive storytelling for literacy-based CT and CS education, and its implementation as a web-based application Unfold Studio.

A well-designed set of research questions that addresses: (1) whether the interactive storytelling approach supports the learning of CS; (2) what CS practices emerge through engagement with other literacies; and (3) how do the tools and pedagogies effectively catalyze the connections across literacies? The PI proposes to extend previous work to establish answers to these questions. The PI aims to conduct studies using Unfold Studio to create authentic learning experiences, address diversity and equity, and broaden participation among underrepresented youth.

The DBR approach proposed, and embedding the work in communities through the workshops run at the Foundry is likely to be quite effective, especially since this project will be connected to the Foundry’s existing programs.

Weaknesses

The connections of this approach to the education of STEM content are not well established. How exactly will CS and CT concepts and practices be built into the Unfold Studio activities?

What specific refinements beyond interface improvements will be made to Unfold Studio as part of this project?

More details about the back-end data collection and what kind of analyses will be performed would have been very useful. How exactly will literacy events and story edits be parsed to study computational skills? What are the details of the analysis methods? For example, is standard regression-based analysis the best way to study the relations between “different forms of CS learning?” It would be very useful to connect the high-level variables in Figure 4 to more specific variables derived from log analysis. There are also more automated path analysis methods, e.g., Ahn (2002) (Ahn, J.: Beyond single equation regression analysis: path analysis and multi-stage regression analysis. Am. J. Pharm. Educ. 66(1), 37–41 (2002)) and Wright (1983) (Wright, S.: On “Path analysis in genetic epidemiology: a critique”. Am. J. Hum. Genet. 35(4), 757–768 (1983)) that may be more applicable to this work. The social network analysis method to answer RQ2 and the conjecture mapping approach for RQ 3 could have been better explained.

Broader Impacts

Strengths

The idea of connecting CS literacy and education to the cultural identities and contextualized story-telling scenarios is appealing

The idea to reach out to three key audiences: workshop participants’ communities, formal and informal educators, and the academic community interested in informal STEM learning.

Website that makes pedagogical and curricular materials available.

Weaknesses

How will Unfold Studio be made available to other researchers and practitioners?

What is the plan to further develop and sustain the international network of teachers and practitioners using Unfold Studio?

To what extent does the proposal identify the characteristics and needs of the targeted underrepresented groups (public or professional) to be served?

The proposal clearly identifies K-12 minority youth population from Buffalo, NY, which is predominantly Black with a recent influx of refugees. This community is largely disconnected from opportunities to learn and profit from CS Research.

To what extent does the proposal include specific plans or strategies for addressing or accommodating the specific interests, community or cultural perspectives, and/or educational needs of participants of the identified underrepresented groups?

The PI proposes to use storytelling as a way of preserving cultural identities and context (primarily geographic) in teaching CS literacy.

Summary

The idea of using overlapping literacies to educate underrepresented students in Computer Science literacies is somewhat novel, and Unfold Studio represents a fairly mature system that can be deployed in both formal and informal learning environments. The proposed research is theoretically well-framed, but some of the details of how the system development will be advanced and the analysis of student work will be performed could have been explained in more detail.

Review 3

Rating: Fair

Intellectual Merit

Strengths

Reconceptualizing Computer Science in terms of students’ cultures and worlds is a new research perspective.
The web application, Unfold Studio, for interactive storytelling could capture students’ different literacy about Computer Science.

Weaknesses

The main purposes are only updating functions of the existing system (i.e., Unfold Studio) and developing analytics platform.
It is not clear how the interactive storytelling activities would practically enhance CS or STEM education in addition to reframing CS literacy.
The differences among cognitive, situated and critical CS learning assessment are not clearly described. Also it would be better to provide developed assessment of cognitive CS learning.
Pedagogical approaches embedded in Unfold Studio are not fully described.

Broader Impacts

Strengths

The ongoing partnership with a community makerspace

Weaknesses

This proposal focuses more on theoretical perspectives about computational literacies.
It is not clear what would be disseminated to other STEM educators.

To what extent does the proposal identify the characteristics and needs of the targeted underrepresented groups (public or professional) to be served? To what extent does the proposal include specific plans or strategies for addressing or accommodating the specific interests, community or cultural perspectives, and/or educational needs of participants of the identified underrepresented groups?

Weaknesses

This proposal does not include specific information about the characteristics and needs of the targeted underrepresented groups

Summary Statement

This proposal is about CS literacies based on three epistemological aspects and aims to improve the web application system for interactive storytelling. The target category is innovations in development of AISL, but the plan focuses mainly on theoretical and research perspectives. Since the PI has been working on the system in many years, more detailed and practical implementation plans for broader audience need to be provided.

Review 4

Rating: Very Good

Intellectual Merit

Strengths

The proposal points out that CS framing is one of the main causes of low participation in the area. The proposed approach is to use interactive storytelling to change computing literacy. While doing so, the PI will study the relationships between the individual, situated, and critical framing of CS learning. The problem is well-document; the solution is original and backed by literature.
The overall plan is sound. There will be eight after-school workshops for participating youths on interactive storytelling. The platform to be expanded and used for the project has already been developed. However, it needs more functionalities to be developed (data analytics, I believe). Storytelling-based teaching of programming concepts is not a new area – we have seen Alice and Scratch. However, those are more visual and focuses heavily on the structural blocks of programming. Instead, the proposed project takes a much straight-forward and relatable approach of text-based gaming. The idea stays away from fancy visual representation and stays true to one of the oldest forms of building human connection – through storytelling. Needless to say, this is an exciting idea that is likely to change how youths and the public view computing (in a more favorable term).
The team is perfect for carrying out the tasks. There are community partners, two graduate students from the departments of Learning and CS. There will be an instructor and an artist from the community partner, Foundry. Each position has tasks they are responsible for.

Weaknesses

The last research question (RQ3) is broad, and the plan to pursue it is not specific. Especially, the ‘pedagogy’ part of the question can benefit from having more details. The other two questions are sound.

Broader Impact

Strengths

The project will be changing the way CS is framed, which will make it more likely for youths to related to computing literacy. This, in turn, will increase participation in computing, especially from minority demographics.
The project will serve up to 128 youths in grade 7-10 in Buffalo, NY – a region that is largely disconnected from computational literary practices. Additional 200-300 members of the extended community will also experience part of the project. There will be art installations open to the community to foster community engagement.
The Unfold Studio platform is already available to the public (free). Pedagogical and curriculum materials will be made public so that other researchers and educators can utilize that.

To what extent does the proposal identify the characteristics and needs of the targeted underrepresented groups (public or professional) to be served? To what extent does the proposal include specific plans or strategies for addressing or accommodating the specific interests, community or cultural perspectives, and/or educational needs of participants of the identified underrepresented groups?

It is a well-document problem that computer science and the field of computing lack diversity. Low participation from minority groups in computing has been a consistent trend for decades. The proposal addresses this area, pointing out one possible cause of how current computing narratives are alienating minority demographics. As a solution, the project proposes an alternative, interactive storytelling-based approach to lower participation barriers.
The proposed plan is sound in terms of reaching the target audience, the youths from 7-10 grades (broken into two groups for their respective developmental states) and the public. The proposal presents a flexible framework to allow participants to relate different literacies (textual, geographical, etc.) with computing literacy. This creative freedom is likely to engage students in exploring computing in a more in-depth manner.

Summary Statement

This is a very well-written, well-planned, and well-reasoned proposal. The proposed work is original and transformative in nature; it aims to use interactive storytelling to interact with the computer via programming. This concept can change the narratives and identities linked with Computer Science that are linked to lower participation from minority groups. The PI has already developed the platform to be used for the project, so it is easy to see the power and potential of the idea. The team is highly qualified and well-equipped to carry out the work. Overall, an excellent proposal that can shape the future direction of CS education in both informal and formal settings.

Panel Summary

Project Overview

This is a four-year-long, collaborative AISL “Innovations in Development” proposal. The University of Buffalo and a community Makerspace, Foundry, come together to study CS learning in the context of overlapping literacies (computational, textual, geographical). The proposal is based on the hypothesis that the low participation in CS is mostly due to the skill and knowledge-based framing of CS. Instead, if there is a culture and identity preserving framework, similar to the one proposed, it is likely to engage the marginalized youths.

Intellectual Merit

Strengths

Teaching computing through computing literacy is an interesting framework. Reconceptualizing Computer Science from students’ point of view is innovative.
Interactive storytelling with the text-based game through the Unfold Studio platform is a novel idea.
The team has the expertise to execute the proposed work. There is an existing collaboration between the university and the community partner, Foundry.
The digital platform is currently available online and functional.
The project will be using a design-based research approach in engaging the community with the project.

Weaknesses

Given that the Unfold studio platform is already there, more details on what would be added to the system would have been helpful. One of the improvements listed for Unfold Studio is related to backend data collection. The proposal barely addresses how the data would be analyzed or what methods will be used. Even though regression-based analyses are mentioned, there are better techniques available (e.g., path analyses) that can yield deeper insights into the data.
It is not clear how the CS learning would take place through Unfold Studio. How would the stories be analyzed for identifying and measuring computing concepts? The connection between storytelling and how it will help to learn STEM and CS would strengthen the proposal.

Broader Impacts

Strengths

Connecting CS literacy with cultural literacy will bring a much bigger audience.
The storytelling approach is relatable. Students can write the story, contextualize the story, and share the story. Another aspect of engagement is the ability for students to extend existing stories (or collaborative storytelling).
The research group is well-defined. The connection with a pre-existing part of a community - the Makerspace group at Buffalo - as opposed to trying to start something from scratch, is a strength of the proposal.
Listed set of deliverable items is rich.

Weaknesses

How will unfold studio be shared with other researchers?
The recruitment plan should be made clear with more details. What will connect the students with the project?
Even though the PI has done similar workshops before, details on the proposed workshops are largely missing.
The sustainability plan of the project should be addressed.

Additional Solicitation-Specific Criteria

The researchers described how Buffalo is underserved from a CS education perspective and how the proposed work will benefit this community.
The proposed plan is well-designed for reaching the intended audience, the youths from 7-10 grades, the public, formal and informal educators, and learning researchers. The proposal presents a flexible framework to allow participants to relate different literacies (textual, geographical, etc.) with computing literacy. This is likely to engage participants with computing.

Summary Statement

The proposed research is theoretically well-framed. The proposal is extending prior research of computing literacy from one area to multiple overlapping areas. The idea of using overlapping literacies to support identity building for the marginalized population in Computer Science is a novel one. The team is well-equipped with the existing storytelling platform. However, more details on system development, program implementation, and student learning assessment would have made the proposal stronger.

The summary was read by the panel. The panel concurred that it accurately reflects the panel discussion.

PANEL RECOMMENDATION: Competitive