Unfold Studio: Supporting computational literacies through interactive storytelling

Submitted to AISL NSF #2215531

Review 1

Rating: Fair

Intellectual merit

What is the potential for the proposed activity to advance knowledge and understanding within its own field or across different fields (Intellectual Merit)? To what extent do the proposed activities suggest and explore creative, original, or potentially transformative concepts? This proposal has strong potential for advancing an innovative approach to CS learning through narrative storytelling that could connect in unique ways to the needs and interests of informal learning communities focused on writing. This proposal could also further what is known about the role of narrative in CS learning.

However, there are a few key weaknesses to the Intellectual Merit in the current proposal. This proposal sits at the intersection of CS and the use of narrative storytelling that has a rich history over several decades that is not well laid out in the proposal. Key questions about this intersection of writing and CS in informal settings are also unanswered (as much of this prior literature is based in compulsory school settings). Consequently, it makes it challenging for the reader to determine the relative contribution of the proposal to this rich history of work (especially how much of the problem is rooted in self-citations). In addition, while there are many starts of excellent ideas throughout the proposal, there are also many ideas that compete for top billing, forcing the narrative to lack a clear rhetorical structure that would build toward a convincing argument. For example, does the proposal advance what we know about informal literacies, about sociocultural and critical perspectives, use of narrative in CS, about use of narrative in makerspaces, etc.? While I understand that the PI would like to advance all of these things, the proposal would benefit from a clearer foundation for a tighter problem space that would argue for the current design and innovation as the focus of the design research and setting are a non sequitur in the current framing. I would like to suggest that the PI partner with a more senior scholar in the field in subsequent proposals to help hone these ideas since they have quite a bit of potential. Lastly and perhaps most importantly, it’s unclear why the program needs a new UI and backend analytics? There is little detail on what this would look like both in practice (how will these workshops operate exactly? How will the designs advance the prior work?). The start of proposal led me to errantly believing that Unfold Studio was a newly proposed design and so was surprised to later learn that the current proposal aims to iteratively refine the current designs and build upon the presumably prior dissertation work. While that could work in a proposal like this, the current proposal does little to outline the new work and the motivation/need for these features to advance our understanding of learning CS in informal settings.

Is the plan for carrying out the proposed activities well-reasoned, well-organized, and based on a sound rationale? Does the plan incorporate a mechanism to assess success? No, it’s really not clear what would be developed and how the workshops would be organized. As outlined above, more details are needed in this section of the proposal as it stops short of outlining what will be newly designed and why. In addition, it’s not yet clear that Unfold Studio engages learners in a broad range of CS concepts. From the model, I see a starting event function and the repeated uses of conditionals. Does this design offer a high ceiling of expressive CS in the story structuring? Does the youth writing lend itself to this type of engagement? A clearer table and analysis of the STEM + CS engagement is needed in a high competitive proposal.

How well qualified is the individual, team, or institution to conduct the proposed activities? It is not clear that the team is capable of doing both the iterative UI and backend analytics design needed. Much seems to rest on the GSR that will be hired to do this design work. The PI’s technical credentials as well as more background on this GSR is needed (or potentially consider another solution to ensure confidence, such as partnering with another faculty member in CS/Software Engineering, sub-contracting with an outside agency, etc.). Not seeing mock-ups of the newly proposed UI or backend analytics, which also diminishes confidence in the proposal.

Are there adequate resources available to the PI (either at the home institution or through collaborations) to carry out the proposed activities? No, it does not appear so. This work would be strengthened through a small pilot investment from the home institution as well as supports for the PI on grant writing activities. These are ideas worthy of these initial institutional investments needed to put together a highly competitive proposal.

Broader impacts

Strengths: The project aims to offer sustained CS learning opportunities to 128 youth at the Foundry. Over 75% of participants in The Foundry’s STEAM education programs live in Buffalo’s highest-poverty zip codes. The study’s focus on narrative storytelling and CS also has implications for broadening participation in STEM more broadly.

Weaknesses: What is less clear about the description of the youth targeted for direct instruction is the exact demographics in terms of race and gender and whether these highest-poverty areas are among the nation’s highest poverty areas. The proposal does not clearly address the information needed to adequately assess. Further, it’s unclear how this work will build toward impact beyond this initial set of youth that would be expected at this stage of an Innovations in Development proposal. Partnering with the National Writing Project or MakerEd to conduct some pilot national workshops in the later stages of the grant would be a valued addition.

Data Management Plan: Sufficiently addresses the types of data collected and the protocols needed to maintain confidentiality throughout the span of data collection, analysis, and later archival plans.

Additional criteria

Does the proposal identify the characteristics and needs of the targeted underrepresented groups (public or professional) to be served?
Does the proposal include specific plans or strategies for addressing or accommodating the specific interests, community, or cultural perspectives, and/or educational needs of participants of the identified underrepresented groups?

While the proposal demonstrates that the PI generally understands the needs of young people targeted in the proposal (though not 100% convinced that this type of “choose your own adventures” are genres of writing still of interest to young people), what is less clear is how the proposal describes both the makerspace and practices of the young people that attend the makerspace. It’s clear that the PI and the makerspace are in the early days of building a research partnership. What is less clear is whether the proposed set of activities will be engaging in this setting (since the Foundry is best known for learner engagement in digital fabrication technologies toward entrepreneurial ends). As the proposal currently stands, this does not seem like the right context for this type of writing workshop (e.g., 826 or fan fiction communities seems like a more natural context for this digital storytelling work) and there is little pilot evidence to support that the youth would be interested in such a workshop. A stronger pilot study is needed for this level of award and 4-year investment.

Summary Statement

This Innovations in Development proposal would fund four years of collaboration with The Foundry, a community makerspace, in researching how interactive storytelling catalyzes computer science (CS) learning through multiple iterations of participatory design-based research. The PI aims to further develop Unfold Studio, a web application for learning CS through interactive storytelling and advance a narrative, text-based approach to programming, which holds the potential to sustain identities and cultures while developing core CS competencies. The key strengths lie in the intellectual merit of the proposal, particularly in its innovative way of weaving together narrative storytelling with CS learning. Key challenges in the current proposal include the lack of clear pilot evidence that these designs would be of interest to the youth in makerspace settings (and that they would sufficiently engage both the writing and the CS), the lack of detail on the proposed design iterations, and the lack of a clear rhetorical argument/contribution to the literature. This work holds great potential and I highly encourage the PI to continue with this line of work, potentially making use of internal university supports for a pilot (particularly given that this is a new and junior faculty member at the institution) and/or apply for an earlier stage AISL proposal, like the Pilot and Feasibility Studies. This would help to move this work beyond the dissertation and better situate it in its new context.

Review 2

Rating: Fair

Intellectual merits

Strengths:

This project addresses the hypothesis that low participation is due to the way CS is presented to underrepresented youth, and that by reframing this presentation using personal storytelling will engage these youth who may not think CS is for them.
The product (Unfold Studio) has been through design-based research and teacher/facilitator materials have been developed and tested.

Weaknesses:

The explanation of how assessment of the computational literacies would be developed and conducted needs more details and research-basis.
The broad claim that this project will “organize the nascent field of computing education research” is far-reaching and not supported in the proposal.

Broader impacts

Strengths: The revisions to develop a “highly-useable” and “freely-available” version of Unfold Studio will provide for a tested tool for integration of storytelling (writing) as a way to teach computer science concepts to students who may be interested in literacy vs. standard STEM methods of teaching CS concepts.

The research regarding the use of storytelling to teach computer science and computer science to teach storytelling could provide data to support integration in the literacy / writing classroom as a tool for reaching ALL. This is especially needed in secondary classrooms where the students can self-select electives and opt-out of CS.

Weaknesses: The “community art installation” in this project seems like an afterthought (p. 12 and 16) and disconnected from the research project.

Data Management Plan: The data management plan addresses how data will be stored, organized, and made available for re-use/re-distribution and archival. Any user data (log-in, artifacts, …) is addressed, especially with respect to data privacy, through the IRB-protocol and consent of workshop participants. There is no specific information on how student participant data (work products) other than “IRB-protocol”. This is sufficient, but could also include a description of the participating student age range and the need for FERPA compliance and consent/assent from parents/guardians as part of the data management plan.

Additional criteria

Does the proposal identify the characteristics and needs of the targeted underrepresented groups (public or professional) to be served?

Yes, a thorough discussion of the needs of the underrepresented (public) groups are described on p. 12.

Does the proposal include specific plans or strategies for addressing or accommodating the specific interests, community, or cultural perspectives, and/or educational needs of participants of the identified underrepresented groups?

Cultural awareness of the community is central to the project through use of storytelling to learn CS.

Summary Statement

This proposed project builds on design-based iterations to develop Unfold Studio, integrating/teaching CS through literacy and storytelling. Cultural relevance is baked into the storytelling workshop through participants personal interactive story telling. There is strong potential for broad impact with research outcomes regarding student engagement and self-efficacy in CS based on the storytelling curriculum. However, in this project students self-select participation and therefore, there is a selection bias inherent in out-of-school and elective activity choice that affects the study population and results.

Strengths: The revisions to develop a “highly-useable” and “freely-available” version of Unfold Studio may provide for a tested tool for integration of storytelling (writing) as a way to teach computer science concepts to students who may be interested in literacy vs. standard STEM methods of teaching CS concepts.

Weaknesses: The “community art installation” in this project seems like an afterthought (p. 12 and 16) and disconnected from the research project.

Does the proposal identify the characteristics and needs of the targeted underrepresented groups (public or professional) to be served?

Yes, a thorough discussion of the needs of the underrepresented (public) groups are described on p. 12.

Does the proposal include specific plans or strategies for addressing or accommodating the specific interests, community, or cultural perspectives, and/or educational needs of participants of the identified underrepresented groups?

Cultural awareness of the community is central to the project through use of storytelling to learn CS.

However, in this project students self-select participation, and therefore, there is a selection bias inherent in out-of-school and elective activity choice that affects the study population and results.

Review 3

Rating: Fair

Intellectual merit

Strengths: This study investigates how youth’s engagement with Unfold Studio,-an interactive storytelling web program, facilitates CS learning. The proposal seeks to operationalize the computational literacies framework in informal settings. This proposal offers a creative way to engage young people in multiple literacies and has the potential to advance knowledge and understanding in the field of informal CS education.

The proposed activity will be implemented in collaboration with The Foundry a local youth serving organization. The project design is guided by a clear logic model and includes improving UnfoldStudio and creating an analytic platform into the web program to collect data from youth grades 7-10. The project builds on prior work conducted by the PI that demonstrates simulative learning is associated with cognitive CS learning in the use of Unfold Studio.

Weaknesses: Information on how The Foundry will select youth participants is needed.

The project implementation describes semester-long workshops for youth, and information on the length and frequency of these is absent. Additional information on support provides to staff at The Foundry who will be on-site during program implementation is needed.

Information on the supports provided for youth who serve as peer mentors could strengthen the proposal. Evidence that grade level equates skill and knowledge in regard to peer mentoring with this software is needed.

The project timeline indicates workshops will be implemented before the assessment tools are improved and validated. Information on the rationale for this decision and when these assessments will be given to participating youth is needed.

The advisory board offers varying knowledge and skills to complement this project. Evidence that members of the advisory board have the expertise to serve to conduct a rigorous evaluation is needed. Information on who in the board will be the lead evaluator will strengthen this proposal.

Resources and collaborations with the home institution and community partners are adequate to implement this project.

Broader impacts

Strengths:

The UnfoldStudio platform will be available to the public and has the potential to engage youth at the national and international levels. Curricula to support the implementation of this educational intervention will be publicly available.
The PI will manage the project with The Foundry director. This partnership leverages local resources to achieve project’s proposed outcomes and strengthens this partnership.
Project results will be disseminated in journal articles for scholars, conferences, and professional development for practitioners. UnfoldStudio will serve as a research platform for others.

Weaknesses:

Project will engage a limited number of youth during implementation. At best, it will reach 128 youth if young people are not return participants, otherwise the number will be lower.
A collaboration with the schools and schoolteachers where potential youth participants attend, would leverage additional resources and partnerships, and support the achievement of the proposal’s goals.
The proposal offers a critique of computer software companies that invisibly watch users every move, collect and disseminate data. In the data collection section, the PI proposes collecting data on how users interact with Unfold Studio and indicates this data may be shared with others. Information on the rationale to share this data with whomever requests it would strengthen the proposal.

Additional criteria

Does the proposal identify the characteristics and needs of the targeted underrepresented groups (public or professional) to be served?

The proposal describes the potential participation of 128 youth in grades 7-10. Most youth live in low-income communities. An unknown estimated number of youth may be immigrants.

Does the proposal include specific plans or strategies for addressing or accommodating the specific interests, community, or cultural perspectives, and/or educational needs of participants of the identified underrepresented groups?

The proposal describes a collaboration with WEDI, a non-profit organization that supports refugee youth as a possible strategy to accommodate youth’s needs. The proposal describes the software platform is designed to respond to young people’s interests and cultural perspectives as youth create their own stories.

Summary Statement

This study investigates how youth’s engagement with Unfold Studio,-an interactive storytelling web program, facilitates CS learning. This proposal offers a creative way to engage young people in multiple literacies and has the potential to advance knowledge and understanding in the field of informal CS education. The project design is guided by a clear logic model and includes improving UnfoldStudio and creating an analytic platform into the web program to collect data from a small sample of youth.

Additional information on participant selection, length and frequency of workshops, support provided for youth who serve as peer mentors could strengthen the proposal. Evidence the advisory board has the expertise to conduct a rigorous external evaluation is needed.

Review 4

Rating: Fair

Intellectual merit

Strengths:

The theoretically based computational literacies framework focusing on identity-building and the strong justification for its inclusion in informal learning contexts is an important strength.
The PI includes a thoughtful justification for the age groups focused on in the proposal and how they are divided into two different tracks for workshops.
The Foundry seems like a strong community partner that will contribute to the project’s success.
The PI appears to have significant development of this project completed/underway as part of a previous project, as well as pilot data demonstrating its successful use.

Weaknesses:

The operationalization of the research questions and the methods used to address them are underspecified. For example, there is no comparison group to contrast CS learning from the Unfold Studio workshop so it’s not clear how the PI will answer the question about “tradeoffs” in learning (RQ2) and the regression models for this RQ are not specified. Similarly, the proposal states that research question three will be addressed via a mixed methods approach but the steps/components of this approach are not described, outside of an example from prior work. Interviews are to be conducted but it is not clear how many or how interviewees will be selected (if not all participants). The proposal mentioned formalization of research questions at the end of the second year, but not having more specifics available at this stage makes it difficult to evaluate the rigor and intellectual merit of the proposed research.
The PI states that youth will be encouraged to re-enroll in subsequent workshop sessions, but it’s not clear to what extent this will benefit students if they have already been exposed to the curriculum. To highlight the role of experience students as mentors, it would be a boon to the proposal to create a formal mentor program separate from the enrolled students.
The timeline for the workshops and iterative development was not clear. How many workshop cycles will take place and how long will the PI and his team have to iterate in between cycles? It also seems as if the backend development will take place concurrently with the workshop piloting and iteration; will the new features be able to be used/tested in any of the workshop cycles? The relationship between these two aspects was not clear, making it difficult to evaluate the value of incorporating them both in the project.
The proposal (and budget justification) mentions an art installation but there is little detail included about this aspect of the project or how it contributes to addressing the research questions.
Megan McNally, Executive Director of The Foundry seems like an important partner and is listed as a subaward but not as a co-PI and none of her time is funded on this project. Given the importance of her role (and that the Educational Director position is currently in transition), it would inspire confidence to see more information about her commitment to this project and the extent to which she will be involved in the planning/development/implementation.
Two (different) sub-award budget justifications are included.
More information is needed about the facilities sand resources available at UB and The Foundry to support the research team. This will be especially important to be able to evaluate The Foundry’s role as a partner in this project. For example, how much space do they have, what other staff are employed by the organization, what office space and computer/office equipment are available for the project personnel to use?

Broader impacts

Strengths:

The project includes a stipend for high school youth to encourage enrollment in the program rather than taking an afterschool job. This is an inclusive practice that should allow participation by those who would otherwise not be able to benefit from this type of program.

Weaknesses:

Workshop participants and leaders will be invited to the advisory board meeting, but no funding is included to compensate these participants for their time.
Drop-in hours and a performance are mentioned but there is little detail about these aspects of the project or what evidence led to the estimate that these components will reach 200 to 300 community members.
It is not clear whether the book to be published will be commercially-available (sold) or available to the public for free.
The proposal mentions Unfold Studio being used as a research platform in the future, but more information about how the PI foresee researchers using the platform would help in evaluating this aspect as a broader impact.

The data management plan describes the different types of data that will be created as part of the project and how they will be stored and shared. No significant weaknesses were noted.

Additional criteria

Does the proposal identify the characteristics and needs of the targeted underrepresented groups (public or professional) to be served?

The proposal identifies minoritized youth as the target for broadening participation. They note that this group is largely disconnected from opportunities to learn and profit from CS.

Does the proposal include specific plans or strategies for addressing or accommodating the specific interests, community, or cultural perspectives, and/or educational needs of participants of the identified underrepresented groups?

The proposal plans to use interactive storytelling to reframe CS for this group and make the field more relevant and enriching for this population.

Summary Statement

This proposal has important strengths, including a promising partnership with a community organization and significant development of the product and pilot data as part of a previous project. However, significant limitations limit enthusiasm for the project, including lack of operationalization of research questions and measures, lack of clarity of the timeline for implementation/iteration, and lack of a clear role definition/time commitment from the Executive Director of the partner community organization.