Title: 2020 TC Talk
Date: 2020-02-26
template: slides

.title[
## SUPPORTING CRITICAL COMPUTATIONAL LITERACIES THROUGH INTERACTIVE STORYTELLING
]

Chris Proctor 
February 26, 2020 
Teachers College, Columbia University

???
- Back-and-forth: Critique and design

---
.full-bleed[
.half-left[<img height="100%" src="/media/slides/2019-buffalo/sketch_2_pointing.png">]
.half-right[
.half-frame[
# Outline

**Unfold Studio** Designing with a theory of learning

**Critical computational literacies** Contrasting theories of learning

**Designing and defining CS education** Broader implications

]]]

---
## Design Problem

Many rationales for K-12 Computer Science .cite[1, 2, 3, 4]

But what is Computer Science? And how will we teach it? .cite[5, 6]

Instead of one answer, there will be a "patchwork." .cite[7]  K-12 education and CS
have much to gain from each other, but communicating across disciplines is hard.

Learning Sciences may play a central role, with important lessons from Literacies.
.cite[8, 9]

.refs[
.anchor[1] Papert (1980) .anchor[2] Wing (2006) .anchor[3] Blikstein (2018) .anchor[4] Santo, Vogel, & Ching (2019) .anchor[5] Barr & Stephenson (2011) .anchor[6] Proctor, Bigman, & Blikstein (2018) .anchor[7] Sfard (1998) .anchor[8] Vossoughi & Gutiérrez (2016) .anchor[9] Vakil (2018)
]

???
This is in the context of our educational system which today remains deeply
inequitable and which is still largely unresponsive to the digital
transformation of our society.

People have been dreaming of K-12 computer science for half a century, but we
have seen a dramatic increase of interest over the last few years. 
There are many different rationales for teaching K-12 Computer Science,
including economic opportunities, democratic participation, and social justice.

There are two core questions--what is CS and how will we teach it. For a long
time, the CS ed community has stressed the need for an "operational definition"
of CS around which we can organize curricula, teacher licensing, and other
resources.

However, with my background as an English teacher, I am skeptical about the
project of defining the right way to do sensemaking and communication.
Efforts to narrowly define Standard English have done a
lot of harm in marginalizing nondominant language practices. Instead, there 
will be a patchwork of understandings, reflecting
the diversity of communities which practice various forms of computational
literacy.

This is broadly where CS and K-12 have a lot to learn from each other--CS has
been grounded in university; K-12 grounded in diverse communities where
schooling means many things, not always opportunity.

This is where the learning sciences, with their focus on learning in context,
have an important role to play. And literacies, with important perspectives on
identity, power, and privilenge.

---
## Unfold Studio

Web application for interactive storytelling using Ink .cite[1] programming language

Print text and social media affordances support connecting computation to existing
literacies, enact *literacy place* .cite[2, 3]

Workshops at Computer Science Teachers Association, MozFest, Philly National Writing Project. Used in 6 states, internationally; nearly 10k student-authored stories

.refs[
.anchor[1] Inkle (2016) .anchor[2] Gee (2004) .anchor[3] Dourish (2006)
]

???
So, what might literacy-based computer science education look like? There has
been a lot of research on interfaces for beginners, but not for these goals. 
I wanted a medium that supported integrating computation into print text and
social media, which could create communities of computational practice grounded
in the identities and cultures that already matter to them.

My dissertation comes at the tail end of three years of design-based research
exploring what kinds of tools and what kinds of pedagogy might support
literacy-based computer science. Through this work, I developed a web
application for interactive storytelling called Unfold Studio. In Unfold Studio, 
students can write interactive stories in a programming language called Ink,
which you can see on the left. They enact choose-your-own-adventure-style games,
which you can see running on the right. Users of Unfold Studio can write and
share stories, follow users, and keep up with their community via a social feed.

---
background-position: top
background-image: url(/media/slides/unfold.png)
background-size: contain

???
Here is a story written by a high school senior in a sociology class. Note the three framings of
practice. Note the infrastructural affordances. This goes beyond
personally-meaningful projects, to projects with rhetorical and critical
possibilities. "Critical discourse model" (Proctor & Blikstein, 2019). 
At the same time, builds on familiar infrastructural affordances.
Has been used in 6 US states; almost 10k student stories.

---
# Designing Unfold Studio

- 3 iterations of design-based research.cite[1] in middle- and high-schools.
- Affordances and their design rationales
- Design methodology

.refs[
.anchor[1] Proctor & Blikstein, 2019
]

---
.full-bleed[
.half-left[
.half-frame[
# Critical computational literacies

]]
.half-right[<img height="100%" src="/media/slides/2019-buffalo/sketch_0.png">]
]

---
## Critical computational literacies .cite[1, 2]

.refs[
.anchor[1] Lee & Garcia (2014) .anchor[2] Vee (2017) 
*situated cognition*: Brown, Collins, & Duguid, (1989); Collins & Greeno (2011) 
*material intelligence*: diSessa (2001)
]

???
The central construct in my research is "critical computational literacies." 
Literacy is a widely-used construct in education, so I want to explain 
what I mean by it. Broadly, I see literacy as closely-aligned with other
learning sciences constructs for thinking about how our practices and identities
 are shaped by context and medium. For example, activity systems, situated
cognition, figured worlds, distributed and embodied cognition. There are two
main reasons why I prefer literacy: first because it builds a bridge to
critical work from Litearcies, and second because it emphasizes the need for
dialogue between different frames.

I think of literacy in terms of two dimensions. The first dimension is radial, different scales of 
contextualized practice. The everyday idea of literacy is knowing how to read and write. However, we
know that reading and writing involves more than just the cognitive skills of
getting meaning from text and encoding meaning into text. It means participating
in a community which practices reading and writing in particular ways, and which
value particular identities.

Becoming a lawyer means learning how to read and write legal text in the world
of the law, and how to perform a lawyerly identity. You can be disbarred if you
don't act the way lawyers are supposed to act.

Zooming out even further, these worlds of practice do not have any sort of
objective legitimacy. To be clear, there is no objectively right way to do
English. Instead, the recognition and normative legitimacy of literacy practices
is contested and political. Some people argue that the primary social function 
of literacy is to enact and justify hierarchy.

When I talk about critical literacy, I mean becoming
aware that our worlds of practice, the systems of signification and value, and
the identites enacted within them, could be otherwise. When literacy is
oppressive, this is the first step to finding strategies for resistance.

- Second axis: diSessa's material intelligence

So that's the first, radial, axis of literacy. The second axis maps practice to
infrastructure. Here, we need to expand our notion of literacy again, because we
read and write with many different technologies. I'm most interested in print
text, computer code, and digital interfaces implemented by code. We can think
about the relationship between practice and infrastructure at each level of
practice.

Just as there are specific cognitive skills associated with reading and writing,
so there are different specific cognitive skills associated with computer programming.
Andy diSessa calls this ability to interact with an external literacy medium 
"material intelligence", and he distinguishes between cognitive material
intelligence and social material intelligence. We can also trace how critical 
practice is shaped by its infrastructure.

So these two axes are how I think about literacy. This is informed by congruent
work by Jim Wertsch, Paulo Freire, Annette Vee, Jim Gee, Andy diSessa, Roz Ivanic, 
Kris Gutierrez, and especially Dorothy Holland.

---
### Three framings of computational thinking

.refs[
Kafai, Proctor, & Lui (2019)
 International Computing Education Research **Best Paper Award**
]

???
One way this framework has been useful is in helping to sort out what we mean by
"computational thinking." If you aren't familiar with this particular debate,
as CS Ed has begun to get real traction at the K-12 level, computer scientists
have been trying to articulate what it is about computing that makes it so powerful. 
Yasmin Kafai and I gave a paper, which won the Chair's award, 
at this year's International Computing Education 
Conference in Toronto. We argued that we can recognize three distinct framings of
computational thinking: cognitive, situated, and critical, and that we need our
research community to engage in theory dialogue between them.

The vast majority of research, assessments, and pedagogy in CS education is
cognitive. As a former English teacher, I am aware of how powerful it can be to
weave together all three framings.

---
background-position: top
background-image: url(/media/slides/2019-buffalo/sketch_1_designing_futures.png)
background-size: contain

.bottom-third[
.half-frame[
### Is participation in interactive story-based literacy associated with CS learning?.cite[1]
]
]

.refs[
.anchor[1] Proctor, Zheng, & Blikstein (2020)
]

---
## Data source: CS learning

Interactive story portfolio submission, assessed using a rubric aligned with K-12 CS Framework's
.cite[1] Control and Variables concepts.

.rubric[
Level | Control criteria | Variable criteria
----- | ------------- | --------------
Advanced (4 points) | Meets criteria for Proficient AND use of control adds meaning to the story. Uses an advanced control structure. | Meets criteria for Proficient AND use of state adds meaning to the story. Uses at least one declared variable.
Proficient (3 points) | Uses diverts correctly and meaningfully to control story execution. | Uses variables (either built-in or declared) to keep track of something in the story and using it to change what happens in the future.
Basic (2 points) | The use of control might be based closely on another story. The use of control might “check the boxes” but not have much effect on the story. May include minor errors in usage. | The use of state might be based closely on another story. The use of state might “check the boxes” but not have much effect on the story. May include minor errors in usage.
Below basic (1 point) | Does not meet criteria for Basic. | Does not meet criteria for Basic.
]

.refs[
.anchor[1] *K-12 CS Framework* (2016)
]

???
Don't need to read the whole table. But this is what I used to assess students'
summative performance. Students were familiar with the rubric; we worked with it
in class.

---

.full-bleed[
.half-left[<img height="100%" src="/media/slides/2019-buffalo/literacy_log.png">]
.half-right[
.half-frame[
## Data Source: Literacy participation

500k events from Unfold Studio log

Build literacy graph: interactions between users and stories

Author score: Number of other user interactions with a user's stories

Audience score: Number of interactions with other users' stories

]]]

???
We operationalize literacy events as actions taken by users in the process of reading and writing stories, as well as browsing, searching, following other users, and commenting on stories. In this study, we consider only those literacy events in which one user views, loves, or forks (makes a copy of) another user’s story. These interactions feature two important, reciprocally-connected roles, those of author and audience. As described in the background, we view these as important learning opportunities within a literacy place grounded in, but extending beyond, the classroom. Each literacy event can be considered as a link in a bipartite network of authors and stories. We define a user’s author score as the number of literacy events in which another user interacted with one of the user’s stories. Similarly, a user’s audience score is the number of literacy events in which that user interacted with a story written by another user. Figure 2 shows a histogram of participants’ author and audience scores.

Thirty of the fifty study participants have author scores of zero because they chose not to make any of their stories publicly visible to their peers. (While this group wrote fewer stories on average than authors with positive author scores, they still wrote an average of 8 stories, including stories for the summative portfolio.) Note that the sums of all author and audience scores are not equal because these scores consider interactions with all "Literacy App" users. Some participants wrote stories which became popular on the site beyond the classes involved in this study, and they were occasionally inspired by stories written by external authors. For example, a student at another school wrote a story in which the player walks through an imagined monument to LGBTQ heroes from history. Several students pointed this story out as they were planning their own writing.

---
class: full-bleed-layout

<video controls autoplay="true" src="/media/slides/2019-buffalo/literacy_graph.mp4" width="100%" style="margin-top: -20px;">
.caption-bottom-left[
&#11044; User 
&#11044; Story
]

---
## Authorship and audience as forms of participation

---
count:false

## Authorship and audience as forms of participation

---
.full-bleed[
.half-left[
.half-frame[
## Data source: Stories
<img src="/media/slides/2019-buffalo/state_flow_hist_vertical_a.png" style="margin-top: -24px;">
]]
.half-right[<img width="100%" src="/media/slides/2019-buffalo/unfold_zdev_code_focus_a.png" style="margin-top: 36px;">]
]

???
Finally, we consider the content of students’ interactive stories, which are the primary artifacts created on "Literacy App". Over the course of the unit, the 48 authors participating in the research wrote 640 stories. In this study, we conduct static program analysis of the code from the final state of each story. (The left half of Figure 1 shows an excerpt of a story’s code.) Following a common strategy of counting syntactic elements which map to concepts (e.g. Brennan & Resnick, 2012; Fields, et al., 2016), we count the use of syntactic elements which correspond to flow and state, the two primary content knowledge goals of the unit.

We chose to count the number of diverts in each story as a measure of practicing flow. An interactive story can be visualized as a directed graph, where each knot, or chunk of textual content, is connected to other knots by edges. Each divert (->) implements an edge, so the number of diverts in a story corresponds to the number of edges in its story graph. We defined a students' flow practice score as the logarithm of the maximum number of diverts in any of an author’s stories. (Using the sum across an author’s stories would be artificially inflated when authors repeatedly forked their own stories, and using an average would be artificially deflated for authors who made numerous throwaway stories for notes or to test out constructs.). We conducted a similar analysis for stories' use of state which will be reported in a subsequent publication.

---
count:false

.full-bleed[
.half-left[
.half-frame[
## Data source: Stories
<img src="/media/slides/2019-buffalo/state_flow_hist_vertical_b.png" style="margin-top: -24px;">
]]
.half-right[<img width="100%" src="/media/slides/2019-buffalo/unfold_zdev_code_focus_b.png" style="margin-top: 36px;">]
]

???
---
## Results

Is participation in interactive story-based literacy associated with CS learning?

???
Our first research question asks whether there is an association between participation in the literacy place, either as author or as audience, and performance on the summative assessment. Using standard OLS regression, we found a statistically-significant association between both author and audience scores and summative performance. Plots of these associations are shown in Figure 3 and regression tables are shown in Table 2. We additionally tried several models including measures of students’ prior interest and experience with Computer Science and English/Language Arts (using the survey instrument from blinded). These covariates both had statistically-significant associations with summative performance. However, when they were added to the models shown in Figure 3 and Table 2, author and audience scores remained statistically-significant and their coefficients did not change much. Therefore, we do not include these covariates in the following results.

Figure 3 shows the positive association between technical score and both author score and audience score. Students who participated more in the literacy place, as authors and as audience, tended to have higher scores on the summative assessment of Computer Science content. This suggests that writing for an audience, or participating as an audience of others’ work, was associated with better performance on the technical summative assessment. There was a substantial correlation between author score and audience score (r2 = 0.36), which explains the collapse of model (3) in Table 2 due to collinearity. In other words, students with high author scores were reasonably likely to also have high audience scores. Intuitively, this is not surprising, as we hypothesize that these are reciprocal, dialogic relationships.

---
count:false

## Results

Is participation in interactive story-based literacy associated with CS learning?

---
background-position: center
background-image: url(/media/slides/2019-buffalo/results_1_path.png)
background-size: contain

## Results

.caption-bottom-left[
\*p < 0.1 
\*\*p < 0.05 
\*\*\*p < 0.01 
]

---
background-position: center
background-image: url(/media/slides/2019-buffalo/results_2_path.png)
background-size: contain

## Results

---
background-position: center
background-image: url(/media/slides/2019-buffalo/results_author_context_path.png)
background-size: contain

## Authorship and CS learning Controlling for prior interest .cite[1]

.refs[
.anchor[1] Friend (2015)
]

---
background-position: center
background-image: url(/media/slides/2019-buffalo/results_audience_context_path.png)
background-size: contain

## Audience and CS learning Controlling for prior interest .cite[1]

.refs[
.anchor[1] Friend (2015)
]

---
## Discussion

Literacy participation, as an author and as audience, was associated with better performance on a summative assessment of Computer Science content.
Both associations were significantly mediated by flow practice.

Supports hypothesis that a literacy-based approach to introductory Computer
Science can be an effective learning environment. Part of a larger argument that
this can also support criticality.

???
In this paper, we have shown that literacy participation, as an author and as audience, was associated with better performance on a summative assessment of Computer Science content. Furthermore, we showed that both associations were mediated by flow practice, a measure of students individually engaging with computer science concepts in their own stories. These results support our broad hypothesis that a literacy-based approach to introductory Computer Science can be an effective learning environment. The fact that these results were not substantially affected by the inclusion of covariates measuring students’ prior interest in Computer Science and writing suggests that this approach could be particularly effective for broadening participation in computing practice. Indeed, in the exit survey, numerous students related that they had not expected to enjoy programming. Even though these associations remain when controlling for prior interest in Computer Science and English/Language Arts, it is possible that we have missed hidden variables accounting for both students’ participation and their scores on the summative assessment. Moreover we have so far only provided a sketch of an argument for these results’ external validity. Our next steps will involve rigorous qualitative analysis
showing that the measures used here accurately describe students’ experiences of participation and learning.
In the course of this analysis, we developed author score and audience score as measures of participation in the classroom literacy place. In this study, we treated these as contextual factors and the
summative portfolio evaluation as the final measure of learning. In that respect, this study is similar to other research showing the importance of sociocultural factors. However, we can also view these results as showing alignment between a traditional cognitive (or competency-based) measure of learning, and two measures based on students' participation in a community of computational practice. In future research, we intend to center participation in a community of practice as a primary form of learning, producing quantitative measures which can be held up against cognitive assessments. The challenge then will be to justify that the participation, the community of practice, and participants' enacted identities are legitimate forms of Computer Science. Social learning analytics combined with qualitative analysis will be invaluable tools in this task, as they will provide a high-granularity view of the nature of students' practice. It seems likely that author and audience scores are a coarse view on emergent dynamics in students’ trajectories of participation; our future research will further explore these dynamics.

---
## Implications

Author and audience scores likely a coarse view on emergent dynamics in
students’ trajectories of participation. Future work using learning analytics, 
social network analysis.

There has been a lot of research showing **that** sociocultural factors are
important, but very little quantititative research assessing participation as
learning.

In this study, we still used a cognitive measure as the outcome variable. Future
research might assess participation and identity-building as outcomes in and of
themselves.

???
TODO clean up the text here

---
class: center
background-position: top
background-image: url(/media/slides/2019-buffalo/sketch_3_talk.png)
background-size: contain

.bottom-third.center[
# Designing and defining CS education
]

---
# Background

Defining: What is computer science?
- Need to unite behind “operational definition” of K12 CS.cite[1]
- CS defined in terms of content, big ideas, practices, competencies.

Designing: How should CS be taught?
- Importance of interaction with computers
- Computational thinking as interdisciplinary skillset
- Confronting stereotypes

What process should be used to answer these questions?
- Arguments for top-down and bottom-up
- Largely implemented via centralized decision-making

.refs[
.anchor[1] Barr & Stephenson (2011)
]

---
# Methods

Context
- Small wealthy school district in California, existing high school CS
- Committee worked for three years designing K12 CS implementation
- Members: teachers, parents, students, administrators, community

Data sources
- Nine in-depth interviews (plus follow-ups)
- Transcripts of public comments and debate from two school board meetings 
- Process documents generated by committee

Analysis
- Iterative open coding via grounded theory

---
# Tension 1: What is computer science?

- Everyone agreed CS is important, but rationales differed.
- Fundamental disagreements about what constitutes CS; not a settled issue. 
- Division between those emphasizing inclusion and preparation for college CS.
- Epistemological divide between big ideas accessed through programming and practices.
- Challenge of building an operational definition

---
###"If you’re teaching origami in computer science class and some guy out in [a wealthy city] is teaching kids how to actually write programs, I’ve got very little doubt which kid is likely to be in better shape coming out of those two classes in terms of being able to function well at the next level. I think it’s creating a chasm between the haves and the have-nots."

---
# Tension 2: How should computer science be taught?

- Agreement on interdisciplinary CT approach at elementary, middle levels
- Disagreement over whether to have CS department at high school level
- Central disagreement over whether to have a required high school class
- Little room in curriculum for new requirements
- Students may not be exposed unless CS is required

---
###"If we believe that computer science is a critical skill for being a citizen in the modern world, then... we really can’t accept a situation in which our female students and the underrepresented minority students are not participating in that at the same levels as other students. That really should be unacceptable to us, and I think a graduation requirement is a way of addressing that."

---
# Tension 3: What process should be used?

- Committee members’ approaches situated in their own experiences
  - Some emphasized programming experiences
  - Some emphasized relationships and communities
- Equity issues present in construction of expertise and legitimacy
- Disagreement over how inclusive the process should be
  - Some felt it was too inclusive, unfocused, and slow
  - Others felt inclusion was necessary to build political consensus, and for democratic public education

---
###"It was just what you would expect. It was nerdy kids and nerdy me and very playful, very advanced math kind of kids and I kept wondering why we weren’t getting more girls because I thought it was fun and I was a girl."

---
###"I worked for 15 years in the tech industry before I decided to teach elementary schools... My background is engineering but not computer science, though the work I was doing was programming. And so after fifteen years in the tech industry, I lost my job, I went back to school, I did my multiple subject credential, and I started teaching as an elementary school teacher.”

---
### "It’s tough because, you know, I felt like [male teacher] and I were the two subject matter experts in the room other than a couple of parents who were in the business. And I think both [male teacher] and I at different times during the past couple years have, you know, threw our arms up."

---
### A turning point for the Learning Sciences

"We [the Computer-Supported Collaborative Learning research community] should be honest with ourselves: if we have not managed to achieve any measurable impact by this point, why should we believe that we will be able to do so in the future?".cite[1]
 
"We applaud this vision of a scientifically literate citizenry in which many and
diverse people act in socially compassionate and democratically responsible
ways. However we disagree with the implicit assumption that
teaching key concepts and methods will
necessarily lead to socially responsible use or to a larger and more diverse
citizenry who participate in discussion and debate of scientific issues."
.cite[2]

.refs[
.anchor[1] Wise & Schwarz (2017) .anchor[2] Eisenhart, Finkel, & Marion (1996)
]

???

---
.title[
## SUPPORTING CRITICAL COMPUTATIONAL LITERACIES THROUGH INTERACTIVE STORYTELLING
]

Chris Proctor 
February 26, 2020 
Teachers College, Columbia University