Title: 2021 CSE
Date: 2021-04-30
template: slides

.title[
## ASSESSMENT FOR LEARNING IN K-12 COMPUTER SCIENCE
]

Chris Proctor 
April 30, 2021

.refs[
chrisproctor.net/slides/2021-cse.html 
Illustrations by Chris Proctor, based on fieldnotes and video
]

???

Hi, I'm Chris Proctor, an assistant professor at UB's department of Learning and Instruction. 
I'm here today to share some recent research with you, but also because I want to build a stronger 
relationship between CSE and the Graduate School of Education.

I am a computer scientist and a literacy educator by training. I taught high school English for 
four years in California and Texas, and I taught middle school CS for two years. I have also worked as a 
lead software developer at an ed-tech nonprofit.

I identify primarily as a Learning Scientist. If you are not familiar with Learning Sciences, it is an 
offshoot of educational psychology which became primarily interested in context: how learning is situated and
distributed in groups of people, in specific environments, and in tools. Distributed cognition, 
thinking together and with computers, emerged in nodes of computer scientists working closely with computers:
Palo Alto, MIT, CMU, and also here at UB. I work in the tradition of Constructionism, trying to create some of 
these experiences in childrens' learning environments.

My primary research methodology is design-based research, building stuff with kids and discovering together
what works and what matters. 
If you're not a Learning Scientist, I would like to say a word about
design-based reserach. DBR is a research methodology which emerged from the
Learning Sciences' desire for external validity. Design-based research aims to
solve practical problems--innovating and improving learning environments--
but it also makes theoretical contributions. If we understand learning as taking
place in complex systems, everything potentially affects everything else in
nonlinear ways. Sometimes this can be decomposed into subsystems which can be
modeled in a manageable way, but often they can't.

Here at UB, I am the program director for the CS teacher preparation program which will soon be offering 
advanced certificates to in-service teachers and initial/professional certification to undergraduate CSE majors 
interested in becoming teachers. But more on that later.

---
.full-bleed[
.half-left[<img height="100%" src="/media/slides/2019-buffalo/sketch_2_pointing.png">]
.half-right[
.half-frame[
# Outline

Research overview
- K-12 CS Epistemologies
- Tools for teaching and learning

Study: Comparing cognitive and sociocultural assessments of learning in middle-school CS

Computer Science / Education at UB

]]]

???

Research overview will locate the present study in my research trajectories, 
and will also serve as some of the background.

---
background-position: top
background-image: url(/media/slides/2019-buffalo/sketch_3_talk.png)
background-size: contain

.bottom-third[
.half-frame[
### Research Overview K-12 epistemologies Tools for teaching and learning
]
]

---
### Three framings of computational thinking

.refs[
Kafai, Proctor, & Lui (2019)
 International Computing Education Research **Chair's Award**
]

???
One way this framework has been useful is in helping to sort out what we mean by
"computational thinking." If you aren't familiar with this particular debate,
as CS Ed has begun to get real traction at the K-12 level, computer scientists
have been trying to articulate what it is about computing that makes it so powerful. 
Yasmin Kafai and I gave a paper, which won the Chair's award, 
at this year's International Computing Education 
Conference in Toronto. We argued that we can recognize three distinct framings of
computational thinking: cognitive, situated, and critical, and that we need our
research community to engage in theory dialogue between them.

The vast majority of research, assessments, and pedagogy in CS education is
cognitive. As a former English teacher, I am aware of how powerful it can be to
weave together all three framings.

---
## Critical computational literacies

.refs[
Freire & Madedo (1987), Ivanič (1998), diSessa (2001), Holland, Lachicotte,
Skinner, & Cain (2001), Wertsch (2009), Lee & Garcia (2014), Vee (2017)
]

???

This diagram is my attempt to bring together conceptions of literacy from the
learning sciences and from the field of new literacies. At the top you can see
practices at the cognitive, the situated, and the critical scales. One axis of
literacy is radial. We can ask questions about literacy practices at and across different scales.

Literacy is about communities of practice reading and writing texts, broadly understood. It's about symbolic systems.
Different infrastructural technologies--books, Twitter, the law--have different
possibilities for practice at each scale. So a second axis is the relationship
between infrastructure and practice.

I want to show you three ways this construct can be useful.

First, I want to argue that literacy is a meta-construct that can encourage
interdisciplinary theoretical dialogue between different research projects. 
Computer scientists have been dreaming for seventy years
of creating comptuer cultures. Well now the computers are everywhere and we a computer
culture, but it is neither inclusive nor just. Various projects described as
"computational thinking" have tried to promote various cultural practices by
learning how computers work. We organized these into three framings of
computational thinking, and suggested litearcy might be a more effective
controlling metaphor.

Second, it shows modes of criticality:
- Culturally-sustaining pedagogy
- Identity authorship and voice
- Critical discourse models

---
## Making With Code

- Constructionist 9th & 10th-grade CS curriculum..cite[1] Goals:
  - Computing culture (authentic practices)
  - Material intelligence (authentic tools)
  - Liberatory pedagogy
- Central research questions: 
  - How might we prepare youth to understand and participate in this computational world?
  - Can a curriculum grounded in open-ended inquiry-based projects also lead to excellent performance on standardized assessments?

.refs[
.anchor[1] Proctor, Han, Wolf, Ng, & Blikstein, 2020
]
---
## Making With Code

<img src="/media/slides/making_with_code_context.png" style="width: 700px; margin: 0 auto;" class="center">
---
class:full-bleed-layout invert-slide-number-color

???
Let's see what this looks like in practice. Here is a story, "Egg-Hatching Simulator," by zdev. You can see the code (left) and the running story (right) of “Egg Hatching Simulator,” a story by zdev (a pseudonym chosen by the student). In this game the player hatches new pets from eggs, inspired by Pokémon. While it is not necessary to read the code in detail, the code does illustrate two elements of syntax which will be analyzed later. Divert statements (->) redirect the story’s flow to another part of the story. Lines beginning with a tilde (~) contain code which interacts with the execution environment, rather than emitting story output. Most often, code lines are used to manipulate state: initializing, updating and checking variables to keep track of what has happened in the story.

The code excerpt in Figure 1 generates a random number between 0 and 1 and then cascades through cases to determine which pet the player receives. If the random number is above 0.999, the player sees “Soo, this is the secret pet. You got an Electric Shock. This is not meant to be in the game yet. If you hatch this and have proof EXAMPLE: Take Screenshot. Come find me, i will give you 10 Bear Paws.!” The story then redirects to the ending, which outputs, “If you made it to this, the Ending you are the luckiest person ever. The chances of hatching this were 1 in 1,000 (I think)............. Props to you!!!!!! .” This text would indeed be shown as output one time in a thousand. Therefore, this text is likely intended to be read by peers who choose to read the game’s source code in addition to playing. Important computational concepts are expressed and framed in the context of speaking to an audience of gamer-programmers, as insiders in-the-know. In positioning the player as being extremely lucky (“1 in 1,000”), zdev makes a probabilistic assertion grounded in a fairly complex code structure, and does so in an interactional context which positions him as an authoritative explainer and the reader as an interested colleague. In the rest of this paper we argue that these literacy interactions, in which students are positioned as authors and as audience, were the basis for a kind of computer science meaning-making for and with others.

---
background-position: center
background-image: url(/media/slides/2019-buffalo/unfold_zdev.png)
background-size: contain
class: invert-slide-number-color
count: false

???
A static slide in case there's trouble loading Unfold Studio.

Let's see what this looks like in practice. Here is a story, "Egg-Hatching Simulator," by zdev. You can see the code (left) and the running story (right) of “Egg Hatching Simulator,” a story by zdev (a pseudonym chosen by the student). In this game the player hatches new pets from eggs, inspired by Pokémon. While it is not necessary to read the code in detail, the code does illustrate two elements of syntax which will be analyzed later. Divert statements (->) redirect the story’s flow to another part of the story. Lines beginning with a tilde (~) contain code which interacts with the execution environment, rather than emitting story output. Most often, code lines are used to manipulate state: initializing, updating and checking variables to keep track of what has happened in the story.

---
background-position: top
background-image: url(/media/slides/2019-buffalo/sketch_1_designing_futures.png)
background-size: contain

.bottom-third[
.half-frame[
### Comparing cognitive and sociocultural assessments of learning in middle school computer science
]
.refs[Proctor, Zheng, & Blikstein, 2020]
]

???

So given my interest in reconceptualizing the goals of K-12 CS education and in developing tools for teaching and learning, assessment becomes a priority.
---

## Context

10-week classroom study (27 hours total) set in "Riverton," a small urban/rural midwestern city.

5 sections of 6th grade students, 50/149 students participating in research.

Worked closely with 2 teachers & district CS education specialist.

School population: 57% white, 25% black, and 10% two or more races. Title I (55% of students eligible for free or reduced lunch). Tenth percentile on state test results. Very little prior exposure to CS.

.refs[
Upadhyaya, McGill, & Decker (2020)
]

???
The students had computer science for an 80-minute block period two or three times a week, for a total of 27 hours of classroom time for each student. The school’s students are reported as 57% white, 25% black, and 10% two or more races. 55% of students are eligible for free or reduced lunch and the school is in the tenth percentile for state test results. 50 out of 149 students across six sections, participated. Very few had prior exposure to computer science.

Very little research takes place in this kind of context, and I think my
literacy framework is particularly important for studying the challenges of
actually connecting these students with opportunity through CS.

---
.full-bleed[
.half-left[
.half-frame[
# Research questions

1. Is participation in interactive story-based literacy associated with computer science learning?
 - Authorship? 
 - Audience?
2. If so, is this association mediated by individual student practice in writing their own stories?
]]
.half-right[<img height="100%" src="/media/slides/2019-buffalo/sketch_0.png">]
]

???
With this context, I would like to share with you a particular research study.
Late-state design-based research tends to move more toward an emphasis on
instumentation and research efficiency (CITE Schwartz...)

I'm focused on two research questions here: 
1. Is participation in interactive story-based literacy associated with computer science learning?
2. If so, is this association mediated by individual student practice in writing their own stories?

---
background-position: center
background-image: url(/media/slides/2019-buffalo/rq1_path.png)
background-size: contain

## Research question 1

???
Here's RQ1

---
background-position: center
background-image: url(/media/slides/2019-buffalo/rq2_path.png)
background-size: contain

## Research question 2

???
And here's RQ2.

---
## Data source: CS learning

Interactive story portfolio submission, assessed using a rubric aligned with K-12 CS Framework's
.cite[1] Control and Variables concepts.

.rubric[
Level | Control criteria | Variable criteria
----- | ------------- | --------------
Advanced (4 points) | Meets criteria for Proficient AND use of control adds meaning to the story. Uses an advanced control structure. | Meets criteria for Proficient AND use of state adds meaning to the story. Uses at least one declared variable.
Proficient (3 points) | Uses diverts correctly and meaningfully to control story execution. | Uses variables (either built-in or declared) to keep track of something in the story and using it to change what happens in the future.
Basic (2 points) | The use of control might be based closely on another story. The use of control might “check the boxes” but not have much effect on the story. May include minor errors in usage. | The use of state might be based closely on another story. The use of state might “check the boxes” but not have much effect on the story. May include minor errors in usage.
Below basic (1 point) | Does not meet criteria for Basic. | Does not meet criteria for Basic.
]

.refs[
.anchor[1] *K-12 CS Framework* (2016)
]

???
Don't need to read the whole table. But this is what I used to assess students'
summative performance. Students were familiar with the rubric; we worked with it
in class.

---

.full-bleed[
.half-left[<img height="100%" src="/media/slides/2019-buffalo/literacy_log.png">]
.half-right[
.half-frame[
## Data Source: Literacy participation

500k events from Unfold Studio log

Build literacy graph: interactions between users and stories

Author score: Number of other user interactions with a user's stories

Audience score: Number of interactions with other users' stories

]]]

???
We operationalize literacy events as actions taken by users in the process of reading and writing stories, as well as browsing, searching, following other users, and commenting on stories. In this study, we consider only those literacy events in which one user views, loves, or forks (makes a copy of) another user’s story. These interactions feature two important, reciprocally-connected roles, those of author and audience. As described in the background, we view these as important learning opportunities within a literacy place grounded in, but extending beyond, the classroom. Each literacy event can be considered as a link in a bipartite network of authors and stories. We define a user’s author score as the number of literacy events in which another user interacted with one of the user’s stories. Similarly, a user’s audience score is the number of literacy events in which that user interacted with a story written by another user. Figure 2 shows a histogram of participants’ author and audience scores.

Thirty of the fifty study participants have author scores of zero because they chose not to make any of their stories publicly visible to their peers. (While this group wrote fewer stories on average than authors with positive author scores, they still wrote an average of 8 stories, including stories for the summative portfolio.) Note that the sums of all author and audience scores are not equal because these scores consider interactions with all "Literacy App" users. Some participants wrote stories which became popular on the site beyond the classes involved in this study, and they were occasionally inspired by stories written by external authors. For example, a student at another school wrote a story in which the player walks through an imagined monument to LGBTQ heroes from history. Several students pointed this story out as they were planning their own writing.

---
class: full-bleed-layout

<video controls autoplay="true" src="/media/slides/2019-buffalo/literacy_graph.mp4" width="100%" style="margin-top: -20px;">
.caption-bottom-left[
&#11044; User 
&#11044; Story
]

---
## Authorship and audience as forms of participation

---
count:false

## Authorship and audience as forms of participation

---
.full-bleed[
.half-left[
.half-frame[
## Data source: Stories
<img src="/media/slides/2019-buffalo/state_flow_hist_vertical_a.png" style="margin-top: -24px;">
]]
.half-right[<img width="100%" src="/media/slides/2019-buffalo/unfold_zdev_code_focus_a.png" style="margin-top: 36px;">]
]

???
Finally, we consider the content of students’ interactive stories, which are the primary artifacts created on "Literacy App". Over the course of the unit, the 48 authors participating in the research wrote 640 stories. In this study, we conduct static program analysis of the code from the final state of each story. (The left half of Figure 1 shows an excerpt of a story’s code.) Following a common strategy of counting syntactic elements which map to concepts (e.g. Brennan & Resnick, 2012; Fields, et al., 2016), we count the use of syntactic elements which correspond to flow and state, the two primary content knowledge goals of the unit.

We chose to count the number of diverts in each story as a measure of practicing flow. An interactive story can be visualized as a directed graph, where each knot, or chunk of textual content, is connected to other knots by edges. Each divert (->) implements an edge, so the number of diverts in a story corresponds to the number of edges in its story graph. We defined a students' flow practice score as the logarithm of the maximum number of diverts in any of an author’s stories. (Using the sum across an author’s stories would be artificially inflated when authors repeatedly forked their own stories, and using an average would be artificially deflated for authors who made numerous throwaway stories for notes or to test out constructs.). We conducted a similar analysis for stories' use of state which will be reported in a subsequent publication.

---
count:false

.full-bleed[
.half-left[
.half-frame[
## Data source: Stories
<img src="/media/slides/2019-buffalo/state_flow_hist_vertical_b.png" style="margin-top: -24px;">
]]
.half-right[<img width="100%" src="/media/slides/2019-buffalo/unfold_zdev_code_focus_b.png" style="margin-top: 36px;">]
]

???
---
## RQ1 Results

Is participation in interactive story-based literacy associated with CS learning?

???
Our first research question asks whether there is an association between participation in the literacy place, either as author or as audience, and performance on the summative assessment. Using standard OLS regression, we found a statistically-significant association between both author and audience scores and summative performance. Plots of these associations are shown in Figure 3 and regression tables are shown in Table 2. We additionally tried several models including measures of students’ prior interest and experience with Computer Science and English/Language Arts (using the survey instrument from blinded). These covariates both had statistically-significant associations with summative performance. However, when they were added to the models shown in Figure 3 and Table 2, author and audience scores remained statistically-significant and their coefficients did not change much. Therefore, we do not include these covariates in the following results.

Figure 3 shows the positive association between technical score and both author score and audience score. Students who participated more in the literacy place, as authors and as audience, tended to have higher scores on the summative assessment of Computer Science content. This suggests that writing for an audience, or participating as an audience of others’ work, was associated with better performance on the technical summative assessment. There was a substantial correlation between author score and audience score (r2 = 0.36), which explains the collapse of model (3) in Table 2 due to collinearity. In other words, students with high author scores were reasonably likely to also have high audience scores. Intuitively, this is not surprising, as we hypothesize that these are reciprocal, dialogic relationships.

---
count:false

## RQ1 Results

Is participation in interactive story-based literacy associated with CS learning?

---
background-position: center
background-image: url(/media/slides/2019-buffalo/results_1_path.png)
background-size: contain

## RQ1 Results

.caption-bottom-left[
\*p < 0.1 
\*\*p < 0.05 
\*\*\*p < 0.01 
]

---
background-position: center
background-image: url(/media/slides/2019-buffalo/results_2_path.png)
background-size: contain

## RQ2 Results

---
background-position: center
background-image: url(/media/slides/2019-buffalo/results_author_context_path.png)
background-size: contain

## Authorship and CS learning Controlling for prior interest .cite[1]

.refs[
.anchor[1] Friend (2015)
]

---
background-position: center
background-image: url(/media/slides/2019-buffalo/results_audience_context_path.png)
background-size: contain

## Audience and CS learning Controlling for prior interest .cite[1]

.refs[
.anchor[1] Friend (2015)
]

???
Supports the hypothesis that a literacy-based approach to introductory Computer
Science can be an effective learning environment. Part of a larger argument that
this can also support criticality.

---
## Discussion

(RQ1) Literacy participation, as an author and as audience, was associated with better performance on a summative assessment of Computer Science content.

(RQ2) Both associations were significantly mediated by flow practice.

Supports hypothesis that a literacy-based approach to introductory Computer
Science can be an effective learning environment. Part of a larger argument that
this can also support criticality.

Developed *author score* and *audience score* as measures of participation.

???
In this paper, we have shown that literacy participation, as an author and as audience, was associated with better performance on a summative assessment of Computer Science content. Furthermore, we showed that both associations were mediated by flow practice, a measure of students individually engaging with computer science concepts in their own stories. These results support our broad hypothesis that a literacy-based approach to introductory Computer Science can be an effective learning environment. The fact that these results were not substantially affected by the inclusion of covariates measuring students’ prior interest in Computer Science and writing suggests that this approach could be particularly effective for broadening participation in computing practice. Indeed, in the exit survey, numerous students related that they had not expected to enjoy programming. Even though these associations remain when controlling for prior interest in Computer Science and English/Language Arts, it is possible that we have missed hidden variables accounting for both students’ participation and their scores on the summative assessment. Moreover we have so far only provided a sketch of an argument for these results’ external validity. Our next steps will involve rigorous qualitative analysis
showing that the measures used here accurately describe students’ experiences of participation and learning.
In the course of this analysis, we developed author score and audience score as measures of participation in the classroom literacy place. In this study, we treated these as contextual factors and the
summative portfolio evaluation as the final measure of learning. In that respect, this study is similar to other research showing the importance of sociocultural factors. However, we can also view these results as showing alignment between a traditional cognitive (or competency-based) measure of learning, and two measures based on students' participation in a community of computational practice. In future research, we intend to center participation in a community of practice as a primary form of learning, producing quantitative measures which can be held up against cognitive assessments. The challenge then will be to justify that the participation, the community of practice, and participants' enacted identities are legitimate forms of Computer Science. Social learning analytics combined with qualitative analysis will be invaluable tools in this task, as they will provide a high-granularity view of the nature of students' practice. It seems likely that author and audience scores are a coarse view on emergent dynamics in students’ trajectories of participation; our future research will further explore these dynamics.
---
.full-bleed[
.half-left[<img width="100%" height="100%" src="/media/slides/2021_LAI_686_flyer.jpg">]
.half-right[
.half-frame[
### Computer Science / Education at UB

- K-12 Computing education research group
- Interdisciplinary grants
- CS teacher preparation

]]]

---
# K-12 Computing education research group

- BPS research-practice partnership
- Design-based research 
- Interdisciplinary courses
- Undergraduate research

---
# Interdisciplinary grants

- 2021 Google CS-ER (sole-PI; $80k): Computational literacies: Supporting cognitive, situated, and critical learning in L-12 Computer Science
- 2021 Google CS-ER (co-PI; $120k) Exploring place-based learning in K-12 CS education
- 2021 NSF AISL (sole PI; $1.2M) Weaving Computer Science into the fabric of cultural worlds through interactive storytelling 
- 2020 NSF RETTL (co-PI; $800k): Augmented intelligence for learning and teaching assistant via relational graph temporal attention network
- 2020 NSF IUSE (co-PI; $600k): Personalized learning for STEM undergraduates through interactive and adaptive AI systems
- 2020 NSF INCLUDES (s.p.; $10M): NSF INCLUDES Alliance: Redesigning for inclusion, success, and equity among underreprepresented populations

---
# CS teacher preparation

---
.title[
## ASSESSMENT FOR LEARNING IN K-12 COMPUTER SCIENCE
]

Chris Proctor 
April 30, 2021

.refs[
chrisproctor.net/slides/2021-cse.html 
Illustrations by Chris Proctor, based on fieldnotes and video
]