Title: 2018 ICLS Computational Thinking
Date: 2018-06-24
template: slides

class: center, middle

# How Broad is Computational Thinking?<br/>A Longitudinal Study of Practices Shaping Learning in Computer Science 
Chris Proctor & Paulo Blikstein<br/>
Stanford Graduate School of Education

.refs[
Slides & resources at chrisproctor.net
]

---
# Primary & secondary CS:<br/> The uptake has outpaced the research

In the US, adoption via: 
- District-wide adoptions
- As full courses.cite[1]
- In bite-sized chunks.cite[2]
- Through informal communities

Substantial disagreements about what constitutes computer science and how to assess it.cite[3]. 
Computational thinking is often a central concept in frameworks and assessments.cite[4].

Block-based interfaces often used in introductory courses.cite[5].

.refs[
.anchor[1] Goode & Margolis, 2011
.anchor[2] Wilson, 2014
.anchor[3] Guzdial, 2008; NRC 2010, 2011; Grover & Pea, 2013; Blikstein, 2018
.anchor[4] K-12 CS Framework, 2016
.anchor[5] Weintrop & Wilensky, 2015
]

???
As we saw in the symposium ysterday, there is a wide variety of ways computational thinking 
is defined and measured. For some stakeholders, such as curriculum designers, this is frustrating.
For others, such as teachers and schools focused on growing locally-situated and responsive 
practices, this pluralism may be welcome. As a side note, I felt the diversity in yesterday's 
symposium reflected some of the potentially subversive qualities of CS, ways it can productively 
challenge entrenched practices of School.

---
class: center

# Does block-based programming help beginners learn computer science?

???
The introductory strategy is to refine an intuitive question until it's rigorous, 
framing the background along the way.

---
class: center

# Does .new[working with] block-based programming help beginners learn computer science?

???
As Pierre said, you can't say that a technology just has some effect. So what 
kinds of practices with block-based programming help beginners?

---
.center[
# Does working with block-based programming help beginners .new[develop computational thinking]?
]

### What do we mean by computational thinking?
- Broad definition of CT: participation.cite[1], literacy.cite[2], situated practices involving identity
  and culture.cite[3]
  - Often assessed via portfolios, interviews, authentic assessments.cite[4]
- Narrow definition of CT: competencies, "ability accompanied by sensibilities".cite[5]
  - Often assessed via standardized assessments.cite[6]

.refs[
.anchor[1] Kafai & Peppler, 2011
.anchor[2] diSessa, 2001
.anchor[3] Papert, 1980; Margolis, 2003; Barron, 2004; Margolis, et al, 2010
.anchor[4] Werner, et al, 2012; Brennan & Resnick, 2012; Barron, et al, 2013; Fields, et al, 2016
.anchor[5] Denning, 2017
.anchor[6] Tew & Guizdal, 2011; Tew & Dorn, 2013
]

???
So this is the basic question of this study. But it turns out to also be an interesting exploration of
how defining computational thinking different may be more or less useful.

Others define computational thinking more narrowly, as the distinctive skills and knowledge gained through programming experience, and which comprise the disciplinary subject matter of computer science. In this view, computational thinking is primarily concerned with designing, using, and reasoning about computational models.

In particular, the jobs-oriented argument for expanding access to computer science tends to emphasize skills valuable to employers and not practices which deepen self-knowledge, civic engagement, or critical awareness.

So we have some options for how to measure computational thinking.

---
# Does working with block-based programming help beginners develop computational thinking: Prior work

- Blocks often perceived as easier to use, but less-powerful, more verbose, and inauthentic.cite[1]
- Little evidence of novices learning across programming interfaces.cite[2]
- Little evidence that programming experience develops generalized mental functions.cite[3]

This study's contribution is a longitudinal analysis of how programming with block-based interfaces is associated with later performance on an assessment of computational thinking.

.refs[
.anchor[1] Weintrop & Wilensky, 2015
.anchor[1] Armoni, Meerbaum-Salant, & Ben-Ari, 2015; Weintrop, 2016
.anchor[2] Pea & Kurland, 1984
]

???
A lot of work has already been done on this question, prominently including people in this room.
These studies are mostly over short time-spans, using decontextualized assessments. The present study's contribution 
is a longitudinal study over three years, anayzing student work artifacts.

---
# Context

n=48 middle-school students in an all-girls’ private school with a required 3-year CS sequence.

???
Students work primarily in Scratch for the first half, then primarily in Python. Focus shifts from self-expression, storytelling, art, to science, math, and algorithmic questions.

---
# Scratch Project Metrics

Static code analysis of Scratch projects.cite[1]

<table class="rubric">
<tr><td>Elaboration</td><td>log(# blocks)</td></tr>
<tr><td>Computational content</td><td>(# data, events, control, sensing, and function blocks)/(# blocks).cite[2]</td></tr>
</table>

.refs[
.anchor[1] Brennan & Resnick, 2012; Boechler, et al, 2014; Fields, et al, 2016; Moreno-Leon, et al, 2017
.anchor[2] Brennan & Resnick, 2012
]

???
Is automatic assessment legit? Moreno-Leon, et al, 2017 found a close correlation between automatic assessment
and expert evaluation. The teacher's assessment was coarser, but also followed the same trend.

I'm going to show you a couple of examples from different ends of the distribution

---
.full-bleed[
.half-left[<img src="/media/slides/scratch_portrait_code.png" style="height:100%;">]
.half-right[
.half-frame[
## Low computational content (z= -2.03),<br/>high elaboration (z=1.49)
<img src="/media/slides/scratch_portrait.png" style="width: 100%;">
]]]

???
Here's an example that might not look very successful from a computer science point of view--
there is very basic decomposition, but basically everything is an imperative list. At the same time, 
though, this took a lot of work. And any time an early adolescent is drawing a self-portrait, representing
her body position, facial expression, clothing, and gesture, she's going to be paying a lot of attention
to just where the lines go.

---
.full-bleed[
.half-left[<img src="/media/slides/scratch_house_code.png" style="height:100%;">]
.half-right[
.half-frame[
## high computational content (z=1.40),<br/>medium elaboration (z=0.34)
<img src="/media/slides/scratch_house.png" style="width: 100%;">
]]]

???
This example is a fairly formulaic drawing, almost exactly like the teacher's drawing. Much more attention
to crafting an elegant algorithm. Note, for example, how functions build up incrementally. This is a quality 
of some really nicely-written libraries, like underscore.js

---
# Summative task

.full-bleed[
.half-left[
    <img src="/media/slides/scratch_summative_task.png" style="width:100%;">
]
.half-right[
.half-frame[

```
moneytoSpend = 200
priceList = [
    145, 28, 76, 93, 140, 200, 183,
    34, 182, 23, 46, 31, 16, 33, 
    190, 62, 189, 102, 90, 130, 139,
    116, 192, 95, 175, 137, 108, 197,
    13, 109, 125, 161, 120, 60, 158, 
    43, 12, 186, 180, 117
]
```
]]]

???
Motivated by a desire to find out what people would choose to use. (Spoiler: almost nobody used Scratch. They said it didn't seem like the right tool; they hadn't used it in a while.)

---
# Summative task rubric

<table class="rubric">
<tr><td>Level 0 </td><td> Worked by hand or using a calculator. No evidence of a computational strategy.(Ex: guessed pairs of numbers over and over)</td></tr>
<tr><td>Level 1 </td><td> Worked by hand or using a calculator. Used a computational strategy. (Ex: decomposed the problem; systematically tested cases)</td></tr>
<tr><td>Level 2 </td><td> Used an ad-hoc  tool (Ex: Word, Excel) to implement a computational strategy (Ex: decomposed the problem; sorting; searching)</td></tr>
<tr><td>Level 3 </td><td> Attempted to implement an algorithm using Scratch or Python, but did not solve all the two-item cases.</td></tr>
<tr><td>Level 4 </td><td> Successfully implemented an algorithm using Scratch or Python to solve all the two-item cases.</td></tr>
<tr><td>Level 5 </td><td> Successfully implemented a generalized algorithm solving a more complex case as well.</td></tr>
</table>

---
# Results

- Of students who used code (58%), all but two used Python. The most common reasons for not using Scratch were because they had not used it in a long time and because it felt like the wrong tool. 
  - Suitability to task: "Python over Scratch because Python seems to be better at number manipulation."
  - Comfort/familiarity: "I chose the tools I used based on how comfortable I felt with them. For example, I felt more comfortable with Python than with Scratch, so I used it more."
- 21% of students used ad-hoc computational strategies such as "find" function in a word-processor or sorting numbers in a spreadsheet

.center[
![rubric score distributions](/media/slides/summative_measure.png)
]

???
Show correlation between projects; correlation of scores to summative assesssment

---
# Results
.full-bleed[
.half-left[<img src="/media/slides/ct_summative.png" style="width: 100%;">]
.half-right[<img src="/media/slides/elab_summative.png" style="width: 100%;">]
]

---
# Results

<table class="regression">
<tr>
<th></th>
<th colspan="4" style="text-align: center;">Summative assessment rubric score</th>
</tr>
<tr class="overbar">
    <td>Computational content</td>
    <td>0.80*</td>
    <td></td>
    <td>0.57</td>
    <td>0.01</td>
</tr>
<tr>
    <td>Elaboration</td>
    <td></td>
    <td>1.21***</td>
    <td>1.10***</td>
    <td>1.01**</td>
</tr>
<tr>
    <td>Prior CT</td>
    <td></td>
    <td></td>
    <td></td>
    <td>-0.21</td>
</tr>
<tr>
    <td>6th grade quant evaluation</td>
    <td></td>
    <td></td>
    <td></td>
    <td>0.49</td>
</tr>
<tr class="overbar underbar">
    <td>Adj. r<sup>2</sup>
    <td>0.082</td>
    <td>0.244</td>
    <td>0.278</td>
    <td>0.300</td>
</tr>

.refs[
Note: \*p<0.05; \*\*p<0.01; \*\*\*p<0.001
]

---
# Discussion: Choice of tools

- Experienced students gave more instrumental explanations for their choice of tools than beginners.cite[1]
- Ad-hoc appropriation of computational tools
  
.refs[
.anchor[1] Weintrop & Wilensky, 2015
]

???
Because almost all students chose something other than Scratch, this became a study of how Scratch may prepare students
for future learning.

---
# Discussion: Elaboration over computational content

- Importance of personally-meaningful projects? Computational literacy practices?
- A broader definition of CT may be more useful for characterizing learners' early trajectories in CS.
- Future research will use students' reflective writing to interpret this finding.

---
class: center, middle

# How Broad is Computational Thinking?<br/>A Longitudinal Study of Practices Shaping Learning in Computer Science 
Chris Proctor & Paulo Blikstein<br/>
Stanford Graduate School of Education

.refs[
Slides & resources at chrisproctor.net
]