Participatory Learning and Assessment in Competency-Based Online Learning (253586350)

Participatory Learning and
Assessment in Competency-Based
Contexts
Daniel T. Hickey
Associate Professor of Learning Sciences
Center for Research on Learning and Technology
Indiana University

Research Context
2009 in Sakai
@IU
2012 in
CourseBuilder
BOOC

2013 in Canvas
@ IU & IUHS
4

Use public contexts give
meaning to knowledge tools
Reward productive
disciplinary engagement
Grade artifacts through local
reflections
Let individuals assess their
understanding privately
Measure aggregated
achievement discreetly

5

James
Greeno

• Ed Psych 540: Learning &
Cognition in Education
• Challenging to Teach
– Diverse experience & ambitions
among students
– Ambitious but busy instructor!
– Started adjuncts in 2011

• Typical Accountability
– Use textbook item bank
– Include a pretest

• Online in Sakai in 2009
– Canvas in 2015

Gina Howard

Andi Rehak

Joshua Quick

The P540 Team

Lauren Smith

Xinyi Shen
7

Firat Soylu

• Educational Assessment BOOC
– P507 Assessment in Schools embedded for credit

• Offered to 500 students for free in 2013 with a grant
from Google using CourseBuilder
– 460 registered, 160 started, 60 finished, 9 in P507

• Offered a smaller streamlined version in 2014
– Added videos and quizzed and automated key features
– 60 started, 22 completed, 12 in P507

Use “public” contexts to give
meaning to knowledge tools
• Problematize knowledge from
learners’ perspective
– Publically open up the discipline

• Learners define (and redefine) a
personally-meaningful context
– Can be disciplinary problem, project,
investigation, etc.
– Embodies their experience, interests,
& aspirations
11

Rogers
Hall

15

16

20

Reward productive
disciplinary engagement
• DE involves declarative knowledge and
cultural practices
• PDE connects knowledge and practices,
makes connections, brings in resources,
etc.
• Support PDE so it can be rewarded

Randi
Engle

– Comment directly & publically on wikifolios
– Work in the open and encourage “lurking”
– No grades or mandatory posts
25

28

29

30

Grade artifacts through
“local” reflections
• Wikifolios generate lots of text and
comments
– Grading artifacts and comments
undermines participation

• Grade reflections on PDE.
– Critical engagement (suitability of context
for learning concepts)
– Collaborative engagement (what you
learned from others)
– Consequential engagement
(consequences of new knowledge)

31

Melissa
Gresalfi

33

Let individuals assess
understanding privately
• Offer ungraded open-ended
items and performance tasks
• Cover all of the big ideas in the
assignment
• Use to re-engage rather than
remediate.
– Items themselves have little formative
value beyond the correct answer

Pamela
Moss

34

35

Measure achievement
discreetly
• Standards-oriented tests
– Measure gains using pretests, compare
instruction, document improvement
– Primary student function is motivating
prior engagement

• Refine tests (but don’t teach to them)
– Use the item analysis routines in your LMS

• Protect test security
– Withhold item-level feedback

• Randomly selected timed items
– Proctor or use unsearchable items

Me!

Results:
Overall Public Engagement in P540
Average Words per Wikifolio & Words per Groupwiki
5000
4000
3000
Wikifolios
2000

Groupwikis

1000
0
1

2

3

4

5 6&7 8

9

10 11 12 13 14 15

• Wikifolios averaged 1569 words (144-3291)
• Groupwikis averaged 3477 words (2705-4630)

Average Number Comments per Wikifolio
15
10

Unthreaded

5

Threaded

0
1

2

3

4

5 6&7 8

1. Wikifolios averaged
8.9 comments & 95
words/comment
2. Groupwikis forums
averaged 61 posts
3. Groupwiki averaged
40 comments & 92
words/comment
4. 2062 comments
included 182 badges

9

10

11

12

13

14

15

Forum Posts by Group and
Comments by Groupwiki
100
80
60
40
20
0

Forum
Posts

Groupwiki
Comments

Agreement & Disagreement
in Badges vs. Comments
100

Percentage Frequeincy

90
80
70
60
50
40

Comments

30

Badges

20
10
0
Agreement

Disagreement

Perspective Reinforcement
Change
Content (potentially co-ccuring)

P540 Exam Scores (percent correct)
Mean Exam Scores (Percent Correct)
100
90
80
70
60
50
40
30
20
10
0
MC Pretest MC Midterm Essay Midterm MC Final
Essay Final
(25 items, 50 (40 minutes, (50 minutes, 5 (60 minutes, (120 Minutes,
minutes, 0
20 items, 10 items 5 points) 30 items 9
6 items, 6
points)
points)
points)
points)

Course Evaluation Results
Campus-Wide Items
Overall, I would rate the quality of this
COURSE as outstanding.

Overall, I would rate this INSTRUCTOR as
outstanding.
My instructor treats students with
RESPECT
My instructor is WELL-PREPARED for class
meetings.
0
(11/15, 5-point Likert, SD to SA)

1

14 Other Sections

2

3

This Course

4

5

Course Evaluation Results
School-Wide Online Learning Items

The instructor used the available TECHNOLOGY well.
I would take ANOTHER ONLINE COURSE from the
School of Education
The level of INTERACTION in this distance course
was sufficient.

The ONLINE FORMAT was well suited for this course.
The online technology used in this course was
RELIABLE
This distance education format helped me feel IN
CONTROL of my own learning
0
(11/15, 5-point Likert, SD to SA)

1

14 Other Sections

2

3

This Course

4

5

Issues with Semi-Formal (Cohorted)
Course Format
 Assumes regular instructor effort
 Is not what online learners expect
 Not flexible
 Can’t get behind

 Led many students to drop out of the BOOC
 Led many IUHS students to stay with
distance ed courses

Moving to Self-Paced Courses
 Not simple to get interaction
 Need to find like-minded peers
 Need to find peers working on same unit
 Need a large number of learners

 Currently creating a self-paced BOOC
 New participant list shows current learners
 New wikifolio feature allowing archiving
 Learners can look at archived work

 Currently creating self-paced IUHS courses
 Modules with exams and Canvabadge

Moving to Open Learning Resources
 Need to automate many features
 Automated registration and homepage
 Real-time participatory feedback
 Exam scoring and badges issuer

 Relying extensively on open ed resources
 Use ranking feature with OERs

 Pending NSF proposals for IUHS and IU

Questions?
Thank you very much!
More info at www.RemediatingAssessment.blogspot.com
Hickey, D. T. (in press). Situative approaches to aligning summative, formative, and
transformative assessment functions. Assessment in Education: Principles,
Policies, and Practices.
Hickey, D. T., & Rehak, A. (2013). Wikifolios and participatory assessment for
engagement, understanding, and achievement in online courses. Journal of
Educational Media and Hypermedia, 22 (4), 229-263.
Hickey, D. T., & Zuiker, S. J. (2012). Multi-level assessment for discourse,
understanding, and achievement in innovative learning contexts. The Journal of
the Learning Sciences, 22, (4) 1-65
Hickey, D. T., Kelly, T. A, & Shen, X. (2014). Small to big before massive: Scaling up
participatory learning and assessment. Proceedings of the Fourth International
Conference on Learning Analytics and Knowledge, Indianapolis, IN (pp. 93-97)