Monthly Archives: September 2011

Making assessment work like the web

From the book jacket to Cathy Davidson's Now You See It

Cathy Davidson‘s comment on my last post about Open Badges — and her recent op ed in the Washington Post — get to the heart of an exciting shift taking place in learning and assessment. A shift where assessment is no longer seen as separate, standardized and external (first you learn, then you externally measure it). But instead, where assessment and feedback are baked right into the learning process, in a much more transparent, social and participatory way.

Assessment that’s social, transparent and participatory

Cathy’s view is that the process of collectively deciding what’s worth measuring — whether for learners, communities of peers, or organizations — represents a crucial learning opportunity in its own right, “as an exercise to engage all an institution’s members in thinking about what it wants to credit and why.”

It’s a fantastic exercise, in other words, in institutional self-reflection, self-evaluation.

My big “Now You See It” lesson is that, until we go through this preliminary step of thinking deeply together about who and what we are, who we want to be, what matters to us, and why it is important to know who contributes to our network and how, then we cannot even think about moving forward in open, innovative new directions.

The problem with an inherited system — whatever that system is: it comes with parameters already defined. To me, the most important thing about this badge experiment is it is an opportunity for a community to explore and understand what its own parameters are.

Media literacy badges from Global Kids

Assessment as a social act

This suggests that the process of deciding what “counts” towards a given goal, competency or achievement is, essentially, a social act — something decided on by communities of people, rather than by some top-down cathedral or gatekeeper. And that peer-to-peer assessment is a vital 21st century skill in its own right.

Standardized testing and other pre-defined yardsticks — which typically offer assessment models that are opaque, hard for many to fully understand, and handed down to you in advance, without discussion or community goal-setting — deny us this learning opportunity. And in many ways, are out of touch with a world where working collaboratively — which includes tactfully evaluating the work of peers, and assessing how well that work relates to larger shared goals — is increasingly important.

Excerpt from iRemix's badge framework

Peer-to-peer evaluation as a 21st century skill

We often talk about the need to assess and recognize “21st century skills” — competencies and achievements difficult for traditional institutions to recognize, because they’re new, reflect changing literacies, and evolve at a rapid rate that’s hard to capture.

Cathy’s point suggests that getting good at “social assessment” and peer-to-peer evaluation — helping to collaboratively set goals and translate them into metrics, being a fair judge of others’ work, collectively defining mastery, understanding nuance or levels of proficiency, designing processes and social mechanisms to work all this stuff out  — is an important 21st century skill in its own right.

Here at Mozilla, for example, we use “code review” to assess the quality of a given community member’s contribution. It’s important that the process is transparent (the criteria are clearly stated and understood), participatory (in that anyone can participate, and *everyone* is subject to code review — there’s no free pass for senior staff or the specially anointed),  and social (in that these reviews are done in the open by recognized peers, rather than some inscrutable gatekeeper).

What drives this approach is not naive idealism or some commitment to kumbaya values — it’s a form of assessment that’s agile, pragmatic and effective. Without these social and transparent elements, our global community would wither and our software would break.

Screen grab from Mozilla's beta1 "Badge Backpack"

“Gatekeeper credentialing” vs. “social proof” that you know what you know

For me, this emphasis on the social aspect of recognition and assessment — assessment as a social act — is crucial. It shifts us from the cathedral (a certificate or credential handed down from on high) to the bazaar (social proof that you know what you know, backed by communities of peers and shareable artifacts.)

In a fantastic online discussion with educators and ed tech innovators organized by Bryan Alexander, this point came up in the context of whether or not badges represent the “commodification” of learning — the reduction of complex human abilities into standardized little bits and pieces that can be plugged into the vagaries of “the market.”

It’s a question that is probably deserving of another post in its own right. But the social dimension of assessment that Cathy and others talk about seems key to me in this regard. The outward piece of badges you can see — the “display” layer, showing what you know — may ultimately prove to be only the small, visible tip of a much larger and more important iceberg: a paradigm shift in assessment. A shift that embraces more social, transparent and participatory models as a vital compliment to (or eventual replacement for) the standardized models we’ve inherited from the industrial era.

Breaking down the barrier between doing and evaluating

We’re seeing an explosion in this kind of assessment, whether it’s stealth assessment, portfolio-based assessment, peer-based assessment, micro-credentialing, or even (shudder) gamification — which done badly, represents gimmicky attempts to sugarcoat or apply simplistic models of human motivation. But as Cathy points out elsewhere, done properly, we can learn something from the best games — where doing and evaluating are not separate, they’re one and the same. When you’re playing a game, you don’t play for a bit and then stop and measure what you did, as a break from the action. Evaluation is woven seamlessly into the experience. Through constant feedback and heads-up displays that are clear, granular, and allow for continuous improvement and self-correction.

Similarly, learners at the School of Webcraft, for example, or iRemix may be earning recognition from their peers for being a great collaborator, team player or communicator — without even knowing it. Rather than teaching or learning to a test, they’re just doing it — working on projects, tackling specific challenges together, and producing shareable artifacts where the social proof is in the pudding.

Prototype community and peer badges from Mozilla and P2PU's School of Webcraft

Assessment that works more like the web

As Cathy’s book details, the old standardized systems of measuring ability — like multiple choice and IQ tests — were never originally intended to play the massive role they currently play in our educational system, and were shaped by a dominant technology: the assembly line. Now that we’re in a new age, with new technological and social paradigms, how can we adopt forms of assessment that tap our full human potential and bear greater relevance in the real world? If badges help accelerate that kind of experimentation and innovation, they’ll form the tip of a much larger and more exciting iceberg.

What’s your elevator pitch for badges?

A lot of the early conversation from the Open Badges launch is rich with ideas we can use to strengthen the story and messaging going forward.

Doug Belshaw, for example, makes a great point about trying to clarify a fuzzy part of the story on OpenBadges.org:

We’ve got the learner stories, all the stuff on the Mozilla wiki and some great blog posts and articles.

Are we missing a coherent ‘elevator pitch’ here that catches (some of) the nuance between credentialising achievement and ‘assessment’?

Sounds like a good idea. The Open Badges concept is meant to be bottom-up, peer-to-peer, and aimed at making assessment and recognition a lot more transparent, social and participatory. But that may not be coming across clearly enough on the front page. <Doug & others: care to help fix that?>

How can we improve our messaging around badges and the competition?

In general, it’s a good opportunity to look for gaps in the general story and message. How can we digest some of the first round of conversation, and use it to clarify the story? What else is missing? What gaps can we fill over the next several weeks?

The current Open Badges pitch:

Here’s the high-level Open Badges story as it stands now (from the OpenBadges.org front page):

What is Mozilla’s Open Badges project?

Learning today happens everywhere, not just in the classroom. But it’s often difficult to get recognition for skills and achievements that happen outside of school. Mozilla’s Open Badges project is working to solve that problem, making it easy for anyone to issue, earn and display badges across the web — through a shared infrastructure that’s free and open to all. The result: helping learners everywhere display 21st century skills, unlock career and educational opportunities, and level up in their life and work.

Stuff we should consider adding to the next version:

  • The peer-based / social / bottom-up element. The stuff Doug mentions.
  • The $2 million “Badges for Lifelong Learning Competition” — and the fact that you can participate in it. As an educator, designer, developer, or organization. This is of course featured prominently in the page — but would be good to explicitly include in the project description right now.
  • The “who.” Who is this for? Who is the primary audience / users / customer for Open Badges software — online courses? Schools? (The short answer is “anybody,” since it’s 100% free, open source, and available to anyone to pick up and start using. But saying “anybody and everybody” isn’t that helpful to readers.)

What else?

How could we make the Open Badges and competition story story and front pages clearer?  Please share ideas as comments on this  thread. We’ll also be chatting about this in each Monday’s Mozilla Drumbeat call – which of course are open to all. So let’s talk.

Open Badges launch: rocket fuel, reaction and resources

Mozilla’s Open Badges project reached a major milestone last week, with the launch of the MacArthur Foundation and HASTAC’s “Badges for Lifelong Learning” Competition. The competition includes collaborators like NASA, the Department of Education, the Corporation for Public Broadcasting and a number of other organizations large and small — all testing the potential of badges to assess, recognize and reward learning.

Rounding up resources and early reaction

This post summarizes some of the key assets and early conversation from the launch, as we push toward the next major competition milestone: the deadline for Badge Content and Program proposals on October 14.

Arne Duncan, U.S. Secretary of Education, speaking at the September 15 launch of the "Badges for Lifelong Learning" competition

Video, transcripts and other useful resources

[ustream vid=17290131 hid=0 w=480 h=296]

Video streaming by Ustream

Press coverage

  • TechShout: “Mozilla is simplifying the online process of issuing and sharing digital learning badges through the Open Badge Infrastructure project”
  • Education Week: “The John D. and Catherine T. MacArthur Foundation is launching a $2 million open competition for ideas relating to badge development.”
  • The Inquirer: “Mozilla offers developers a badge system”
  • SD Times: “Mozilla’s plan to support life-long and professional learning”

Blog posts: Good high-level summaries

  • Cathy Davidson, HASTAC: “Why Badges? Why Not?
    • “Our current, standardized systems of credentialing  are very rigid and often restrictive.  Badges allow groups of people—organizations and institutionsto decide what counts for them and how they want to give credit.”
  • Mark Surman, “Mozilla Launches Open Badges Project
    • “If we’re successful, the benefits to learners will be tremendous. Open Badges will let you gather badges from any site on the internet, combining them into a story about what you know and what you’ve achieved. There is a real chance to create learning that works more like the web.”

Education and learning innovators

  • Rafi Santo:  “#DMLBadges and Shifting the Overton Window on Learning
    • “I’m not here though to discuss the initiative itself, or to weigh in with my opinion on badges and their potential to shift the educational landscape in positive ways, or not. What I do want to talk about is how I see the current conversation around badges being a positive thing for that landscape.”
  • Jason Baird Jackson, “Badges!
    • “One of the best things about badge programs is that they can be organized by a diversity of groups and agencies (unlike formal higher education, which is built around colleges and universities and their slow moving practices).”
  • Stevie Rocco: “Mozilla’s Open Badges Project
    • “To me, this is a really exciting concept for a number of reasons, not the least of which is the idea of learners being able to more adequately express their skills and abilities in ways that haven’t been possible before.”

Tech community

  • Ostatic: “Mozilla’s Open Badges Offer Ways for You to Showcase Your Skills
    • “It is essential that academic degrees are not the only markers for tech-focused achievements in today’s world…. It would be especially great to see Open Badges offer ways for people with specialized open source skills showcase what they know.”

Thoughtful criticism and questions

  • Bud Hunt, “Digging Out My Sash
    • “If the DML competition encourages thinking and writing and exploration and action around ideas like the idea that any accountability system, or accreditation system, is ultimately a subjective system, made by people, however we design it, then I say, let’s rock. But let’s do so carefully.”
  • Andrea Zellner, “Thoughts on Badges for Learning
    • “I don’t feel as if I understand the theoretical basis for the research questions around badges for learning.”
  • Alex Reid, “Welcome to Badge World” + “three learning assessment alternatives #dmlbadges
    • As I view it, the badges/assessment thing tries to address four separate issues that really need to be disconnected. In fact, one could argue that their interconnection has produced the serious problems education faces today.
      1. How do we inspire students and/or support their intrinsic motivations to learn?
      2. How do we measure/value our students’ achievements?
      3. How do we measure/value the success of pedagogies, programs, institutions?
      4. How do we capitalize on the values of #2 and #3 in the job market?

Some early analysis and response (more coming)

  • Carla Casilli, “Fear of a Badges Planet
    • Mozilla’s Open Badge effort is not a gamification of anything. Instead, the Open Badge system is an opportunity to reimagine personal communication of social representation. Think of it as an entirely new, authenticatable, verifiable, dependable means to an end: a brand new vision for the old resume/curriculum vitae.”
  • Audrey Waters, More Thoughts on Badges
    • “Mozilla is part of the force that keeps the Web open and free. So I like them. I do. And I like the idea behind having an open source system by which people can showcase what they’ve learned, what they’ve achieved.”

Blog posts: global