Things I Know 273 of 365: Value added isn’t

Value-added assessment is a new way of analyzing test data that can measure teaching and learning. Based on a review of students’ test score gains from previous grades, researchers can predict the amount of growth those students are likely to make in a given year. Thus, value-added assessment can show whether particular students – those taking a certain Algebra class, say – have made the expected amount of progress, have made less progress than expected, or have been stretched beyond what they could reasonably be expected to achieve.

– The Center for Greater Philadelphia

Professor Andrew Ho came and spoke to my school reform class tonight about the idea of value added and its space in the conversation on American education.

We started looking at a scatterplot of local restaurants situated by their Zagat rating and the Zagat average price per meal.

Ho then plotted a regression line through the scatterplot and took note of one restaurant that had a higher score than predicted for it’s cost.

The temptation was to claim our overachieving restaurant was a good buy for the money. Who’d expect a restaurant with such inexpensive food to have such a high rating?

Then he asked us what we didn’t see.

Portions, ambiance, quality, location, service, selection, etc.

Any of these is familiar to someone who’s debated with a group of friends when attempting to select a restaurant.

His point was simple. Expectations changes based on what you base expectations on.

Ho relabeled the axes – this year’s test results, previous year’s test results.

He asked us what we didn’t see.

Content, delivery, socioeconomic status, race, home life, sports, after-school activities, tutoring, mentoring, etc.

This is to say nothing of the fact that perhaps there is a natural spread to knowledge and growth that is beyond the influence of a teacher or the fact that different combinations of teachers in the life of a student in a given year could have varying effects on achievement.

A psychometrician, statistician and policy researcher, Ho then laid some data on us from the research on value added:

  • Estimates of value added are unstable across models, courses that teacher might teach, and years.
  • Across different value-added models, teacher effect. ratings differ by at least 1 decile for 56%-80% of teachers and by at least 3 deciles for 0%-14% of teachers (this is reassuring).
  • Across courses taught, between 39% and 54% of teachers differ by at least 3 deciles.
  • Across years, between 19% and 41% of teachers differ by at least 3 deciles.

He then made a point that’s come up time and again in my statistics course, “Any test measures, at best, a representative sample of the target domain.”

But we’re not seeing samples that are representative. According to Ho, “In practice, it is an unrepresentative sample that skews heavily toward the quickly and cheaply measurable.” We’re not learning about the population. Put differently, we can’t know all that we want to know. Anyone who says differently is selling something.

When questioned on teacher assessment in his recent Twitter Town Hall, Sec. Duncan said he favored multiple forms of assessment in gauging teacher effectiveness. Nominally, Ho explained, this makes sense, but in effect it can have unintended negative consequences.

Here too, Ho cautioned against the current trend. Yes, value added is often used in concert with observation data or other similar measures. If those observations are counted as “meets expectations” or “does not meet expectations” and all teachers meet expectations, though, we have a problem. The effect is to mute the impact of this measure in the composite. While it may be nominally weighted at 50%, if value added is the only aspect of the composite accounting for variance, “the contribution of these measures is usually much higher than reported, as teacher effectiveness ratings discriminate much better (effective weights) than other ratings.”

Ho’s stated goal was to demystify value added. In that he succeeded.

He left us with his two concerns:

  • The current incentive structures are so obviously flawed, and the mechanisms for detecting and discouraging unintended responses to incentives are not in place.
  • The simplifying assumptions driving “value added,” including a dramatic overconfidence about the scope of defensible applications of educational tests (“numbers is numbers!”), will lead to a slippery slope toward less and less defensible accountability models.

I’d hate to think we’re more comprehensive in our selection of restaurants than teacher assessment.

Advertisements

Things I Know 235 of 365: Bank of America might be losing my business

The lower- to middle-income groups will be most affected due not only to the fee but the higher minimum balances required to avoid the fees.

John Kottmeyer

Adjunct professor at Samford University’s Brock School of Business

I’ve been a Bank of America customer since 2003, but that might be changing. It’s not me Bank of America should worry about; it’s all of my options.

A few weeks ago, I signed up for mint.com as a way to manage my money while I’m here at school. Mint, along with its free iPhone app, helps me keep track of my spending and sends me alerts when it notices increased spending in a specific area or I com close to exceeding my budgeted amount.

As part of mint’s services, I was also given recommendations for banking and credit options that would save me money in the long-run compared to my current accounts.

Because BoA has branches or ATMs in almost any location I travel, I skipped the recommendation window and carried on with my budgeting.

Today, a change.org alert arrived in my inbox.

Seems my bank is going to start charging me a monthly fee for using my debit card. Admittedly, $5/month isn’t much in the grand scheme of things, but the fee runs contrary to my principles and shows a lack of technological trend understanding on the part of the bank.

According to the BBC, BoA received $20 billion in the banking bailout as well as $18 billion in guarantees against bad assets. I don’t know the exact math, but I’m guessing my portion of that bailout more than covers my $5/mo. fees for the rest of my natural life.

This is to say nothing of the fact that charging new loophole fees in the face of federal regulations designed to stop banks from charging predatory fees is bad PR. Charging businesses for running credit/debit transactions as well as customers for using debit cards makes it easy to paint BoA as greedy, uncaring and unscrupulous. I don’t know that this is the case, but the new fee doesn’t make it difficult to connect the dots.

Ethics and PR aside, the fee has me worried that BoA has no eye on the future of transactions. In five years (as a generous estimate) my debit card won’t exist. My transactions, whether they take place in a physical or virtual space, will happen through my phone. My personal QR code or whatever comes after QR codes will be my method of payment. My wallet will be where I carry my license and maybe a business card. Companies like levelup and Starbucks are already setting the stage for the transition.

Creating a fee scheme around a card destined for extinction is shortsighted and a waste of corporate momentum.

In a pre-digital world, such a move made sense. I needed to travel to a physical space to shut down my account and then another physical space to open a new account. My bank can be virtual. BoA can lose my business without my ever having to talk to another human being. Of course, I will talk to human beings, namely my friends and family as I share why I made the change, how easy it was and how much I’m projected to save over the next three years.

For now, my money is staying put and my accounts are open.

I’ve added my name to the change.org petition and I’m hopeful the right people at BoA are checking their e-mail. If not, my new bank is only a few clicks away.

I’ll let you know.

Things I Know 186 of 365: The teaching is ubiquitous

We seek not rest but transformation.
We are dancing through each other as doorways.

– Marge Piercy

I logged in to the dying social network today and found a message from a former student with the subject line “Blogging Advice”:

Hi Mr.Chase hope you are having a great summer. I am going to be blogging from california in a couple of days and was wondering if you could give me any advice. Thanks in advance!

I responded that I’d be happy to help and asked where would be best to have the conversation. I offered Facebook, IM, phone call, and texting.

I expected a quick IM conversation or phone call.

The student opted for texting, explaining she had no computer access at the moment.

I told her that would be great. A few moments later, I received the first text via my Google Voice number in my e-mail inbox.

I responded and archived the message. This continued back and forth, as you can see below, for a total of 25 messages.

All the while, I was working on other projects at my desk.

A question would pop up on my computer and I would reply to her phone.

It looked like this:

Student: Chase!!!!!

Me: What’s up, kid? Ok. Probably, the best place to start is you to come up with specific questions you have about blogging.

Student: Well, I guess my first question would be about the difference between a more journalist approach to blogging versus a more a free write style of blogging.

Me: Great question. Journalism is going to make sure you’ve got the who, what, when where, why and how in there. The goal is to communicate the story or event to people who weren’t there.

Me: For the journaling piece, it functions more as a personal record that is public. Something for you and your memories that is available to others.

Student: Ok, that makes sense. So what is the best way to establish the so what factor for both of blogging? I get that the journaling type of blogging is more personal, but if you are posting don’t you want people to get something out of it?

Me: The something they get out of it are the stories and thoughts you put into words. Sometimes, I’ll write from the perspective of, “I want readers to do X because of this post.” Often, I just want to tell a good story and make people think.

Student: Makes sense. Does that apply to journalist writing style too?

Me: Yes.

Me: When you’re writing to inform, the goal is to make sure you’re offering information people would want to have.

Student: Wait, that confuses me.

Student: What if it’s something they could care less about until you informed them?

Me: Your job as a writer is to make them care.I would imagine it’s the same as your job as a poet.

Student: You’re right. I would think it’s like writing a persuasive essay but i’m pretty sure it’s different. What the difference between essay and the structure of a blog?

Me: Think of a blog as fitting the information of an essay into a more informal storytelling structure.

Student: So there are no set rules?

Me: Nah.

Just tell the story of the piece.

Then, revise.

Then, proofread.

Then, revise.

Then, post.

Me: My best writing comes from reading blogs. See if you can check out some poetry blogs and get a feel for what others are doing. This will help you develop your taste.

Student: You make sound easy Chase. lol

Me: It’s quite difficult at times. I find the easiest recipe is to find something you want to say and commit to saying it. Again, not always easy, but always good.

Student: Well, I think i’m out of questions. Thanks for taking the time to help me. Hope you have a great summer.               Love, Chella

Student: P.S- I know you are going to be amazing at Harvard!

Me: It’s been my pleasure, kid. If any other questions pop up, don’t hesitate to hit me up.

Me: I’m going to try my best to make you proud.

Student: You already have!

The conversation did two things for me.

First, it made me realize I’m still a teacher. I know that sounds odd, but it’s been a huge fear since leaving the classroom. As confident and dedicated as I am to helping people learn, I was still mentally tied to the idea that the classroom or the official title was somehow tied to my powers of pedagogy. This lesson was just in time and just in need for my student and it showed me I am still a teacher.

Second, it made me think about what was necessary for the conversation to take place. Yes, the technology made it happen. I mean, it was a conversation about using technology as a forum for creation. It also could have happened without anything electronic. My understanding is there used to be these things called letters or missives. If my understanding is correct, my student could have sent me a letter with her questions and then I could have replied with my answers and questions. This process could have continued, similar to the one we used, interminably.

So, it wasn’t the technology that led to this learning.

I needed to know her. She needed to know me. Most importantly, she needed to know I cared and would be there if she had a question. I don’t remember making any statements as I was leaving SLA that I’d be willing to help kids with anything they needed. I’d like to think I didn’t have to. I’d like to think they knew.

Today’s conversation helped reinforce that belief.

As I continue to build systems and structures of care in my life, I will focus on and highlight the tools at my disposal for connecting and maintaining connections to people. Always and forever, I will highlight and nurture the caring necessary for community. Even if they’re multi-medium communities of two.

Keeping Tabs 7/11: 5 Sites Taking Up Space in my Browser this Week

Some sites get written about. Some sites get looked at and then forgotten. The five sites below have been open on my browser for at least a week. I’ll be bookmarking them and closing their tabs in my browser as soon as I post this.

What MySpace’s Tom Anderson Thinks of Google+

More people than I care to count, including me, have been pontificating on the possible impact of Google+ this week as the launch begins to go global. It all feels a little like critiquing a newborn’s progress toward an eventual Ph.D. Even so, looking through the whole thing through the eyes of someone who tried to build a social network and then took many admitted wrong turns was interesting. Anderson at once holds a sort of humility and optimism as he writes. The piece has a definite tone of, “If not me, then I’m glad it’s these guys.”

The New Aesthetic

No designer am I. It’s probably why I’m so curious to learn how design works and what the edge of a field I know little about might look like. This tumblr page has yet to fail at giving me a new angle from which to view the world or at least providing me with better questions. It’s getting thrown in delicious as well as my reader. If you’re going to check it out, be sure to see where it all began.

eLearning Africa News Portal

After two summers working with teachers in Kenya and South Africa, I’ll admit to a bit of withdrawal. For as much as teachers in North America like to talk about helping their students connect globally, I’m struck by our general lack of knowledge of what education looks like in various African nations. Fifteen minutes skimming this portal could prove prospective changing.

WikiViz 2011: Visualizing the impact of Wikipedia

I like a challenge. More importantly, I like a challenge that requires literacy. The WikiViz challenge is a competition calling for visualizations telling the story of Wikipedia’s impact. I’ve seen countless keynotes and listened to even more podcasts explaining the site’s importance. I’m keen to see what happens when the story leaves traditional narrative arc behind and gives us something to see. If you’re interested, the deadline is August 19.

BO.LT

I love this. Copy and edit any page on the web. It reminds me of those booths at the state fair when I was younger that let you superimpose yourself onto the cover of Time or People. I wonder if BO.LT let’s you put the site on a T-Shirt after you’re done.

Things I Know 177 of 365: It’s everyone’s idiot box

All television is educational television. The question is: what is it teaching?
– Nicholas Johnson

Being back in Illinois has meant an inordinate amount of television viewing. I have been known to marathon view entire runs of television shows from time to time, but those are usually through iTunes or Netflix.

No, being home has meant honest-to-goodness television (though I’m uncertain how much honesty or goodness I’ve been taking in.

Yesterday, this commercial for State Farm Insurance ran during a show I was watching with my mom:

“Why is it only the guys who get the cool stuff,” my mom asked.

“Because the women are out attending and rating the weddings of three strangers,” I explained, referencing the TLC program Four Weddings we were watching at the time.

I wasn’t worried that the men were able to afford the toucans and moose heads as their beleaguered wives looked on, because the husbands of the four brides competing on the show were looking on in a similar manner.

I’ve been away from broadcast television for a while. It’s nice to see were doing a better job of making everyone appear foolish.

Except Flo:

Thing I Know 164 of 365: Learning is good

I am learning all the time. The tombstone will be my diploma.

– Eartha Kitt

A few weeks ago, I received an e-mail with what was described as a very, very, very unofficial suggested reading list for my program. The books contained therein were those most frequently appearing on course syllabi. Though sleep was the most emphatically suggested way to prepare for our forthcoming studies, I’m a sucker for a good reading list.

I’ve decided to integrate some of the books into the already extensive reading list I’ve built up for myself over the last year. My list is comprised of those books that sound fantastic, but that teaching crowds out.

I plan on alternating books from the suggested reading list and the reading list of suggestions.

Friday I started Peter Senge’s The Fifth Discipline. I understand all the lauding the initial publication of the book received. Senge writes some things about organizational learning that are both intuitive and sadly underpracticed.

Most notable thus far has been the connection he’s drawn between practices of the standard classroom and practices within the modern board room or office. If we want employees to seek out possible problems rather than working for the praise of their employers, we need to stop stop training students to find the right answer kept in seclusion by the teacher.

I’ll have more to write as I continue deeper into Senge’s work. The passage that follow’s though, struck a chord and is worth reading for anyone whose ever learned anything:

The problem with talking about “learning organizations” is that the “learning” has lost its central meaning in contemporary usage. Most people’s eyes glaze over if you talk to them about “learning or “learning organizations.” The words tend to immediately evoke images of sitting passively in schoolrooms, listening, following directions, and pleasing the teacher by avoiding making mistakes. In effect, in everyday use, learning has come to be synonymous with “taking information in.” Yes, I learned all about that at the training yesterday.” Yet, taking in information is only distantly related to real learning. It would be nonsensical to say, “I just read a great book about bicycle riding – I’ve now learning that.”

Real learning gets to the heart of what it means to be human. Through learning, we re-create ourselves. Through learning we become able to do something we never were able to do. Through learning we reperceive the world and our relationship to it. Through learning we extend our capacity to create, to be part of the generative process of life. There is within each of us a deep hunger for this type of learning.

– Peter M. Senge, The Fifth Discipline