View Thread > H2O Meta > Interesting Uses > Automation without evaluation?

There have been bugs with the ratings, rendering them useless at best, misleading at worst.  (At least, since I got no response to my queries, I have to assume they were bugs.)  So that potentially useful feature just didn't pan out.

When would I want to run/host an online community, encourage a thoughtful level of discussion, and yet allow a bot to randomly assign who replies to whom?

If it were a large class, or a professional organization, or nearly any formal circumstance I can think of, the community would be better served by an active moderator assigning the replies -- as well as posting comments themselves, encouraging further clarification or extemporization (while, presumably, attempting not to show favoritism or other undue bias).

Especially in the case of a class discussion, where as an instructor I would want to evaluate the quality of writing -- it makes no sense to leave the discussion to chance, merely to step in at the end, finally reading all the posts, and assigning evaluation scores.  Why not evaluate the posts as they go, providing feedback along the way?

For an informal group or discussion, where evaluation is irrelevant, and stimulation of ideas is paramount, the limiting reply format of the rotisserie seems to cut off people who want to respond to a useful idea elsewhere in the thread (unless they digress from their reply to A, and really reply to B or C).

So the question for me remains this:  what benefit do I see of the rotisserie format over other free-form formats?  And the only answer I can think of is that the discussions I've participated in here have been of a uniformly higher caliber and quality than those elsewhere.  But I honestly do not believe this is due to the format, so much as the identities of the other participants.

Part of the high discussion quality level is due to the fact that we are not posting under fully anonymous guidelines; our profiles often tell others about our serious interests.  We represent ourselves as professionals, even if as professionals expressing opinions outside of our primary field.  But each of us clearly takes this (voluntary) assignment seriously, and so reponds accordingly.

I think we could all participate in a free-form online community with nearly identical results -- if not better ones when freed from our restrictions to reply.

Hmmm -- well, perhaps there is one benefit to the rotisserie.  It would encourage those who are shy, or those who are extroverted, to post exactly one well-reasoned reply each round.  Shy people are not allowed to lurk (read without posting), and extroverts aren't allowed to overly dominate the conversation by replying to every single message.  So there is some social control being exercised.

Yet I think for, say, a classroom situation where participation is mandatory (post one message and two replies in each week's discussion thread), the requirement of the course ought to be enough to stimulate discussion.

Not to harp on it, but I thought the rate-each-post aspect was the most promising feature, and am sorely disappointed that I didn't see it work properly.  That to me was the most useful novelty of the rotisserie system, and so if I were to take anything away (to develop my own BBS, for example), it would be to try to implement that in some format.

WDave

That is my favorite thing about the Rotisserie - no one person or small group can dominate the conversation.  There are often serious problems with such activities in a more traditional threaded discussion.

I disagree that "...we could all participate in a free-form online community with nearly identical results."  I think the structure has everything to do with how successful the Development and the Internet project has turned out to be, as I am able to compare a threaded discussion board and the Rotisserie which are available to the same group of people.

I don't trust that people, even if their grades count on it, will be such good participators in a free-form situation.  Also, having a human moderator in a Rotisserie with 1000 potential users is a full-time job - who can pay someone to run this when it can be so easily automated?

I do wonder, though, what was WDave's problem with the ratings?  I've never had a problem...

I agree with what Wendy is saying -- though I haven't used Rotisserie enough to offer many examples.

Free-form discussion does different things than here -- much better for back-and-forth argument among three or so main participants, for example. But Rotisserie assures that everyone (who bothers) can be heard, can get at least one response, and so on. (And the e-mail reminders help to insure that potential participants do bother.)

OTOH, it's harder to get a sense of "the whole" here, and there isn't any one thematic thread to the conversation in these staged posts and replies.

To address Wendy's question, I posted a fairly long message detailing the rating behavior I had seen to one of the moderators of a tech support area here -- and received no response.

In a nutshell, I thought based on the system documentation available at the time (and nobody has yet seen fit to correct me, so I don't know if this is right or not) that if 5 of us posted in round 1, each of us would have 4 points to rate the other four posts, before writing our replies in round 2.  But I'd often get only 1 point, and only one post to rate, immediately before composing my reply.  And then immediately afterward, finding out that there were in fact 4 other round 1 posts, not just one or two others.

Then looking at the ratings given to each of those earlier posts, or even after several rounds, I saw a bunch of strange numbers -- averages that didn't look possible given the number of participants.

That said, just now, I was given 1 point to rate 1 post, immediately before I replied to that exact same post.

If that's correct system behavior, then I'm not sure how it's actually useful.  If it's a bug, then my original comments stand (that I wish I could see it work correctly).

Hope that clears things up, and I apologize if this repetition comes across as complaining; it's not intended to.  (And if anyone would care to point me to newer documentation or bug reports that tells me what I've misunderstood, I would be glad to read it, and/or take this part of the discussion offline.)

Lot's of stuff here, so I'll address each point individually.

Ratings: Where did you send your problems with the ratings system?  I searched through the stuff submitted via the feedback page and found nothing with your name on it.  Did you submit it as another user or not logged in?  In any case, could you please resubmit it via the feedback page (link at the bottom of the page) while logged in?

Manual Routing: One way of thinking about the rotisserie is that it tries to copy what a good seminar leader does, but it does so in a scalable way.  If the teacher has the time and gumption, doing a manual rotisserie is always preferable (and reasonably easy if not quick to do using a traditional message board).

But it takes a lot of time to read all of the responses and do the routing.  For reference, the rotisserie was originally conceived for 100+ person law student courses, in which each student was submitting responses of up to 500 words in length.  That's roughly 200 pages of material to sift through and make sense of.  It's just not reasonable to expect a teacher to take on that load every week, but the cost of hosting a rotisserie every week is trivial.  Even for a class of 20 students, the time involved to do the work manually is not trivial.  

Certainly a teacher wants to evaluate the writing of her students, but the teacher need not limit student writing to what she has time to evaluate.  One of the main uses of the rotisserie is to give the students the opportunity to do more writing than the teacher has time to evaluate but provide them with feedback via their peers.

Now imagine using this tool for really large groups, which several groups are considering doing.  It'd be nearly impossible to have someone manually assign a 1,000 person rotisserie, much less a 15,000 person one.

Also note that we have non-random routing methods and that the choice of routing methods will continue to grow as we continue to improve the system.  Currently, in addition to random routing, we offer bounce-back routing, which creates conversation between pairs of people (Joe responds to Sue who responds to Joe who responds to Sue, etc) and poll-based routing, which allows the leader to base the routing on the subtantive content of the post, as indicated by the poster in response to a poll question (pro file sharing respondants can be routed to anti file sharing respondants or the converse).

Limiting format: The rotisserie is clearly somewhat limiting and is not meant to be a plugin replacement for all threaded messaging.  There are plenty of things that standard threaded messaging is better for (free flowing brainstorming or informal disussion around large topics, for instance).  It's not quite as limiting as you make it out to be, though.  We do support thread comments, which allow any participant to respond to any post in the discussion while the discussion is still active (as I am doing in this comment!)

Quality of Posts:  You haven't addressed one major feature of the rotisserie, which is its use of semi-synchronous rounds, which lets uesrs to take more time to write and edit thoughtful posts rather than racing to post quickly.  This is one of the big reasons that we think rotisserie discussions work better in many cases that completey asynchronous discussions.  It is a piece of commonly help wisdom that online communications suffer in quality because they are too fast.  The rotisserie mitigates this problem by slowing things down a bit.

Another key to the quality of rotisserie posts is that the knowledge that every post will be specifically assigned to at least one other participant serves as both a carrot and a stick for crafting better posts.  Participants write more carefully because they know there is a much greater chance that they will get useful feedback and also because they know that they will be critiqued for anything they write that cannot be defended.

Required participation: In fact, the best non-rotisserie online discussions I've participated in have operated under rotisserie like rules imposed by the leader outside of the system (everyone needs to respond by this date and then respond to at least one other student by this date).  The rotisserie just handles and enforces all of the details within the system, making the process easier (and therefore more likely to happen more often) for everyone involved.