Reviewing the Review Process for Agile 2009
Sorry I’ve been more than a bit busy lately, working as an independent coach is very rewarding and time consuming. For the past couple of weeks I’ve been doing work reviewing sessions for the Manifesting Agility Stage at Agile2009: “This Stage is all about tools and techniques for rapidly developing a deep understanding of the empirical Agile mindset, and then rapidly applying it—as individuals, and as groups.” As a you can imagine even with only 87 proposals this is a very time consuming process. Our stage has 1440 minutes allotted and 3750 minutes of proposals.
Its been hard work, first up in theory we should provide every proposal a proper review even if its only a few sentences. While I tried that and I was able to provide comments (visible to the public) or reviews (visible only to the author and other reviewers) – I commented on or reviewed 50+ sessions (across all the stages). Sorry too everyone who’s session didn’t get a review. There are only three of us who are actually doing the work on this stage and we just can’t provide notes to all of you. Consider these notes your comments.
My number one discovery many proposal authors aren’t providing enough detail for either me or the attendees. Consider for me a reviewer I need to decide why to give you a chunk of our tiny stage. I need to see what the subject is, how you will present it. You need to sell. I need to understand how you will present – will it be a straight talk? will there be an interactive session? What will the interactive session be etc.? The longer your proposed session the more you need to sell. For a 45 minute experience report you can say less – but if you expect 90 minutes or even 180 you had better be clear and I need to see what value people will get. If its just a straight talk and its 90 minutes long what can you tell me about why audience will sit and listen for the whole time. People I seen present before will have an easier time here – I know that Mary Poppendieck can be very captivating even when she’s just reading slides. The rest of us are not. I know that Jean Tabaka always has engaging and interesting sessions.
If you asked for 180 minutes – we had 17 proposals of that length, but each will consume 1/8th of our stage budget. For something this long it had better be amazing especially since attendees will have to contend with some of their workshop partners leaving halfway through. We had 4-5 that would all be amazing but for the time constraints can only pick one.
Experience Reports: “captures the story of a real agile project, summarizing what happened on the project and the key learning points” – these are generally 45 minutes long so when you propose a 90 minute experience we want to know why will the audience sit that long to hear your story. This really is a case of less is more.
Also remember what you write in your proposal not only has to sell the reviewers it also has to sell the attendees. This is all they will have to decide whether your session will be a exciting or a snooze.
Mechanics
Onto the mechanics and my discoveries. First I liked last years approach with the whole community being able to vote on sessions and using the wisdom of crowds for best to be found. However I gather there were problems, in some cases authors had their friends vote up their sessions thereby gaming the system. In addition a system like that means that the more well known a person the better their sessions will do. So Jeff S. and Mary P. will get their sessions voted up almost no matter what they write. Whereas completed unknowns had a much tougher time. This years system while harder work for the reviewers at least levels the playing ground a bit. Hopefully we will find a more elegant approach in future years.
On the whole the submission system with reviews and comments has worked fairly well, but here’s what I wish were different.
- Authors need to know that proposals are being reviewed/commented and that replies/updates are a critical part of the acceptance process.
- Authors need to be notified when a comment/review is added.
- People who write comments/reviews need to know when replies are made
- RSS is not the the only notification system some of use still prefer email.
- The difference between comments and reviews needs to be made more clear.
- “Browse session proposals” page needs to display the length of the sessions. It makes it easier to make decisions if we know their length.
- Proposal database should exportable to spreadsheet. A web page is handy but once the hard work begins it would be alot easier if I could just work in Excel or even Google Docs.
- I don’t want recommendations visible to the proposal author. These are often friends and colleagues who I want to work with again. This maybe true today but I can’t tell.
- …
I realize alot of work has gone into making a usable submission system and my suggestions just create more but these would help the authors by knowing what matters next year and the reviewers by reducing the burden. I also understand that many great sessions won’t get accepted because not enough detail was provided.
Next blog posting will be back on track with something of more interest to most.
If you enjoyed this post, subscribe now to get free updates.
Did you love this post?
Then please share it!
March 25, 2009 at 5:13 pm | Elisabeth Hendrickson
I was with you all the way up until “I don’t want recommendations visible to the proposal author. These are often friends and colleagues who I want to work with again.”
You’re a reviewer on a session that I proposed. I can see your review, though if you have a separate field for an accept/reject recommendation, I can’t see that. (And btw, I posted a comment answering a question you had in your review.)
However, I want to note that even if I *could* see a big old “REJECT” recommendation, it would be OK. Honest.
I am grateful to you for serving on the conference committee. It’s a hard, thankless, time-consuming job. I know; I served on the committee for 2 years.
I understand that this is an excruciatingly difficult decision process and that there are far more good submissions than there are slots. And I understand that your recommendation is not about whether or not you like me, or whether you want to work with me again. This is a decision about which sessions have the greatest probability of serving the audience for the conference. Ultimately, that’s about serving the customers: the conference delegates.
Greater transparency gives me more feedback from which to learn for next time.
So I hope the conference review system allows for even more transparency in coming years. I might not like the news, but I’d rather know.
March 26, 2009 at 10:06 am | Mark Levison
After a conversation with Elisabeth I’m convinced, be open and transparent is the right thing. Proposal authors should be informed of what you have to say. There is a small risk that a few noses will get out of joint but that’s a risk we have to take.
March 27, 2009 at 7:58 am | Chris Matts
Great post.
Dave Nicollete and I are thinking of running an informal session at the end of Agile 2009 where a few reviewers could give some guidance to people submitting next year. Interested in joining the informal panel?
July 29, 2009 at 12:09 pm | J. B. Rainsberger
I finally took the time to respond to this, Mark. I like your suggestions and really hope that Agile 2010 with implement them. I especially think they need to improve the flow of information during the proposal period. If they do a good job of that in 2010, then perhaps as early as 2011 we can reduce the selection process to scheduling and managing conflicts, because we will have done all the reviewing and rating leading up to the submission deadline.
I have a number of suggestions, too, but I fear that if the stage producers and submitters don’t jointly take the reins here, then nothing will change. We know how much energy one needs to produce a stage and review submissions. It exhausts us. If we truly want to change things, then we must start now.
I like Chris and Dave’s idea about providing guidance to submitters. I think we should also guide them in demanding feedback! I think that would provide a great first step. I think the submitters and reviewers both need to act as customers for the submission system. I volunteer to own a feature request. Who else does?
July 29, 2009 at 1:34 pm | Dave Nicolette
Mark,
I agree with one or two others who have commented that authors need to see any recommendations that are made. The recommendations are the way we improve our proposals. If recommendations are secret, then submitters have no basis to improve their submissions. We’re just tossing darts in the general direction of the dart-board while blind-folded. I don’t think this has anything to do with noses getting out of joint. After all, we /want/ our sessions to be interesting and valuable.
Think about the way sessions are evaluated for the XP Days Benelux conference. FWIW I think it’s the best-organized agile event on the calendar; the organizers actually apply agile thinking to the task of organizing the conference. It’s the only event I know of that uses a truly open review process, although there are several others that solicit feedback from “anyone” rather than just official reviewers. Submitters and other interested parties openly help improve all the submissions. This not only makes individual session proposals better, but also enables people to eliminate duplicate content, combine proposals, and refine the content of sessions so that there is a coherent flow to the whole event, and so that presenters can be sure they are addressing particular areas of interest to participants. Participants can experience a certain level of continuity and consistency throughout a day’s worth of sessions presented by different people – sessions that weren’t originally conceived as part of a continuum of related content. The result tends to be much better than could be achieved if each submitter worked in isolation and only a few people made decisions about acceptance.
If we talk about all this in Chicago, with Chris and that other Dave guy whose surname is spelled almost the same as mine, maybe we can include one or more of the people involved with that event to see if they might have some ideas we can use for Agile 2010.
Cheers,
Dave