Reviewing the Review Process for Agile 2009

Sorry, I’ve been more than a bit busy lately. Working as an independent coach is very rewarding and time consuming. For the past couple of weeks, I’ve been reviewing sessions for the Manifesting Agility Stage at Agile2009: “This Stage is all about tools and techniques for rapidly developing a deep understanding of the empirical Agile mindset, and then rapidly applying it—as individuals, and as groups.” As you can imagine, even with only 87 proposals, this takes a lot of time. Our stage has 1,440 minutes allotted and 3,750 minutes of proposals.

It’s been hard work. In theory, we should first give every proposal a proper review, even if it’s only a few sentences. While I tried that and was able to provide comments (visible to the public) or reviews (visible only to the author and other reviewers),  commented on or reviewed 50+ sessions (across all the stages). Sorry, too, that everyone whose session didn’t get a review. There are only three of us who actually do the work on this stage, and we just can’t provide notes to all of you. Consider these notes your comments.

My number one discovery is that many proposal authors aren’t providing enough detail. As a reviewer, I need to decide why I should give you a chunk of our tiny stage. I need to see what the subject is and how you will present it. You need to sell. I need to understand how you will present. Will it be a straight talk? Will there be an interactive session? What will the interactive session be, etc.? The longer your proposed session takes, the more you need to sell. For a 45-minute experience report, you can say less, but if you expect 90 minutes or even 180, you had better be clear, and I need to see what value people will get. If it’s just a straight talk and it’s 90 minutes long, you need to tell me why audience should sit and listen for the whole time. People I have seen present before will have an easier time here. I know that Mary Poppendieck can be very captivating, even when she’s just reading slides. The rest of us are not. I know that Jean Tabaka always has engaging and interesting sessions.

If you asked for 180 minutes, we had 17 proposals of that length, but each will consume 1/8th of our stage budget. For something this long, it had better be amazing especially, since attendees will have to contend with some of their workshop partners leaving halfway through. We had 4 or 5 that would be amazing, but for time constraints, we can only pick one.

Experience Reports: “captures the story of a real Agile project, summarizing what happened on the project and the key learning points”—these are generally 45 minutes long, so when you propose a 90-minute experience, we want to know why the audience will sit that long to hear your story. This really becomes a case when less is more.

Also remember that what you write in your proposal has to sell not only the reviewers but also the attendees. This is why they have to decide whether your session will be exciting or just a snooze.

Mechanics

On to the mechanics and my discoveries. First, I liked last year’s approach, with the whole community being able to vote on sessions and using the wisdom of crowds to find the best. However, I gather there were problems. In some cases, authors had their friends vote up their sessions, thereby gaming the system. In addition, a system like that means that the more well known a person is the better their sessions will do. So Jeff S. and Mary P. will get their sessions voted up almost no matter what they write, whereas completed unknowns had a much tougher time. This year’s system while harder work for the reviewers—at least levels the playing field a bit. Hopefully, we will find a more elegant approach in future years.

On the whole, the submission system with reviews and comments has worked fairly well, but here’s what I wish were different:

• Authors need to know that proposals are being reviewed/commented and that replies/updates are a critical part of the acceptance process.
• Authors need to be notified when a comment/review is added.
• People who write comments/reviews need to know when replies are made.
• RSS is not the only notification system; some of us still prefer email.
• The difference between comments and reviews needs to be clearer.
• The “Browse session proposals” page needs to display the length of the sessions. It makes it easier to make decisions if we know their length.
• The proposal database should be exportable to a spreadsheet. A web page is handy, but once the hard work begins, it would be a lot easier if I could just work in Excel or even Google Docs.
• I don’t want recommendations visible to the proposal author. These are often friends and colleagues who I want to work with again. This may be true today, but I can’t tell.

…

I realize a lot of work has gone into making the submission system usable, and my suggestions just create more work, but knowing what matters next year would help the authors and the reviewers by reducing the burden. I also understand that many great sessions won’t be accepted because not enough detail was provided.

Image via: Agile 2009 Conference