Sorry, I’ve been more than a bit busy lately. Working as an independent coach is very rewarding and time consuming. For the past couple of weeks, I’ve been reviewing sessions for the Manifesting Agility Stage at Agile2009: “This Stage is all about tools and techniques for rapidly developing a deep understanding of the empirical Agile mindset, and then rapidly applying it—as individuals, and as groups.” As you can imagine, even with only 87 proposals, this takes a lot of time. Our stage has 1,440 minutes allotted and 3,750 minutes of proposals.
It’s been hard work. In theory, we should first give every proposal a proper review, even if it’s only a few sentences. While I tried that and was able to provide comments (visible to the public) or reviews (visible only to the author and other reviewers), commented on or reviewed 50+ sessions (across all the stages). Sorry, too, that everyone whose session didn’t get a review. There are only three of us who actually do the work on this stage, and we just can’t provide notes to all of you. Consider these notes your comments.
My number one discovery is that many proposal authors aren’t providing enough detail. As a reviewer, I need to decide why I should give you a chunk of our tiny stage. I need to see what the subject is and how you will present it. You need to sell. I need to understand how you will present. Will it be a straight talk? Will there be an interactive session? What will the interactive session be, etc.? The longer your proposed session takes, the more you need to sell. For a 45-minute experience report, you can say less, but if you expect 90 minutes or even 180, you had better be clear, and I need to see what value people will get. If it’s just a straight talk and it’s 90 minutes long, you need to tell me why audience should sit and listen for the whole time. People I have seen present before will have an easier time here. I know that Mary Poppendieck can be very captivating, even when she’s just reading slides. The rest of us are not. I know that Jean Tabaka always has engaging and interesting sessions.
If you asked for 180 minutes, we had 17 proposals of that length, but each will consume 1/8th of our stage budget. For something this long, it had better be amazing especially, since attendees will have to contend with some of their workshop partners leaving halfway through. We had 4 or 5 that would be amazing, but for time constraints, we can only pick one.
Experience Reports: “captures the story of a real Agile project, summarizing what happened on the project and the key learning points”—these are generally 45 minutes long, so when you propose a 90-minute experience, we want to know why the audience will sit that long to hear your story. This really becomes a case when less is more.
Also remember that what you write in your proposal has to sell not only the reviewers but also the attendees. This is why they have to decide whether your session will be exciting or just a snooze.
Mechanics
On to the mechanics and my discoveries. First, I liked last year’s approach, with the whole community being able to vote on sessions and using the wisdom of crowds to find the best. However, I gather there were problems. In some cases, authors had their friends vote up their sessions, thereby gaming the system. In addition, a system like that means that the more well known a person is the better their sessions will do. So Jeff S. and Mary P. will get their sessions voted up almost no matter what they write, whereas completed unknowns had a much tougher time. This year’s system while harder work for the reviewers—at least levels the playing field a bit. Hopefully, we will find a more elegant approach in future years.
On the whole, the submission system with reviews and comments has worked fairly well, but here’s what I wish were different:
- Authors need to know that proposals are being reviewed/commented and that replies/updates are a critical part of the acceptance process.
- Authors need to be notified when a comment/review is added.
- People who write comments/reviews need to know when replies are made.
- RSS is not the only notification system; some of us still prefer email.
- The difference between comments and reviews needs to be clearer.
- The “Browse session proposals” page needs to display the length of the sessions. It makes it easier to make decisions if we know their length.
- The proposal database should be exportable to a spreadsheet. A web page is handy, but once the hard work begins, it would be a lot easier if I could just work in Excel or even Google Docs.
- I don’t want recommendations visible to the proposal author. These are often friends and colleagues who I want to work with again. This may be true today, but I can’t tell.
…
I realize a lot of work has gone into making the submission system usable, and my suggestions just create more work, but knowing what matters next year would help the authors and the reviewers by reducing the burden. I also understand that many great sessions won’t be accepted because not enough detail was provided.
Mark Levison has been helping Scrum teams and organizations with Agile, Scrum and Kanban style approaches since 2001. From certified scrum master training to custom Agile courses, he has helped well over 8,000 individuals, earning him respect and top rated reviews as one of the pioneers within the industry, as well as a raft of certifications from the ScrumAlliance. Mark has been a speaker at various Agile Conferences for more than 20 years, and is a published Scrum author with eBooks as well as articles on InfoQ.com, ScrumAlliance.org an AgileAlliance.org.
I was with you all the way up until “I don’t want recommendations visible to the proposal author. These are often friends and colleagues who I want to work with again.”
You’re a reviewer on a session that I proposed. I can see your review, though if you have a separate field for an accept/reject recommendation, I can’t see that. (And btw, I posted a comment answering a question you had in your review.)
However, I want to note that even if I *could* see a big old “REJECT” recommendation, it would be OK. Honest.
I am grateful to you for serving on the conference committee. It’s a hard, thankless, time-consuming job. I know; I served on the committee for 2 years.
I understand that this is an excruciatingly difficult decision process and that there are far more good submissions than there are slots. And I understand that your recommendation is not about whether or not you like me, or whether you want to work with me again. This is a decision about which sessions have the greatest probability of serving the audience for the conference. Ultimately, that’s about serving the customers: the conference delegates.
Greater transparency gives me more feedback from which to learn for next time.
So I hope the conference review system allows for even more transparency in coming years. I might not like the news, but I’d rather know.
After a conversation with Elisabeth I’m convinced, be open and transparent is the right thing. Proposal authors should be informed of what you have to say. There is a small risk that a few noses will get out of joint but that’s a risk we have to take.
I finally took the time to respond to this, Mark. I like your suggestions and really hope that Agile 2010 with implement them. I especially think they need to improve the flow of information during the proposal period. If they do a good job of that in 2010, then perhaps as early as 2011 we can reduce the selection process to scheduling and managing conflicts, because we will have done all the reviewing and rating leading up to the submission deadline.
I have a number of suggestions, too, but I fear that if the stage producers and submitters don’t jointly take the reins here, then nothing will change. We know how much energy one needs to produce a stage and review submissions. It exhausts us. If we truly want to change things, then we must start now.
I like Chris and Dave’s idea about providing guidance to submitters. I think we should also guide them in demanding feedback! I think that would provide a great first step. I think the submitters and reviewers both need to act as customers for the submission system. I volunteer to own a feature request. Who else does?
Mark,
I agree with one or two others who have commented that authors need to see any recommendations that are made. The recommendations are the way we improve our proposals. If recommendations are secret, then submitters have no basis to improve their submissions. We’re just tossing darts in the general direction of the dart-board while blind-folded. I don’t think this has anything to do with noses getting out of joint. After all, we /want/ our sessions to be interesting and valuable.
Think about the way sessions are evaluated for the XP Days Benelux conference. FWIW I think it’s the best-organized agile event on the calendar; the organizers actually apply agile thinking to the task of organizing the conference. It’s the only event I know of that uses a truly open review process, although there are several others that solicit feedback from “anyone” rather than just official reviewers. Submitters and other interested parties openly help improve all the submissions. This not only makes individual session proposals better, but also enables people to eliminate duplicate content, combine proposals, and refine the content of sessions so that there is a coherent flow to the whole event, and so that presenters can be sure they are addressing particular areas of interest to participants. Participants can experience a certain level of continuity and consistency throughout a day’s worth of sessions presented by different people – sessions that weren’t originally conceived as part of a continuum of related content. The result tends to be much better than could be achieved if each submitter worked in isolation and only a few people made decisions about acceptance.
If we talk about all this in Chicago, with Chris and that other Dave guy whose surname is spelled almost the same as mine, maybe we can include one or more of the people involved with that event to see if they might have some ideas we can use for Agile 2010.
Cheers,
Dave