*I had drafted this earlier this summer but never got around to finishing and posting this. Am doing so now*
I just finished reviewing for a conference and in looking over my reviews was reminded of the following comment by a senior colleague to me at ICML this year. He has been program chair for several conferences and said that he thought that RL reviewers were the hardest on their own community that he had ever seen. I thought back to my turns as senior PC or area chair at ML conferences and realized that I had felt the same way in those cases.
So, assuming you agree with the supposition, why is it the case? In looking back at years of reviewing, I think the reasons are:
1. There is a large subset of the RL community that is at best skeptical of papers that don't do "real applications". Certainly there is good reason to be hard on shoddy empirical work or claims. Nevertheless this subset goes too far perhaps?
2. Assuming that a reviewer is willing to accept simulated domains, in the absence of widely accepted benchmarks there is no agreement to a standard suite of problems and so different reviewers set their own standards of acceptable empirical test sets. Many reviewers reject "gridworld"-types of tasks for example.
3. There is also a healthy skepticism of theoretical results in a section of the RL community. And indeed, some theory while quite possibly true has little significance. But again, perhaps these papers are treated too harshly?
4. Perhaps the most important reason however is that a significant part of RL research is focused on issues other that strict performance. Arguably, the focus of machine learning on benchmarks and performance is misplaced for RL? This is to say that while some part of the RL community needs to focus on engineering and performance, others should be allowed and even encouraged to explore the more difficult to quantify AI issues.
Of course it is entirely possible that we are not producing enough good papers as a community and thus being harsh is the right thing. I don't believe this :)
Any comments?
(In a later post, I will make specific suggestions to address the issues above)
Subscribe to:
Post Comments (Atom)
4 comments:
Interesting... as a semi-outsider, it always seemed to me that RL is one of the more amicable fields. So the discrepancy between that and reviewing practices is even more surprising (at NIPS it is well known that you don't want to put your paper in the RL category lest you get those harsh reviewers).
I think of myself as a rather harsh reviewer (and many will vouch for the fact that I am a critical person in general), but when reviewing for ICML this year (for the first time; mostly RL papers), I was surprised to find myself the least critical on the papers I reviewed, in (almost?) all cases.
I felt good with myself for being somewhat forgiving :-). Is the RL community not shooting itself in the foot a bit?
Hi Yael. Indeed, modern RL is a most amicable field and much of the credit for that goes to our founders, Andy Barto and Rich Sutton. This comity is really quite prevalent within the field. But reviewing by its very nature is intended to bring out our ability to discriminate between good and bad work, and so it is not surprising that reviewing is where the issue under discussion arises.
I agree with your notion that RL might be shooting itself in the foot by this reviewing attitude. This impact is most in evidence at NIPS I think. I intend to write a next post suggesting how we, as writers of submitted papers, can help the reviewers understand the diversity of research-goals in RL.
Yes, that would be good also re the discussions recently on the RL mailing list. Some of us are interested in real-world applications, some in theoretical advances and some in animal behavior... (to name only three). Reviewers can't expect (and should not require, I think) that each paper address all three (or more) -- especially given the fact that conference papers are limited to 8 pages! I am definitely *not* advocating losing the ability to sift good from bad work, or becoming a field of mediocre work, but recognizing diversity of goals and approaches can help get more good work out there...
Keeping the field moving forward is a delicate process and tough reviews do have a role to play. We certainly don't want to become complacent. At ICML this year, reviewers across the board will be urged to not got bogged down in finding picky little things to complain about, but to focus on the ideas opened up by the work. We'd rather see some papers that give us really interesting things to think about than to publish a series of flawless incremental work. (False choice, I know, but I'm just trying to convey the preferences.)
Post a Comment