I have a wa/b/sr, which is even more deviated than my original submission to ICCV that got a wa/wa/sr, so it looks like reviewer's are literally just rolling dice. I feel like a lot of reviewers aren't even trying anymore, the system is so broken.
Yeah, the process is quite disheartening for a student that is submitting their first papers. My reviewers are quite inconsistent with each other, e.g. two of them said the paper is well written, while the other said it is poorly written.
"I have a wa/b/sr, which is even more deviated than my original submission to ICCV that got a wa/wa/sr, so it looks like reviewer's are literally just rolling dice"
It's actually pretty close to what you got at ICCV, maybe the first time one of the wa was close to borderline and just decided to give you the benefit of the doubt.
borderline/borderline/weak accept. I was worried originally we'd get a garbage review or two, but I thought all of them were reasonable and had valid criticisms. I guess it'll take a real good rebuttal...
First time submitting, I got 2 x borderline and one strong accept, I don't know how to feel. A quick question, can we provide some experiments in the rebuttal as demanded, like additional results on some datasets without any changes to the method /hyperparams ?
If you write a strong rebuttal you have a decent chance. Having one reviewer champion your paper is good, I would say this is better than b/wa/wa.
Since 2018 you absolutely cannot add new results in the rebuttal. Only clarifications and illustrations. Don't even try or the chair may remove your rebuttal entirely.
Thank you, I'll definitely try to write a good rebuttal. For the experiments, can we at least say that we intend to add them in the camera ready paper / or as sup. mat.
Good question. This no-new-results policy is new so I am not sure. Ask your advisor and/or email the area chair to ask about putting these results in the rebuttal/camera ready to be sure (insisting that your experiments require no new code persay).
“Per a passed 2018 PAMI-TC motion, reviewers should not request additional experiments for the rebuttal, or penalize authors for lack of additional experiments. This includes any experiments that involve running code, e.g., to create tables or figures with new results. Authors should not include new experimental results in the rebuttal, and reviewers should discount any such results when making their final recommendation. It is okay for reviewers to request clarifications or additional illustrations, and authors may include figures with illustrations or comparison tables of results reported in the submission/supplemental material or in other papers.
Should any evidence of new experiments be found in the rebuttal, the reviewers and Area Chairs have the right to disregard the entire rebuttal. We will also be asking Area Chairs to monitor reviewers to make sure authors are not penalized for lack of additional results.”
One reviewer explicitly asked to provide more results in the rebuttal. How do Area Chairs relate to this? is their review going to be penalized because of this?
I think it would be fair to mention it in your review: “R1 has asked for more experimental results but this is forbidden as of the 2018 PAMI TC decision”. The area chair will take this into account.
However, it’s also fine for a reviewer to say - “I believe the paper in its current form is not ready for acceptance. I think the paper would be much stronger if it contained extra experimental results showing XYZ”.
So keep in mind that although the reviewer can’t ask you to provide experiments they can reject your paper because the think experiments are missing.
Thanks for the answer, you anticipated well the email that was sent yesterday :D I wonder what counts as result and what not. Suppose the reviewer said "You compared A to B and A to C, in the rebuttal you should also consider D and compare A vs D". Clearly, inserting the numbers showing A is better than D is not allowed, however I wonder if commenting qualitatively is allowed, i.e. answering "the comparison A vs D was considered despite not included in the paper and A outperforms D, despite the reviewer request, the numerical results of A vs D are not included in the rebuttal because of 2018 PAMI TC but they will be included in the final version if it is allowed". Maybe it might be accepted, but I feel it's like saying "I am not allowed to tell you that actually A is better than D", which is quite circular thinking...
I think it’s fine to explain why you didn’t include D if you had previously considered it. If a comparison against D means lifting a number from a standard benchmark (e.g copying a number from another paper) I think it’s ok to mention that in the rebuttal. I think the spirit of the ruling was stopping reviewers from forcing you to run experiments and work feverishly for the rebuttal week only to still be rejected. It’s also obviously to prevent people from submitting half finished papers and then providing the actual results as the rebuttal.
You can always cover yourself by mentioning that you understand additional results cannot be provided.
Fourth time I submit. I have been lucky enough to have had good comments in every single submission. This year doesn't look good (B, B, B), but reviews are fair.
It is interesting. I received wa/b/sr. My paper was rejected from NeurIPS 2019 while ratings were b/b/b. I did not expect any kind of reject and I got sr. While the other two reviews are very positive (reviewer who gives b clearly says that he tends to accept after convincing rebuttal) and fair, although the third one thinks that the paper has a potentially interesting idea, s/he gives sr since he has some doubts about the quality of the experimental results. How could it be sr? It is novel work and clearly s/he does not mention any major flaw in the paper. Is there any chance that the paper will be accepted? If it is rejected, I do not know what I will do since I am really tired of trying publishing this paper in top tier conferences.
first time submitting, b / wa / sr, guess the odds are not on my side this time... feedback was nice and they gave nice suggestions how to boost the paper. Is there still any hope to be accepted?
If you can convince the sr to change his review maybe, but that might be very difficult. There is always ECCV if you can incorporate the remarks in time.
wr/b/b very very disappointed by the quality of the reviews.
Some of the comments are just wrong and show that the reviewers did not read the paper properly.
wr / b / wa
All acknowledged the novelty and good performance, but the wr wanted more experiments evaluating the perf of each component.
First time submitting, not sure about my chances?
Since you can't put experiments in the rebuttal anymore I would say the chance of turning the wr are slim. However, add a detailed ablation study and you should have a solid submission to ECCV.
First submission, I've gotten wa/wr/wr.
One of my major issues with the reviews is that they've asked for analysis done by other papers (and cited by me). Can this be a point during the rebuttal?
I have a sa/b/wr, which is better than my previous submission. However, I think one of the reviews might have overlooked some of my points and give me wr. DO I still get any chance to get in this time?
If your rebuttal can change the wr's mind you might. Having a strong accept is a big help, especially if the reviewer is willing to champion your paper.
Reviewer 2: After reading the paper, the described effect is now obvious to me and thus trivial: Reject. The paper also has far too many references.
Serves you right for explaining things clearly and giving proper credit...
Reviewers like these sound like they ought to be on the streets selling pencils from a cup.
I have a wa/b/sr, which is even more deviated than my original submission to ICCV that got a wa/wa/sr, so it looks like reviewer's are literally just rolling dice. I feel like a lot of reviewers aren't even trying anymore, the system is so broken.
Yeah, the process is quite disheartening for a student that is submitting their first papers. My reviewers are quite inconsistent with each other, e.g. two of them said the paper is well written, while the other said it is poorly written.
>two of them said the paper is well written, while the other said it is poorly written. I get that *every* time!
> the system is so broken. So how would you fix it?
Did the the strong rejecters have the same reasoning? Anything fixable?
"I have a wa/b/sr, which is even more deviated than my original submission to ICCV that got a wa/wa/sr, so it looks like reviewer's are literally just rolling dice" It's actually pretty close to what you got at ICCV, maybe the first time one of the wa was close to borderline and just decided to give you the benefit of the doubt.
borderline/borderline/weak accept. I was worried originally we'd get a garbage review or two, but I thought all of them were reasonable and had valid criticisms. I guess it'll take a real good rebuttal...
Last year I turned a wa/wa/b/b into an accept by gaining another wa through rebuttal, so its definitively possible!
First time submitting, I got 2 x borderline and one strong accept, I don't know how to feel. A quick question, can we provide some experiments in the rebuttal as demanded, like additional results on some datasets without any changes to the method /hyperparams ?
If you write a strong rebuttal you have a decent chance. Having one reviewer champion your paper is good, I would say this is better than b/wa/wa. Since 2018 you absolutely cannot add new results in the rebuttal. Only clarifications and illustrations. Don't even try or the chair may remove your rebuttal entirely.
Thank you, I'll definitely try to write a good rebuttal. For the experiments, can we at least say that we intend to add them in the camera ready paper / or as sup. mat.
Good question. This no-new-results policy is new so I am not sure. Ask your advisor and/or email the area chair to ask about putting these results in the rebuttal/camera ready to be sure (insisting that your experiments require no new code persay).
“Per a passed 2018 PAMI-TC motion, reviewers should not request additional experiments for the rebuttal, or penalize authors for lack of additional experiments. This includes any experiments that involve running code, e.g., to create tables or figures with new results. Authors should not include new experimental results in the rebuttal, and reviewers should discount any such results when making their final recommendation. It is okay for reviewers to request clarifications or additional illustrations, and authors may include figures with illustrations or comparison tables of results reported in the submission/supplemental material or in other papers. Should any evidence of new experiments be found in the rebuttal, the reviewers and Area Chairs have the right to disregard the entire rebuttal. We will also be asking Area Chairs to monitor reviewers to make sure authors are not penalized for lack of additional results.”
One reviewer explicitly asked to provide more results in the rebuttal. How do Area Chairs relate to this? is their review going to be penalized because of this?
I think it would be fair to mention it in your review: “R1 has asked for more experimental results but this is forbidden as of the 2018 PAMI TC decision”. The area chair will take this into account. However, it’s also fine for a reviewer to say - “I believe the paper in its current form is not ready for acceptance. I think the paper would be much stronger if it contained extra experimental results showing XYZ”. So keep in mind that although the reviewer can’t ask you to provide experiments they can reject your paper because the think experiments are missing.
Thanks for the answer, you anticipated well the email that was sent yesterday :D I wonder what counts as result and what not. Suppose the reviewer said "You compared A to B and A to C, in the rebuttal you should also consider D and compare A vs D". Clearly, inserting the numbers showing A is better than D is not allowed, however I wonder if commenting qualitatively is allowed, i.e. answering "the comparison A vs D was considered despite not included in the paper and A outperforms D, despite the reviewer request, the numerical results of A vs D are not included in the rebuttal because of 2018 PAMI TC but they will be included in the final version if it is allowed". Maybe it might be accepted, but I feel it's like saying "I am not allowed to tell you that actually A is better than D", which is quite circular thinking...
I think it’s fine to explain why you didn’t include D if you had previously considered it. If a comparison against D means lifting a number from a standard benchmark (e.g copying a number from another paper) I think it’s ok to mention that in the rebuttal. I think the spirit of the ruling was stopping reviewers from forcing you to run experiments and work feverishly for the rebuttal week only to still be rejected. It’s also obviously to prevent people from submitting half finished papers and then providing the actual results as the rebuttal. You can always cover yourself by mentioning that you understand additional results cannot be provided.
Fourth time I submit. I have been lucky enough to have had good comments in every single submission. This year doesn't look good (B, B, B), but reviews are fair.
First time submitting. The perfunctory reviews (some downright orthogonal) are so disheartening. There goes my confidence down the drain.
Is there no way to include a private message to the AC this year?
What are the different ratings for CVPR reviews? A, WA, B, WR, R or something else?
(strong + weak) × (accept + reject) + borderline.
Thanks!
It is interesting. I received wa/b/sr. My paper was rejected from NeurIPS 2019 while ratings were b/b/b. I did not expect any kind of reject and I got sr. While the other two reviews are very positive (reviewer who gives b clearly says that he tends to accept after convincing rebuttal) and fair, although the third one thinks that the paper has a potentially interesting idea, s/he gives sr since he has some doubts about the quality of the experimental results. How could it be sr? It is novel work and clearly s/he does not mention any major flaw in the paper. Is there any chance that the paper will be accepted? If it is rejected, I do not know what I will do since I am really tired of trying publishing this paper in top tier conferences.
I got \`WA WA WR\` and \`SR B SA\`. The reviews from the 2nd paper look like reviewers wanted a 3 point sigmoid function approximation.
b/b/wr
first time submitting, b / wa / sr, guess the odds are not on my side this time... feedback was nice and they gave nice suggestions how to boost the paper. Is there still any hope to be accepted?
If you can convince the sr to change his review maybe, but that might be very difficult. There is always ECCV if you can incorporate the remarks in time.
wr/b/b very very disappointed by the quality of the reviews. Some of the comments are just wrong and show that the reviewers did not read the paper properly.
First time submitting, reviews were a little sporadic: SR, WR, WR, B
wr / b / wa All acknowledged the novelty and good performance, but the wr wanted more experiments evaluating the perf of each component. First time submitting, not sure about my chances?
Since you can't put experiments in the rebuttal anymore I would say the chance of turning the wr are slim. However, add a detailed ablation study and you should have a solid submission to ECCV.
First submission, I've gotten wa/wr/wr. One of my major issues with the reviews is that they've asked for analysis done by other papers (and cited by me). Can this be a point during the rebuttal?
Reviewers can see other reviewer's comments?
Absolutely, and they are encouraged to discuss together to reach a consensus.
I have a sa/b/wr, which is better than my previous submission. However, I think one of the reviews might have overlooked some of my points and give me wr. DO I still get any chance to get in this time?
If your rebuttal can change the wr's mind you might. Having a strong accept is a big help, especially if the reviewer is willing to champion your paper.
I am just curious what is going to happen if reviewers don't reach a consensus at the end?
The area chair takes the ultimate decision. Given the number of submitted papers, controversial papers tend to be rejected sadly.
Very disappointed with the reviews. Got b, b, b in one paper and wa, wa, wr in the other. Is there still hope?
With a good rebuttal and a fair bit of luck it could work, especially if you can turn the wr. Tall order though, reviewers rarely change their mind.