Once a reviewer of one of our papers wrote that she/he did not understand our mathematical model. That's a honest submission. No one expect that every one will understand everything. But had not that affected the decision made on our paper? May be. May be not.
But every reviewer faces this problem. As science is getting more and more interdisciplinary, one often find some part of the paper bit difficult to understand and review. I am not talking of complete ignorance. Neither talking of a badly written paper. Am talking of a situation where you broadly understand the concepts and issues, but lacks clarity on particulars in that paper. The best option then is to ask the authors to explain and help you understand their paper better.
Once you have understood the paper, with clarity, then only you can make a rational judgment on the paper. Isn't that obvious? But not in practise. Journals does not allow you to post queries or make comments on a manuscript without making a judgment out of three choices: minor revision, major revision, or reject.
There is no scope of a dialogue, albeit with anonymity, between the people who did the science and those who did the vetting. Yes, there exist the practise of post-review rebuttal. But that's only after the reviewer has made the decision.
The purpose of publishing scientific papers has changed with time. So has changed the practise and culture of peer-review. Journal editors complain of shortage in serious reviewers, authors complain of lackluster reviews, reviewers complain of lack of professional incentive in reviewing papers.
Even then, there are people who review each others papers and do that with all earnest. They still believe in the elementary purpose of peer-review of a scientific paper: to improve the manuscript and to improve the work reported there.
Won't it be wiser to help this lot scientists to do the job better? One step towards better review would be to provide a fourth option to a reviewer. Let the reviewers post questions or start a thread of discussion with the authors, before they decide on the manuscript.
Obviously, such interactions would be considered as part of the review documents and has to bounded by a specific duration. It would also be bounded by all legal and ethical guidelines of peer-review.
Am not sure, how many of my peers will use this option. But letting some use it judiciously, wont harm science, but make it better.