Hello itkolkata and near_the_pits,
I am happy to inform you that eBay is working on some changes to the Feedback System to address precisely the issue you are discussing. One of the changes is that if a member leaves feedback for a transaction and does not participate in the Unpaid Item (UPI) process, eBay will neutralize their feedback. This means the rating (whether positive or negative) will not affect the recipient's feedback score.
You can read more about these proposed changes here - http://www2.ebay.com/aw/in/200506.shtml#2005-06-24192504
Regards,
Rohit
Community Development Team
Thanks and Regards,
Rohit,
Buyer Engagement Team