Withholding forgiveness makes that person the sinner
“Yom Kippur does not atone for transgressions committed against a fellow man unless we conciliate him.. The person whose forgiveness is sought should grant forgiveness willingly and wholeheartedly and not be obstinate.. if a person harbours enmity in his heart, his prayers on Yom Kippur will not be accepted, God forbid, but the one who is magnanimous and forgiving, will have all his sins forgiven.” (Code of Jewish Law 131:4) This authoritative rabbinic text says that only if the person who was wronged forgives the wrongdoer, will God forgive the aggrieved person. Again this moves the definitive judgment from the human to the Divine realm.
https://web.archive.org/web/20160324043723/http://yonanewman.org/kizzur/kizzur131.html
The Concise Code of Jewish Law, the Kitzur Shulchan Aruch, is sometimes considered too concise. So it is valuable to find the sentiment expressed in the earlier and more thorough Shulchan Aruch of Reb Schneur Zalman of Liadi, the ‘Alter Rebbe’ of Lubavitch:
“Section 606:4.[One who is asked to] forgive should not be callous and withhold forgiveness.6 On the contrary, he should forgive immediately, unless his intent is for the benefit of the person asking forgiveness, to humble his brazen heart.
Footnote 6: Indeed, a person who does so is now considered the sinner (Rambam, loc. cit.). To illustrate this principle, Rambam notes in the following law (2:10) that because the Gibeonites did not forgive the family of King Shaul, they were ostracized: “The Gibeonites are not among the Children of Israel” (II Shmuel 21:2).”
http://www.chabad.org/library/article_cdo/aid/3385299/jewish/Shulchan-Aruch-Chapter-606-Requirement-to-Appease-One-Another-on-Erev-Yom-Kippur.htm