Striking a Balance: Alleviating Inconsistency in Pre-trained Models for Symmetric Classification Tasks