The vicar of Swaffham has disagreed with a local town councillor over a proposal for church bells to ring out to celebrate Brexit on January 31.