Our Bible reading calls Joseph a "righteous man." which sounds like his being a close-to-perfect Baptist. I prefer the time-honored option of calling Joseph a "Just Man," or one who gave another whatever the other person has a right to.
Joseph, while seeing that his espoused wife was pregnant when he was the only one to know he wasn't the father, felt that he had no right to claim the child.
On the other hand, Mary, whose sinless life gave her the right to praise from all sides, could not be seen to have had adultery.
My mother used to say, "You are damned if you do; you are damned if you don't." A modern way of puttimg that is to say "Joseph was in a bind." When have you ever been in a bind?