Even when I turn on snapping, if I drag an object away from it's location, I want the object to not end up in it's old location. Otherwise I would not have started dragging it.

Therefore, stop making the original location available a snap target. As a movement / cardinal direction target, sure. Not as a "return to where you're from" target, this makes no sense.