Version 24.0 (64-bit)

When applying an effect to a path in Illustrator, checking the "Preview" checkbox will preview that effect, but unchecking the box does not return the path back to its original shape. To test:

Use the pen tool to draw a multi-segment shape.
Select the shape
Click Effect->Distort & Transform->Roughen
Click the Preview checkbox (the effect should be applied)
Click the Preview checkbox again (the effect will not be removed)

I've tested this with most of the previewable effects in the Effects menu and they all behave the same.