Introduce a tolerance for Snap to Grid / Pixel, so that snapping happens only when points are close enough to grid intersections
When Snap to Grid is enabled, the snapping is hard — a dragged point is always constrained to the grid intersections. While sometimes it’s handy, at other times is just too rigid, especially if you zoom in and try to make tiny adjustments. Paired with the fact we can’t toggle snapping temporarily on the fly, this makes it into 'try, curse, undo, disable, try again' routine. Same applies to Snap to Pixel.
Compare how Affinity Designer handles it (the turquoise GIF) with the way Ai uses (the orange GIF).

-
Plus you might wanna be interested in this request as well:
Allow to exclude handles from being snapped to grid
http://illustrator.uservoice.com/forums/333657/suggestions/49616768