So for others with this problem what I discovered is if I have previously opened property box in any other perspective and then went back to debug it is slow again. Each other perspective where the properties gets opened I need to make sure I close it before I begin debugging again. Basically I've been toggling back to modeler perspective a lot also and that was causing problem too.
PS: And yes I know Applebee will say don't use Modeler perspective, it's just habit!! Trying to break. 
I can just stay within SAP HANA Developer perspective for what I'm doing. I have habit of going back to modeler to check/tweak my view that I'm processing in my procedure.