That sounds very strange
It makes no difference for me whether I have Reference images loaded or not. Nor should it make a difference for you!
Do you see this problem when using a specific reference image or is it happening with any reference image?
If you start a new document with different reference images dose it still happen to you?