Working out of your head is good too, but knowing how to creatively work with reference is a neat trick.

First you learn how to make things look real (which then frees you up to later make unreal things look real as well because you're learning how light and color and texture work).

Then it's also hard to paint things without reference, at least if you're painting realistically. And even if you are putting a lot of things together that weren't together in the real world, you can still apply it all to a style or technique and translate it into a consistent finish. (example, if you key everything to blue and green. You still would know how to make it read properly.)

I understand what you mean though that you don't want to just reproduce photographs as they appear in the photo. But remember that using reference is still legitimate, even as merely a starting point from which you can deviate.

Nothing is wasted.