My computer now has a whole extra gig of ram, totalling in at 2 gigabyte.
The brush doesn't lag on 500 DPI images anymore. Layer transform is still a problem though.
Layer transform at 500 DPI doesn't crash the program anymore, but one full screen stroke with the resize tool only resizes a millimeter or so. And it isn't exactly gradual or precise.
I'm perfectly willing to call this a bug... You somehow need to optimize Layer transformation. I just bought a whole extra gig of ram and while it stops the program from crasching it still didn't make Layer transform usefull in high resolutions.