I noticed that if I create a canvas of 8000x8000, ArtRage (eventually) complies. Things seem to work fine. (This puts the working set over 1 GB of memory.)
If I try to add another layer, there's a misleading visual feedback: the pad of paper thingy shows the "add a layer" animation of a new sheet sliding into place. But no new layer is actually added, and the user gets no feedback that it really failed. If I paint, I am painting on the original layer.
If I happen to have the column of layers view open, I might notice the failure to add a new layer.
I would expect that the same situation would happen if I use smaller layers and try to add the fiftieth layer which just happens to fail malloc() due to the 32bit OS process limit.
If the app can't add a layer, the user should find out about it.