Your are having trouble because the ".col" files are meant to be used with the "Color Samples Panel" not the "Color Picker Panel".
To access the color samples panel and use a .col color file:
1 - Select "Tools" and then click on "Show Color Samples Panel" from the menu (NOTE: Shortcut or this on a PC is hitting the "S" key on your keyboard).The Color Picker allows you to load custom colors as well, but handles things differently. For the Color Picker, you need to use image files (.jpg, .png, .bmp etc...). For that, you'd do just what you were trying to do with the .col files, but use an image instead.
2 - With the Sample panel now opened, click on the small black triangle in the upper left corner of the panel.
3 - Select "Load Colors" and locate your .col file.
4 - Select your .col file and click on "open" and you should be good to go.
I made a quick video tutorial for how to use color panels in ArtRage for someone else who had asked a while back. To watch it, please Click Here. It goes over how to both the Color Samples Panel and the Color Picker Panel, as well as demonstrating how to use a reference image as a color source.