When using the type of end mill holder that uses a grub screw to hold the EM in place, it's important to make sure the slot in the EM is pulled up against the screw.? Otherwise, as I learned the hard way, the end mill still can be pulled down and ruin your work.? You want the screw to act as a stop as well as a holding device.
This said, I have to wonder about smaller end mills that don't have a slot ground in their shank.? I think a collet-type holder probably would hold them more firmly.? I definitely would use a collet for smaller carbide end mills -- the point force of the grub screw could break the cutter.?
At work we had a machine for cutting small pockets in things like integrated circuit packages and it could spin the cutters up to 150,000RPM's.? As you can imagine it used a very well balanced collet.
Mark
