Whats the Mask to Selected Polys checkbox for?
The Mask to Selected Polys checkbox allows you limit your vertex color changes to only those faces you have selected. Your changes will then have a hard edge at the selected face boundry and a soft edge within the selected face(s). The checkbox itself will only become enabled when an object has both a vertex and face selection defined. For more information on this topic, please refer to the following page in the help section.