After you click the red QuickMask button, a translucent red mask appears around your selection. This red mask represents the outside of your selection. When you click the dashed QuickMask button to go back into normal selection mode, anything red will not be selected, and anything clear will be.

So, let's start playing with this mask. First, right click on the image and select Filters > Distorts > Waves from the menu. Apply the filter, and watch what happens to the mask.