No. You only lose points in the category you drew. Your other points for that species will carry over. For example, you draw an Antlerless Deer Special Permit. You only lose your Antlerless Deer points. Your points in Quality Deer, Buck Deer or Second Deer carry over for future drawings.
3rd of January, 2014