Yes. You earn one point for each year you do not draw a permit. Keep in mind that Multiple Season deer and Multiple Season elk are two different categories.
Your Special Permit points return to zero when drawn, but only for that category. Points never expire and you do not lose your points for not buying or applying each year.
10th of December, 2013