No. You do not lose your points for not buying or applying each year. Points never expire. Your Special Permit points return to zero when drawn, but only for that category. Your other points remain and carry over.
Keep in mind that you only earn points by buying applications each year. You earn one point for each Special Permit application you buy. You can buy one application for each category each year.
12th of December, 2013