If you are invited to the company holiday party, you should attend. While it's natural to consider declining an invitation, it's not considered proper etiquette to do so in a professional setting.
“If I’m not good enough for you to cash in a holiday day on, then fine – just decline the invitation- but it’s bloody rude to put me in a position where I’m now stuck holding the day for ...