You don’t need to respond to all invitations, but yes, you can decline them. In the For Me tab, tap the three dots next to the invitation, and tap decline. You can give a reason (minimum 80 characters) if you’d like, but it is not required.