Yes, tap on the three points in the invitation under "For me". You can give a reason why you are rejecting the invitation, or you can reject it without giving a reason. If you give a reason, the user who invited you will receive a message.

The message must be at least 80 characters long. This is to prevent only impersonal messages like "I don't have time" from being sent.