They have a right to be upset. Not everyone enjoys having spells cast on them. Or... uh... things that are pretty similar to spells? I'm not sure if ki is magic. I'm pretty sure it can be Counterspelled, so it might be magic.

[Shrug.]

But I would rather have people taking things out on me than anyone else. I mean, if it has to be anyone, why not the person with a couple hundred pounds of armor on?