It's Tachibana's thoughts that slam into Jules far before the words do and make him realize that he'd messed up. Minsid hell, this is one of the reasons why he doesn't hang around other people. Initially he's angry at himself, though his years of keeping his emotions and thoughts well concealed serves him well once again and no sign of that anger comes out.

Tachibana's words, however, and the accusation behind them break past Jules' barriers and he flinches visibly. He's getting far too soft if he can't even take that without showing that it hurts.

"As an unpleasant side effect."

It's only the fact that Jules can still hear what's swirling around in Tachibana's head that keeps him from retorting in his usual sarcastic and antagonistic style.