"That isn't what you said", he counters, then turns away. "Forget it. I shouldn't have started this."
Other options:
no subject
"That isn't what you said", he counters, then turns away. "Forget it. I shouldn't have started this."