Will I be charged for kicking the guy in the knee cap?

Possible self defense claim, his size and yours don't matter. He technically bothered you but you kicked him in anger and retaliation, basic assault if you ask me. Good luck with the case if it comes down to it however I doubt he'll take it that far. I feel you were right in defending yourself.