I’ve had this issue but in my case it was the foam tips I was using. I think they were causing a sight allergic reaction in my ears. I switched to another pair and the issue improved.

I think sweat getting trapped in there also exacerbates the issue (at least for me)

But I agree this is an issue and one Apple should probably address.