Thorn in one's side
I believe the answer is:
bane
'thorn in one's side' is the definition.
(I've seen this in another clue)
This is all the clue.
(Other definitions for bane that I've seen before include "Cause of distress, ruin" , "Something causing misery or distress" , "Source of misery" , "Cause of trouble or worry, like bean" , "Cause of great distress in a bean" .)