Brush shifting

I seem to encounter a problem when I use the brush tool that occassionally it seems to switch to a different tool. I don't know if anyone else has this problem but sometimes it forgets that I am using the brush end and acts like an eraser, or it sticks to the last tool I used.