Removing bugs in my herb garden?

I have black ants crawling all over my rosemary. yes I know I can just wash them off after I remove a brnach as I use it but is there a safe spray I can also use to kill them off? The main reason is we always have ants in the house so any time I can kill them off in the garden might help infesting the house.