Should I remove the gravel from my garden soil?

Image by clattermonger
I dug down into the soil beside the pen wall to the level of the bottom of the lowest breeze block. I can now get in to remove the wall without worrying about the weight of the soil.

Question by tagsbear: Should I remove the gravel from my garden soil?
I am a beggining gardener. I am making a garden, next to a driveway that was gravel. I turned over the dirt and there is ALOT of gravel in the soil. Should I try and remove it? I want to plant perennials and annuals.

Add your own answer in the comments!