Preheat oven to 400 degrees.
Cut orange in half and slice 6 thin slivers to add to the olives. Repeat with the lemon. (Save remaining orange to squeeze over the olives after baking.) Place olives in an oven proof baking dish. Nestle orange and lemon slices slices, bay leaf and rosemary sprigs around the olives. Drizzle olive oil over the olives. Sprinkle on red pepper flakes, if you like a little spice. Bake olives in the hot oven for 10-12 minutes until warmed through. Squeeze fresh orange juice over the hot olives and serve hot or at room temp.