Preheat the oven to 350°F (175°C). Grease a 9x5-inch loaf pan.
In a small saucepan, heat the milk and lavender buds over medium heat until just simmering. Remove from heat and let steep for 10 minutes. Strain and set aside.
In a large bowl, cream together the butter and sugar until light and fluffy.
Beat in the eggs, one at a time, until well incorporated.
In another bowl, whisk together the flour, baking powder, and salt.
Alternately add the dry ingredients and the lavender-infused milk to the butter mixture, beginning and ending with the flour mixture.
Pour the batter into the prepared loaf pan and smooth the top.
Bake for 40-45 minutes, or until a toothpick inserted into the center comes out clean.
Allow the bread to cool in the pan for 10 minutes, then transfer to a wire rack to cool completely.