Preheat oven to 350 degrees F (180 degrees C). Grease a loaf pan.
In a large mixing bowl, cream together the butter and sugar until light and fluffy. Beat in the eggs one at a time, mixing well between additions. Beat in the vanilla extract, mashed bananas, and peanut butter until fully incorporated. Mix in the flours, baking soda, baking powder, and salt. Stir in the milk.
Transfer batter to the prepared loaf pan and bake for 55 to 65 minutes, or until a toothpick inserted into the center comes out clean. Cool loaf in pan for 10 minutes before removing and transferring to a cooling rack to cool completely.