Mix brown and white sugar with the milk in large pot; bring mixture to a boil. Stir in peanut butter, reduce heat to medium and bring the mixture back to boil while stirring constantly. Remove the pot from heat when a drop of the mixture forms a ball in a glass of cold water. Stir butter and vanilla into the mixture; stir vigorously until the fudge hardens. Stir always in same direction. Pour fudge onto buttered plates or waxed paper. Let cool and cut into 1-inch pieces.