For mixing paint I think you have to take the average of the colors:
((100,200,150) + (50,60,70)) / 2 = (75, 130, 110)
BTW: mixing paint is NOT the same as mixing light like 255,0,0 (red) + 0,255,0 (green) + 0,0,255 (blue) = 255,255,255 (white)
try it yourself: take the 3 colors paint (red green blue) and paint 3 parts of a circle each with a different color, then spin the circle (using a pencil)
take the same colors and mix then with same amounts, you won't get the same color as with spinning the circle.