About the first ‘computer’ that I got my hands on was a mechanical, hand powered desktop computer that would add, subtract, multiply and divide. I think that it was produced about 1925. It required about 50 Joules to perform a multiplication. (multiplying two 10 digit numbers might require 50 rotations of the hand crank.)
0.02 multiplies per Joule.

In 1954 IBM delivered the 701 computer which consumed about 100Kw and performed a multiplication in 400 μsec. Thus 2,500 multiplies consume 100,000 Joules, or about 40 Joules per multiply.
0.025 multiplies per Joule.

Stretch from IBM using somewhat more power multiplied in 2.4 μsec in 1961. Perhaps about 3 multiplies/Joule.

Steady progress has been made and that according to DARPA we get 109 multiplies per Joule today—better than a 1011 efficiency improvement. DARPA is shooting in 2012 for nearly 1011 multiplies per Joule.

watt = joule/sec