There is a great conversation going on the nuke list about learning math for compositing.
Here are some links to what came up in the discussion:
Mathematics for Computer Graphics
The Art and Science of Digital Compositing
Mathematics for Visual Effects and Design
Fxphd also has a class called PHY101: Introduction to Physics the link is not coming up. I can recommend all of those books. I wrecked my car driving home from a talk and book signing Ron Brinkmann did at AFI. Compositing is all about math. Make no mistake.