I'm trying to figure out a formula... for how much a picture (rectangle) would have to be scaled up during a rotation (at any rotation amount) so that you don't see the edge of the picture in the square of the bounding box.

If the bounding box is square it might be a simpler formula. But when the box is rectangle it seems like it gets harder to figure out. Although it might be a different formula if the width>height, than if the height>width. The graphic above shows 30 degrees which took 170%. I would think 45 degrees would be the worst case scenario then from there to 90 it would go down again.
To clarify, I am looking for a formula to calculate a scale factor to use given an angle a height and a width. We can assume the size and aspect of the bounding box to be equal to the size and aspect of the photo. So the input is Angle, Height, and Width.. the output is Scale Factor.
Or actually, maybe it can be figured out using the Angle and an Aspect Ratio.. where aspect ratio is width/height.
In practice this will be done in code.. so it doesn't have to be a single formula. It can be different formulas for different scenarios.. such as if the w>h or w=h or w

If w>h, the scaling factor is (w/h) sin(angle) + cos(angle).
If h>w, the scaling factor is (h/w) sin(angle) + cos(angle).
– badweasel Jul 08 '13 at 07:32