Steven, maybe I can offer an alternate suggestion. It will mean an
extra trip through VP, but I think it will be less headache than
trial-and-error.
Since your floor and background will not usually be the same color,
run a VP sequence that alphas the background color against a color
that matches your floor. This will effectively *replace* the
background with the floor color.
You might have to sample a pixel from the floor in a paint program to
get the RGB value for best match. Watch those gammas going in and
out.
Next trip through VP. Now that background and floor match exactly,
with exception of the shadowed areas, you can alpha against the new
uniform background with the background of your room or whatever.
I have not verified this idea: "It just came to me."