I have a problem where I have a set of points in one domain (a camera focal image plane) and a polynomial transformation that puts those points into another domain (real-world coordinates) and another set of polynomials that does the reverse. I have different polynomials for different cameras. I would like to work out the transformation from one camera to the other. If $x$ and $y$ are the coordinates in the original domain, and $A$ and $B$ are the real-world coordinates, then the math looks something like this:
$A=P_A(x,y)\qquad B=P_B(x,y)$
$x'=P_x(A,B)\qquad y'=P_y(A,B)$
with all of the polynomials being 4th order, and I have been given the 15 coefficients for each step. I feel like I should be able to combine the polynomials into a single step transformation:
$x'=P_x(P_A(x,y), P_B(x,y)) \qquad y'=P_y(P_A(x,y), P_B(x,y))$
but working out how the coefficients should combine by brute force becomes quickly intractable with paper and pencil (or whiteboard and pen). I also feel like this is the sort of thing that could be done with some linear algebra (which would be nice since I intend to ultimately code this up in MATLAB). However, it has been 20 years since my last linear algebra course and even longer since calculus and so I don't even know what language to use for searching on Google or SE.
My questions:
What is this sort of problem called? I tried "nested polynomials", "composite polynomials", "polynomial mapping", "polynomial matrix" among others, and all pointed to something not at all like this problem (though "polynomial matrix representation" gave hints that there might be something there).
Where should I start in tackling this problem? Is there a good algorithm for simplifying nested polynomials like this, or am I better off sticking with the two-step transformation?