Questions like this are really exercises in guessing what the asker was thinking. It is not possible to determine the answer mathematically. Instead, you have to come up with some definition of $a \mathbin\triangle b$ that agrees with the information given. There are many such definitions. Here is one:
Let us represent the pair $(a,b)$ by $2^a 3^b$. Using the points $(2^1 3^0,3)$, $(2^1 3^2,12)$, $(2^0 3^2,5)$, $(2^1 3^5,96)$ and $(2^2 3^3, 96)$, we find the unique interpolating polynomial of minimal degree:
$$ f(x) = \frac{150671}{272276484480}x^4 - \frac{19120391}{54455296896}x^3 + \frac{15349819}{378161784}x^2 - \frac{7010527}{56023968}x + \frac{72147351}{23343320}.$$
Now define $a \mathbin\triangle b = f(2^a3^b)$. We obtain $2 \mathbin\triangle 4 = -540445677/333476$.
Note that the polynomial $f$, called the Lagrange polynomial, can be found algorithmically, and so we have a general technique to automatically produce answers to questions like these.