Here are two ways to think about the problem . . .
The first way matches the calculations you had issues with.
Let $B$ be the event that the first ball drawn is ball $2$, and let $B'$ be the event that the first ball drawn is one of the other $3$ balls.
For the first draw, all $4$ balls are equally likely, hence $P(B)=\frac{1}{4}$, and $P(B')=\frac{3}{4}$.
If event $B$ occurs, $A_2$ can't occur, so $P(A_2|B)=0$.
If event $B'$ occurs, there are $3$ balls left, one of which is $A$, and all are equally likely, hence $P(A_2|B') = \frac{1}{3}$.
Then we get
\begin{align*}
P(A_2) &= P(B)P(A_2|B)+P(B')P(A_2|B')\\[4pt]
&=\left({\small{\frac{1}{4}}}\right)(0)+\left({\small{\frac{3}{4}}}\right)\left({\small{\frac{1}{3}}}\right)\\[4pt]
&={\small{\frac{1}{4}}}\\[4pt]
\end{align*}
In concept, in order for $A_2$ to occur, first $B'$ must occur, and next, given that $B'$ has occurred, ball $2$ must be the next ball of the $3$ remaining balls.
The other way is even simpler . . .
Ball $2$ must occur on some draw.
By symmetry, all draw orders of the $4$ balls are equally likely, so for $1\le k\le 4$, the probability that ball $2$ occurs on draw $k$ is $\frac{1}{4}$.
Hence $P(A_2)=\frac{1}{4}$.
