Don't assume $y=360-x$. I'm not sure why you assumed that, but unfortunately, it's wrong.
Instead, for the time that Jimmy starts, let $x$ be the angle measure of the angle from $12$ to the minute hand, going clockwise, and let $y$ be the angle measure of the angle from $12$ to the hour hand, going clockwise. This is basically what you said, but more specific.
Since we know that Jimmy starts between $9:00$ and $10:00$, we can deduce that $y$ is at least $\frac{9}{12}*360^\circ=270^\circ$. Then, since the minute hand goes $360^\circ$ (one full rotation) in one hour and the hour hand goes $30^\circ$ (one twelfth of a rotation, from $9$ to $10$) in one hour, the hour hand must move $(\frac{x}{12})^\circ$ after $9:00$. Thus, we have the following equation:
$$y=270+\frac{x}{12}$$
Now, for the time that Jimmy ends, we know that the minute hand and the hour hand switch. Thus, we just need to switch our definitions: $y$ is now the angle measure of the angle from $12$ to the minute hand, going clockwise, and $x$ is now angle measure of the angle from $12$ to the hour hand, going clockwise.
Since we know that Jimmy ends between $10:00$ and $11:00$, we can deduce that $x$ is at least $\frac{10}{12}*360^\circ=300^\circ$. Then, since the minute hand goes $360^\circ$ (one full rotation) in one hour and the hour hand goes $30^\circ$ (one twelfth of a rotation, from $10$ to $11$) in one hour, the hour hand must move $(\frac{y}{12})^\circ$ after $10:00$. Thus, we have the following equation:
$$x=300+\frac{y}{12}$$
Substitute the first equation into the second equation:
$$x=300+\frac{270+\frac{x}{12}}{12}$$
Multiply everything by $12$:
$$12x=3600+270+\frac{x}{12}$$
Multiply everything by $12$ again:
$$144x=43200+3240+x=46440+x$$
Subtract both sides by $x$:
$$143x=46440$$
Divide both sides by $143$:
$$x=\frac{46440}{143}$$
Now, we have found $x$. Using this, we can find the number of minutes before $10:00$ in the starting time and the number of minutes past $10:00$ in the ending time.
Starting Time:
The minute hand moves $360^\circ$ in one hour and there are $60$ minutes in one hour. This means that if the minute hand has moved $x^\circ$, $\frac x 6$ minutes have passed. Thus, we just need to divide $x$ by $6$ to find the number of minutes past $9:00$. This turns out to be $7740/143$. However, we want to find the number of minutes before $10:00$. Since the difference between $9:00$ and $10:00$ is just one hour and there are $60$ minutes in an hour, we just need to subtract $7740/143$ from $60$, which yields $840/143$ minutes before $10:00$.
Ending Time:
First, subtract $300^\circ$ from $x$ so we can figure out how much the hour hand has moved since $10:00$. This yields $\frac{3540}{143}$. The hour hand moves $30^\circ$ in one hour and there are $60$ minutes in an hour. This means that if the hour hand has moves $x^\circ$ since the hour, $2x$ minutes have passed. Thus, we just need to multiply $\frac{3540}{143}$ by $2$, which yields $\frac{7080}{143}$ minutes past $10:00$.
Final Answer:
The difference between the starting time and ending time is the sum of the number of minutes the starting time was before $10:00$ plus the number of minutes the ending time was after $10:00$. This sum is $\frac{840}{143}+\frac{7080}{143}=\frac{7920}{143}$. However, we want our answer in hours, so we need to divide this by $60$, which yields $\frac{132}{143}$ which can be simplified to $\frac{12}{13}$. Thus, Jimmy spent $\frac{12}{13}$ hours painting.
