Assuming that the true average rating (i.e. without any rounding) was exactly 4, then the total rating for all 20 people is $4 \times 20 = 80$. So the total rating after adding the 21st rating in is $80 + 3 = 83$, so the average rating across 21 people is $83/21 \approx 3.95$. If you don't store either an accurate value for the average, or the total of all ratings, you're going to have trouble adjusting the rating as new ones come in since rounding will tend to push everything back to the previous value.
EDIT for example:
Let's suppose you already have 100 ratings, with a total of 374, so the average is 3.74. Then if the 101st user gives a rating of ...
1, the new average will be $375/101 = 3.71$
2, the new average will be $376/101 = 3.72$
3, the new average will be $377/101 = 3.73$
4, the new average will be $378/101 = 3.74$
5, the new average will be $379/101 = 3.75$
If you just store that value rounded to the nearest whole number, you'll say all of those are 4. Then as new ratings come in, you'll probably keep rounding the result to 4 for as long as you like - even if a million people all give a rating of 1, if you update after every rating you'll never see the value shift. Storing to 2 decimal places means that the same problem will happen once you reach a few thousand ratings.