Let's introduce some notation to describe your question. Suppose we perform $N$ measurements of a time-dependent quantity $x(t)$. Denote the observed value of $x$ at time $t_i$ as $x_i=x(t_i)$ for $i \in \{1,2,...,N\}$. Then define $r_{i+1}=\frac{x_{i+1}}{x_i}$ as the fractional increase of $x$ between times $t_i$ and $t_{i+1}$ for $i \in \{1,2,...,N-1\}$. Moreover, it is convenient to assign $r_1=1$. It follows by iterative application of this definition that $x_i=x_1 \prod_{j=1}^{i} r_j$, for all $i \in \{1,2,...,N\}$. The (unweighted) average of $x$ across the $N$ measurements is $$\bar{x}=\frac{1}{N} \sum_{i=1}^{N} x_i=\frac{x_1}{N} \sum_{i=1}^{N} \prod_{j=1}^{i} r_j\tag{1}.$$ The metric you are considering for the average fractional change is then $$\bar{r}=\frac{\bar{x}}{x_1}=\frac{1}{N} \sum_{i=1}^{N} \prod_{j=1}^{i} r_j \tag{2}.$$ For computational purposes, it is useful to note that $\sum_{i=1}^{N} \prod_{j=1}^{i} r_j =r_1\left(1+r_2\left(1+...r_{N-1}\left(1+r_N\right)\right)\right)$. In terms of your example, we would have that $N=4$, $x_1 = 1$, and $\vec{r}=(1,1.02,1.03,1.02)$. Then we would calculate $\bar{r}=1+r_2(1+r_3(1+r_4))=1.035553$ and $\bar{x}=\bar{r}x_1=1.035553$. It should hopefully be clear from $(2)$ why $\bar{r}$ is not simply the arithmetic, geometric, or harmonic mean of the components of $\vec{r}$.
In fact, I would argue that $\bar{r}$ as defined in $(2)$ should not be understood as an average of the components of $\vec{r}$. The prototypical property of an averaging function $\bar{s}:S^N \to S$ for $N$ elements of a set $S$ is that for all $s \in S$, we have that $\bar{s}(s,s,...,s)=s$. In the case that $S=(0,\infty)$ and $r_i=r \in S$ for all $i \in \{1,2,...N\}$, $$\bar{r}=\frac{1}{N} \sum_{i=1}^{N} \prod_{j=1}^{i} r = \frac{1}{N} \sum_{i=1}^{N} r^i = \frac{r}{N} \lim\limits_{r' \to r} \frac{1-r'^N}{1-r'},$$
where we have used the standard formula for a finite geometric series (the limit being needed for when $r=1$ or $N=1$). For all $r \neq 1$ and $N\geq 2$, it is then apparent that $\bar{r} = \frac{r}{N}\frac{1-r^N}{1-r} \neq r$, and even more alarming, $\lim\limits_{r \to \infty} \frac{\bar{r}}{r} = \frac{1}{N} \lim\limits_{r \to \infty} \frac{1-r^N}{1-r}=\infty$.
I would instead think of $\bar{r}$ as the fractional change between the first measurement of $x$ and the sample mean $\bar{x}$ as it is defined in $(2)$. I am afraid we do not have reason to extend its meaning beyond this.
