In $\mathbb R^n$, a rectangular domain simply means a domain of the form: $$ \tag{$\ast$}(a_1,b_1)\times \cdots \times(a_n,b_n)$$ with $a_i<b_i$ for each $i=1,\dots,n$. For $n=2$ this is literally a rectangle and for $n=3$ it is a rectangular prism.
It is common to also call any domain in the form of ($\ast$) except with any combination of open intervals, closed interval, or half-open intervals rectangular i.e. $[0,1] \times [0,1)$ is considered rectangular.
Now for separation of variables, I don't entirely agree with how your lecture notes has phrased it - I think it is better to say that separation of variables is possible for domains that are rectangular possibly after a change of coordinates. For example, your lecture notes seems to suggest that a disk is rectangular! This is silly - what it means is that after converting to polar coordinates, a disk is transformed in to a rectangular region.