While not english-speaking, in France you can do part of that.
Universities
On principle, Universities are open to everyone. You have the "right" to assist to any course any time you want. Specific sessions with TA are restricted however to people enlisted.
In practice, there are access restriction in some cases, like entering the buildings or premises. It may require to state your case and desire to be granted access, and pass through the control (and the people enforcing it might not be aware of this "fundamental" right). This is usually easier in early years of teaching, with very large attendance, that it is with classes of a dozen people all on the teacher attendance list.
Collège de France
This is an institution whose purpose is both research and teaching at very high level. You can assist lectures freely (sometimes you need to register yourself in advance). However the topics are not following the flow of a course, and can be on any subject, so you can't rely on it to build your knowledge logically and reliably. This is for cherry picking!
PHD lectures
Like all "exams" in France, you have the right to assist to it if you don't disrupt. It is, of course, very specific and very much like lecture at the Collège de France.
Caveats and workarounds
With all of the above, you can't take exams. Exams are restricted to people enlisted in the Universities.
However, in France Universities fees are usually fairly small (a few hundreds euros per year for most Universities), and there is no age restriction.
There might be restriction based on: (i) level (you need to have initial years passed - or equivalent level - before taking more advanced courses) or (ii) number of people wanting to take the class (more a local problem for some specific courses, and usually only for first and second year of University)
For English courses, I will let the Americans, British, Irish, Scottish, Welsh, Australians... speak for it!
