The Earth is a huge and interconnected ecosystem, would something as small as a habitat provide a self sustaining ecosystem? The best known experiment is Biosphere2, which attempted to create an entirely self sustaining closed environment.
This is generally regarded as a failure. Oxygen and carbon dioxide levels fluctuated, they did not grow enough food (although almost enough) and all the non-human vertebrates and pollinating insects died, while others (cockroaches and ants) were all too successful.
In fact, I would regard it as a partial success, given the limitations on the experiment:
The area was very small compared to the proposed space habitats.
The human volunteers were short of scientific and technical knowledge, and bickered amongst themselves (which is what happens when you confine a few people to a small space for a long time…)
It’s the first serious experiment of its kind, and it is rare to get things right the first time
Some external input would be acceptable in practice e.g. some scrubbing of air to get rid of excess CO2, and not leave it all to plants. After all, that keeps the ISS habitable.
On Earth we go to considerable lengths already to “artificially” recycle solid and aqueous wastes, particularly in urban areas in the developed world, and the environmental pressures to do this are increasing.
More research is needed, but I would judge that sustainable ecosystems, on the scale of the habitats discussed here, are quite achievable. It does however imply that the bigger the habitat, the easier it would be.
Getting Around in Space