Tip: Use docker images for pyspark and postgres for testing

[NickCrews]

Ibis uses docker and docker-compose to host the various backends. I thought this was a great idea and wanted to point it out.

Doing this would

• pro: make it easier to have repeatable dev environments
• pro: easier to resolve python dependencies. Now you don't have to bring in all the transitive python deps for pyspark or psyopg2!
• con: require devs/CI to have docker
• con: maybe warrant some more scripting to admin this stuff

[NickCrews]

This lib looks like it makes this trivial: https://testcontainers-python.readthedocs.io/en/latest/README.html

[RobinL]

Thanks - sounds like a good idea to me too. Just need to find the time to do it!

It's interesting they don't seem to use docker for Spark , at least I can't see it in the docker compose. But we certainly could