AWS Glue Pyspark Fetch Databases and Tables Metadata

example AWS Glue pyspark job script that fetches all the catalog databases and tables metadata.

see main.py

• first method uses spark sql
• second method uses python boto3 Glue client to interact with Glue API directly

Notes

ensure --enable-glue-datacatalog is enabled for glue job to allow spark sql to access metadata catalog

Glue Console Script View