AWS Glue Pyspark Fetch Databases and Tables Metadata
Written in 2021. Tools, APIs and prices may have changed since.
Lab note What I wrote down while learning this by building with it. Short by design; the code is in pfeilbr/aws-glue-pyspark-fetch-databases-and-tables-metadata-playground.
example AWS Glue pyspark job script that fetches all the catalog databases and tables metadata.
see main.py
- first method uses spark sql
- second method uses python boto3 Glue client to interact with Glue API directly
Notes
ensure --enable-glue-datacatalog is enabled for glue job to allow spark sql to access metadata catalog
Glue Console Script View
