HeadlinesBriefing favicon HeadlinesBriefing.com

AWS Glue Python Shell for Long-Running Batch Jobs

DEV Community •
×

AWS Community Builder Aki tested whether Glue Python Shell can reliably run for 48 hours, a common need for monthly aggregations or backfills. The experiment confirmed the service handles extended execution without interruption, using a simple script that logged hourly progress to CloudWatch. This addresses a key architectural question for teams managing infrequent, heavy batch workloads.

The 48-hour run consumed 48 DPU-hours, costing $21.12. While this is more expensive than running a comparable EC2 m6i.xlarge instance on Spot ($3.97), it eliminates operational overhead like OS patching, instance management, and handling capacity shortages or Spot interruptions. The trade-off is cost versus operational simplicity and reliability.

Glue Python Shell supports up to 168-hour executions, making it viable for weekly or longer batch jobs. However, it lacks OS-level customization and has fixed DPU-based resource scaling. For workloads requiring always-on execution, extreme runtime, or fine-tuned hardware, EC2 remains the better fit. The choice hinges on prioritizing predictable execution over raw cost.