Jul 26, 2023
Good question. This is where you need to intelligently group your tables into several tasks. My general rule is to run 5-10 tables per task ad throttle the number of row fetched using CommitRate.
Written by Manoj Kukreja
Author, Big Data Engineering, Data Science, Data Lakes, Cloud Computing and IT security specialist.