I’m writing a series of irregular short posts on some tips and tricks on using Geospatial Python. They may not be best practice, but I hope they are of help. I’ll try and give some context to when I’ve used the ‘tip’.
Use tqdm as a diagnostic tool to understand your iterables
You are going to need to install tqdm – follow guidance here
By all means use tqdm for your progress bars in your code – that is what it is meant for. This tip though is for those that want to understand why their code might be slow and secondly to visualise a loop within a loop. It might help you re think the way you’ve set your code up.
Ultimately all you need to do is wrap you iterable (in this example a for loop) with tqdm.
from tqdm import tqdm
import time
# just wrap any iterable with tqdm(iterable)
for i in tqdm (range(100)):
time.sleep(0.01)
This code is a for loop of 100 steps and in each step it sleeps for 0.01 second. Its fast!
100%|██████████| 100/100 [00:01<00:00, 64.00it/s]
What happens when I do the same for a for loop within a for loop?
for i in tqdm (range(100), desc="Loading..."):
# perhaps don't do this in real life!
for k in tqdm(range(100)):
time.sleep(0.01)
time.sleep(0.01)
Crickey.. that is slow
Loading…: 3%|▎ | 3/100 [00:04<02:32, 1.57s/it]
0%| | 0/100 [00:00<?, ?it/s]
7%|▋ | 7/100 [00:00<00:01, 64.91it/s]
14%|█▍ | 14/100 [00:00<00:01, 64.80it/s]
21%|██ | 21/100 [00:00<00:01, 64.41it/s]
28%|██▊ | 28/100 [00:00<00:01, 64.19it/s]
35%|███▌ | 35/100 [00:00<00:01, 63.75it/s]
42%|████▏ | 42/100 [00:00<00:00, 63.86it/s]
49%|████▉ | 49/100 [00:00<00:00, 64.36it/s]
56%|█████▌ | 56/100 [00:00<00:00, 64.43it/s]
63%|██████▎ | 63/100 [00:00<00:00, 64.44it/s]
70%|███████ | 70/100 [00:01<00:00, 64.39it/s]
77%|███████▋ | 77/100 [00:01<00:00, 64.48it/s]
84%|████████▍ | 84/100 [00:01<00:00, 64.44it/s]
91%|█████████ | 91/100 [00:01<00:00, 64.21it/s]
100%|██████████| 100/100 [00:01<00:00, 64.22it/s]
or if I remove the tqdm part in the second loop
for i in tqdm (range(100), desc="Loading..."):
# perhaps don't do this in real life!
for k in range(100):
time.sleep(0.01)
time.sleep(0.01)
Loading…: 9%|▉ | 9/100 [00:14<02:22, 1.57s/it]
So consider using this in your for loops – almost certainly you will understand why your code is running slower, and hey if you’ve never used tqdm before perhaps I’ve just given you a progress bar you never knew you could have.
I’ve added comments where needed to the code here. I hope this has been of use.
I’ll be adding more tips and tricks
Image credit https://unsplash.com/photos/MJAoiige14E
I am a freelancer able to help you with your projects. I offer consultancy, training and writing. I’d be delighted to hear from you.
Feel free to connect or follow me; I am always keen to talk about Earth Observation.