tqdm: A fast, extensible progress bar

taqadum: progress (from Arabic: تقدّم).

tqdm: I love you so much (from Spanish: te quiero demasiado).

Instantly make your loops show a smart progress meter - just wrap any iterable with tqdm(iterable), and you're done!

Casper O. da Costa-Luis. PyData London, May 2019.

• https://pypi.org/project/tqdm (pip page)
• https://github.com/tqdm (everything you could want*)
• https://tqdm.github.io (snazzy new layout)
• https://tqdm.github.io/PyData2019/slides.html (these slides)

*source code, documentation, examples, wiki, release history, issue tracker, divine inspiration, solution to global warming

Before

from time import sleep
for i in range(100):
    sleep(0.01)
print("Er, how long did that take?")

Er, how long did that take?

After

from tqdm import tqdm       # <-- yes
from time import sleep
for i in tqdm(range(100)):  # <-- magic
    sleep(0.01)

100%|██████████| 100/100 [00:01<00:00, 97.05it/s]

trange(N) can be also used as a convenient shortcut for tqdm(xrange(N)).

from tqdm import trange
from time import sleep
for i in trange(100):
    sleep(0.01)

100%|██████████| 100/100 [00:01<00:00, 96.91it/s]

from tqdm import trange
from time import sleep
for i in trange(100, desc="hello", unit="epoch"):
    sleep(0.01)

hello: 100%|██████████| 100/100 [00:01<00:00, 96.89epoch/s]

It can also be executed as a module with pipes:

! seq 999999 | python3 -m tqdm | wc -l

999999it [00:00, 2330224.81it/s]
999999

! seq 9999999 | python3 -m tqdm --bytes | wc -l

75.2MB [00:00, 236MB/s]
9999999

! seq 9999999 | python3 -m tqdm --bytes --total 80000000 | wc -l

 99%|██████████████████████████████████████▍| 75.2M/76.3M [00:00<00:00, 245MB/s]
9999999

Why?

• Easy

• Quick
  • <100ns per iteration overhead
  • Unit tested against performance regression
  • Skips unnecessary iteration displays (no console spamming)

• Intelligent ETA (remaining time)

• Cross-platform (Linux, Windows, Mac, FreeBSD, NetBSD, Solaris/SunOS)

• Console, terminal, GUI, IPython/Jupyter

• Dependency-free

Usage

(It's versatile)

Iterable-based

Wrap tqdm() around any iterable:

text = ""
for char in tqdm(["a", "b", "c", "d"]):
    sleep(0.25)
    text = text + char

100%|██████████| 4/4 [00:01<00:00,  3.97it/s]

Instantiation outside of the loop allows for manual control over tqdm():

pbar = tqdm(["a", "b", "c", "d"])
for char in pbar:
    sleep(0.25)
    pbar.set_description("Processing %s" % char)

Processing d: 100%|██████████| 4/4 [00:01<00:00,  3.94it/s]

Manual

Manual control on tqdm() updates by using a with statement:

with tqdm(total=100) as pbar:
    for i in range(10):
        sleep(0.1)
        pbar.update(10)

100%|██████████| 100/100 [00:01<00:00, 98.28it/s]

• If given a total (or an iterable with len()), predictive stats are displayed.
• with is also optional (but don't forget to del or close() at the end)

pbar = tqdm(total=100)
for i in range(10):
    sleep(0.1)
    pbar.update(10)
pbar.close()

100%|██████████| 100/100 [00:01<00:00, 98.11it/s]

Module

Simply inserting tqdm (or python3 -m tqdm) between pipes will pass through all stdin to stdout while printing progress to stderr.

Note that the usual arguments for tqdm can also be specified.

# count lines of code in all *.py
find . -name '*.py' -type f -exec cat \{} \; \
   | wc -l
# ... wait ages ...
4318982

# the tqdm way
find . -name '*.py' -type f -exec cat \{} \; | tqdm --unit loc --unit-scale \
   | wc -l
4.32Mloc [00:06, 705kloc/s]
4318982

# boring backup
tar -zcf - ~/dloads > backup.tgz
# ... are we there yet? ...

# with tqdm
tar -zcf - ~/dloads | tqdm --bytes --total `du -sb ~/dloads | cut -f1` \
    > backup.tgz
 32%|██████████▍                      | 8.89G/27.9G [00:42<01:31, 223MB/s]

7z a backup.7z docs/ | grep Compressing | \
    tqdm --total $(find docs/ -type f | wc -l) --unit files >> backup.log
100%|███████████████████████████████▉| 8014/8014 [01:37<00:00, 82.29files/s]

Documentation

from tqdm import tqdm
help(tqdm)

tqdm?

! python3 -m tqdm --help

Usage:
  tqdm [--help | options]

Options:
  -h, --help     Print this help and exit.
  -v, --version  Print version and exit.
  --desc=<desc>  : str, optional
            Prefix for the progressbar.
  --total=<total>  : int or float, optional
            The number of expected iterations. If unspecified,
            len(iterable) is used if possible. If float("inf") or as a last
            resort, only basic progress statistics are displayed
            (no ETA, no progressbar).
            If `gui` is True and this parameter needs subsequent updating,
            specify an initial arbitrary large positive number,
            e.g. 9e9.
  --leave  : bool, optional
            If [default: True], keeps all traces of the progressbar
            upon termination of iteration.
            If `None`, will leave only if `position` is `0`.
  --ncols=<ncols>  : int, optional
            The width of the entire output message. If specified,
            dynamically resizes the progressbar to stay within this bound.
            If unspecified, attempts to use environment width. The
            fallback is a meter width of 10 and no limit for the counter and
            statistics. If 0, will not print any meter (only stats).
  --mininterval=<mininterval>  : float, optional
            Minimum progress display update interval [default: 0.1] seconds.
  --maxinterval=<maxinterval>  : float, optional
            Maximum progress display update interval [default: 10] seconds.
            Automatically adjusts `miniters` to correspond to `mininterval`
            after long display update lag. Only works if `dynamic_miniters`
            or monitor thread is enabled.
  --miniters=<miniters>  : int or float, optional
            Minimum progress display update interval, in iterations.
            If 0 and `dynamic_miniters`, will automatically adjust to equal
            `mininterval` (more CPU efficient, good for tight loops).
            If > 0, will skip display of specified number of iterations.
            Tweak this and `mininterval` to get very efficient loops.
            If your progress is erratic with both fast and slow iterations
            (network, skipping items, etc) you should set miniters=1.
  --ascii=<ascii>  : bool or str, optional
            If unspecified or False, use unicode (smooth blocks) to fill
            the meter. The fallback is to use ASCII characters " 123456789#".
  --disable  : bool, optional
            Whether to disable the entire progressbar wrapper
            [default: False]. If set to None, disable on non-TTY.
  --unit=<unit>  : str, optional
            String that will be used to define the unit of each iteration
            [default: it].
  --unit-scale=<unit_scale>  : bool or int or float, optional
            If 1 or True, the number of iterations will be reduced/scaled
            automatically and a metric prefix following the
            International System of Units standard will be added
            (kilo, mega, etc.) [default: False]. If any other non-zero
            number, will scale `total` and `n`.
  --dynamic-ncols  : bool, optional
            If set, constantly alters `ncols` and `nrows` to the
            environment (allowing for window resizes) [default: False].
  --smoothing=<smoothing>  : float, optional
            Exponential moving average smoothing factor for speed estimates
            (ignored in GUI mode). Ranges from 0 (average speed) to 1
            (current/instantaneous speed) [default: 0.3].
  --bar-format=<bar_format>  : str, optional
            Specify a custom bar string formatting. May impact performance.
            [default: '{l_bar}{bar}{r_bar}'], where
            l_bar='{desc}: {percentage:3.0f}%|' and
            r_bar='| {n_fmt}/{total_fmt} [{elapsed}<{remaining}, '
              '{rate_fmt}{postfix}]'
            Possible vars: l_bar, bar, r_bar, n, n_fmt, total, total_fmt,
              percentage, elapsed, elapsed_s, ncols, nrows, desc, unit,
              rate, rate_fmt, rate_noinv, rate_noinv_fmt,
              rate_inv, rate_inv_fmt, postfix, unit_divisor,
              remaining, remaining_s, eta.
            Note that a trailing ": " is automatically removed after {desc}
            if the latter is empty.
  --initial=<initial>  : int or float, optional
            The initial counter value. Useful when restarting a progress
            bar [default: 0]. If using float, consider specifying `{n:.3f}`
            or similar in `bar_format`, or specifying `unit_scale`.
  --position=<position>  : int, optional
            Specify the line offset to print this bar (starting from 0)
            Automatic if unspecified.
            Useful to manage multiple bars at once (eg, from threads).
  --postfix=<postfix>  : dict or *, optional
            Specify additional stats to display at the end of the bar.
            Calls `set_postfix(**postfix)` if possible (dict).
  --unit-divisor=<unit_divisor>  : float, optional
            [default: 1000], ignored unless `unit_scale` is True.
  --write-bytes  : bool, optional
            If (default: None) and `file` is unspecified,
            bytes will be written in Python 2. If `True` will also write
            bytes. In all other cases will default to unicode.
  --lock-args=<lock_args>  : tuple, optional
            Passed to `refresh` for intermediate output
            (initialisation, iterating, and updating).
  --nrows=<nrows>  : int, optional
            The screen height. If specified, hides nested bars outside this
            bound. If unspecified, attempts to use environment height.
            The fallback is 20.
  --colour=<colour>  : str, optional
            Bar colour (e.g. 'green', '#00ff00').
  --delay=<delay>  : float, optional
            Don't display until [default: 0] seconds have elapsed.
  --delim=<delim>  : chr, optional
            Delimiting character [default: '\n']. Use '\0' for null.
            N.B.: on Windows systems, Python converts '\n' to '\r\n'.
  --buf-size=<buf_size>  : int, optional
            String buffer size in bytes [default: 256]
            used when `delim` is specified.
  --bytes  : bool, optional
            If true, will count bytes, ignore `delim`, and default
            `unit_scale` to True, `unit_divisor` to 1024, and `unit` to 'B'.
  --tee  : bool, optional
            If true, passes `stdin` to both `stderr` and `stdout`.
  --update  : bool, optional
            If true, will treat input as newly elapsed iterations,
            i.e. numbers to pass to `update()`. Note that this is slow
            (~2e5 it/s) since every input must be decoded as a number.
  --update-to  : bool, optional
            If true, will treat input as total elapsed iterations,
            i.e. numbers to assign to `self.n`. Note that this is slow
            (~2e5 it/s) since every input must be decoded as a number.
  --null  : bool, optional
            If true, will discard input (no stdout).
  --manpath=<manpath>  : str, optional
            Directory in which to install tqdm man pages.
  --comppath=<comppath>  : str, optional
            Directory in which to place tqdm completion.
  --log=<log>  : str, optional
            CRITICAL|FATAL|ERROR|WARN(ING)|[default: 'INFO']|DEBUG|NOTSET.

Examples and Advance Usage

• See the examples folder;
• import the module and run help();
• consult the wiki
  • this has an excellent article on how to make a great progressbar, or
• run this DEMO.ipynb file!

Description and additional stats

Custom information can be displayed and updated dynamically on tqdm bars with the desc and postfix arguments:

from tqdm import trange
from random import random, randint
from time import sleep

with trange(10) as t:
    for i in t:
        t.set_description('GEN %i' % i)
        # formatted automatically based on argument's datatype
        t.set_postfix(loss=random(), gen=randint(1,999), str='h', lst=[1, 2])
        sleep(0.1)

GEN 9: 100%|██████████| 10/10 [00:01<00:00,  9.45it/s, gen=396, loss=0.439, lst=[1, 2], str=h]

with tqdm(total=10, bar_format="{postfix[0]} {postfix[1][value]:>8.2g}",
          postfix=["Batch", dict(value=0)]) as t:
    for i in range(10):
        sleep(0.1)
        t.postfix[1]["value"] = i / 2
        t.update()

Batch        4

Nested progress bars

tqdm supports nested progress bars. Here's an example:

from tqdm import trange
from time import sleep

for i in trange(4, desc='1st loop'):
    for j in trange(5, desc='2nd loop'):
        for k in trange(50, desc='3nd loop', leave=False):
            sleep(0.01)

• On Windows colorama will be used if available
• For manual control over positioning (e.g. for multi-threaded use), specify position=n

Hooks and callbacks

tqdm can easily support callbacks/hooks and manual updates.

import urllib, os

urllib = getattr(urllib, 'request', urllib)
eg_link = "https://caspersci.uk.to/matryoshka.zip"
urllib.urlretrieve(eg_link, filename=os.devnull, data=None);

class TqdmUpTo(tqdm):
    def update_to(self, blocks_so_far=1, block_size=1, total=None):
        if total is not None:
            self.total = total
        # will also set self.n = blocks_so_far * block_size
        self.update(blocks_so_far * block_size - self.n)

with TqdmUpTo(unit='B', unit_scale=True, miniters=1,
              desc=eg_link.split('/')[-1]) as t:  # all optional kwargs
    urllib.urlretrieve(eg_link, filename=os.devnull,
                       reporthook=t.update_to, data=None)
    t.total = t.n

matryoshka.zip: 100%|██████████| 262k/262k [00:00<00:00, 556kB/s]  

It is recommend to use miniters=1 whenever there is potentially large differences in iteration speed (e.g. downloading a file over a patchy connection).

Machine Learning

from tensorflow import keras
model = keras.Sequential()

# ... etc ...

model.fit(x, y, epochs=10);

Epoch 1/10
1000/1000 [==============================] - 1s 1ms/step - loss: 123450.4777 - mean_absolute_error: 351.3478
Epoch 2/10
1000/1000 [==============================] - 0s 44us/step - loss: 122634.9961 - mean_absolute_error: 350.1859
Epoch 3/10
1000/1000 [==============================] - 0s 52us/step - loss: 121823.3951 - mean_absolute_error: 349.0256
Epoch 4/10
1000/1000 [==============================] - 0s 47us/step - loss: 121018.2029 - mean_absolute_error: 347.8706
Epoch 5/10
1000/1000 [==============================] - 0s 52us/step - loss: 120219.2435 - mean_absolute_error: 346.7207
Epoch 6/10
1000/1000 [==============================] - 0s 49us/step - loss: 119424.1497 - mean_absolute_error: 345.5727
Epoch 7/10
1000/1000 [==============================] - 0s 45us/step - loss: 118632.8348 - mean_absolute_error: 344.4263
Epoch 8/10
1000/1000 [==============================] - 0s 45us/step - loss: 117845.8773 - mean_absolute_error: 343.2822
Epoch 9/10
1000/1000 [==============================] - 0s 56us/step - loss: 117062.4701 - mean_absolute_error: 342.1397
Epoch 10/10
1000/1000 [==============================] - 0s 45us/step - loss: 116284.3545 - mean_absolute_error: 341.0009

from tqdm.keras import TqdmCallback
model.fit(x, y, epochs=10, verbose=0, callbacks=[TqdmCallback()]);

100%|██████████| 10/10 [00:00<00:00, 24.24epoch/s, loss=1.09e+5, mean_absolute_error=330]

Pandas Integration

Due to popular demand (ergh).

import pandas as pd
import numpy as np
from tqdm import tqdm

df = pd.DataFrame(np.random.rand(5, 10))

# Register `pandas.progress_apply`, `pandas.Series.map_apply`, etc with `tqdm`
# (can use `tqdm_gui`, `tqdm_notebook`, optional kwargs, etc.)
tqdm.pandas(desc="my bar!")

# `map` => `progress_map`
# `apply` => `progress_apply`
df.progress_apply(lambda x: x**2)
# can also groupby:
# df.groupby(0).progress_apply(lambda x: x**2)

my bar!: 100%|██████████| 10/10 [00:00<00:00, 1626.01it/s]

	0	1	2	3	4	5	6	7	8	9
0	0.343456	0.333026	0.013214	0.109925	0.027551	0.015859	0.084797	0.055252	0.276558	0.538887
1	0.491100	0.007598	0.008333	0.127863	0.325472	0.157241	0.555851	0.028682	0.145064	0.956230
2	0.002674	0.481197	0.693777	0.396851	0.129166	0.317659	0.007226	0.009895	0.678273	0.434440
3	0.002297	0.267686	0.163257	0.919152	0.340434	0.820713	0.495618	0.879623	0.688926	0.474775
4	0.589205	0.077341	0.020816	0.439388	0.063140	0.003705	0.061417	0.029562	0.473743	0.424798

IPython/Jupyter Integration

IPython/Jupyter is supported via the tqdm_notebook submodule:

from tqdm.notebook import tnrange, tqdm_notebook
from time import sleep

for i in tnrange(3, desc='1st loop'):
    for j in tqdm_notebook(range(100), desc='2nd loop'):
        sleep(0.01)

• native Jupyter widget
• fully working nested bars
• colour hints (blue: normal, green: completed, red: error/interrupt, light blue: no ETA)

• Can let tqdm automatically choose between CLI or notebook versions

from tqdm.autonotebook import tqdm  # may raise warning about Jupyter
from tqdm.auto import tqdm  # who needs warnings
tqdm.pandas()

• Cannot distinguish between jupyter notebook and jupyter console