apache_beam.utils.byte_limited_queue module

A thread-safe queue that limits capacity by total byte size.

class apache_beam.utils.byte_limited_queue.ByteLimitedQueue

Bases: object

A fair queue that limits by both element count and total byte size.

A single element is allowed to exceed the maxbytes to avoid deadlock.

Initializes a ByteLimitedQueue.

Parameters:

  • maxsize – The maximum number of items allowed in the queue. If 0 or negative, there is no limit on the number of elements.

  • maxbytes – The maximum accumulated bytes allowed in the queue. If 0 or negative, there is no limit on the total bytes of the elements.

blocked_byte_size()

Return the total byte size of elements in the queue that are blocked.

byte_size()

Return the total byte size of elements in the queue.

get(*, block=True, timeout=None)

Remove and return an item from the queue.

If the queue is empty, block until an item is available, unless block is false or a timeout occurs.

Parameters:

  • block – If True, block until an item is available. If False, raise queue.Empty immediately if the queue is empty.

  • timeout – If block is True, wait for at most timeout seconds. If None, block indefinitely.

Returns:

The item removed from the queue.

Raises:

  • ValueError – If timeout is negative.

  • queue.Empty – If the queue is empty and block is False or the timeout occurs.

get_nowait()

Remove and return an item from the queue without blocking.

max_bytes
max_elements
put(item, item_bytes, *, block=True, timeout=None)

Put an item into the queue.

If the queue is full, block until a free slot is available, unless block is false or a timeout occurs.

Parameters:

  • item – The item to put into the queue.

  • item_bytes – The size of the item.

  • block – If True, block until space is available. If False, raise queue.Full immediately if the queue is full.

  • timeout – If block is True, wait for at most timeout seconds. If None, block indefinitely.

Raises:

  • ValueError – If timeout or item_bytes is negative.

  • queue.Full – If the queue is full and block is False or the timeout occurs.

qsize()

Return the total number of elements in the queue.