FFCV Fields

Fields for the fastxtend Loader

fastxtend provides multiple FFCV fields, including existing FFCV fields as a reference, and a modified RGBImageField with Pillow support for dataset creation.

FFCV Field Reference

These fields are from FFCV. You can find the original documentation at the FFCV API Reference.

Each field has at least one decoder to use in a FFCV Pipline.

BytesField

 BytesField ()

A subclass of :class:~ffcv.fields.Field supporting variable-length byte arrays.

Intended for use with data such as text or raw data which may not have a fixed size. Data is written sequentially while saving pointers and read by pointer lookup.

The writer expects to be passed a 1D uint8 numpy array of variable length for each sample.

Pair with the BytesDecoder.

IntField

 IntField ()

A subclass of :class:~ffcv.fields.Field supporting (scalar) integer values.

Pair with the IntDecoder.

FloatField

 FloatField ()

A subclass of :class:~ffcv.fields.Field supporting (scalar) floating-point (float64) values.

Pair with the FloatDecoder.

NDArrayField

 NDArrayField (dtype:numpy.dtype, shape:Tuple[int,...])

A subclass of :class:~ffcv.fields.Field supporting multi-dimensional fixed size matrices of any numpy type.

Pair with the NDArrayDecoder.

JSONField

 JSONField ()

A subclass of :class:~ffcv.fields.BytesField that encodes JSON data.

The writer expects to be passed a dict that is compatible with the JSON specification.

.. warning :: Because FFCV is based on tensors/ndarrays the reader and therefore the loader can’t give return JSON to the user. This is why we provide :class:~ffcv.fields.JSONField.unpack which does the conversion. It’s up to the user to call it in the main body of the loop

Pair with the BytesDecoder.

Warning

Because FFCV is based on tensors/ndarrays the reader and therefore the loader can’t give return JSON to the user. This is why we provide JSONField.unpack which does the conversion. It’s up to the user to call it in the main body of the loop.

JSONField.unpack

 JSONField.unpack (batch)

Convert back the output of a :class:~ffcv.fields.JSONField field produced by :class:~ffcv.Loader into an actual JSON.

It works both on an entire batch and will return an array of python dicts or a single sample and will simply return a dict.

TorchTensorField

 TorchTensorField (dtype:torch.dtype, shape:Tuple[int,...])

A subclass of :class:~ffcv.fields.Field supporting multi-dimensional fixed size matrices of any torch type.

Pair with the NDArrayDecoder.

Modified Fields

FFCV’s RGBImageField with Pillow support for dataset creation.

source

RGBImageField

 RGBImageField (write_mode='raw', max_resolution:int=None,
                min_resolution:int=None, smart_threshold:int=None,
                jpeg_quality:int=90, compress_probability:float=0.5,
                interpolation=3, resample=<Resampling.LANCZOS: 1>,
                pillow_resize:bool=False)

A subclass of :class:~ffcv.fields.Field supporting RGB image data.

	Type	Default	Details
write_mode	str	raw	How to write the image data to the dataset file. Should be either ‘raw’ (`uint8` pixel values), ‘jpg’ (compress to JPEG format), ‘smart’ (decide between saving pixel values and JPEG compressing based on image size), and ‘proportion’ (JPEG compress a random subset of the data with size specified by the `compress_probability` argument). By default: ‘raw’.
max_resolution	int	None	If specified, resize images to have maximum side length equal to this value if maximum side length is larger. By default: None
min_resolution	int	None	If specified, resize images to have minimum side length equal to this value if minimum side length is larger. By default: None
smart_threshold	int	None	When `write_mode='smart`, will compress an image if RAW byte size is larger than `smart_threshold`.
jpeg_quality	int	90	The quality parameter for JPEG encoding (ignored for `write_mode='raw'`). By default 90
compress_probability	float	0.5	Ignored unless `write_mode='proportion'`; in the latter case it is the probability with which image is JPEG-compressed. By default 0.5.
interpolation	int	3	The OpenCV interpolation flag for resizing images with OpenCV. By default INTER_AREA.
resample	Resampling	Resampling.LANCZOS	The Pillow resampling filter for resizing images with Pillow. By default LANCZOS.
pillow_resize	bool	False	Use Pillow to resize images instead of OpenCV. By default False (OpenCV).
Returns	None

Pair with one of the following three decoders: