# DataLoaders


<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! -->

``` python
bs = 4
letters = list(string.ascii_lowercase)
```

## DataLoader helpers

fastai includes a replacement for Pytorch’s *DataLoader* which is
largely API-compatible, and adds a lot of useful functionality and
flexibility. Before we look at the class, there are a couple of helpers
we’ll need to define.

------------------------------------------------------------------------

<a
href="https://github.com/fastai/fastai/blob/main/fastai/data/load.py#L51"
target="_blank" style="float:right; font-size:smaller">source</a>

### fa_collate

``` python
def fa_collate(
    t
):
```

*A replacement for PyTorch `default_collate` which maintains types and
handles `Sequence`s*

``` python
#e.g. x is int, y is tuple
t = [(1,(2,3)),(1,(2,3))]
test_eq(fa_collate(t), default_collate(t))
test_eq(L(fa_collate(t)).map(type), [Tensor,tuple])

t = [(1,(2,(3,4))),(1,(2,(3,4)))]
test_eq(fa_collate(t), default_collate(t))
test_eq(L(fa_collate(t)).map(type), [Tensor,tuple])
test_eq(L(fa_collate(t)[1]).map(type), [Tensor,tuple])
```

------------------------------------------------------------------------

<a
href="https://github.com/fastai/fastai/blob/main/fastai/data/load.py#L59"
target="_blank" style="float:right; font-size:smaller">source</a>

### fa_convert

``` python
def fa_convert(
    t
):
```

*A replacement for PyTorch `default_convert` which maintains types and
handles `Sequence`s*

``` python
t0 = array([1,2])
t = [t0,(t0,t0)]

test_eq(fa_convert(t), default_convert(t))
test_eq(L(fa_convert(t)).map(type), [Tensor,tuple])
```

------------------------------------------------------------------------

<a
href="https://github.com/fastai/fastai/blob/main/fastai/data/load.py#L66"
target="_blank" style="float:right; font-size:smaller">source</a>

### SkipItemException

``` python
def SkipItemException(
    *args, **kwargs
):
```

*Raised to notify
[`DataLoader`](https://docs.fast.ai/data.load.html#dataloader) to skip
an item*

------------------------------------------------------------------------

<a
href="https://github.com/fastai/fastai/blob/main/fastai/data/load.py#L71"
target="_blank" style="float:right; font-size:smaller">source</a>

### collate_error

``` python
def collate_error(
    e:Exception, batch
):
```

*Raises error when the batch could not collate, stating what items in
the batch are different sizes and their types*

------------------------------------------------------------------------

<a
href="https://github.com/fastai/fastai/blob/main/fastai/data/load.py#L88"
target="_blank" style="float:right; font-size:smaller">source</a>

### DataLoader

``` python
def DataLoader(
    dataset:NoneType=None, bs:NoneType=None, num_workers:int=0, pin_memory:bool=False, timeout:int=0,
    batch_size:NoneType=None, shuffle:bool=False, drop_last:bool=False, indexed:NoneType=None, n:NoneType=None,
    device:NoneType=None, persistent_workers:bool=False, pin_memory_device:str='', wif:NoneType=None,
    before_iter:NoneType=None, after_item:NoneType=None, before_batch:NoneType=None, after_batch:NoneType=None,
    after_iter:NoneType=None, create_batches:NoneType=None, create_item:NoneType=None, create_batch:NoneType=None,
    retain:NoneType=None, get_idxs:NoneType=None, sample:NoneType=None, shuffle_fn:NoneType=None,
    do_batch:NoneType=None
):
```

*Inherit from this to have all attr accesses in `self._xtra` passed down
to `self.default`*

Arguments to
[`DataLoader`](https://docs.fast.ai/data.load.html#dataloader):

- `dataset`: dataset from which to load the data. Can be either
  map-style or iterable-style dataset.
- `bs` (int): how many samples per batch to load (if `batch_size` is
  provided then `batch_size` will override `bs`). If `bs=None`, then it
  is assumed that `dataset.__getitem__` returns a batch.
- `num_workers` (int): how many subprocesses to use for data loading.
  `0` means that the data will be loaded in the main process.
- `pin_memory` (bool): If `True`, the data loader will copy Tensors into
  CUDA pinned memory before returning them.
- `timeout` (float\>0): the timeout value in seconds for collecting a
  batch from workers.
- `batch_size` (int): It is only provided for PyTorch compatibility. Use
  `bs`.
- `shuffle` (bool): If `True`, then data is shuffled every time
  dataloader is fully read/iterated.
- `drop_last` (bool): If `True`, then the last incomplete batch is
  dropped.
- `indexed` (bool): The
  [`DataLoader`](https://docs.fast.ai/data.load.html#dataloader) will
  make a guess as to whether the dataset can be indexed (or is
  iterable), but you can override it with this parameter. `True` by
  default.
- `n` (int): Defaults to `len(dataset)`. If you are using iterable-style
  dataset, you can specify the size with `n`.
- `device` (torch.device): Defaults to
  [`default_device()`](https://docs.fast.ai/torch_core.html#default_device)
  which is CUDA by default. You can specify device as
  `torch.device('cpu')`.

Override `create_item` and use the default infinite sampler to get a
stream of unknown length (`stop()` when you want to stop the stream).

``` python
class RandDL(DataLoader):
    def create_item(self, s):
        r = random.random()
        return r if r<0.95 else stop()

L(RandDL())
```

    []

``` python
L(RandDL(bs=4, drop_last=True)).map(len)
```

    []

``` python
dl = RandDL(bs=4, num_workers=4, drop_last=True)
L(dl).map(len)
```

    [4, 4, 4, 4]

``` python
test_num_workers = 0 if sys.platform in ("win32","darwin") else 4
test_eq(dl.fake_l.num_workers, test_num_workers)
with dl.fake_l.no_multiproc(): 
    test_eq(dl.fake_l.num_workers, 0)
    L(dl).map(len)
test_eq(dl.fake_l.num_workers, test_num_workers)
```

``` python
def _rand_item(s):
    r = random.random()
    return r if r<0.95 else stop()

L(DataLoader(create_item=_rand_item))
```

    [0.7325140636747459, 0.830728338963398, 0.4511918915376345, 0.5045324614875057, 0.7998277103483771, 0.06672730120594539]

If you don’t set `bs`, then `dataset` is assumed to provide an iterator
or a `__getitem__` that returns a batch.

``` python
ds1 = DataLoader(letters)
test_eq(L(ds1), letters)
test_eq(len(ds1), 26)

test_shuffled(L(DataLoader(letters, shuffle=True)), letters)

ds1 = DataLoader(letters, indexed=False)
test_eq(L(ds1), letters)
test_eq(len(ds1), 26)

t2 = L(tensor([0,1,2]),tensor([3,4,5]))
ds2 = DataLoader(t2)
test_eq_type(L(ds2), t2)

t3 = L(array([0,1,2], dtype=np.int64),array([3,4,5], dtype=np.int64))
ds3 = DataLoader(t3)
test_eq_type(L(ds3), t3.map(tensor))

ds4 = DataLoader(t3, create_batch=noop, after_iter=lambda: setattr(t3, 'f', 1))
test_eq_type(L(ds4), t3)
test_eq(t3.f, 1)
```

If you do set `bs`, then `dataset` is assumed to provide an iterator or
a `__getitem__` that returns a single item of a batch.

``` python
def twoepochs(d): return ' '.join(''.join(list(o)) for _ in range(2) for o in d)
```

``` python
ds1 = DataLoader(letters, bs=4, drop_last=True, num_workers=0)
test_eq(twoepochs(ds1), 'abcd efgh ijkl mnop qrst uvwx abcd efgh ijkl mnop qrst uvwx')

ds1 = DataLoader(letters,4,num_workers=2)
test_eq(twoepochs(ds1), 'abcd efgh ijkl mnop qrst uvwx yz abcd efgh ijkl mnop qrst uvwx yz')

ds1 = DataLoader(range(12), bs=4, num_workers=3)
test_eq_type(L(ds1), L(tensor([0,1,2,3]),tensor([4,5,6,7]),tensor([8,9,10,11])))

ds1 = DataLoader([str(i) for i in range(11)], bs=4, after_iter=lambda: setattr(t3, 'f', 2))
test_eq_type(L(ds1), L(['0','1','2','3'],['4','5','6','7'],['8','9','10']))
test_eq(t3.f, 2)

it = iter(DataLoader(map(noop,range(20)), bs=4, num_workers=1))
test_eq_type([next(it) for _ in range(3)], [tensor([0,1,2,3]),tensor([4,5,6,7]),tensor([8,9,10,11])])
```

Iterable dataloaders require specific tests.

``` python
class DummyIterableDataset(IterableDataset):
    def __iter__(self):
        yield from range(11)

ds1 = DataLoader(DummyIterableDataset(), bs=4)
# Check it yields fine, and check we can do multiple passes
for i in range(3):
    test_eq_type(L(ds1), L(tensor([0,1,2,3]),tensor([4,5,6,7]),tensor([8,9,10])))

# Check `drop_last` works fine (with multiple passes, since this will prematurely terminate the iterator)
ds1 = DataLoader(DummyIterableDataset(), bs=4, drop_last=True)
for i in range(3):
    test_eq_type(L(ds1), L(tensor([0,1,2,3]),tensor([4,5,6,7])))
```

``` python
class SleepyDL(list):
    def __getitem__(self,i):
        time.sleep(random.random()/50)
        return super().__getitem__(i)

t = SleepyDL(letters)

%time test_eq(DataLoader(t, num_workers=0), letters)
%time test_eq(DataLoader(t, num_workers=2), letters)
%time test_eq(DataLoader(t, num_workers=4), letters)

dl = DataLoader(t, shuffle=True, num_workers=1)
test_shuffled(L(dl), letters)
test_shuffled(L(dl), L(dl))
L(dl)
```

    CPU times: user 12.4 ms, sys: 2.15 ms, total: 14.6 ms
    Wall time: 337 ms
    CPU times: user 9.25 ms, sys: 1.8 ms, total: 11 ms
    Wall time: 314 ms
    CPU times: user 9.41 ms, sys: 1.91 ms, total: 11.3 ms
    Wall time: 264 ms

    ['c', 'a', 'e', 'f', 'j', 'l', 'v', 'n', 'y', 'h', 'd', 'o', 'p', 'r', 'u', 's', 'w', 'z', 'i', 'm', 'k', 't', 'b', 'q', 'g', 'x']

``` python
class SleepyQueue():
    "Simulate a queue with varying latency"
    def __init__(self, q): self.q=q
    def __iter__(self):
        while True:
            time.sleep(random.random()/100)
            try: yield self.q.get_nowait()
            except queues.Empty: return

q = Queue()
for o in range(30): q.put(o)
it = SleepyQueue(q)

if not ((sys.platform == "win32" and IN_NOTEBOOK) or sys.platform == "darwin"):
    %time test_shuffled(L(DataLoader(it, num_workers=4)), L(range(30)))
```

``` python
class A(TensorBase): pass

for nw in (0,2):
    t = A(tensor([1,2]))
    dl = DataLoader([t,t,t,t,t,t,t,t], bs=4, num_workers=nw)
    b = first(dl)
    test_eq(type(b), A)

    t = (A(tensor([1,2])),)
    dl = DataLoader([t,t,t,t,t,t,t,t], bs=4, num_workers=nw)
    b = first(dl)
    test_eq(type(b[0]), A)
```

``` python
list(DataLoader(list(range(50)),bs=32,shuffle=True,num_workers=3))
```

    [tensor([30, 19,  0, 42, 44, 12, 18, 25, 41,  1, 39, 34, 32, 10, 35, 40,  6, 23,
             45,  9, 48, 37,  5, 46, 36, 29, 43, 20, 38, 28,  3,  4]),
     tensor([ 8, 47, 17, 24, 26,  2, 27, 31, 49, 16, 21, 15, 11, 14,  7, 33, 13, 22])]

``` python
class A(TensorBase): pass
t = A(tensor(1,2))

tdl = DataLoader([t,t,t,t,t,t,t,t], bs=4, num_workers=2, after_batch=to_device)
b = first(tdl)
test_eq(type(b), A)

# Unknown attributes are delegated to `dataset`
test_eq(tdl.pop(), tensor(1,2))
```

Override `get_idxs` to return the same index until consumption of the
DL. This is intented to test consistent sampling behavior when
`num_workers`\>1.

``` python
class AdamantDL(DataLoader):
    def get_idxs(self):
        r=random.randint(0,self.n-1)
        return [r] * self.n

test_eq(torch.cat(tuple(AdamantDL((list(range(50))),bs=16,num_workers=4))).unique().numel(),1)
`