How to create/work with hyperstacks too large for RAM

stacks
memory
Tags: #<Tag:0x00007fd542e566f8> #<Tag:0x00007fd542e564a0>

#1

Hi, I have acquired a time series of 3D images (all present as single PNG files) that exceeds 20GB in size. I am wondering how to display it. I’ve created a hyperstack by opening a subset of time steps as a stack and concatenating those. Is it somehow possible to create and display hyperstacks that are too large to fit into RAM?


#2

Hi @speedymcs, and welcome to the forum.

You can do this with Virtual stacks, which are easiest to open if every slice/timepoint is its own 2d image.

John


#3

Hi John, thanks for your response! In my case, every timepoint is a stack of single images already, so I want to create a 4D stack in the end. Is this possible as a virtual stack? I tried opening every timepoint as a virtual stack, but when choosing ‘Concatenate’, an error occures due to full memory.


#4

Ah, I see. As far as I know, there is not an “easy” (read: without some coding) way to do that at the moment.

I’m not sure what your background is, but since your dataset is quite big, I’d recommend looking into bigdataviewer for display (and imglib2 which is its backend).

I think this thread, and this post in particular:


which talks about loading many sub images into a single CellImg which is basically what you want to do.

If you are willing to resave your data into a different format (such as hdf5 - see the link above) then opening and browsing with bigdataviewer will be quite easy. If you need to keep your current file organization, then it will probably be necessary to do some coding.

A big info dump, I know, but hope its helpful…

John