Wrap ImagePlus VirtualStack into imglib2

imglib2
bigdataviewer
Tags: #<Tag:0x00007fd69cf0e9a0> #<Tag:0x00007fd69cf0e798>

#18

Hi @ctrueden I finally got around to testing CellImg code on a 10000+++ frame .tif image and so far it works fine. I seem to be able to go to the middle of the image, process a few frames, and get a result, with little overhead.

Thanks to everybody who contributed to this thread, it was very helpful.


#19

@Christian_Tischer Finally, I have an exhaustive answer for Virtual Stack / CellImg related questions…

The following describes the state of things in imglib2 4.3.0 and imglib2-cache 1.0.0-beta-7, which should make it into Fiji soon, hopefully.


First, regarding how CellImg works:
CellImg, or rather AbstractCellImg, is an imglib2 Img that divides its underlying storage into Cells.
Each Cell represents a (hypercube) block of the full image. It has flat data (like an ArrayImg) and knows where its coordinates are in the full image. The accessors (Cursor, RandomAccess) of the CellImg know in which cell they currently “are” (which cell feeds the pixel data) and how to move between cells when you move them across the whole image.

Technically, the cells of a CellImg are in an image (RandomAccessibleInterval) whose pixel type is Cell.

For the virtual stack, and caching in general, LazyCellImg is interesting. The “image of cells” of a LazyCellImg is lazily evaluated. It relies on an implementation of the LazyCellImg.Get interface

public interface Get< T >
{
	T get( long index );
}

which provides the cell (T) for a given flattened index. Whenever an accessor walks into a cell, this is where it asks for data.

This opens the way for cached CellImgs that implement Get in various ways. These cached images live in imglib2-cache and can be build using ImgFactorys as usual. For example,

		final long[] dimensions = new long[] { 640, 640, 640 };
		final Img< ARGBType > img = new DiskCachedCellImgFactory< ARGBType >()
				.create( dimensions, new ARGBType() );

creates a 640^3 image that is backed by a disk cache:
Initially, cells don’t really exist, so the image takes up no memory (although the full image would be 640^3 * 4 = 1000 MB)
A new empty cell is created whenever an accessor “walks into it”. You can read/write the image as any other image. Cells keep track of whether they are “dirty”, i.e. have been modified from their initial empty state. A cache keeps all the created cells until memory runs full. Then, if a dirty cell is evicted from the cache, it (its data) is written to a temporary folder. When an accessor walks into that cell the next time, it is restored from disk.
imglib2-cache takes care of all of that, you just have to use DiskCachedCellImgFactory, that’s it.

It becomes even more useful by the fact that you can provide a CellLoader which is used to fill the data of “empty” cells when they are initially created. (After that, the disk cache takes over to track modifications.)

imglib2-cache comprises various variations on that scheme. You can use the factories, maybe with your own CellLoaders or build it all up from scratch using the lower-level building blocks of imglib2-cache.


On to virtual stacks.

What you want to do here, is to make a CellImg where each cell represents exactly one plane of the image (you can do that by setting cellDimensions to w*h*1) and provide a CellLoader that gets the data from the planes of the virtual stack. Here is how to do it:

final ImagePlus imp = IJ.openVirtual( "16bit.tif" );

// assuming we know it is a 3D, 16-bit stack...
final long[] dimensions = new long[] {
		imp.getStack().getWidth(),
		imp.getStack().getHeight(),
		imp.getStack().getSize()
};

// set up cell size such that one cell is one plane
final int[] cellDimensions = new int[] {
		imp.getStack().getWidth(),
		imp.getStack().getHeight(),
		1
};

// make a CellLoader that copies one plane of data from the virtual stack
final CellLoader< UnsignedShortType > loader = new CellLoader< UnsignedShortType >()
{
	@Override
	public void load( final SingleCellArrayImg< UnsignedShortType, ? > cell ) throws Exception
	{
		final int z = ( int ) cell.min( 2 );
		final short[] impdata = ( short[] ) imp.getStack().getProcessor( 1 + z ).getPixels();
		final short[] celldata = ( short[] ) cell.getStorageArray();
		System.arraycopy( impdata, 0, celldata, 0, celldata.length );
	}
};

// create a CellImg with that CellLoader
final Img< UnsignedShortType > img = new ReadOnlyCachedCellImgFactory().create(
		dimensions,
		new UnsignedShortType(),
		loader,
		ReadOnlyCachedCellImgOptions.options().cellDimensions( cellDimensions ) );

This works.

Some variations:

  1. You want to be able to write to the resulting img. Then you should use a DiskCachedCellImgFactory instead of the ReadOnlyCellImgFactory used in the example. The cells (planes) are then initialized from the virtual stack but all your modifications will be tracked and swapped to disk if necessary.
  2. You don’t want to copy the data, but directly use the short[] arrays underlying the virtual stack. Then you need to use a CacheLoader<Long,Cell<VolatileShortArray>> instead of the CellLoader. But this goes deeper into the imglib-cache internals and maybe this leads too far here. (If you want an example, I can post the code. The code is not much longer, but the explanation of what is going on would be…)

Finally, https://github.com/imglib/imglib2-cache-examples has a lot of examples going from extremely easy to quite low-level.


Read a single plane from a multi-dimensional image
ImgLib2 Virtual Stack of multiple images
How to create/work with hyperstacks too large for RAM
#20

Thanks @tpietzsch

Hoping to piggyback on this a little as it feels related

Disabling (or clearing) the cache
Is there a way to either disable caching (so that only the current cell is the only one in memory and once the cursor finds itself in another cell the previous one is removed). I know this seems undesirable but due to other processes going on in my program I don’t want my CellImg to hold on to cells, given I know I am not going back to them as I am just iterating over the images once. As you say imglib2-cache takes care of evicting and loading cells to and from the cache, but due to other concurrent processes instantiating objects at the same time I don’t want to be teetering near my memory limit. If I could disable caching that would be great, if not if there is a way to call a method to flush the cache that could work also.

ReadOnlyCachedCellImgFactory does not extend ImgFactory
ReadOnlyCachedCellImgFactory does not extend ImgFactory. I am loading my images using SCIFIO and passing a ImgFactoryHeuristic to the config as a way of selecting what ImgFactory to use when SCIFIO loads an image. This requires a return type of ImgFactory. This isn’t a huge issue as of yet as following the examples I have been using DiskCachedCellImgFactory, but figured I would just mention this as it might be an issue for someone not wrapping a VirtualStack (which I am aware was the topic at hand) but instead using SCIFIO to obtain a CellImg that acts like a virtual stack (without consuming resources beyond its active slice).

As a newcomer I just want to say great library, and thanks for any help you can give.


#21

You can set the cache to only retain a fixed number of cells using DiskCachedCellImgOptions, e.g., this

final DiskCachedCellImgOptions options = options()
		.cellDimensions( cellDimensions )
		.cacheType( CacheType.BOUNDED )
		.maxCacheSize( 100 );

sets up a cache that only retains 100 cells (can grow beyond that if you have >100 accessors that are in different cells).
See, for example https://github.com/imglib/imglib2-cache-examples/blob/a20220cb0f34c01dccd30e662255f88b8701c206/src/main/java/net/imglib2/cache/example02/Example02.java#L36-L39

Yes, that is on purpose. ReadOnlyCachedCellImgFactory cannot create an Img without a CellLoader so it cannot satisfy the interface in a meaningful way


#22

@tpietzsch First of all, I would like to apologize for my very late answer, but back then, when you posted the answer, I was not good enough in java and imglib2 to actually use it. However, today, I tested it and got it working and it is great!

@ctrueden I also started testing what happens to input images that are given to an IJ2 plugin (where they are converted to a ‘dataset’) and found the following:

  1. ImagePlus.ImageStack => dataset with PlanarImg inside
  2. CachedCellImg => dataset with CellImg inside
  3. ImagePlus.VirtualStack => dataset with CellImg inside

Case 1

Everything fine

Case 2

What I do not understand is whether the CellImg that is provided by the IJ2 plugin actually is the original CachedCellImg that has been somehow cast to a CellImg, or whether it really has been changed. The fact that the cellDims changed makes me suspect that it actually is not the original CachedCellIImg (see screenshots).

image

image

Case 3

Depends on the answer to case 2, maybe it is already working.
Anyway, I guess the idea would be that the IJ2 plugin builds a CachedCellImg from the VirtualStack, using code such as Tobias provided above, right?

In case this is not implemented yet, I am happy to try to write code (like below) that builds a CachedCellImg from any VirtualStack, i.e. covering the 3 different types (unsigned byte, unsigned short, and float), and also dealing with the 5 dimensions that the VirtualStack (ImagePlus) could have.


#23

I wrote a wrapper for ImagePlus VirtualStack.
It works as ImageJFunction.wrap() but uses imglib2-caches to lazily load the image.
https://github.com/imglib/imglib2-ij/pull/12


#24

The package https://github.com/imagej/imagej-legacy does the automatic conversion between ImagePlus to Dataset. We would need to change this code, if we want the automatic wrapping of VirtualStack to Dataset to be done lazily.

But the code in imagej-legacy confuses me a lot, for the moment, I don’t understand it well enough to make the required changes. Why are the displays needed. Why do we need Harmonizers


#25

@maarzt ( @tpietzsch )
Thanks so much for pulling this off.
This is really great!

@ctrueden
I hope you have/had a good travel back home!
I am herewith doing the job you gave me, which is to bug you to implement the next step :slight_smile: Namely using ImageJFunction.wrap at the place of (or instead of?) the Harmonizer.


#26

I agree that it is convoluted, and would benefit from simplification.

Because in ImageJ1, an ImagePlus can have visualization settings such as position in ZCT, current LUT, current selection, and an overlay containing ROIs. Whereas in ImageJ2, a Dataset cannot have any of those things—it is a wrapper around an ImgLib2 accessible without any associated current position. A DatasetView wraps a Dataset and has visualization settings; an ImageDisplay has a list of DataViews which can include linked Dataset and Overlay (i.e. ROI) objects. So it came to be that ImageDisplay was the correct choice for synchronization with ij.ImagePlus.

Because not everything in the ImageJ1 API can be offered dynamically. ImageJ1 has some fields internally, and a radically different API than ImageJ2, so there are places where copying/translating/adapting data was needed, rather than only wrapping it. That said, we need to be wrapping things in more situations. But you cannot e.g. wrap ImageJ2 Overlay objects into ImageJ1 ROIs, because ImageJ1 ROIs are not interface-driven.

Thank you @Christian_Tischer. I am sorry to say I might be quite slow with progress on this front, although if @maarzt and I stay in frequent contact, perhaps we can move things ahead more quickly. If @fjug and @maarzt agree: perhaps @maarzt and I could have a weekly video chat to keep the improvements to ImageJ Legacy flowing? Otherwise, I fear my time is going to be saturated with the SciJava Common overhaul, ImageJ Ops, the ImageJ Launcher, continued efforts toward eliminating the ImageJ Jenkins, …


#27

Would it be an option, for the time being, to add below lines of code to IJ2 plugins?
Like this, the wrapping that @maarzt developed would be applied and one would just have to make sure that the image that is passed on to the actual computations is the img rather than the dataset.

  1. What do you think?
  2. Is is already predictable when the new ImageJFunctions.wrap will be available to the community?
@Plugin( initializer = "init" )

@Parameter
    private ImagePlus imagePlus;

Img img;

protected void init()
{
        if ( imagePlus != null )
        {
            img = ImageJFunctions.wrap( imagePlus );
        }
}

#28

For the meantime you can write a plugin for wrapping the VirtualStack into an Img/Dataset:

@Plugin( type = Command.class , menuPath = "Image > Wrap Virtual Stack")
public class WrapVirtualStack implements Command
{
	@Parameter
	ImagePlus in;

	@Parameter(type = ItemIO.OUTPUT)
	Img<?> out;

	@Override public void run()
	{
		out = VirtualStackAdapter.wrap(in);
	}
}

Currently you would need to use VirtualStackAdapter instead of ImageJFunctions.
The VirtualStackAdapter can be found here: https://github.com/maarzt/imglib2-ij/blob/virtualstack/src/main/java/net/imglib2/img/VirtualStackAdapter.java


#29

That’s also an option, but imho the advantage of my approach would be that the users would not have to do the extra step of converting themselves, but it would just happen without them noticing. What do you think?


#30

@maarzt @tpietzsch @ctrueden
Hi All,

I have another question, now regarding the visualisation of an Img.

I am using this code to show the image:

uiService.show( img );

In my case the img is a 3GB xyz image with ~3000 z-planes where each z-planes has a different RealViews.transform() attached to it (as a consequence of a registration). Until I call above line of code everything is super fast, because - afaik - nothing is actually computed. However, once I am showing the result it takes a really long time (indicating that all the transforms are actually applied to all pixels).

Is there some way to make the computation of the transforms lazy, such that they only occur once the user actually browses to a specific z-plane?


What does uiService.show() do?
#31

Using Bdv for showing the data is much faster:

1 second: BdvFunctions.show( img, "", Bdv.options().is2D() );
100 seconds: uiService.show( img ) 

Simple problem now is how to save the data when it is displayed in the BDV :slight_smile:


#32

Hi @Christian_Tischer,

Glad that you found that Bdv helps!

In Bigwarp, I have some code that copies a random accessible into another one - this might be helpful to look at / steal. It obviously has to transform every pixel, but its multi-threaded, so that helps somewhat.

I use this method where the target is a wrapped ImagePlus, then have ImageJ/Fiji save the result to a file.

Hope this will be useful,
John

Edit: Forgot to mention that copying means you will allocate a bunch more memory, but it usually ends up being faster to copy (in parallel), then write, and I’m often happy to trade memory for speed.


IJ2 save an image (with lots of frames) from Command
#33

I thought that was exactly what @maarzt has been working on with the new imglib2-ij wrapping?


#34

Yes, this is true, and that works fine! I think the issue right now is “kind of the other way around”: I am already having an Img and want to show it using uiService.show( img ). Could you say what is happening at this point? Is the Img copied/wrapped into an ImagePlus or does it stay an imglib2 datastructure?

[Edit]: I am thinking about things like this:

  • If the context is ImageJ1 and uiService.show( img ) needs to convert the img into an ImagePlus in order to show it; how does it do it exactly?
    • Does it do it in a VirtualStack style, where the getProcessor() function is only evaluated once the user browses to the respective plane?
    • Or does it copy everything into RAM?
  • The reason all of this is important (to me) is because I am often dealing 500GB to 2.5TB sized data sets; so it is very relevant at which stage something is copied to RAM :slight_smile:

#35

Could you also point me to the code where you generate the wrapped ImagePlus for this specific use case? I guess you initialise it as an empty image with the same dimensions as your input RA? And then for actual saving: do you wrap back to ImagePlus and use IJ1 methods?


#36

Sure, its kind of spread out and with “clutter”, so I’ll try putting the relevant parts directly here with links to where I use it.

// create the image plus image
RandomAccessibleInterval<T> rai;
final T t = rai.realRandomAccess().get();
final ImagePlusImgFactory< T > factory = new ImagePlusImgFactory< T >();
final ImagePlusImg< T, ? > target = factory.create( itvl, t );

/*
 * do stuff...
*/

ImagePlus ip = target.getImagePlus();

see here.

Yea… :confused:


#37

Yes, great. @ctrueden I’m fine with working on this, if we have the weekly video chat, as you suggested. @fjug agreed too. The overall goal would be to make this VirtualStack wrapping possible, and change what’s necessary in imagej-legacy? It would like to this is smaller steps, which we both agree on.