Can't access output after using IJ2 op shearView

imagej-ops
Tags: #<Tag:0x00007fb87d766e60>

#1

I have large light sheet datasets that need to be sheared or deskewed before analysis (the stack was collected with the sample/sheet motion not orthogonal to the XY plane). I’ve been using an ImageJ script or TransformJ affine to perform the deskewing; they work but execution speed isn’t as fast as I’d like. After noticing that the IJ2 ops has a shear transform built in I spent most of today trying to create a small script using it in hopes that it might be faster (or made faster in future). Also it seems there is the possibility of displaying a deskewed version of the dataset without actually resaving which would be nice. But some newbie mistake(s) remain despite my best efforts with the documentation and forum posts.

There are at least 2 things that I’m not sure about (1) how to apply the transform and (2) how to save the result. I see things about RandomAccessibleInterval and the idea of a “burnIn” op in the forum but it’s all going over my head.

Can anybody point me in the right direction? Links to further reading much appreciated. Perhaps more importantly, does this even seem like a good strategy to deskew large datasets?

Here is the simple Groovy script I’m using for testing.

# @ImageJ ij
image = ij.io().open("http://imagej.net/images/fluoview-multi.tif");  // convenient example stack
sheared = ij.op().run("shearView", image, 2, 0);  // shear the data, not sure which axis is which nor a good way to find out but this is my best guess guess
ij.ui().show(sheared);
// I'm not sure how to make a concrete copy of the view that can be saved by itself

I get a warning in the console and log: Ignoring unsupported output: result [net.imglib2.view.TransformView] and then nothing appears. I can show the original image just fine. I’m using FIJI on Win10 with Java8 and ImageJ 1.51t.

Thanks for your help!


#2

Good day,

TransformJ should do the job and I don’t think that there is a faster method showing the same interpolation quality. What interpolation scheme did you use?

regards

Herbie


#3

@Herbie: I find that TransformJ affine is slower than my script that enlarges the X dimension of the image and translates the pixels of each slice along the X proportional to Z. I suspect the reason TransformJ affine is slower is because its fully general and thus performing needless computations (no narrower TransformJ skew exists). I’d like to be able to choose interpolation scheme, including none, depending on the application.


#4

Just a remark:

Without interpolation your images wont be suited for reasonable evaluation.

Be successful

Herbie


#5

@jondaniels

I am pretty sure the author of those ops - @tibuch - would be best to help you regarding your script, etc.

eta :slight_smile:


#6

@jondaniels I played around with the shear op for a few minutes this morning. I still don’t fully understand how it is suppose to work, however it looks like it wraps an integer version of shear, thus does not require an interpolater.

I am working on general transform op which should be able apply a general shear if you construct the transform matrix. This op would take in an optional interpolate factory parameter. I hope to get a chance to write some tests for the shear case later this weekend or next week. The general transform op won’t be released at least for a few weeks (next version of ops), but it should handle your use case.

For now I think the reason you cannot see the image, is that the TransformView does not have an interval associated with it.

I changed your script slightly to take in an input image, and tested a 2D case (I had errors when I tried your 3D image). In the first case I use Views.Interval to specify the interval the sheared image should be displayed over. Passing in the input image, means it will use the input interval. In the second case I used a slightly different different version of the op that takes in an input interval, and seems to automatically set the output interval.

// @ImageJ ij
// @Img image
import net.imglib2.view.Views;

//image = ij.io().open("http://imagej.net/images/fluoview-multi.tif");  // convenient example stack

sheared = ij.op().run("shearView", image, 0, 1);  // shear the data, not sure which axis is which nor a good way to find out but this is my best guess guess
ij.ui().show(Views.interval(sheared,image));

sheared2 = ij.op().run("shearView", image, image, 0, 1);  // shear the data, not sure which axis is which nor a good way to find out but this is my best guess guess
ij.ui().show("sheared2", sheared2);

input
image
sheared 1
image
sheared 2
image


#7

@bnorthan, thanks that is helpful! I was missing Views.interval().

I confirmed your script works in my environment. However, if the two dimensions passed to the shearView op are interchanged then I get a java.lang.ArrayIndexOutOfBoundsException, e.g. with sheared = ij.op().run("shearView", image, 1, 0);.

Good catch that perhaps the shearView op is only intended for 2D images and not 3D. So maybe I need to spend some time digging deeper and figure out how to write an op for the 3D deskew.


#8

So I copied and pasted some stuff from the transform op I’m writing and “groovied” it and came up with this…

I’m no expert in imglib2, I just wrote this based on suggestions from people, and by looking at examples. Others might have suggestions as to how to make it better/faster.

// @ImageJ ij

import net.imglib2.view.Views;
import net.imglib2.FinalInterval;
import net.imglib2.interpolation.randomaccess.LanczosInterpolatorFactory;
import net.imglib2.outofbounds.OutOfBoundsConstantValueFactory;
import net.imglib2.realtransform.AffineTransform3D;
import net.imglib2.realtransform.AffineGet;
import net.imglib2.realtransform.RealViews;
import net.imglib2.type.NativeType;
import net.imglib2.util.Util;
import net.imglib2.view.Views;

image = ij.io().open("http://imagej.net/images/fluoview-multi.tif");  // convenient example stack

transform=new AffineTransform3D();
		
zero = Util.getTypeFromInterval(image).copy();
zero.setZero();
outOfBoundsFactory = new OutOfBoundsConstantValueFactory(zero);
		
interpolator =new LanczosInterpolatorFactory();
		
double[][] mat=[[1,0.45,0,0],[0,1,0,0],[0,0,1,0],[0,0,0,1]];
transform.set(mat);
		
outputInterval=new FinalInterval([0,0,0] as long[],[2*image.dimension(0)-1,image.dimension(1)-1,image.dimension(2)-1] as long[]);
		
Views.interval(Views.raster(RealViews.affineReal(Views.interpolate(
				Views.extend(image, outOfBoundsFactory), interpolator),
				(AffineGet) transform)), outputInterval);

#9

Hi @jondaniels

I looked into the op-implementation and it is just a wrapper for the integer based shear from imglib2. That is why no interpolator is needed and this is most likely the reason for the fast computation.

For a general shear I would do something like @bnorthan proposed above.


#10

Hi @tibuch,

A quick non-expert glance at the imglib2 ShearTransform code suggest it was written to handle arbitrary number of dimensions. However I have been unable to get the op version “shearView” to work except with individual images (2D) and then only with a specific axis ordering as noted earlier.

Is this expected or is this a bug that I should report to someone?

I realize that the shear can be computed with a general affine transform (and greatly appreciate the example @bnorthan provided) but that approach seems less efficient than using special-purpose code that avoids calculating all the terms that are known to be zero.


#11

For me the script works with a 3D image. But only with parameters ij.op().run("shearView", image, 1, 0);. Any other combination of dimensions is not working. Maybe @hanslovsky can help us out?

The Views.shear() implementation, which is the one used in the imagej-ops, is very simple and a more sophisticated implementation would be nice! Do you have by any chance one which could be added to imagej-ops?


#12

@tibuch

The Views.shear() implementation, which is the one used in the imagej-ops, is very simple and a more sophisticated implementation would be nice! Do you have by any chance one which could be added to imagej-ops?

No I don’t have any experience writing code for image manipulation beyond chaining ImageJ commands in a script.

So at this point we are stuck on an apparent bug in the shearView code where only one combination of dimensions is working.


#13

@jondaniels, @tibuch,

Nice find. I decided to look into this issue a little.

Interestingly, it doesn’t seem to be an issue with Views.shear only but rather with some combination of that and the visualization code.

I think this because visualizing arbitrary shears with bigdataviewer looks correct, i.e.:

Img<T> img = ...
IntervalView< T > imgShear12 = Views.interval( 
                Views.shear( Views.extendZero( img ), 1, 2 ),
                img );

BdvFunctions.show( imgShear12, "shear 12" );  // looks good
// ImageJFunctions.show( imgShear12 ); // CRASHES (see below)

will continue looking into this.

I’m curious what you mean by this exactly - does it crash? If so, is the error at all similar to what I’m observing (see below).

John

Exception in thread "zSelector" java.lang.ArrayIndexOutOfBoundsException: -186
    at net.imglib2.img.basictypeaccess.array.FloatArray.getValue(FloatArray.java:62)
    at net.imglib2.type.numeric.real.FloatType.get(FloatType.java:115)
    at net.imglib2.type.numeric.real.FloatType.getRealFloat(FloatType.java:126)
    at net.imglib2.converter.RealFloatConverter.convert(RealFloatConverter.java:51)
    at net.imglib2.converter.RealFloatConverter.convert(RealFloatConverter.java:46)
    at net.imglib2.display.projector.IterableIntervalProjector2D.map(IterableIntervalProjector2D.java:151)
    at net.imglib2.img.display.imagej.ImageJVirtualStack.getProcessor(ImageJVirtualStack.java:174)
    at ij.ImagePlus.setSlice(ImagePlus.java:1561)
    at ij.gui.StackWindow.setSlice(StackWindow.java:305)
    at ij.gui.StackWindow.run(StackWindow.java:236)
    at java.lang.Thread.run(Thread.java:745)

Edit:
Specifically, its weird to me that IterableIntervalProjector2D is called here - I don’t understand how this class could know about the shear in 3D. :confused:


#14

Wow! That is interesting. Thanks for the investigation @bogovicj!

Yes, I get ArrayIndexOutOfBoundsExceptions too.