Memory not clearing over time

memory
bio-formats
Tags: #<Tag:0x00007fb886820e38> #<Tag:0x00007fb886820cf8>

#1

Hi,

Some of my users are using one of my ImageJ macros with ImageJ 1.51d. They are finding that over time the memory is running out despite the images being closed. The images are quite big (approx. 500 mb) but otherwise just a normal macro. We never had this issue prior to recently upgrading to 1.51d. The macro uses the TransformJ Translate plugin but other than that nothing special. I’ve had tried cleaning the memory by clicking the toolbar or by running the macro command but the memory is still full until Fiji restart. Any ideas?

Best,
Dominic.


#2

Hi @dwaithe,

are you running one macro which iterates over a huge number of images or is the execution of the macro manually started after a user loaded images individually? In any case: would you mind sharing the macro here on the forum so that the developers could actually trace the issue?

Cheers,
Robert


#3

You can monitor you Java application with Visual VM to inspect which part of the macro is responsible for the memory bug.

E.g., make a heap dump of the running application.

I guess that some image references are not dereferenced and could’nt be garbage collected by the VM.

See:


#4

Hi,
Thanks for your suggestions so far. Robert, the original script is pretty long so I have created a minimal script that recreates the same phenomenon:

path = “/path/to/folder/”;
open(path+“170516-RNA-DNA-RaserFISH-MFL0h-aRNAg-pAr-pE647_01_001_R3D.dv”);
close();
open(path+“170516-RNA-DNA-RaserFISH-MFL0h-aRNAg-pAr-pE647_01_002_R3D.dv”);
close();
open(path+“170516-RNA-DNA-RaserFISH-MFL0h-aRNAg-pAr-pE647_01_003_R3D.dv”);
close();
open(path+“170516-RNA-DNA-RaserFISH-MFL0h-aRNAg-pAr-pE647_01_004_R3D.dv”);
close();

Also thank you for the suggestion Bio7 I have used the VisualVM to profile the heap usage whilst running the above script.

I have circled the times at which I ran the above script. You can see that the script quickly starts to fill the heap, despite every image being closed. Looks like a bug potentially when opening .dv files? We have a work-around which is just to close Fiji every now and then, but I think it important to bring to your attention. Please suggest better workaround or fix the bug in next update.

Best,
Dominic.


#5

Hi Dominic,

I have the same issue at the moment. I have ~50 images with a size of 1GB each and run out of memory every 3-4 images. I spend a lot of time searching for a solution and you can try run("Collect Garbage");
This can solve the problem but it seems there is more than one “garbage collector” and some work better than others



They suggest using the -XX:+UseParNewGC but I haven’t figured out how to implement this. Maybe this helps you.

Best,
Joko


#6

Since no one else mentioned it yet, I just wanted to point out the dedicated Troubleshooting section of the web site, which goes over some ways to diagnose these sorts of issues:


#7

Thanks for posting this. I have already set my memory to 20GB and run the garbage collector without any success. I use Fiji on MacOS. Do you know where I can change the default garbage collector?


#8

Do you know where I can change the default garbage collector?

While tuning the garbage collector is possible, it will not address the likely root cause of your problem, which is probably a memory leak. If you have technical aptitude, you could try debugging it using a profiler like JVisualVM as suggested by @Bio7.

Regarding your macro: are you running in batch mode? And on which platform do you run? And which version of Java? There is a known memory leak in Java 8 on OS X when opening and closing lots of windows.


#9

I am running in batch mode (all windows are closed) and I have the same issue when I set batch mode = false.
I have Java 8 and run OS X El Capitan Version 10.11.5
I monitor the memory while I run the script with the built-in Memory monitor in Fiji and no memory gets cleared after closing the last image and every 2nd or 3rd image I reach more than 90% and the Macro stops


#10

Thanks for the additional information, and sorry for not reading the preceding messages in the thread carefully before.

I tried to reproduce your issue (I also run Java 8 + El Capitan), and was able to do so using another format supported by Bio-Formats.

At first glance, it looks like the ImagePlus objects (i.e.: ImageJ images) are indeed not being garbage collected. Something is holding references to the image planes. I am digging further in JVisualVM now to learn more.


#11

Here is what I’ve discovered so far:

  • When opening an image using the Bio-Formats plugin (either directly via e.g. File :arrow_forward: Import :arrow_forward: Bio-Formats, or indirectly via File :arrow_forward: Open…), with default settings (Autoscale on, everything else off), memory is not returned to Java after the image window is closed.

  • The problem only happens with Oracle Java (7 or 8; tested with 1.7.0_80, 1.8.0, 1.8.0_77 and 1.8.0_101), not Apple Java (6; tested with 1.6.0_65).

  • It happens with both latest and older versions of Bio-Formats (tested with 5.1.0, 5.1.10 and 5.2.1).

  • It happens with older versions of Fiji (Life-Line from 2014) as well as latest.

  • The bug only manifests with images opened by Bio-Formats, not opened by ImageJ 1.x core—i.e., File :arrow_forward: Open… on a single file, as well as a sequence opened via File :arrow_forward: Import :arrow_forward: Image Sequence….

So… my current guess is that it’s a bug in Oracle Java triggered somehow by the Bio-Formats code?

@dgault @s.besson @melissa Is anyone on the Bio-Formats team able to reproduce this?

My OS X is 10.11.5. My test dataset was a Prairie dataset ~126M in size, although I’d be surprised if this is format specific.

@dwaithe Have you tried downgrading to an earlier version of ImageJ 1.x (Help :arrow_forward: Update ImageJ…) and seeing whether the issue still happens? This would be very useful to know.


#12

I can certainly reproduce the same behaviour you have described and have started attempting to root cause which object is not being garbage collected.

I have opened a card on the Bio-Formats 5.2.x Trello board for this ongoing investigation - https://trello.com/c/jrg0TfwK/44-imagej-plugin-memory-leak


#13

@dgault Thanks. A couple of additional comments:

  • I did not try with plain ImageJ 1.x. If you cannot reproduce in plain ImageJ 1.x, it may be a bug relating to ImageJ2, likely the ImageJ Legacy layer.

  • I did not isolate with certainty that this is a Bio-Formats-specific issue. Just that somehow, naively opening images with ImageJ’s built-in functionality does not trigger the problem. It is possible there is some other non-Bio-Formats-specific incantation that suffers from this issue.

  • I verified that the window frames are being disposed. After they are closed, you can still access them via Frame.getFrames(), with each disposed Frame returning false as expected for isDisplayable().

FWIW, here is a groovy script I was using inside Fiji to test things:

import ij.gui.ImageWindow
frames = java.awt.Frame.getFrames()
for (f in frames) {
	println(f.getTitle())
	c = f.getClass().getName()
	displayable = f.isDisplayable() ? "displayable" : "NOT-displayable"
	println("-> " + f.getClass().getName() + ", " + displayable)
	if (ImageWindow.class.isAssignableFrom(f.getClass())) {
		println("-> Killing it...")
		if (f.isVisible()) ((ImageWindow) f).close()
//		f.setVisible(false)
//		f.dispose()
	}
}

But probably the best route is to use a profiler to track down lingering hard references.


#14

@ctrueden : Thank you for the script and the updates.
We have created a card to record your observations,
and we have started testing the same in better detail,

We look forward to tackling this issue at the earliest.

Best,
Balaji


#15

I just upgraded to macOS Sierra (i.e., 10.12) and it seems like—in my brief tests so far—this issue is gone now. Repeatedly opening and closing images opened via Bio-Formats now seems to free memory as expected.


#16

My users are using the Pet Ct viewer and the same problem of memory not being freed exists there.

To demonstrate the problem I measured the memory use at start (total memory) at 2.6GB.
Then I did a drag and drop of the folders one by one. Each time I told it to make a stack.
I was careful not to open Pet Ct viewer, so as not to complicate the problem.
Memory use went up to 3.3GB.
Then I hit “X” on each of the studies and the memory use remained at 3.3GB.

The problem the users are seeing is that over time there is a Java error saying it is out of memory, so Fiji crashes.

If you say that bio-formats release memory, perhaps I need to port over my application to bio-formats? I use imagePlus objects all over the place and I don’t know what to port them to. There are other ImageJ programs like Gaussian smooth that I use, and it isn’t clear if would be able to continue to use them.


#17

The problem was the opposite: images opened using Bio-Formats somehow did not free memory when closed, when using OS X 10.11 (and possibly other earlier versions of OS X—I did not test that).

My point above is that macOS Sierra (the new OS version 10.12) no longer seems to have this problem.

Are your users using Macs? If so, you could have them try updating to macOS Sierra, although it is still very new so there might be other downsides to that technically.


#18

Off the top of my head I know of 2 of my users who are using Mac, but I don’t know what version.

I heard of the problem from a user of Windows and I can demonstrate that memory is not released on my Linux machine.

So it seems that the problem is deeper than bio-formats. I had misunderstood that bio-formats solves the problem. If possible, we need to find out what is going on even with classic ImageJ. Likewise, at some point I’ll need to port over my plugin to bio-formats. When is a good time to do this is still not clear to me.


#19

Maybe the Debugging memory leaks section of the wiki helps you?

Why?


#20

I looked at the Debugging memory leaks, but in the drag and drop of studies I purposely chose NOT to use my plugin so as to have one fewer variable in the equation. My user on Windows is getting the out of memory exception, but he is obviously using my plugin. He looks at multiple studies at once which also weighs down the problem.
He doesn’t have a clear scenario of when it happens, but he does see memory usage increase and never decrease. With 12GB of memory on this machine it is a bit difficult to run out of memory but I definitely see memory use increase and never decrease. He sees the same perpetual increase, but he runs out of memory after looking at (and closing) many studies.
The work around which I suggested for the moment is just restart Fiji after it crashes. Not very nice, but at least he can work.

On bio-formats I understood it is the wave of the future and essentially everything should go over to it. I didn’t start because I didn’t want to make waves for my users.