Using self written Jython modules in ImageJ

jar
jython
scripting
Tags: #<Tag:0x00007fb87c779cc8> #<Tag:0x00007fb87c779930> #<Tag:0x00007fb87c779728>

#1

With this post I want to share my experiences with self written Jython modules in ImageJ. If you have some hints on optimising my approach, you are welcome to respond.

I started adding Jython scripts to my ImageJ plugins project EFTEMj. I deploy the scripts as 2 jar files. The first file (EFTEMj-pyScripts_.jar) contains the scripts that are discovered by the newly added SJC discovery mechanism. By using the directory structure scripts/Plugins/EFTEMj/ The scripts are listed in the menu Plugins>EFTEMj. ImageJ adds a menu separator to split the scripts from the Java plugins.

A second jar file (EFETEMj-pyLib.jar) contains self written modules that are used by my scripts in EFTEMj-pyScripts_.jar. I found out that the folder jars/Lib is part of the jython classpat in ImageJ. That is why I place EFETEMj-pyLib.jar at this folder. Jython will look inside jar files to discover modules. The jar file contains a module called EFTEMj_pyLibs. In my scripts I can import files from this module by using e.g. from EFTEMj_pyLibs import CorrectDrift as drift.


Use openCV within Jython macro
Rewriting the Jython Scripting wiki page
#2

That all sounds fantastic.

I would like to ask you for a favor: could you please edit the Jython scripting page’s Importing other .py scripts (modules) section to describe your scheme for Jython modules? I think it is superior to the one outlined there, which currently involves hacking the fiji.dir system property—a process which is unfortunately Fiji-specific, and which I have not ever tested to verify whether it still works. Your process would be much nicer to outline on that page, so that everyone can do things the “easy way” without hacks or repeated boilerplate code.

More generally, that Jython page is pretty gargantuan, and could benefit from some trimming. If you see any other places that could be trimmed and/or streamlined, please feel free to go to town on it. :grin:


#3

AFAICT, @tferr uses a similar method in his BAR collection, and documents how to use it here:

https://github.com/tferr/Scripts/blob/master/lib/BARlib.py

As he also mentions, having python modules in a subdirectory of ./Fiji.app/ leads to the compiled modules being put in the same location and not being updated anymore. So your method of keeping them in a jar file benefits from both the dependency management and the “updatability” :thumbsup: (unless the compiled modules are saved anywhere in the Fiji directory?!).


#4

@ctrueden
I started with using the the fiji.dir system property. To make Fiji use changed .py files I included a sys.modules.clear() resulting in the following import code:

from sys import modules, path
# When using own modules it is necessary to use 'sys.modules.clear()'.
# https://groups.google.com/forum/#!msg/fiji-devel/2YshfLDHiIY/MR0LoRJ6tm4J
# https://stackoverflow.com/questions/10531920/jython-import-or-reload-dynamically
modules.clear()
from java.lang.System import getProperty
path.append(getProperty('fiji.dir') + '/plugins/Scripts/Plugins/EFTEMj/')
import HelperDialogs as dialogs
import CorrectDrift as drift

@imagejan
I was not able to find any compiled jython files ($py.class).
Updating the jar file works fine: Extracting a file from the jar, adding a IJ.log(...) and putting it back results in the log message to show up, when importing the modified file.


#5

This is great!
@m-entrup thanks for testing it right away. This is great.
@imagejan thanks for noticing it. I know it has always been a limitation of BAR libs, but at the time, it was really the best way to implement them in my hands, at least. I opened a new issue to migrate everything to the new system.

BTW, Mark had this great idea of porting BAR libs to external ops, that was the entry point to the Adding new ops page. There is now some proof-of-principle (see in the Script Editor, Templates>BAR>) (OR look for BAR in Plugins>Utilities>Find Ops…). Any suggestions on how to extend it further, are more than welcome!


#6

Hi all,
late reply. I faced the same issue of imported modules not being refreshed. However when I clear the modules then I have a java null pointer exception. There are many other modules in sys.modules, so I wonder if fully clearing is too stringent,

If I understood well the other posts, wrapping the script into a JAR could be a solution. I would need some more precision though :
I have first one main script that request the user to choose a method/strategy.
According to the choice of strategy, I import the corresponding module, that pops-up a user input window to configure the chosen strategy. So this imported module is modified every time it is imported in the main script.
My question is if I should wrapped the main script and the submodules in the same JAR or if I should wrapp them separatly (or even the mainscript in one JAR and the submodules in one other JAR).
Thanks


#7

Uh, I’m not sure I understand that. Would you have your code available somewhere online?

Why can’t you just run one script from within the other, instead of importing the module? (Maybe I’m not thinking “Pythonic” enough here…)
But note that we’re working on making cross-import of scripts easier in the future, without having to append the path etc., see here:

But this will likely not be finished before some substantial changes in the SciJava framework in the (hopefully) near future.


It shouldn’t matter, but I’d recommend to put all your scripts in a single project like illustrated by the example-script-collection:


#8

So here is the main script. The idea is that it is requesting the user for an image-processing strategy, then watching a folder for new images, and calling the Run() method from the submodule that correspond to the image-processing strategy chosen.

# @String (label="Smart imaging method", required=true, value="Template matching", choices={"Template matching","Keypoint matching", "Center of mass"}) Method
"""
MAIN script  - Folder watcher used for smart imaging
It first pop up a user input to choose the method to use for smart imaging
Then it calls the corresponding script for smart imaging, that pops up another user input window to configure this particular method
"""

'''
# Some attempt to clean the imported modules (not functionnal, delete the class file instead)
import sys
print sys.modules
sys.modules.clear()
# When using own modules it is necessary to use 'sys.modules.clear()'.
# https://groups.google.com/forum/#!msg/fiji-devel/2YshfLDHiIY/MR0LoRJ6tm4J
# https://stackoverflow.com/questions/10531920/jython-import-or-reload-dynamically
'''

# import the rest of the module after cleaning
from   os.path import join,isdir,isfile
import os, time, sys

# Add plugin folder to search path
#from java.lang.System import getProperty
#sys.path.append(getProperty('fiji.dir') + '\scripts\Plugins\MyPlugins')
sys.path.append(r"C:\Users\Laurent Thomas\Desktop\Fiji.app\scripts\Plugins\MyPlugins")

# Delete the compiled class file otherwise we can not dynamically update the imported module
ScriptPath = r"C:\Users\Laurent Thomas\Desktop\Fiji.app\scripts\Plugins\MyPlugins\MatchTemplate_SI_IJM$py.class"
if isfile(ScriptPath):
	os.remove(ScriptPath)

''' 
Here importing the submodules according to the choice of the input form
''' 
# Set the run method according to choice
if Method == 'Template matching':
	from MatchTemplate_SI_IJM import Run

elif Method == "Keypoint matching" :
	from KeypointMatching import Run

elif Method == "Center of mass":
	from CenterOfMass import Run

# Routine
ListOfImage = ['...Result from folder watching...']
for Image in ListOfImages : 
      Run(Image)

Then the (shortened) template matching submodule is like :

from   fiji.util.gui   import GenericDialogPlus

# Input GUI window  (not using the @ input in case we want to wrap it into a Config() function)
Win = GenericDialogPlus("Enter path to the template image") # Title of the window
Win.showDialog()

'''Configuration of the method prior to the use of Run()'''
# Recover Path to template
if (Win.wasOKed()): 
	PathTemplate = Win.getNextString()
	#print PathTemplate

# Initialise converter
MatToImp = MatImagePlusConverter()
ImpToMat = ImageConverter()

# Open template and convert
Template   = IJ.openImage(PathTemplate)
TemplateCV = ImpToMat.convertTo(Template)  # convert to OpenCV matrice object

def Run(PathImage):
    '''The routine function executed in the main script with each image'''
     matchTemplate('blabla')

I cut a few lines to make it more succinct so it is not functional but you have the idea of the structure.

Initially I though of having another function Config() in the submodules in addition to the Run(), so that I would call Config() once in the main script. But since the submodules are fully executed once when being imported, I though it would not be necessary. And it works indeed if I delete the compiled class file.

But even if I set up this Config() method, since it is is not returning any value that I could store in the main script (and I dont want it anyway) I am not sure that it will fix the problem.
In this case for instance, the Config() would update the value of the variable Template, which is then used in the Run() but due to the compilation the Template in Run() would still probably point to the previous Template used at compilation time.
additionaly : I don’t want Config() to return any value to use in Run() since it would be different for each submodule while the main script is suppose to be generic.

Quite lengthy message sorry, I hope I am clear.
Thanks


#9

About my previous post, I think I had not fully understood how the import would work. So the good thing is that adding a Config() function fixed the “un-updated” piece of code that I wanted to import.

I also successfully compiled my Jython scripts into a .jar thanks to https://github.com/m-entrup/imagej-jython-package :star_struck:
However using https://github.com/imagej/example-script-collection/ I had the following error message and there was no jar file in the target folder

Running ScriptTest
log4j:WARN No appenders could be found for logger (org.bushe.swing.event.EventService).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Apr 26, 2018 9:48:26 AM java.util.prefs.WindowsPreferences <init>
WARNING: Could not open/create prefs root node Software\JavaSoft\Prefs at root 0x80000002. Windows RegCreateKeyEx(...) returned error code 5.
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 2.838 sec <<< FAILURE! - in ScriptTest
testScriptOutput(ScriptTest)  Time elapsed: 2.719 sec  <<< ERROR!
java.lang.NullPointerException
        at ScriptTest.testScriptOutput(ScriptTest.java:45)

Maybe I am missing something…
By the way,@imagejan if you could add the procedure to build the jar to the readme of this repository that would be great, so far there are only the badges so it was not obvious how to use it until I found your related post :


#10

That’s strange, because it builds with passing tests on Travis.

@LThomas did you remove the Convert_to_Lower_Case.groovy script? That might explain the NullPointerException, as the unit test is testing the functionality of that script.
I added the test just to illustrate how you can use continuous integration even when using scripts and not Java plugins, but you can always remove the src/test directory, or disable tests by running:

mvn -DskipTests

Thanks for the suggestion, I’ll do that. BTW, pull requests from the community are of course also welcome to improve these repositories. If you feel something is not well enough documented: just improve it :wink:


#11

Yes indeed I had removed all scripts before putting mine. Sure, I haven’t though of pull request, not used to it yet ! Thanks


#12

I discovered that the updater also allows to directly upload .py scripts or compiled .py$.class ! :star_struck:
I agree that for versioning and dependencies it is maybe not ideal but for quick prototyping it is really nice !
No need to repack the full jar if one wants to update one script…


#13

FYI, I’ve added some initial documentation to the project README:

If you feel something is missing, please help to improve by submitting a PR.


Sure, that was the case long before some people started to package scripts into jars, simply to benefit from dependency management and deployment via Maven. Sorry if this wasn’t clear to you! Can you suggest a place in the documentation where this information is missing and you’d have likely picked it up if it had been there? I’m sure that would help others in the future.

No need, unless you want to keep your scripts and dependencies in a consistent and reproducible state using Maven. But you’re by no means forced to do so.

BTW, by using jar packaging, you don’t lose the ability to edit and run scripts from the editor. Just hold Shift when clicking a (script file-generated) menu entry, and the script source code will open in the script editor. :slight_smile: (or use the search bar in combination with the Source button, of course…)