Trainable WEKA plugin updatable models

weka
Tags: #<Tag:0x00007fd541f34148>

#1

Which models are updatable?..I notice that I can’t load a fastRandomForest model and add a new category for re-training. I need incremental training capabilities as new targets become important as a function of time.


#2

Hello @rondespain,

Here you are information about updatable classifiers in WEKA. As you see, there are only a few such classifiers and I’m not sure they will work when adding new classes.

The Trainable Weka Segmentation plugin is designed to build a new classifier from scratch every time you click on “Train classifier”. I can easily modify the code to update the model if it is updatable. Would you like that?

So far, what you can do is to save the data related to your different stages as ARFF files, and once you get to the point of knowing the final number of classes, merge the ARFF files (outside the plugin) and load them into the GUI.


Trainable WEKA seg options OK button off screen
#3

Including the updatable features in WEKA would be great…I too am unsure if the updatable classifiers will allow new classes. It would be great if the updatable classifiers would allow the addition of additional classes and instanses. The ARFF file format would seem to allow that.

The main problem with the updatable classifiers (from the javadoc) is that they might not be very good ones. I’m having great luck with the fast random forest (did you develop this filter?it works well) and the neural net. so It might be better for me to edit the ARFF files. My images of different diseases are in different images so I will either have to make a montage of several images and do a single training, or train on each image and merge their ARFFs…Which seems more practical to you?

If you make changes to the Trainable Weka Segmentation plugin, please print out the output from the WEKA log to the ImageJ log which contains the cross validation statistics, confusion matrix ad model details.

Thanks
Ron DeSpain


#4

OK, I will do that as soon as I can.

I would recommend merging the ARFF files, yes.

I will consider it. The reason why I didn’t do it in the first place is because the Random Forest classifies perfectly the training data, so the default confusion matrix won’t be very informative. The plugin does not perform cross validation, so there is nothing to show in that sense.


#5

When I ran the Random Forest I saved the ARFF and re-ran it in WEKA with cross validation and got this output and confusion matrix (not perfect)

=== Stratified cross-validation ===FastRandomForest -I 100 -K 0 -S 1
=== Summary ===
Correctly Classified Instances 8839 99.8983 %
=== Confusion Matrix === FastRandomForest -I 100 -K 0 -S 1
a b c <-- classified as
2837 7 0 | a = class 1
2 1578 0 | b = class 2
0 0 4424 | c = class 3

I was able to improve that a little with the Multi Layer Perceptron.

=== Stratified cross-validation === MultilayerPerceptron -L 0.3 -M 0.2 -N 500 -V 0 -S 0 -E 20 -H a
=== Summary ===
Correctly Classified Instances 8843 99.9435 %
=== Confusion Matrix ===
a b c <-- classified as MultilayerPerceptron -L 0.3 -M 0.2 -N 500 -V 0 -S 0 -E 20 -H a
2843 1 0 | a = class 1
3 1577 0 | b = class 2
1 0 4423 | c = class 3

1] It looked like both WEKA runs were 10 fold cross validations. …is that not true?

2] As far as updatable models, I think it would be more useful to modify the plugin to allow the integration of new classes and instances within the plugin by opening the model and the data, adding new stuff by clicking on the image, re-training and then saving the updated model and data. If you can cut and paste the ARFF data, that should be possible within your great interface.

Ron


#6

This is only true in the Weka Explorer, which by default uses a 10-fold cross-validation during training. In the plugin, all user traces are used for training, so there are no folds and the random forest would learn the training set perfectly as I said.

My major concern with that is it would open the door to problems for the vast majority of users, since most of them do not need adding classes on the fly. In their case, loading a file with different classes than the current ones in the GUI would be most likely an error.

It just occurred to me you could use a very simple solution that is already working in the plugin: add initially as many classes as you think you might eventually need. There is no problem if later on some of the classes do not have traces, only two are needed that way.


#7

Adding extra classes that have no data would be a good way to make the model classes updatable (if you can rename the classes), but would that allow adding new traces during an update to the empty classes while preserving the traces already in the training set?


#8

Yes, I just checked it out and it works!


#9

What is your procedure for doing this? When I load a model and data into Trainable WEKA Segmentation plugin the OK button disappears so I can’t change the names of the classes.
Do I just load in the model and proceed to select new data…as this scenerio only moves the OK button half off the screen, so I can still click it after renaming the empty classes.


#10

Please, notice your off-screen window/dialog problems are not related to the Trainable Weka Segmentation but most probably to the Java interaction with your operative system and monitor configuration.

That being said, what I was suggesting to solve your problem is the following:

  1. Start TWS with your first image and add as many classes as you want (more than what you think you might need).
  2. Trace samples of the classes you want and save them as ARFF.
  3. Start TWS with another image and load the ARFF file.
  4. Now you can continue tracing samples of the other classes if you want (you can leave some empty as well).

You can also save and load a classifier between steps 2 and 3 but since the updatable classifiers are not taken into account yet in TWS, it won’t make much sense.


#11

I’ve noticed that when I reload a model and apply it to a new image, the class colors are no longer obvious. The classes are listed on the right of the image, but since there are no added selections, the colors of the text aren’t there. Are the class colors retrievable somehow?


#12

When you load a classifier there are no traces so you need to remember the classes you used. The easiest way is to name your classes with informative names so later you know which classes correspond to which pixels.


#13

You recommended
1Start TWS with your first image and add as many classes as you want (more than what you think you might need).
2Trace samples of the classes you want and save them as ARFF.
3Start TWS with another image and load the ARFF file.
4Now you can continue tracing samples of the other classes if you want (you can leave some empty as well).

When I got to step 3 after closing out and restarting the segmanter, the segmenter complained that the ARFF classes didn’t match, so I loaded the model and I was able to load the ARFF…however the save button at the bottom of the setup window was again off screen, so I wasn’t able to change the model type. I’m running FiJI on a 64bit Win10 PC…any recommendations as to how to get around the setup menu extending off the bottom of the screen?

Thanks
Ron DeSpain


#14

I’m not a Windows user so maybe someone else in the forum can help… Have you tried these solutions?


#15

Thanks for the heads up to the suberuser site. Since the settings button went off screen I thoulght I’d just load the ARFF data in WEKA, change the model, retrain, and save out the model. However, when I tried to load the WEKA generated model into the FIJI WEKA TRAINABLE SEGMENTATION PLUGIN, I got an error reading the .model file. Is there a way for me to modify the model file generated in WEKA, in a text editor, so that it is readable in the FIJI plugin?


#16

What error did you get? Can you post it here or send me the .model file?

No, the model file is more than just text. You can only load it in TWS or WEKA itself to evaluate it or retrain it when possible.


#17

I have the error messages and can share the model and console output, but I don’t know how to attach things here in the forum…Please contact me at rondespain@yahoo.com and I’ll send over the files.

How do you want me to credit you and your great plugin. I’m helping a medical grad student do his dissertation, so this work will eventually get published…I’m doing the work pro-bono (free), so I can’t offer you compensation.

Thank you for your help so far.

I just saw the upload arrow, so I’ll try that.weka model error message in TWSerror2 loading WEKA classifier in WTS

looks like only jpg’s are accepted so here they are: error2 loading WEKA classifier in WTSweka model error message in TWS

Ron DeSpain
321 751 3789


#18

I will send you an e-mail. You can always use something like Dropbox or Google Drive.

A citation of the TWS paper would be appreciated :wink: