As @imagejan said, the out of bag error is a good indicator of your classifier performance on samples that are similar to your training ones. Of course, you won’t have an idea of the performance on samples that the classifier has never seen and are very different from your training samples.
In general, it is a good idea to feed the classifier with representative samples of all the the type of pixels you might encounter on your images.
As a rule of thumb, during the interactive process of adding new samples and retraining
- If the out of bag error decreases, you are going in the right direction and the new samples you added are helping. Therefore, continue adding even more samples.
- If the out of bag error increases, do not worry, it might be because
- the new samples were added erroneously to a class (unlikely if you are being careful),
- the new samples were unseen and have increased the complexity of the classification, so continue adding and training to try to reduce the error.
- If the out of bag error stays the same, the samples you added didn’t make a difference and most probably you can stop training.