Thanks very much for your suggestion. However, I believe that the 2D boundary created by your suggestion alters the ROI surrounding the retinal nuclear layer differently than a rolling circle would. For example, at a single nucleus that “sticks out” from the rest of the nuclear layer, the 2D boundary created by Process>Filters>Maximum is displaced from the boundary of the nucleus, whereas the 2D boundary created by a rolling circle would intersect the boundary of the nucleus (depending on the radius of the rolling circle and local concavity).
I found a related approach with multiple rounds of Process>Binary>Dilate and Process>Binary>Erode was unsatisfactory (protruding nuclei were clipped in the final ROI), although maybe if I tweaked this approach some more it would work. Edit>Selection>Enlarge was similarly unproductive. I also looked at Convex Hull, which worked well for the convex side of the retinal nuclear layer of interest (overall, retinal layers are C-shaped in cross-section) but failed on the concave side. I also looked into various approach that blurred the image before segmenting the nuclear layer, but these typically clipped protruding nuclei or were so large that unwanted nuclei from other layers were included in the ROI. I may be missing something, but a rolling circle approach seems like it might be the most effective solution to smooth without clipping occasional protruding nuclei.
So far a by no means exhaustive search of the internet has not yielded any ImageJ/Fiji algorithms or Matlab implementations of a rolling circle smoothing approach. I 'll take a closer look at your MorphoLibJ collection of plugins for something that may work instead.
Please correct me if I have the wrong ideas!