Introducing QuPath: Open source bioimage analysis & digital pathology


#1

Hi everyone,

I wanted to announce here that I recently put online a new open source software application for bioimage analysis, called QuPath.

You can find the documentation and download it at https://qupath.github.io

QuPath screenshots

QuPath’s focus is on whole slide image analysis (i.e. huge, tiled 2D images), and in particular digital pathology applications - although it can also be useful for other purposes, including fluorescence image analysis.

Features include:

  • A new, multi-threaded whole slide image viewer (written with JavaFX), including extras e.g. color transforms, support for touchscreens / graphics tablets
  • Tools for manual image evaluation, including annotation and counting tools
  • A collection of novel algorithms, e.g. for cell segmentation, dearraying Tissue Microarrays
  • A hierarchical, object-based data model to support working with many hundreds of thousands of image objects (e.g. cells) quickly and efficiently
  • Interactive machine learning for object classification, e.g. using cell features or region textures
  • Ability to combine steps into workflows for a wide range of applications, which can then be automated
  • Support for high-throughput Tissue Microarray scoring, including basic survival analysis based upon biomarker scores
  • Powerful scripting with Groovy (or, alternatively, Jython or Javascript)
  • Extensible design, enabling developers to add features to help share new algorithms and ideas

While developing QuPath, I’ve been hugely influenced by years spent using, teaching and generally relying upon ImageJ and Fiji, and I wanted something a bit ImageJ-like for pathology.

With that in mind, QuPath isn’t intended to compete with the fantastic open source tools for bioimage analysis that are already out there (e.g. ImageJ, Fiji, Icy, CellProfiler, ilastik…), but to add something new and complementary. For example, while it’s possible to use QuPath on its own, you can also use it along with ImageJ - exchanging data between them - to get the best of both together.

QuPath and ImageJ

Anyway, I hope some of you might try QuPath out and find it useful. If you’ve any feedback, questions or suggestions, please let me know. I’ve also set up Gitter and Google groups for any related discussions.

Thanks!

Pete


#2

That looks veeeery good. For sure will try it soon. Thanks!


#3

Thanks Gabriel!

I referred a lot to your colour deconvolution information and implementation along the way, which helped enormously. Hopefully the way it’s done in QuPath meets your approval…

I also hope you don’t mind that I added a link to your site here.

Pete


#4

It seems that someone has spend a lot of time to come up with something very beautiful and most likely very helpful.

Personally I think it would be great to not create software that is yet another disconnected island, but that is most likely only my personal taste. At least there is some interoperability…

Found here: https://github.com/qupath/qupath/wiki/Working-with-ImageJ

Note: Currently, QuPath only integrates directly with the ‘original’ ImageJ - and not with ImageJ2.

…although also this is not suiting my preferences 100%.

Are there plans to integrate with frameworks like imglib2 and the imagej2 plugin mechanism which have proven their success in many areas and applications such that interoperability is guaranteed? Also maybe there is some synergy between imagej-ops and your work where we can avoid redundant implementation efforts.

Keep up the great work!


#5

Hi Florian,

Thanks for your comments. I did anticipate some frustration that I didn’t go down the ImgLib2/ImageJ2 route, so please let me explain my reasoning.

At the time when I started this work, it was not clear that QuPath would become a software application in its own right - nor that it would be open source. Furthermore, ImgLib2 and ImageJ2 were at a considerably earlier stage of development. Since I was working on my own in an applications-focussed group, I had to act quickly to show results with the techniques and technologies I knew best, and which were ready for what I needed.

Given these pressures and uncertainties, and that I am not a group leader with the authority to steer the direction entirely on my own, the practical alternative would not have been a grander, collaborative open source project with interoperability from Day 1, but more likely switching to a commercial solution for our analytical needs. I hope you agree that, even if QuPath doesn’t suit your preferences 100%, this would be worse.

If I were to start it again today, ideally working with a team of people and having agreement from all interested parties that the result would be open source, the ImgLib2 road would certainly be one I’d explore enthusiastically. But I hope you understand that, for a range of practical reasons, it wasn’t an option for this version of QuPath. Maybe it is for a future version - and that’s one of the reasons why I was so keen to push for it to be open source at all.

However, this isn’t the only explanation for my choices. There is also a more technical side that makes me hesitant even now - albeit one that might be addressed if I knew ImgLib2 (for example) better.

This is the issue of determining the right trade-offs. For QuPath’s digital pathology applications, the analysis is typically applied to 2D RGB images, often up to 40 GB in size - and the users are frequently pathologists, or at least not image analysis specialists. While some measure of flexibility is important (and QuPath does have some support for other data types, e.g. z-stacks, fluorescence), I believe it’s essential to handle the kind of data most common for the primary application as well as possible - and it’s also essential that the software can be picked up quickly by the target users, most of whom do not have any interest in linking up to other tools. I would therefore argue that, in this specific case (but not in general), it’s more important that the software is efficient and user-friendly than that it is generic and interoperable. Of course it would be better to achieve all of these, and other people might have chosen different trade-offs from the ones that I did.

With that in mind, QuPath makes the detection and interactive classification of many hundreds of thousands of objects fast, interactive and (relatively) easy. It can generate and summarize millions of measurements in minutes, and display these in a wide variety of ways (including colour-coded measurement maps) while smoothly panning and zooming around an image that’s frequently much too big for the RAM (these videos are a bit out of date, but show the idea).

Achieving this required a lot of attention to optimisation tailored to the application, image caching, representing objects as efficiently as possible etc. What’s more, QuPath can be scripted and linked up with ImageJ, OpenCV, Weka and MATLAB today, and I’d be very happy to link it with other tools in the future. I don’t know if this would have been achieved as quickly, with my finite time and abilities, if I had chosen to base more of the core of the application on existing frameworks (which may not have fully existed when I started). I do expect that the benefits of doing this would also have necessitated some sacrifices, and given the differing goals of QuPath and the other tools built on these frameworks, perhaps even some conflicts.

Certainly I do not intend for QuPath to be a disconnected island. I also don’t think every design decision I made was optimal, and I continue to develop, revise and improve the code. I also hope that community involvement will help strengthen the software in time, and interoperability is one way to do that.

But I also think that much of the bioimaging community is already served quite brilliantly by the tools that exist and are being developed - with ImageJ2 among the most important of these - and it doesn’t need me to try to create another one along these lines. My hope is that QuPath offers something fundamentally different, and which goes some way to addressing the needs of a sufficiently wide range of users for whom there is no other accessible, open source solution currently available. Having worked most of my career with fluorescence microscopy, I can attest that the field of pathology is quite different.

Anyhow, sorry that was a bit long - but hopefully you understand my position, my openness (and enthusiasm) for interoperability where it makes sense, and why I took the decisions that I did so far.

To answer your question, I don’t personally plan the kind of integration you describe imminently, primarily because of a lack of time and other immediate priorities, but it is a discussion I would like to have - if you or any on the imglib2/imagej2 side have an interest as well.

My personal view, which I think is only slightly different to yours, is that it is great to encourage people to create and release open source software that will hopefully be of benefit to others, in whichever way works best for them. For many, this can really be a struggle in an academic context, and I think a lot of researchers find it easier to do so in a less-then-fully-connected way at first. To me, that’s no bad thing - it can also have benefits in terms of offering new and distinctive approaches.

Once a new software island is known to exist, anyone who wants can take a boat there to look around… and afterwards to consider whether it’s worthwhile to work together build a bridge :slight_smile:

Anyway, thanks for your thought-provoking comment… and patience, if you read this far.

Pete


#6

Hi Pete,

I agree with you in many points.
Your efforts to create open software that you share with the community is honorable. I really do mean this! :slight_smile:

It is indeed a problem that it is not easy enough to enter the open source community around Fiji/ImageJ2 and KNIME. We have to work on that so we can show people early one how they might benefit from using existing infrastructure.

One idea: we just launched a project called DAIS and we plan to invite developers and people you ant to become developers in Summer 2017 to Dresden. The event should last maybe a week or 10 days and is maybe best described by the word ’ Learnathon’. (A place to get in tough with the existing stuff from the Fiji/ImageJ2/KNIME ecosystem.)

Maybe that is something you’d be interested in?

Best,
Florian


#7

Hi Pete,

You did a great job with your application it looks impressive! I know how hard it sometimes is if you have limited resources / time to provide a user-friendly and stable open-source software in the academic environment. However, there are always pros and cons for starting a software development from the scratch or try to improve whats already out there. The decision is up to the developer.

Anyway, in the ImageJ2 world there exist several frameworks and tools which might be interesting for you:

  • BigDataViewer to have a look at images up to 100TB.
  • ImgLib2 for n-dimensional image processing
  • ImageJ-Ops to implement image processing algorithms (attention, under heavy development). You can override implementations to be super efficient for certain data- and image-types
  • SciJava-Common to write algorithms in the “write once, run anywhere” style.
  • Scijava-Scripting for scripting of plugins in many many languages
  • KNIME Image Processing for integration of several data analytic and bio-imaging tools and to setup integrated workflows.

If you want to enter this universe, a lot of people would be super exited and we’d be happy to help you getting started. Maybe you can have a look at some components of this world and integrate them into your application. If at some point there exists a converter (or even better a wrapper, to avoid memory duplication) from your data-structures to ImgLib2 and vice-versa, then we can maybe expose some of your algorithms as ImageJ-Ops, ImageJ2-Plugins and KNIME nodes. On the other hand you could reuse existing algorithms from ImageJ-Ops or ImgLib2.

Cheers and welcome to the forum :wink:

Christian


#8

Wow, congratulations, this seems to be a nice project!

Regarding the target audience, it seems to have quite some overlap with Orbit, i.e. whole-slide viewer, machine-learning segmentation of 2D histological images.

Are you supporting (or planning to support) 3D volumes as well?


#9

Hi Florian and Christian,

I would indeed be interested in DAIS - that sounds a good idea, and thanks for mentioning it. Maybe we can discuss the possibility closer to the time.

I’ve also been very interested to observe (albeit distantly) ImageJ2 developments, and have been interested in ImgLib2 back from before the ‘2’. I also checked out the BigDataViewer a few times, although my understanding is that it does not support whole slide / pyramidal images at this time, or at least not without converting the files. Is that right?

Since QuPath already contains converters for ImageJ1 (e.g. to translate between QuPath objects and ImageJ Rois/Overlays), and some similar functionality for working with OpenCV, it should be quite straightforward to use ImgLib2 (and possibly ImageJ-Ops?) when adding future commands.

Thanks for the welcome :wink:

Pete


#10

Hi Jan,

Thanks! And thanks also for the Orbit link - hadn’t see it before, but looks very interesting.

I’ve no plans for 3D volumes at the moment. QuPath can handle z-stacks and time series, including whole slide z-stacks (such as commonly used in cytology) - so browsing slice by slice and annotating already works fine, and commands like cell detection should also work for individual slices of stacks. Therefore having a z-stack shouldn’t prevent anyone from using QuPath.

But that’s all the support that’s planned for now, and I’m mostly focussed on 2D whole slide applications. I’d leave 3D up to the BigDataViewer and other software designed more with that in mind :slight_smile: Still, I’m always open to options and ideas…

Pete


#11

Hi, not at all. And all glory should go to Arnout Ruifrok’s work.


#12

Hi,

I wanted to give a quick update to say there is now a preprint describing QuPath on bioRxiv at http://biorxiv.org/content/early/2017/01/12/099796

This includes several videos and scripts (in the Supplementary Material) showing how different kinds of high-throughput analysis can be automated, and how ImageJ + QuPath can be combined together to add extra features and functionality to both.

Apart from that, lots of new features, fixes and other improvements have been added to QuPath since November, including much better handling of fluorescence images. Therefore even if you don’t happen to work with whole slide or pathology images, perhaps you might still find QuPath worth a look for its other complementary abilities (e.g. cell detection, measurement heatmaps, interactive object classification).


#13

A huge thank you for creating this tool. I am very pleased with the speed on my not-so-new Mac. It has already helped one of my core facility users who was at wits’ end trying to work with a large .scn file with Leica’s offline viewer.


#14

Bonjour, je m’intéresse à Qupath pour compter et estimer la surface des cellules étalées sur lames après un frottis de l’utérus (attaché une image de cellule basale).
Selon les exemples présentés par Qupath, le cellules issues de coupes histologiques sont de petites tailles et l’image présente un très grand nombre de cellules.
Alors que dans notre cas nous avons des images détaillons les cellules (cellules beaucoup plus grandes) dans le but d’estimer le rapport de surface noyau/cytosol.
Le but des d’identifier des cellules anormales ayant en général un rapport de surface Noyau/cytosol élevé.
Merci,
Cordialement


#15

Bonjour, je m’intéresse à Qupath pour compter et estimer la surface des cellules étalées sur lames après un frottis de l’utérus (attaché une image de cellule basale).
Selon les exemples présentés par Qupath, le cellules issues de coupes histologiques sont de petites tailles et l’image présente un très grand nombre de cellules.
Alors que dans notre cas nous avons des images détaillons les cellules (cellules beaucoup plus grandes) dans le but d’estimer le rapport de surface noyau/cytosol.
Le but des d’identifier des cellules anormales ayant en général un rapport de surface Noyau/cytosol élevé.
Merci,

Cordialement