R on Data Science Guts

Deep learning–based automated measurements of the scrotal circumference of Norwegian Red bulls from 3D images

Wed, 25 Jan 2023 01:00:00 +0000

Working on Computer Vision tasks is always exciting for me. During my carrier I was working with many different types of images and was solving many different problems related to them in the fields of biology, medicine, genetics, climatology and many more. Today I would like to tell you about one of most extraordinary use cases I’ve ever worked on.

The problem

Computer Vision can be applied in many different fields, sky is the limit, but to be completely honest I would never guess that someday I would work on automatic measurement of the scrotal circumference of Norwegian Red bulls.

Everything started around year ago when my former student, now a PhD candidate at Inland Norway University of Applied Sciences, Joanna Bremer wrote me an e-mail with a simple question: Can we measure a scrotal circumference from the 3D images using deep learning? From this moment i was hooked!

I turns out that it is a major agricultural trend to automate measurements of different physiological and behavioral traits. Scrotal circumference is an essential part of the selection criteria for bulls in breeding programs. Traditionally circumference is measured manually with the use of scrotal tape. Automation of this process and implementation into feeding stations would be a valuable tool for performance testing stations and bovine semen collection centers and it would improve the safety and welfare of both technicians and animals.

Before we will show you our solution we should mention other members of our team. Beside me and Joanna there was also Elisabeth Kommisrud who is Joanna’s mentor and supervisor and Øyvind Nordbø.

We would also like to thank Hallstein Holen and his team at the Geno performance testing station: Jan Tore Rosingholm, Erik Skogli, Sigmund Høibakken and Stein Marius Brumoen for their time and indispensable help with data collection.

The solution

To crack this case we basically used 3 computer vision algorithms and some fancy mathematics. Our solution can be summarized in this steps:

Semantic segmentation of the scrotum
Connected–component labeling (CCL) algorithm for the segmentation artefacts
Direct Linear Least Squares fitting of an ellipse to the predicted scrotum
Approximation of the scrotal circumference

In the beginning we had nothing except for the images of the bulls taken using 3D camera, that shows how far is the object from the lens. We decided to create segmentation masks for the scrotum and train U-Net to predict the scrotum location.

For most of the images, the predicted segmentation mask contained one solid object, which was expected and desirable. Some artefacts that created a second smaller object for the remaining images were found in the predicted segmentation mask. To solve this problem, we used a connected–component labeling (CCL) algorithm (also known as blob extraction or region labeling) to count the number of solid objects in a predicted segmentation mask. In the case of finding more than one object, only the object with the highest area would stay on the segmentation mask.

After segmentation and cleaning faze, the Direct Linear Least Squares algorithm was used to fit an ellipse onto the boundary of a segmented mask.

In the final step we used Pade approximation combined with the distance and angle per pixel informations to calculate final scrotal circumference.

If you wanr to know more about the process, results and validation of our method check out our paper.

Explaining predictions of Convolutional Neural Networks with 'sauron' package.

Sun, 10 Jan 2021 01:00:00 +0000

Explainable Artificial Intelligence, or XAI for short, is a set of tools that helps us understand and interpret complicated “black box” machine and deep learning models and their predictions. In my previous post I showed you a sneak peek of my newest package called sauron, which allows you to explain decisions of Convolutional Neural Networks. I am really glad to say that beta version of sauron is finally here!

Sauron

With sauron you can use Explainable Artificial Intelligence (XAI) methods to understand predictions made by Neural Networks in tensorflow/keras. For the time being only Convolutional Neural Networks are supported, but it will change in time.

You can install the latest version of sauron with remotes:

remotes::install_github("maju116/sauron")

Note that in order to install platypus you need to install keras and tensorflow packages and Tensorflow version >= 2.0.0 (Tensorflow 1.x will not be supported!)

Quick example: How it all works?

To generate any explanations you will have to create an object of class CNNexplainer. To do this you will need two things:

tensorflow/keras model
image preprocessing function (optional)

library(tidyverse)
library(sauron)

model <- application_xception()
preprocessing_function <- xception_preprocess_input

explainer <- CNNexplainer$new(model = model,
                              preprocessing_function = preprocessing_function,
                              id = "imagenet_xception")
explainer
# <CNNexplainer>
#   Public:
#     clone: function (deep = FALSE) 
#     explain: function (input_imgs_paths, class_index = NULL, methods = c("V", 
#     id: imagenet_xception
#     initialize: function (model, preprocessing_function, id = NULL) 
#     model: function (object, ...) 
#     preprocessing_function: function (x) 
#     show_available_methods: function () 
#   Private:
#     available_methods: tbl_df, tbl, data.frame

To see available XAI methods for the CNNexplainer object use:

explainer$show_available_methods()
# # A tibble: 8 x 2
#   method name                  
#   <chr>  <chr>                 
# 1 V      Vanilla gradient      
# 2 GI     Gradient x Input      
# 3 SG     SmoothGrad            
# 4 SGI    SmoothGrad x Input    
# 5 IG     Integrated Gradients  
# 6 GB     Guided Backpropagation
# 7 OCC    Occlusion Sensitivity 
# 8 GGC    Guided Grad-CAM

Now you can explain predictions using explain method. You will need:

paths to the images for which you want to generate explanations.
class indexes for which the explanations should be generated (optional, if set to NULL class that maximizes predicted probability will be found for each image).
character vector with method names (optional, by default explainer will use all methods).
batch size (optional, by default number of inserted images).
additional arguments with settings for a specific method (optional).

As an output you will get an object of class CNNexplanations:

input_imgs_paths <- list.files(system.file("extdata", "images", package = "sauron"), full.names = TRUE)

explanations <- explainer$explain(input_imgs_paths = input_imgs_paths,
                                  class_index = NULL,
                                  batch_size = 1,
                                  methods = c("V", "IG",  "GB", "GGC"),
                                  steps = 10, # Number of Integrated Gradients steps
                                  grayscale = FALSE # RGB or Gray gradients
)

explanations
# CNNexplanations object contains explanations for 3 images for 1 model.

You can get raw explanations and metadata from CNNexplanations object using:

explanations$get_metadata()
# $multimodel_explanations
# [1] FALSE
# 
# $ids
# [1] "imagenet_xception"
# 
# $n_models
# [1] 1
# 
# $target_sizes
# $target_sizes[[1]]
# [1] 299 299   3
# 
# 
# $methods
# [1] "V"   "IG"  "GB"  "GGC"
# 
# $input_imgs_paths
# [1] "/home/maju116/R/x86_64-pc-linux-gnu-library/4.0/sauron/extdata/images/cat_and_dog.jpg"
# [2] "/home/maju116/R/x86_64-pc-linux-gnu-library/4.0/sauron/extdata/images/cat.jpeg"       
# [3] "/home/maju116/R/x86_64-pc-linux-gnu-library/4.0/sauron/extdata/images/zebras.jpg"     
# 
# $n_imgs
# [1] 3

raw_explanations <- explanations$get_explanations()
str(raw_explanations)
# List of 1
#  $ imagenet_xception:List of 5
#   ..$ Input: num [1:3, 1:299, 1:299, 1:3] 147 134 170 147 134 168 144 134 170 144 ...
#   .. ..- attr(*, "dimnames")=List of 4
#   .. .. ..$ : NULL
#   .. .. ..$ : NULL
#   .. .. ..$ : NULL
#   .. .. ..$ : NULL
#   ..$ V    : int [1:3, 1:299, 1:299, 1:3] 0 0 0 0 0 0 0 0 0 0 ...
#   .. ..- attr(*, "dimnames")=List of 4
#   .. .. ..$ : NULL
#   .. .. ..$ : NULL
#   .. .. ..$ : NULL
#   .. .. ..$ : NULL
#   ..$ IG   : int [1:3, 1:299, 1:299, 1:3] 0 0 0 0 0 0 0 0 0 0 ...
#   .. ..- attr(*, "dimnames")=List of 4
#   .. .. ..$ : NULL
#   .. .. ..$ : NULL
#   .. .. ..$ : NULL
#   .. .. ..$ : NULL
#   ..$ GB   : int [1:3, 1:299, 1:299, 1:3] 0 0 2 0 0 111 0 0 28 0 ...
#   .. ..- attr(*, "dimnames")=List of 4
#   .. .. ..$ : NULL
#   .. .. ..$ : NULL
#   .. .. ..$ : NULL
#   .. .. ..$ : NULL
#   ..$ GGC  : num [1:3, 1:299, 1:299, 1] 7.13e-05 0.00 4.55e-04 7.13e-05 0.00 ...
#   .. ..- attr(*, "dimnames")=List of 4
#   .. .. ..$ : NULL
#   .. .. ..$ : NULL
#   .. .. ..$ : NULL
#   .. .. ..$ : NULL

To visualize and save generated explanations use:

explanations$plot_and_save(combine_plots = TRUE, # Show all explanations side by side on one image?
                           output_path = NULL, # Where to save output(s)
                           plot = TRUE # Should output be plotted?
)

If you want to compare two or more different models you can do it by combining CNNexplainer objects into CNNexplainers object:

model2 <- application_densenet121()
preprocessing_function2 <- densenet_preprocess_input

explainer2 <- CNNexplainer$new(model = model2,
                               preprocessing_function = preprocessing_function2,
                               id = "imagenet_densenet121")

model3 <- application_densenet201()
preprocessing_function3 <- densenet_preprocess_input

explainer3 <- CNNexplainer$new(model = model3,
                               preprocessing_function = preprocessing_function3,
                               id = "imagenet_densenet201")

explainers <- CNNexplainers$new(explainer, explainer2, explainer3)

explanations123 <- explainers$explain(input_imgs_paths = input_imgs_paths,
                                      class_index = NULL,
                                      batch_size = 1,
                                      methods = c("V", "IG",  "GB", "GGC"),
                                      steps = 10,
                                      grayscale = FALSE
)

explanations123$get_metadata()
# $multimodel_explanations
# [1] TRUE
# 
# $ids
# [1] "imagenet_xception"    "imagenet_densenet121" "imagenet_densenet201"
# 
# $n_models
# [1] 3
# 
# $target_sizes
# $target_sizes[[1]]
# [1] 299 299   3
# 
# $target_sizes[[2]]
# [1] 224 224   3
# 
# $target_sizes[[3]]
# [1] 224 224   3
# 
# 
# $methods
# [1] "V"   "IG"  "GB"  "GGC"
# 
# $input_imgs_paths
# [1] "/home/maju116/R/x86_64-pc-linux-gnu-library/4.0/sauron/extdata/images/cat_and_dog.jpg"
# [2] "/home/maju116/R/x86_64-pc-linux-gnu-library/4.0/sauron/extdata/images/cat.jpeg"       
# [3] "/home/maju116/R/x86_64-pc-linux-gnu-library/4.0/sauron/extdata/images/zebras.jpg"     
# 
# $n_imgs
# [1] 3

explanations123$plot_and_save(combine_plots = TRUE,
                              output_path = NULL,
                              plot = TRUE
)

Alternatively if you already have some CNNexplanations objects generated (for the same images and using same methods) you can combine them:

explanations2 <- explainer2$explain(input_imgs_paths = input_imgs_paths,
                                    class_index = NULL,
                                    batch_size = 1,
                                    methods = c("V", "IG",  "GB", "GGC"),
                                    steps = 10,
                                    grayscale = FALSE
)

explanations3 <- explainer3$explain(input_imgs_paths = input_imgs_paths,
                                    class_index = NULL,
                                    batch_size = 1,
                                    methods = c("V", "IG",  "GB", "GGC"),
                                    steps = 10,
                                    grayscale = FALSE
)

explanations$combine(explanations2, explanations3)

explanations$get_metadata()
# $multimodel_explanations
# [1] TRUE
# 
# $ids
# [1] "imagenet_xception"    "imagenet_densenet121" "imagenet_densenet201"
# 
# $n_models
# [1] 3
# 
# $target_sizes
# $target_sizes[[1]]
# [1] 299 299   3
# 
# $target_sizes[[2]]
# [1] 224 224   3
# 
# $target_sizes[[3]]
# [1] 224 224   3
# 
# 
# $methods
# [1] "V"   "IG"  "GB"  "GGC"
# 
# $input_imgs_paths
# [1] "/home/maju116/R/x86_64-pc-linux-gnu-library/4.0/sauron/extdata/images/cat_and_dog.jpg"
# [2] "/home/maju116/R/x86_64-pc-linux-gnu-library/4.0/sauron/extdata/images/cat.jpeg"       
# [3] "/home/maju116/R/x86_64-pc-linux-gnu-library/4.0/sauron/extdata/images/zebras.jpg"     
# 
# $n_imgs
# [1] 3

explanations$plot_and_save(combine_plots = TRUE,
                           output_path = NULL,
                           plot = TRUE
)

Sneak peek into 'sauron' package - XAI for Convolutional Neural Networks.

Sun, 08 Nov 2020 01:00:00 +0000

Explainable Artificial Intelligence, or XAI for short, is a set of tools that helps us understand and interpret complicated “black box” machine and deep learning models and their predictions. Today I would like to show you a sneak peek of my newest package called sauron, which allows you to explain decisions of Convolutional Neural Networks.

What exactly does CNN see?

Let’s start with the basic. We’re gonna need a model, test images for which we want to generate explanations and image preprocessing function (if needed).

library(sauron)

input_imgs_paths <- list.files(system.file("extdata", "images", package = "sauron"), full.names = TRUE)
model <- application_xception()
preprocessing_function <- xception_preprocess_input

There’s a ton of different methods to explain CNNs, but for now with sauron you have access to 6 gradient based ones. You can check full list using sauron_available_methods function:

sauron_available_methods
# # A tibble: 6 x 2
#   method name                  
#   <chr>  <chr>                 
# 1 V      Vanilla gradient      
# 2 GI     Gradient x Input      
# 3 SG     SmoothGrad            
# 4 SGI    SmoothGrad x Input    
# 5 IG     Integrated Gradients  
# 6 GB     Guided Backpropagation

Package is still in development so I won’t talk about the theory of those methods today. I will leave it for another post (or more probably multiple posts :smile: ).

To generate any set of explanations simply use generate_explanations function. Beside, images paths, model and optional preprocessing function you have to pass class indexes for which explanation should be made (NULL means select class with highest probability for this image), some method specific arguments and if you want to generate grayscale o RGB explanation maps.

explanations <- generate_explanations(
  model,
  input_imgs_paths,
  preprocessing_function,
  class_index = NULL,
  methods = sauron_available_methods$method,
  num_samples = 5, # SmoothGrad samples
  noise_sd = 0.1, # SmoothGrad noise standard divination
  steps = 10, # Integrated Gradients steps
  grayscale = FALSE)

Now we can plot our results:

plot_explanations(explanations, FALSE)
# $Input

# 
# $V

# 
# $GI

# 
# $SG

# 
# $SGI

# 
# $IG

# 
# $GB

# 
# $Input

# 
# $V

# 
# $GI

# 
# $SG

# 
# $SGI

# 
# $IG

# 
# $GB

What next?

First of all, as I said in the beginning, sauron is still in development and it should be available at the end of 2020. So if this topic is interesting for you be sure to visit my github from time to time.

Second of all I’m planning to expand sauron capabilities. The first step will be to add methods like: Grad-CAM, Guided Grad-CAM, Occlusion and Layer-wise Relevance Propagation (LRP).

Nucleci segmentation in R with Platypus.

Fri, 16 Oct 2020 01:00:00 +0000

In my previous post I’ve introduced you to my latest project platypus - R package for object detection and image segmentation. This time I will go into more details and show you how to use it on biomedical data.

The data

Today we will work on 2018 Data Science Bowl dataset. You can download images and masks directly form the url or using Kagge API :

kaggle competitions download -c data-science-bowl-2018

After downloading the data, unpack them and move to preferred destination. For this example we will be interested only in stage1_train and stage1_test subdirectories, so you can delete other files if you want.

Before we start, let’s investigate a little bit.

library(tidyverse)
library(platypus)
library(abind)
library(here)

# Print current working directory
here()

# [1] "/home/maju116/Desktop/PROJECTS/Moje Projekty/platypus"

# Set directories with the data and models
data_path <- here("examples/data/data-science-bowl-2018/")
models_path <- here("examples/models/")

# Investigate one instance of data (image + masks)
sample_image_path <- here("examples/data/data-science-bowl-2018/stage1_train/00071198d059ba7f5914a526d124d28e6d010c92466da21d4a04cd5413362552/")

list.files(sample_image_path, full.names = TRUE) %>%
  set_names(basename(.)) %>%
  map(~ list.files(.))

# $images
# [1] "00071198d059ba7f5914a526d124d28e6d010c92466da21d4a04cd5413362552.png"
# 
# $masks
#  [1] "07a9bf1d7594af2763c86e93f05d22c4d5181353c6d3ab30a345b908ffe5aadc.png"
#  [2] "0e548d0af63ab451616f082eb56bde13eb71f73dfda92a03fbe88ad42ebb4881.png"
#  [3] "0ea1f9e30124e4aef1407af239ff42fd6f5753c09b4c5cac5d08023c328d7f05.png"
#  [4] "0f5a3252d05ecdf453bdd5e6ad5322c454d8ec2d13ef0f0bf45a6f6db45b5639.png"
#  [5] "2c47735510ef91a11fde42b317829cee5fc04d05a797b90008803d7151951d58.png"
#  [6] "4afa39f2a05f9884a5ff030d678c6142379f99a5baaf4f1ba7835a639cb50751.png"
#  [7] "4bc58dbdefb2777392361d8b2d686b1cc14ca310e009b79763af46e853e6c6ac.png"
#  [8] "4e3b49fb14877b63704881a923365b68c1def111c58f23c66daa49fef4b632bf.png"
#  [9] "5522143fa8723b66b1e0b25331047e6ae6eeec664f7c8abeba687e0de0f9060a.png"
# [10] "58656859fb9c13741eda9bc753c3415b78d1135ee852a194944dee88ab70acf4.png"
# [11] "6442251746caac8fc255e6a22b41282ffcfabebadbd240ee0b604808ff9e3383.png"
# [12] "7ff04129f8b6d9aaf47e062eadce8b3fcff8b4a29ec5ad92bca926ac2b7263d2.png"
# [13] "8bbec3052bcec900455e8c7728d03facb46c880334bcc4fb0d1d066dd6c7c5d2.png"
# [14] "9576fe25f4a510f12eecbabfa2e0237b98d8c2622b9e13b9a960e2afe6da844e.png"
# [15] "95deddb72b845b1a1f81a282c86e666045da98344eaa2763d67e2ab80bc2e5c3.png"
# [16] "a1b0cdb21f341af17d86f23596df4f02a6b9c4e0d59a7f74aaf28b9e408a4e4c.png"
# [17] "aa154c70e0d82669e9e492309bd00536d2b0f6eeec1210014bbafbfc554b377c.png"
# [18] "acba6646e8250aab8865cd652dfaa7c56f643267ea2e774aee97dc2342d879d6.png"
# [19] "ae00049dc36a1e5ffafcdeadb44b18a9cd6dfd459ee302ab041337529bd41cf2.png"
# [20] "af4d6ff17fa7b41de146402e12b3bab1f1fe3c1e6f37da81a54e002168b1e7dd.png"
# [21] "b0cbc2c553f9c4ac2191395236f776143fb3a28fb77b81d3d258a2f45361ca89.png"
# [22] "b6fc3b5403de8f393ca368553566eaf03d5c07148539bc6141a486f1d185f677.png"
# [23] "be98de8a7ba7d5d733b1212ae957f37b5b69d0bf350b9a5a25ba4346c29e49f7.png"
# [24] "cb53899ef711bce04b209829c61958abdb50aa759f3f896eb7ed868021c22fb4.png"
# [25] "d5024b272cb39f9ef2753e2f31344f42dd17c0e2311c4927946bc5008d295d2e.png"
# [26] "f6eee5c69f54807923de1ceb1097fc3aa902a6b20d846f111e806988a4269ed0.png"
# [27] "ffae764df84788e8047c0942f55676c9663209f65da943814c6b3aca78d8e7f7.png"

As you can see each image has its own directory, that has two subdirectories inside:

images - contains original image that will be the input of the neural network
masks - contains one or more segmentation masks. Segmentation mask is simply telling us which pixel belongs to which class, and this is what we will try to predict.

For the modeling, beside train and test sets, we will also need a validation set (No one is forcing you, but it’s a good practice!):

train_path <- here("examples/data/data-science-bowl-2018/stage1_train/")
test_path <- here("examples/data/data-science-bowl-2018/stage1_test/")
validation_path <- here("examples/data/data-science-bowl-2018/stage1_validation/")

if (!dir.exists(validation_path)) {
  dir.create(validation_path)
  # List train images
  train_samples <- list.files(train_path, full.names = TRUE)
  set.seed(1234)
  # Select 10% for validation
  validation_samples <- sample(train_samples, round(0.1 * length(train_samples)))
  validation_samples %>%
    walk(~ system(paste0('mv "', ., '" "', validation_path, '"')))
}

Semantic segmentation with U-Net

Since we now something about our data, we can now move to the modeling part. We will start by selecting the architecture of the neural network. In case of semantic segmentation there is a few different choices like U-Net, Fast-FCN, DeepLab and many more. For the time being in the platypus package you have access only to the U-Net architecture.

U-Net was originally developed for biomedical data segmentation. As you can see in the picture above architecture is very similar to autoencoder and it looks like the letter U, hence the name. Model is composed of 2 parts, and each part has some number of convolutional blocks (3 in the image above). Number of blocks will be hyperparameter in our model.

To build a U-Net model in platypus use u_net function. You have to specify:

number of convolutional blocks,
input image height and width - must be in the form 2^N!,
will input image be loaded as grayscale or RGB,
number of classes - in our case we have only 2 (background and nuclei)
additional arguments form CNN like number of filters, dropout rate

blocks <- 4 # Number of U-Net convolutional blocks
n_class <- 2 # Number of classes
net_h <- 256 # Must be in a form of 2^N
net_w <- 256 # Must be in a form of 2^N
grayscale <- FALSE # Will input image be in grayscale or RGB

DCB2018_u_net <- u_net(
  net_h = net_h,
  net_w = net_w,
  grayscale = grayscale,
  blocks = blocks,
  n_class = n_class,
  filters = 16,
  dropout = 0.1,
  batch_normalization = TRUE,
  kernel_initializer = "he_normal"
)

After that it’s time to select loss and additional metrics. Because semantic segmentation is in essence classification for each pixel instead of the whole image, you can use categorical cross-entropy as a loss function and accuracy as a metric. Other common choice, available in platypus, would be dice coefficient/loss. You can think of it as of a F1-metric for semantic segmentation.

DCB2018_u_net %>%
  compile(
    optimizer = optimizer_adam(lr = 1e-3),
    loss = loss_dice(),
    metrics = metric_dice_coeff()
  )

The next step will be data ingestion. As you remember we have a separate directory and multiple masks for each image. That’s not a problem for platypus! You can ingest data using segmentation_generator function. The first argument to specify is the directory with all the images and masks. To tell platypus that it has to load images and masks from separate directories for each data sample specify argument mode = "nested_dirs". Additionally you can set images/masks subdirectories names using subdirs argument. platypus will automatically merge multiple masks for each image, but we have to tell him how to recognize which pixel belongs to which class. In the segmentation masks each class is recognized by a specific RGB value. In our case we have only black (R = 0, G = 0, B = 0) pixel for background and white (R = 255, G = 255, B = 255) pixels for nuclei. To tell platypus how to recognize classes on segmentation masks use colormap argument.

binary_colormap

# [[1]]
# [1] 0 0 0
# 
# [[2]]
# [1] 255 255 255

train_DCB2018_generator <- segmentation_generator(
  path = train_path, # directory with images and masks
  mode = "nested_dirs", # Each image with masks in separate folder
  colormap = binary_colormap,
  only_images = FALSE,
  net_h = net_h,
  net_w = net_w,
  grayscale = FALSE,
  scale = 1 / 255,
  batch_size = 32,
  shuffle = TRUE,
  subdirs = c("images", "masks") # Names of subdirs with images and masks
)

# 603 images with corresponding masks detected!
# Set 'steps_per_epoch' to: 19

validation_DCB2018_generator <- segmentation_generator(
  path = validation_path, # directory with images and masks
  mode = "nested_dirs", # Each image with masks in separate folder
  colormap = binary_colormap,
  only_images = FALSE,
  net_h = net_h,
  net_w = net_w,
  grayscale = FALSE,
  scale = 1 / 255,
  batch_size = 32,
  shuffle = TRUE,
  subdirs = c("images", "masks") # Names of subdirs with images and masks
)

# 67 images with corresponding masks detected!
# Set 'steps_per_epoch' to: 3

We can now fit the model.

history <- DCB2018_u_net %>%
  fit_generator(
    train_DCB2018_generator,
    epochs = 20,
    steps_per_epoch = 19,
    validation_data = validation_DCB2018_generator,
    validation_steps = 3,
    callbacks = list(callback_model_checkpoint(
      filepath = file.path(models_path, "DSB2018_w.hdf5"),
      save_best_only = TRUE,
      save_weights_only = TRUE,
      monitor = "dice_coeff",
      mode = "max",
      verbose = 1)
    )
  )

And calculate predictions for the new images. Our model will return a 4-dimensional array (number of images, height, width, number of classes). Each pixel will have N probabilities, where N is number of classes. To transform raw predictions into segmentation map (by selecting class with max probability for each pixel) you can use get_masks function.

test_DCB2018_generator <- segmentation_generator(
  path = test_path,
  mode = "nested_dirs",
  colormap = binary_colormap,
  only_images = TRUE,
  net_h = net_h,
  net_w = net_w,
  grayscale = FALSE,
  scale = 1 / 255,
  batch_size = 32,
  shuffle = FALSE,
  subdirs = c("images", "masks")
)

# 65 images detected!
# Set 'steps_per_epoch' to: 3

test_preds <- predict_generator(DCB2018_u_net, test_DCB2018_generator, 3)
dim(test_preds)

# [1]  65 256 256   2

test_masks <- get_masks(test_preds, binary_colormap)
dim(test_masks[[1]])

# [1] 256 256   3

To visualize predicted masks with the original images you can use plot_masks function.

test_imgs_paths <- create_images_masks_paths(test_path, "nested_dirs", FALSE, c("images", "masks"), ";")$images_paths

plot_masks(
  images_paths = test_imgs_paths[1:4],
  masks = test_masks[1:4],
  labels = c("background", "nuclei"),
  colormap = binary_colormap
)

Platypus - R package for object detection and image segmentation.

Tue, 06 Oct 2020 01:00:00 +0000

Computer Vision tasks

Computer Vision, or CV for short, is a field of computer science focused on development of techniques that will help computers understand the content of digital images in a way similar to human understanding. There’s a lot of different computer vision tasks, but today I want to focus only on four basic ones.

Basic Computer Vision tasks:

Image classification - in this task we want to compute the probability (or probabilities) that the input image is in a particular class. It could be performed with Convolutional Neural Networks using keras package.
Semantic segmentation - very similar to image classification, but instead of classifying the whole image, we want to classify each pixel of this image. Note that we are not saying anything about location of the object.
Object detection - we want to classify and locate objects on the input image. Object localization is typically indicated by specifying a tightly cropped bounding box.
Instance segmentation - it’s a combination of semantic segmentation and object detection. Like in semantic segmentation we want to classify each pixel to a different class, but we also want to distinguish between different objects of the same class.

Platypus

platypus is an R package for object detection and semantic segmentation. Currently using platypus you can perform:

multi-class semantic segmentation using U-Net architecture
multi-class object detection using YOLOv3 architecture

You can install the latest version of platypus with remotes package:

remotes::install_github("maju116/platypus")

Note that in order to install platypus you need to install keras and tensorflow packages and Tensorflow version >= 2.0.0 (Tensorflow 1.x will not be supported!)

Quick example: YOLOv3 bounding box prediction with pre-trained COCO weights

To create YOLOv3 architecture use:

library(tidyverse)
library(platypus)
library(abind)

test_yolo <- yolo3(
  net_h = 416, # Input image height. Must be divisible by 32
  net_w = 416, # Input image width. Must be divisible by 32
  grayscale = FALSE, # Should images be loaded as grayscale or RGB
  n_class = 80, # Number of object classes (80 for COCO dataset)
  anchors = coco_anchors # Anchor boxes
)

test_yolo
#> Model
#> Model: "yolo3"
#> ________________________________________________________________________________
#> Layer (type)              Output Shape      Param #  Connected to               
#> ================================================================================
#> input_img (InputLayer)    [(None, 416, 416, 0                                   
#> ________________________________________________________________________________
#> darknet53 (Model)         multiple          40620640 input_img[0][0]            
#> ________________________________________________________________________________
#> yolo3_conv1 (Model)       (None, 13, 13, 51 11024384 darknet53[1][2]            
#> ________________________________________________________________________________
#> yolo3_conv2 (Model)       (None, 26, 26, 25 2957312  yolo3_conv1[1][0]          
#>                                                      darknet53[1][1]            
#> ________________________________________________________________________________
#> yolo3_conv3 (Model)       (None, 52, 52, 12 741376   yolo3_conv2[1][0]          
#>                                                      darknet53[1][0]            
#> ________________________________________________________________________________
#> grid1 (Model)             (None, 13, 13, 3, 4984063  yolo3_conv1[1][0]          
#> ________________________________________________________________________________
#> grid2 (Model)             (None, 26, 26, 3, 1312511  yolo3_conv2[1][0]          
#> ________________________________________________________________________________
#> grid3 (Model)             (None, 52, 52, 3, 361471   yolo3_conv3[1][0]          
#> ================================================================================
#> Total params: 62,001,757
#> Trainable params: 61,949,149
#> Non-trainable params: 52,608
#> ________________________________________________________________________________

You can now load YOLOv3 Darknet weights trained on COCO dataset. Download pre-trained weights from here and run:

test_yolo %>% load_darknet_weights("yolov3.weights")

Calculate predictions for new images:

test_img_paths <- list.files(system.file("extdata", "images", package = "platypus"), full.names = TRUE, pattern = "coco")
test_imgs <- test_img_paths %>%
  map(~ {
    image_load(., target_size = c(416, 416), grayscale = FALSE) %>%
      image_to_array() %>%
      `/`(255)
  }) %>%
  abind(along = 4) %>%
  aperm(c(4, 1:3))
test_preds <- test_yolo %>% predict(test_imgs)

str(test_preds)
#> List of 3
#>  $ : num [1:2, 1:13, 1:13, 1:3, 1:85] 0.294 0.478 0.371 1.459 0.421 ...
#>  $ : num [1:2, 1:26, 1:26, 1:3, 1:85] -0.214 1.093 -0.092 2.034 -0.286 ...
#>  $ : num [1:2, 1:52, 1:52, 1:3, 1:85] 0.242 -0.751 0.638 -2.419 -0.282 ...

Transform raw predictions into bounding boxes:

test_boxes <- get_boxes(
  preds = test_preds, # Raw predictions form YOLOv3 model
  anchors = coco_anchors, # Anchor boxes
  labels = coco_labels, # Class labels
  obj_threshold = 0.6, # Object threshold
  nms = TRUE, # Should non-max suppression be applied
  nms_threshold = 0.6, # Non-max suppression threshold
  correct_hw = FALSE # Should height and width of bounding boxes be corrected to image height and width
)

test_boxes
#> [[1]]
#> # A tibble: 8 x 7
#>    xmin  ymin  xmax  ymax p_obj label_id label 
#>   <dbl> <dbl> <dbl> <dbl> <dbl>    <int> <chr> 
#> 1 0.207 0.718 0.236 0.865 0.951        1 person
#> 2 0.812 0.758 0.846 0.868 0.959        1 person
#> 3 0.349 0.702 0.492 0.884 1.00         3 car   
#> 4 0.484 0.543 0.498 0.558 0.837        3 car   
#> 5 0.502 0.543 0.515 0.556 0.821        3 car   
#> 6 0.439 0.604 0.469 0.643 0.842        3 car   
#> 7 0.541 0.554 0.667 0.809 0.999        6 bus   
#> 8 0.534 0.570 0.675 0.819 0.954        7 train 
#> 
#> [[2]]
#> # A tibble: 3 x 7
#>     xmin   ymin  xmax  ymax p_obj label_id label
#>    <dbl>  <dbl> <dbl> <dbl> <dbl>    <int> <chr>
#> 1 0.0236 0.0705 0.454 0.909 1.00        23 zebra
#> 2 0.290  0.206  0.729 0.901 0.997       23 zebra
#> 3 0.486  0.407  0.848 0.928 1.00        23 zebra

Plot / save images:

plot_boxes(
  images_paths = test_img_paths, # Images paths
  boxes = test_boxes, # Bounding boxes
  correct_hw = TRUE, # Should height and width of bounding boxes be corrected to image height and width
  labels = coco_labels # Class labels
)

For more details and examples on custom datasets visit platypus page and check out the latest posts.