All detectors are accessible from “Detectors“tab, in the top bar.
You can create a new detector by clicking on the blue “Create a Detector” button, on the top left corner of the Detectors page.
This action will prompt a pop-up window where you can name your detector and select the detection type.
Depending on what you would like to detect, you can create detectors in Count or Segmentation mode:
Additionally, you can also select Change mode to exclusively identify changes over time, within the same geographical location, such as the appearance or disappearance of objects or changes in land cover.
Does the class you are looking after has generally a defined size and/or boundaries?
You should be in “Count” mode if your goal for your detector is to detect and count individual objects. If you imagine yourself as the AI, your goal would be to point at all individual instances of your object with your finger. In machine learning terminology, this mode is the same as “object detection”. However, you will still get the proper mask segment of the object as output (all pixels making the object). If you want a simplified output, such as “oriented bounding boxes” check out the “Output Shape” setting.
You should be in “Segmentation” mode if your goal is more to map areas/regions or in imagery terms, textures. You can imagine the AI’s interaction as brushing or painting over areas of interest that you want it to find.
Finally, if you are primarily interested in identifying and detecting only changes happening between two images / points in time, you can choose to create your detector in change mode and to train a dedicated change detector. Learn more about change detector.
Detecting individual trees or buildings is Count
Mapping dense forest canopy coverage in terms of area is Segmentation, as mapping the presence of waterbodies in mines.
Monitoring deforestation using Change detection.
There are some ambiguous cases where either of the two could be used, but the main thing to remember is that Count does a much better job at separating objects that are potentially close together and that separation is required in order to get an accurate count.
Also in Count, the model needs to see the whole object (e.g. an entire building if counting individual buildings) while in Segmentation the model can see only portion of the pattern of interest (e.g. part of a roof if segmenting roof type materials) So basically it comes down to which metric you are more interested in finding (a count or an area).
In this case with classes having a mix of well defined objects boundaries (e.g. buildings or cars) and open land covers (e.g. tree or road coverage), we suggest to go with “Segmentation” mode to handle the latter.
Note: choosing between count or segmentation does not change the overall model under the hood being still deep learning exploiting convolutional neural networks. It adapts the model architecture mainly towards its last layers and its loss function to better handle the separation of objects. This also involves a speed difference when training or running a detector in Count (longer) or Segmentation (faster) mode.
From the list, you can see some of the info of a detector.
This opens a popup, which contains details about your detectors, like the sharing status, creator and id (useful for interaction via the Public API), but also some editable fields, such as the name and the tag (beta feature, which allows to filter detectors when listing them via the Public API).
The Detector Settings enable you to finely adjust and optimize your detectors to meet specific project requirements and use cases beyond just adding more annotations. See sub-pages below for details on each detector setting available.
Basic settings:
Your detector’s resolution specifies the spatial resolution that all training images in your detector will be automatically scaled to when both training and detecting. This is to ensure object and texture size consistency (in pixels) in both your data set and project imagery for better results. When the image resolutions are changed they are simply scaled (like in a simple image editing software). It does not add new details or information to your imagery (a blurry object will remaining a blurry object after scaling). Therefore, it is usually recommended that you only train on and run your detector on, resolutions that are very close to its resolution setting.
When you first create your detector, the resolution will be set the resolution of the first image you add to your detector. It can be then changed in the detector settings under the ‘Spatial Resolution” setting later on if necessary.
We recommend that you use training images that have similar spatial resolutions. You can then set your detector spatial resolution to the average resolution of your training images.
It can sometimes be helpful to change the detector resolution further, for example if the objects you are detecting are either:
Example
We provide here an example to give an idea of the target resolution setting effects:
The detector when observing the images looks at them by dividing each image into smaller tiles, and two undesired scenarios can happen.
Too high-resolution images
Ideally in a tile, you should be able to see your entire object, plus some of its surroundings. If it’s not the case you can use the resolution detector settings to downscale your images so that your objects fit better.
By downscaling our imagery (so setting a lower spatial resolution setting) we see the following, which is much more understandable by the model:
Too low-resolution images
You may also have the opposite problem where your objects are too small (for example 2px by 2px objects are quite challenging for the detector to pick up). Especially if these small objects are adjacent to each other the problem becomes even more challenging for the detector to separate them. You may want to consider upscaling if that is the case using the detector resolution settings. Currently upscaling is limited to 2x.
With the native resolution, we get the results on the right:
Now we can scale up the resolution by 2x to get the following results, much better:
In order to see these kinds of images applied to your case, you can click on the Training Report icon (green icon) to be found in the top bar.
In this way, you can judge if your objects are too big or too small for the detector and consequently adjust the image resolution.
Anything else you should know?
There are two possibilities for your Detector Type, Count and Segmentation.
There are currently two output shape options available in Picterra:
Note that in either case you can visually your the centers of your results outside of the detector settings in your results view.
The size filter simply filters the results by a specified area (in square meters).
There are 3 different settings
Note that Auto is only available in Count mode.
The training time (or steps) is a more advanced machine learning parameter.
When a machine learning model trains it is sending in batches of image/annotation information to the detector in an iterative manner, one batch at a time, where each batch is a small subset of your full dataset. The number of training steps is basically the number of batches being sent to the model.
If your dataset is large (more annotations, more training areas), it means you should use more training steps so that your detector has the chance to be fully exposed to your entire dataset. By default we set the number of steps to 500 but as a rough rule of thumb, if you surpass 200 annotations you should consider adding more steps. You may also want to add more steps if you have more classes, (we will change this setting automatically as you add new classes)
In the future we will be working on a way to better decide an optimal number of training steps, but for the moment you will have to do a bit of experimentation!
Advanced settings:
Whenever you train a detector you are feeding your training areas to a deep learning model one after the other. These models need a fixed image size to be passed into them, which is just the nature of how these models work in training. Let’s call this fixed size image a “tile”. For us the default tile is a 256 x 256 sample of one of your training areas, which is quite standard in the machine learning computer vision field. If your training areas are larger than this, a single tile will only encompass a subpart of that area. This is fine because we will select a tile from each of your training areas multiple times during training and each time we randomly sample the location of the tile in your training area, the idea being that eventually after multiple training steps all training area contents will be seen.
Training tiles as seen in the training report
However, sometimes this tile size is simply too small. Say your object is too large to fit into a 256 x 256 tile and the full object is never seen in one tile by your model. This is like trying to learn what a human face looks like but you’ve only ever seen separately a nose, or an eye, or a mouth, but never all of it together. It can be achieved, but it makes the task harder. The way we’d get around this problem with Picterra previously was to use the detector resolution setting to scale down your images, which can work fine in many cases but in doing so you also lose detail in your image. If that detail is important in producing correct detections (which depends on your use case), then it may not be such a good idea to scale down that imagery. However, if we just increase the tile size then perhaps we can fit the entire object in a single tile without losing detail, and that’s what the tile size setting does. instead of scaling your imagery down by two, you can increase the tile size from 256 x 256 to 512 x 512 for example.
Here is an example dataset of where this setting has helped in improving the results, producing a full 5% increase in performance (from 63% to 68%) on a difficult and complex problem.
Missing some paving on the left as well as a falsely detected rooftop. Improved results on the right (no rooftop detection at all).
Falsely detected road on the left, correct on the right.
Despite not looking for a specific object (this is a segmentation detector), the tile size is still helpful here because the “paving” in here is defined as basically being not the main road for vehicles, which means the class contains multiple different appearances and textures. Of course there are still rules in what materials are used for which and so scaling the imagery down can be detrimental because you lose detail of the texture of the materials.
However, a major factor in how we as humans would determine whether or not something is pavement is not just by its material but also its surrounding context.
We can tell something is a sidewalk because it’s a narrow linear element next to a larger wider linear element, the road. Since the context is important, having more of it is helpful. So how can we increase the context while still maintaining the texture detail? By increasing the tile size. This dataset in particular could also use more data, and perhaps multiple classes, to improve the score further, but that’s a different story.
It sounds great on paper but of course there are a few drawbacks to be aware of. It may not always perform better as expected. Also it may not even be the details that are important for the model to learn what your object is anyways (in the end it’s not us who decides what details in the images to use during training, it’s the model’s responsibility to do that automatically). As mentioned earlier, it really depends on your use case. In addition increasing the tile size means increasing the GPU memory of your model. A bump from 256 x 256 to 512 x 512 means 4x the GPU memory. This means your training will also take 4x longer. In short this setting can be quite helpful but it’s important to experiment and play around with it to see if it can be of benefit to your detector.
This setting involves the feature extractor portion of the network. Deep learning models have a start and an end. The start is where you provide the input to the model (the images) and the end is where you get the output (the detections). At the start of the network there is a sub-network dedicated to extracting features from your imagery. Let’s not confuse this with the word “feature” used in GIS lingo, which refers typically to actual objects or elements in your image. A feature here is basically a list of numbers per pixel that encodes some kind of information about the image, they are defining characteristics of your images and objects of interest, but turned into something less qualitative and more quantitative. It’s using these features that the rest of the deep learning model decides whether or not a pixel belongs to your object of interest or not and from that can produce detections.
The backbone model is exactly this first feature extractor sub network and it can be helpful sometimes to swap it out with different sub networks. Our default backbone is Resnet34. There are variations of this backbone model that we can use as well, like Resnet18 and Resnet50, the latter being a lighter faster to train model meant for smaller datasets, whereas the former being heavier, slightly longer to train, but also better for more complex scenarios and larger datasets. It really shines during multiclass applications where your detector needs a bit more bandwidth to help separate similar looking objects into many different classes.
Here is an applicable case where we have a 10 class dataset on planes, where indeed some of the classes can look quite similar. We trained the detector on both Resnet34 (the default) and Resnet50. Let’s compare some of the mistakes that the Resnet34 version made to the corrected results using the Resnet50 backbone.
Resnet34 on the left, these should all be the same class of plane. On the right we see results from switching to Resnet50, where the model was able to better learn the difference between similar plane types and correctly classify all the planes in this area.
These planes should all be classified into the “cyan” class. On the left with Resnet34 you see that it mistakenly classified most of them into the “yellow” class which was a plane that was similar in size but a different wing angle. On the right, you see that with the Resnet50 model most of them managed to be corrected.
In the future we may consider adding more different varieties of backbone models, such as Efficient Nets but these will require further testing to see whether or not they are actually worth adding in terms of potential performance gains. What we’re trying to avoid here is having a long laundry list of backbones just for the sake of having them.
Background sampling ratio defines how many tiles per batch (always set to 32) are dedicated to sampling over background. What does this mean exactly? In the FAQ about the training tile setting we explained that we randomly sample tiles from within your training areas. Some tiles or regions of tiles will contain fewer annotations and more background than others, or perhaps no annotations at all. It’s important to strike a balance between how many foreground and background samples you have to make sure your detector sees a proper amount of each. This “proper” amount however is something that can vary from use case to use case. It is highly dependent on the complexity of your imagery and what your detector is having difficulty with. For example, if you are having trouble getting a lot of false positives (bad precision) maybe you want your detector to sample more background regions to make it understand better that these regions are really just “background” and shouldn’t be detected. Or perhaps you’re having the opposite problem with false negatives (bad recall), then maybe you want to decrease this value so that you see more “foreground” (or annotated) pixels. As before it’s not an exact science, but something to experiment with if you see a clear trend in how your detector is outputting results or a large difference in the precision and recall scores that’d you’d like to bring closer together.
In this next example the client was attempting to detect small bodies of water in a mountainous region (the alps) from low resolution satellite infrared imagery. The bodies of water themselves are not particularly complex objects. They basically just look like dark blue blobs. But we need to make sure that we don’t falsely detect any other “dark-ish” looking blobs as lakes. Comparing the results of the background sampling ratio at 0.1 to 0.5, we can see a clear difference in results.
Using a 0.1 background sampling ratio
Using a 0.5 background sampling ratio
We can see that the IOU score got a pretty big bump, but in addition the precision also had an increase of about 5%, which means we probably have less false positive detections. Comparing the results visually we do indeed see examples of this.
False positive detections on the left using a background sampling ratio of 0.1, not present on the right where we use 0.5.
False positive detections on the left using a background sampling ratio of 0.1, not present on the right where we use 0.5.
When your detector outputs your results it first creates a raw pixel mask of your output and then finds turns that mask into a polygon by specifying which vertices are needed to sufficiently outline that mask.
There is a bit of flexibility here with how closely we match the mask shape. If we match it too closely your polygon might have way more vertices than you need, making it very difficult to edit later and possibly difficult to even load in your browser if the polygon is very large. We could also simplify the polygon a lot but then you might lose detail in the shape of your output which is also not always desirable if you for example care about the area coverage of your output.
Therefore we have introduced an advanced setting called “simplification” that allows you to choose which level of polygon simplification to apply to your results. The “Normal” setting is what your detectors have been using in the past for simplification. There are also “Low“, “High” and “Very high” settings.
The “Low setting” for example could be useful if you have high resolution imagery of tree tops and you want to accurately map the canopy area coverage of each individual tree.
|
|
The higher settings could be used for example for road mapping where you don’t care that much about the area of coverage of the road but rather the overall path. This then makes it much easier to edit the road later if the detector missed regions as there will be fewer vertices to modify.
|
|
Training a detector is an iterative process, influenced by the nature of your project, resolution of your imagery, clarity of the object of interest, and accuracy level you aim to achieve. Here are the basic steps to follow in order to create an efficient detector:
Please note:
Creating high-quality and robust detectors is essential for achieving accurate results across the whole lifecycle of your geospatial analysis projects. The Picterra platform offers powerful features to help you accomplish this.
Here are some best practices and recommendations to consider when using these tools:
The model tries to make sure it outputs distinct, separate objects, but depending on how crisp the edges in the imagery itself are, it may still make some mistakes here and there and merge nearby objects. If your detector is having issues with this, you can additionally try to inset your annotations from the edge of the objects. Your detector will try to learn to output the insetted part of your objects, so this may not be the best choice if you also care about the area of coverage of your objects but if you just care about the count, this is a valid strategy.
To ensure your detector functions correctly, it’s crucial to review your training areas and verify that all outlines are complete and accurate. Missing annotations can significantly impair your detector’s performance.
Additionally, outline the entire object rather than just a portion when annotating. Incomplete annotations can be as problematic as missing outlines, as your detector might interpret some valid pixels as non-annotated. For example, ensure you outline all houses, not just a few. Or contour the whole tree crown, rather than just roughly circling it, if you are in segmentation mode and interested in retrieving the tree crown area.
Learn more about creating annotations
Ensure a balanced set of annotations between your different classes to improve the performance and accuracy of your detector. An unbalanced number of annotations can be spotted in the training report’s “Dataset statistics” section. Additionally, quality is more important than quantityquality. Adding fewer examples but of many different appearances of your class in varying backgrounds is much more helpful than annotating any instances of only a few appearance types.
The background is considered as a class.
What do we mean by background? Anything that is not part of the outlines but still included in the training area. But don’t forget not to put too much. You can check its impact on the training report and check the number of times your annotations have been seen by the model. Two hyperparameters will impact the portion of your background seen by the model. The first one is the background ratio, which, if you increase, will reduce the number of annotations and increase the number of backgrounds during training. The second is the number of training steps. By increasing it, your model will see more and more background.
Learn more about best practices to add annotations
Ensure that objects are well-contained within the training areas and visible at the end of the training report. If objects are not entirely visible, consider downscaling the detector resolution or increasing the tile size in the advanced detector settings. This does not apply to land segmentation, where only portions of land of interest are annotated.
Learn more about training areas
Regularly reviewing training warnings will help you maintain a high-quality set of annotations and point you to potential false positives (wrong detections) or false negatives (missed detections). In the first case, consider adding empty training areas over some of these regions where false positives occur to refine your detector. In the second case, consider adding new training areas and annotations over missed detections to improve the detector recall. Training warnings can also help you find errors in your annotations, such as overlapping annotations in count mode or training areas and accuracy areas that overlap.
Learn more about training Warnings
Use the dataset recommendation report to auto-generate markers pointing to regions where your dataset could have more training coverage. Wherever a marker is recommended, you can consider adding a training/accuracy area centered around this marker.
Following this process will allow you to diversify training and accuracy areas on a variety of backgrounds while avoiding redundancy, ensuring that your training dataset is representative of different conditions without being too large.
Additional annotations and training areas are needed if your detector lacks recall and misses too many objects. Follow the dataset recommendations markers for optimal results. You may also check that you have not bumped your training steps (detector settings) too much, the default being 500 steps.
Learn more about Dataset recommendations
Increasing the background sampling ratio in the detector advanced settings may help reduce false positives if your detector lacks precision. Suppose you need to refine your object class because the detector lacks precision when detecting similar objects. In that case, a strategy is to create a second class for those objects, forcing the detector to specialize in discriminating between these two besides the background. Be aware that you will still need a certain amount of annotations, balanced with the first class, for this strategy to work.
Adjusting the resolution can significantly improve your model’s performance. Sometimes, your initial resolution might be too high for your model’s capabilities. For example, detecting houses with a spatial resolution set at 1 centimeter would be impractical. Imagine the model’s perspective: at 1 cm resolution, it only sees a small portion of roofs and walls in a 256×256 pixel area, approximately 5 square meters. This limited view makes it challenging to understand the complete shape of a building or its surroundings.
In contrast, at 50 cm resolution, the same area shows a 128×128 meter section, capturing more roof variations, background details, and neighboring buildings. Thus, reducing your training sample’s resolution from 1 cm to 50 cm enhances context and can significantly improve your model’s ability to accurately predict houses.
You can monitor what the model sees during training in the training report to ensure your detector’s resolution is appropriately set.
You may find that some detectors are having trouble differentiating one object for another. They may be less precise because there is a similar-looking common object that they are mistaking for your class. One thing that could help is to add another class for these misleading objects to make the detector more explicitly learn the difference between them and your desired class. This will affect your detector score since it averages the score between classes, but you can look at the training report for individual class scores.
Confidence maps of your accuracy areas are helpful for understanding where your detector may be having issues. The bright yellow areas indicate where your detector may be confused and unsure about its output. This could help debug issues like class confusion, resolution issues, or difficult objects and textures. Note that it’s likely impossible to eliminate all confidence in your detector, but visualizing these maps may give you some hints on how to reduce it. Also, note that the edges of your objects, by definition, will always be uncertain as that is the boundary between where your objects transition into the background; you can ignore these bright yellow outlines in the confidence map.
Learn more about confidence map
Use the training area sampling table for class balance and training steps.
The training area sampling table in the training report tells you roughly how many times a class is seen during the training process. You can use this either to see whether or not some classes are being seen more than others due to class balance issues or if you see that not all classes are being seen more than ten times at least, you should increase the number of training steps so that the detector can learn from all your annotations an adequate number of times.
The training toolbar and training sidebar allow you to create annotations and areas used by the machine learning model, to learn from.
In the Training toolbar on the left, you’ll find the commands to add annotations (Drawing tools and Magic wand) and create areas (Training, Testing and Accuracy areas). It will allow you to import polygons (Import) and add comments (Markers) directly on your dataset.
The sidebar on the right, allows you to manage your layers (Classes), comments (Markers), and provides you insights for improving your model (Warnings and Dataset recommendations).
Multi-class extends the detector building process in Picterra Foirge by enabling the annotation of multiple types of objects/segmentations within a single detector. Unlike the traditional method of creating individual detector for each class, multi-class functionality empowers users to include diverse object or segmentation types within a unified workflow.
In addition to workflow improvements it also produces both cleaner (no overlapping detections between classes) and better overall results in terms of accuracy, as the detector can learn to better differentiate between similar looking classes.
For a more in-depth guide with visuals, you can read about it in our multiclass blog post.
Note that the way the model will be fed with training patch of images during the training steps is by sampling uniformly across the classes. Therefore a less represented class (in terms of pixels covered) will be more sampled than the others. The training report shows the “Training area sampling stats” which will allow you to see how the sampling performed and which classes were seen during training (more info about training report). Finally, the advanced detector setting of “Background Sample Ratio” remains valid in controlling how much of pure background tiles vs. foreground (classes) are seen during training. (more info about Advanced Detector Settings)
Picterra Forge provides two pathways for change detection:
The Detector Insights tools are crafted to streamline the iterative process of refining and enhancing your models, ensuring highest quality results. You can access a range of advanced Detector Insights tools conveniently located in the menu panel of the Detector Training UI.
Localized training warnings are designed to offer you precise feedback on potential issues or mistakes in your dataset. Rather than sifting through tons of data to locate a small discrepancy, these warnings will directly point you to the problematic areas. Find out more about warnings
Please note that there are also some warnings in the training report but these are not localized on your imagery.
What is dataset recommendation?
Dataset recommendation is a tool that helps you during the training phases of your detector. It generates markers similar to the markers you can create yourself. However, these auto-generated Recommendations markers denote regions where your dataset could use more coverage. Wherever there is a recommendation marker, you will want to consider adding a training/accuracy area centered around this marker.
Another valuable tool for identifying areas where model improvement may be needed is the confidence map.
The confidence map displays a heat map overlay onto the images, with brighter colors indicating higher uncertainty levels in the model’s detections. This could represent detections the model is unsure about or ones it almost detected but lacked confidence to output. By comparing these heat maps to the results, you can deduce where your detector may be encountering confusion.
How to activate a confidence map? To activate a confidence map, simply click on a button in the area panel for an immediate view of the model’s confidence levels in these specific zones.
How to use confidence maps? The main benefit of confidence maps is their ability to pinpoint areas in the dataset or detector model that may require refinement. Start by examining accuracy scores in specific areas. If the scores are low, use the confidence maps to delve deeper and understand the underlying issues.
How to interpret confidence maps? It’s important to acknowledge that some uncertainty is normal, especially around object edges, which are naturally more ambiguous for detection models and you can safely ignore those. Focus on significant bright spots within the maps, indicating areas of higher uncertainty and potential for model improvement.
The training report is available after each training iteration is complete and can be accessed using the button on the right of the “Train Detector” button as seen below.
The training report is an extremely useful tool to give you more detailed insights into your detector and help you with improvements and additional experimentation. It contains multiple sections detailed below.
In more details, here are the different situations you can be facing:
Detector merging simply copies every raster (along with its annotations/areas) belonging to each of the “source” detectors into the “target” detector. At this point, when training the latter, our ML model will learn from all this information, plus the original one (on target).
In case you need to annotate in parallel on a single detector, the suggested workflow is
Picterra Forge allows multiple users the ability to work on the specific detector simultaneously. All detectors shared across the organization support the collaboration feature which allows multiple users to train and annotate the detector.
Tree detections with high simplification
T
Roads results at normal simplification level
Roads results at very high simplification level
