Segmentation Masks and Metadata

Celantur container can generate two different segmentation masks and metadata per processed image:

• Binary Segmentation

• Instance Segmentation

It's activated with the --save-mask {all, instance, binary} parameter. The segmentation is saved as a PNG file.

Anonymization, binary segmentation and instance segmentation applied to an image with Celantur software.

Binary Segmentation

The binary segmentation mask consist of two colors:

• Background is black

• Anonymized segments are white

The file will be saved as image-name_bin_mask.png.

Instance Segmentation

v26.02.1 and later

In instance segmentation masks, the RGB color values are used to differentiate to individual instances/objects.

• The R (red) channel is 0.

• The G (green) channel encodes individual instances / objects.

• The B (blue) channel encodes the object type:

Object type	Blue channel value	Blue channel value in binary
Person	128	1000 0000
License plate	64	0100 0000
Face / head	32	0010 0000
Vehicle	16	0001 0000

When objects overlap, their color codes are combined using the binary OR operation, e.g. overlap of license plate and vehicle results in 80 (16 + 64), overlap of two persons remains 128.

E.g. [0, 85, 16] is a vehicle. 16 identifies the object type, 85 identifies the instance from others.

The file will be saved as image-name_ins_maks.png.

Pre v26.02.1

In instance segmentation masks, the RGB color values are used to differentiate to individual instances/objects.

• The R (red) channel is 0.

• The G (green) channel encodes individual instances / objects.

• The B (blue) channel encodes the object type:

  • Person: 128

  • License plate: 64

  • Face: 192

  • Vehicle: 255

E.g. [0, 85, 192] is a face.

The file will be saved as image-name_ins_maks.png.

Scale Down Mask Files

By adding the optional --mask-scale {0-100} (CLI) or /v1/file/1/instance-mask?mask-scale={0-100} (Container API) parameter, mask files will be scaled down by the specified ratio.

Metadata

Image metadata

Metadata about detected instances/objects are stored in the corresponding image-name.json file.

Image metadata example

{
    "id": "image-name.jpg",
    "detections": [
    {
        "id": 0,
        "parent_image": "image-name.jpg",
        "offset": [
            1560,
            744
        ],
        "bbox": [
            1957,
            744,
            3003,
            1855
        ],
        "type": 103,
        "score": 0.9993754029273987,
        "is_anonymised": true,
        "type_label": "face",
        "color": null
    },
    "size": [
        3456,
        5184
    ],
    "duration": 1.8533296539999355,
    "filename": "image-name.jpg",
    "folder": null
}

Video metadata

Metadata is generated for individual frames and provided as a range of mulitple frames.

• Batch/Stream mode: The information is stored in filename-[startframe]-[endframe].json in the output directory. The maximum number of frames covered by a single file is 500.

• REST API mode: The infromation can be retrieved via the Download video metadata (JSON) endpoint.

Video metadata example

[
  {
    "id": 0,
    "detections": [
      {
        "id": 0,
        "parent_image": 0,
        "offset": [
          3,
          4
        ],
        "bbox": [
          3,
          4,
          201,
          171
        ],
        "type": 103,
        "score": 0.8267934918403625,
        "is_anonymised": true,
        "type_label": "face",
        "color": null
      }
    ],
    "size": [
      178,
      320
    ],
    "duration": 0.35388135999892256
  },
  {
    "id": 1,
    "detections": [
      {
        "id": 0,
        "parent_image": 1,
        "offset": [
          0,
          6
        ],
        "bbox": [
          0,
          20,
          137,
          175
        ],
        "type": 103,
        "score": 0.8945223689079285,
        "is_anonymised": true,
        "type_label": "face",
        "color": null
      }
    ],
    "size": [
      178,
      320
    ],
    "duration": 0.2979645720006374
  },
  ...
]

Metadata attribute reference

Attributes of a image or video frame

Attribute	Description
id	The id of the image (file name) or video frame (sequential number)
detections	List of detections, see Detected instances/objects provided as a list under the detections attribute
size	Size of the image or frame in [width, height]
duration	Duration of processing (inference and anonymization) an image or video frame. Does not include IO, e.g. read/write from hard drive.
filename	Name of the file
folder	Name of the folder (relative to root input folder)

Detected instances/objects provided as a list under the detections attribute

Attribute	Description
id	The id of the detection, a sequential number starting from 0.
parent_image	The name of the image or the id of the video frame the detected instance/object was found on
offset	The offset of the detection's bounding box from the upper left corner of the image (x/y coordinates in pixels)
bbox	The coordinates of the detection's bounding box (x1, y1, x2, y2)
score	The detection's confidence score. States how confident the model is about the detection being a specific label (see type_label) between 0.0 and 1.0.
is_anonymized	Specifies whether the detection was anonymized (or detected with method = detect).
type_label	The detection's label assigned by the model. E.g. face, license plates, etc.
type	Numerical representation of the type_label.
color	The detections color (RGB) in the instance segmentation mask.
duration	Processing duration for a video frame (only for videos).