Stream video example shows switched RGB colors (red <-> blue) #41
Comments
|
We need to:
|
Understand how the Color Space is configured on the Canvas APIPart of the canvas API is addressed in #39 (comment). The We can check the original color space of this 2D rendering context by drawing a rectangle ( function tmp_visual() {
if(g_img != undefined)
{
virtual_img.src = yarp.getImageSrc(g_img.compression_type,g_img.buffer);
// video_stream_element.getContext('2d').drawImage(virtual_img,0,0,video_stream_element.width,video_stream_element.height);
video_stream_element.getContext('2d').fillStyle = "#FF0000";
video_stream_element.getContext('2d').fillRect(0, 0, video_stream_element.width, video_stream_element.height);
var myImageData = video_stream_element.getContext('2d').getImageData(0, 0, video_stream_element.width, video_stream_element.height);
}
setTimeout(tmp_visual,33);
}We get an myImageData = {
colorSpace: "srgb",
width: 300,
height: 150,
data: {255,0,0,255, 255,0,0,255, 255,0,0,255, ...}
}The data field translates [Red,100% alpha]. We use the same approach to check the green and blue color mapping. |
Check where in the transmission between the source file and final rendering the Color Space is modifiedWe define three test image files for checking color space changes between the source and destination images. Each test image is a rectangle filling the canvas, either red, green or blue. The test images are encoded in PPM* P6 format, manually edited in P3 (ASCII image data) and then converted to P6 (binary image data) in Gimp (saved as "raw" format). PPM formathttp://paulbourke.net/dataformats/ppm/ RGB color charthttps://www.rapidtables.com/web/color/RGB_Color.html Example - Red rectangle(*) NoteFrom the Yarp source code, we see that the fakeFrameGrabber only supports PPM and PNG formats. By testing we verified that it supports only PPM P1, P5 and P6 (binary or byte formats) and no PNG. We get the following source-destination color mapping:
The change in the values of ±1/255 R, G, or B amplitude is probably due to the compression/decompression. Blue and Red colors are clearly switched. If we load the purple rectangle file formatted as JPEG directly from the example yarp.js/examples/stream_video/index.html Lines 169 to 181 in d2e4609 by var virtual_img = new Image();
virtual_img.src = "/testImage-xxxRectangle.jpg";
var write_time = document.getElementById('write-time');
function tmp_visual() {
if(g_img != undefined)
{
video_stream_element.getContext('2d').drawImage(virtual_img,0,0,video_stream_element.width,video_stream_element.height);
}
setTimeout(tmp_visual,33);
}we obtain an unchanged image on the canvas with the original color order. In this case, the image source URI Note on the metadataWe tried to check the Exif metadata directly with the NodeJS package Note on other ways of comparing the imagesI also tried to compare the image data URI generated through an online 3rd party tool from the saved JPEG file "/testImage-xxxRectangle.jpg" and the data URI generated within https://ezgif.com/image-to-datauri |
Check where in the transmission between the source file and final rendering the Color Space is modified [..Continued]Let us recap the pipeline for rendering the rectangle image in the browser, either through the <fakeFrameGrabber> and <YarpJS Example Server>, either by loading directly the file in the browser ( Through the YarpJS stream video exampleConfig file fakeFrameGrabber_testImage.ini:Run command:Transmission pipeline:[testImage-xxxRectangle-P6.ppm] ⟶ <fakeFrameGrabber> ⟶
Loading directly the file in the browser[testImage-xxxRectangle.jpg] ⟶ <HTML page>
Comparing the Image file data VS image data received on the Yarp socketWe compared in NodeJS command line the image appearance and metadata from the file "testImage-xxxRectangle.jpg" against what we get from the image data received by the client:
testImage-xxxRectangle.jpgRGBA color {200,0,100} testImage-xxxRectangleCopyFromFaultyBuffer.jpgRGBA color {100,0,200} Comparing the Image file data VS image data received on the Yarp port
|
Check where in the transmission between the source file and final rendering the Color Space is modified [..On and On]When reading the Yarp port We get the output This is the header When reading the image data from port Focusing now on the analysis of the processing done in |
Check where in the transmission between the source file and final rendering the Color Space is modified [processing done in
|
|
The analysis in #41 (comment) narrowed down the bug location within the JPEG compression function called in yarp.js/YarpJS/src/YarpJS_Image.cpp Line 68 in d2e4609 This is enough to proceed with the performance analysis and improvement in ami-iit/yarp-openmct#104 by trying a new transmission protocol (mjpeg), before actually implementing a fix here, since that issue has more priority and the compression used with the MJPEG protocol shall replace the JPEG compression implemented in |
|
CC @traversaro |
|
Great investigation! |
|
Actually, the compression function call points to yarp.js/YarpJS/src/YarpJS_Image.cpp Line 39 in d2e4609 which is implemented in an OpenCV image encoding/decoding library (
/** @brief Encodes an image into a memory buffer.
The function imencode compresses the image and stores it in the memory buffer that is resized to fit the
result. See cv::imwrite for the list of supported formats and flags description.
@param ext File extension that defines the output format.
@param img Image to be written.
@param buf Output buffer resized to fit the compressed image.
@param params Format-specific parameters. See cv::imwrite and cv::ImwriteFlags.
*/
CV_EXPORTS_W bool imencode( const String& ext, InputArray img,
CV_OUT std::vector<uchar>& buf,
const std::vector<int>& params = std::vector<int>()); |
Convert the images to OpenCV JPEG format/** @brief Encodes an image into a memory buffer.
The function imencode compresses the image and stores it in the memory buffer that is resized to fit the
result. See cv::imwrite for the list of supported formats and flags description.
@param ext File extension that defines the output format.
@param img Image to be written.
@param buf Output buffer resized to fit the compressed image.
@param params Format-specific parameters. See cv::imwrite and cv::ImwriteFlags.
*/
CV_EXPORTS_W bool imencode( const String& ext, InputArray img,
CV_OUT std::vector<uchar>& buf,
const std::vector<int>& params = std::vector<int>());/** @brief Saves an image to a specified file.
The function imwrite saves the image to the specified file. The image format is chosen based on the
filename extension (see cv::imread for the list of extensions). In general, only 8-bit
single-channel or 3-channel (with 'BGR' channel order) images
can be saved using this function, with these exceptions:
- 16-bit unsigned (CV_16U) images can be saved in the case of PNG, JPEG 2000, and TIFF formats
- 32-bit float (CV_32F) images can be saved in PFM, TIFF, OpenEXR, and Radiance HDR formats;
3-channel (CV_32FC3) TIFF images will be saved using the LogLuv high dynamic range encoding
(4 bytes per pixel)
- PNG images with an alpha channel can be saved using this function. To do this, create
8-bit (or 16-bit) 4-channel image BGRA, where the alpha channel goes last. Fully transparent pixels
should have alpha set to 0, fully opaque pixels should have alpha set to 255/65535 (see the code sample below).
- Multiple images (vector of Mat) can be saved in TIFF format (see the code sample below).
If the image format is not supported, the image will be converted to 8-bit unsigned (CV_8U) and saved that way.
If the format, depth or channel order is different, use
Mat::convertTo and cv::cvtColor to convert it before saving. Or, use the universal FileStorage I/O
functions to save the image to XML or YAML format.
The sample below shows how to create a BGRA image, how to set custom compression parameters and save it to a PNG file.
It also demonstrates how to save multiple images in a TIFF file:
@include snippets/imgcodecs_imwrite.cpp
@param filename Name of the file.
@param img (Mat or vector of Mat) Image or Images to be saved.
@param params Format-specific parameters encoded as pairs (paramId_1, paramValue_1, paramId_2, paramValue_2, ... .) see cv::ImwriteFlags
*/
CV_EXPORTS_W bool imwrite( const String& filename, InputArray img,
const std::vector<int>& params = std::vector<int>());enum ImwriteFlags {
IMWRITE_JPEG_QUALITY = 1, //!< For JPEG, it can be a quality from 0 to 100 (the higher is the better). Default value is 95.
IMWRITE_JPEG_PROGRESSIVE = 2, //!< Enable JPEG features, 0 or 1, default is False.
IMWRITE_JPEG_OPTIMIZE = 3, //!< Enable JPEG features, 0 or 1, default is False.
IMWRITE_JPEG_RST_INTERVAL = 4, //!< JPEG restart interval, 0 - 65535, default is 0 - no restart.
IMWRITE_JPEG_LUMA_QUALITY = 5, //!< Separate luma quality level, 0 - 100, default is 0 - don't use.
IMWRITE_JPEG_CHROMA_QUALITY = 6, //!< Separate chroma quality level, 0 - 100, default is 0 - don't use.
IMWRITE_PNG_COMPRESSION = 16, //!< For PNG, it can be the compression level from 0 to 9. A higher value means a smaller size and longer compression time. If specified, strategy is changed to IMWRITE_PNG_STRATEGY_DEFAULT (Z_DEFAULT_STRATEGY). Default value is 1 (best speed setting).
IMWRITE_PNG_STRATEGY = 17, //!< One of cv::ImwritePNGFlags, default is IMWRITE_PNG_STRATEGY_RLE.
IMWRITE_PNG_BILEVEL = 18, //!< Binary level PNG, 0 or 1, default is 0.
IMWRITE_PXM_BINARY = 32, //!< For PPM, PGM, or PBM, it can be a binary format flag, 0 or 1. Default value is 1.
IMWRITE_EXR_TYPE = (3 << 4) + 0, /* 48 */ //!< override EXR storage type (FLOAT (FP32) is default)
IMWRITE_EXR_COMPRESSION = (3 << 4) + 1, /* 49 */ //!< override EXR compression type (ZIP_COMPRESSION = 3 is default)
IMWRITE_WEBP_QUALITY = 64, //!< For WEBP, it can be a quality from 1 to 100 (the higher is the better). By default (without any parameter) and for quality above 100 the lossless compression is used.
IMWRITE_PAM_TUPLETYPE = 128,//!< For PAM, sets the TUPLETYPE field to the corresponding string value that is defined for the format
IMWRITE_TIFF_RESUNIT = 256,//!< For TIFF, use to specify which DPI resolution unit to set; see libtiff documentation for valid values
IMWRITE_TIFF_XDPI = 257,//!< For TIFF, use to specify the X direction DPI
IMWRITE_TIFF_YDPI = 258, //!< For TIFF, use to specify the Y direction DPI
IMWRITE_TIFF_COMPRESSION = 259, //!< For TIFF, use to specify the image compression scheme. See libtiff for integer constants corresponding to compression formats. Note, for images whose depth is CV_32F, only libtiff's SGILOG compression scheme is used. For other supported depths, the compression scheme can be specified by this flag; LZW compression is the default.
IMWRITE_JPEG2000_COMPRESSION_X1000 = 272 //!< For JPEG2000, use to specify the target compression rate (multiplied by 1000). The value can be from 0 to 1000. Default is 1000.
};The current parameters yarp.js/YarpJS/src/YarpJS_Image.cpp Lines 34 to 35 in d2e4609 No additional parameter here is meant to convert the channel order.
cvtColor(...)
Tried a quick dirty fix replacing yarp.js/YarpJS/src/YarpJS_Image.cpp Line 39 in d2e4609 by... cv::Mat internalImageBGR;
cv::cvtColor(internalImage,internalImageBGR,cv::COLOR_RGB2BGR,0);
cv::imencode(encodeString,internalImageBGR, internalBuffer, p);and it's working fine |
|
As per @S-Dafarra's suggestion, check if we can use directly the automatic conversion from Yarp. |
Shall be handled in #43 . |
|
switched RGB colors fixed by #40 . |
When running the stream video example and streaming the Framegrabber "bouncing" ball or the scrolling "line", we can clearly see that the RGB colors are swapped from RGB to BGR.
Originally posted by @nunoguedelha in ami-iit/yarp-openmct#101 (comment)
The text was updated successfully, but these errors were encountered: