On the non-gaming side, the objective of video display architecture is to resolve the display limitations and solutions with minimal inputs of additional cost, energy, and time (time here refers to the response time of the display not necessarily the time to develop the solution). The solution is, of course, different depending on the display and the application. Even within a particular type of display, a multi-domain TFT LCD has different characteristics from an IPS mode LCD, so developing a “one size fits all” solution is difficult and producing a completely optimized solution is impossible without supplementing the display Electronic Digital ID (EDID) with additional information about display type and performance.
In TV display architectures, when a given image is provided, the normal process is to convert that image from whatever format into a yUV image to do cosmetic corrections. The usual process is to first do:
• Noise combing,
• edge identification for edge enhancement,
• motion estimation:
......for response judder recognition and elimination and
......response time improvement,
• facial identification:
......for ideal facial color mapping and
......facial blurring,
• blues stretch (enabling perceived blues outside of the normal color gamut of the display),
• green stretch,
• dynamic contrast enhancement (backlight dimming).
After the image modification is complete then it is converted to an rgb image and sent via the LVDS to the digital display. While all of this is going on the audio is being processed then held in a delay loop (the video processing for a real image takes long enough that there would be a noticeable timing difference between the audio and the video so the audio is held until the video is ready... gamers will notice that without a gaming mode that turns of the image processing and audio delay, an LCD TV responds at least 1/2 second behind their inputs, spoiling their performance). Once the video is received by the display, some further response time improvement is provided by the overdrive in the display’s embedded processor. Because different displays have different response times and because there are two processing circuits (one on the device side, one on the display side... that do not communicate with each other) handling response time improvement, there is necessarily some tuning that must be done to ensure that the combination of the device side response time improvement and the display side response time improvement do not produce any notable visual artifacts. This is particularly true if the device would have to interface with both computing LCDs (normally off-state white) and entertainment displays (usually off-state black). Finally, if the display has a “blinking backlight” or some form of dynamic contrast embedded in its own processing, then this becomes a further area of conflict between the device and the display that must be tuned further.
Saturday, April 24, 2010
The Impact of Digital v. Analog Management on Display Image Quality
By Norman Hairston
normh@alum.mit.edu
“Is this the real life? Is this just fantasy?
Caught in a landslide, No escape from reality
Open your eyes, look up to the skies and see….”
Queen “Bohemian Rhapsody” Lyrics
Introduction
Visual artifacts have always plagued the display industry. In the CRT era there were two; each one tied to the digitization of the image. Moiré resulted from the “beating” of the pixilation of the displayed image with any close periodicity in the captured image, e.g. plaid suits. Temporally, the time slicing digitization of the image frequently resulted in backward spinning wheels on autos or other anomalous imaging resulting from temporally resonant activity. Interestingly, even though the advertising industry is so obsessive that it demands that the ends of any length of spaghetti be buried in any image, it permits the backward spinning wheels on million dollar auto commercials because there is nothing they can do.
In the modern era, as digitization has progressed, more importantly as digital image manipulation progresses, the number and extent of digital artifacts has grown. This article discusses digital artifacts and their proliferation as the display industry has migrated from being analog/visual (human factors centered), to being computing intensive. Instead of building a better display, the industry now looks to software modifications to improve the image.
In the history of the color CRT, there was the invention of the Trinitron, Black Matrix, high voltage phosphors, and high black level glass; improvements to the native display. Early in the development of LCD technology there was the development of IPS (by RCA) and its resurrection (by Hitachi); also fundamental improvements to the basic display device. Although other technologies have subsequently been developed that match IPS, it was IPS that showed the potential of LCD technology to equal or exceed the CRT. Interestingly, at the time Hitachi LCD was a part of Hitachi Electron Devices (HED), the old Hitachi CRT organization, rather than a part of the Hitachi semiconductor or any other fundamentally digital organization. Although there have been subsequent improvements in image quality, and certainly drastic improvements in cost and product quality, this was the last quantum leap in LCD visual quality.
The issue with LCD display technology development has been an increasing reliance on software modifications and further digitization to improve the image. The industry increasingly relies on “driver firmware upgrade 9.991” rather than looking at how to improve the base technology. The result has been incremental improvements in the image when displaying unchallenging HDTV content, but increasing visual artifacts when displaying challenging images. Further, the LCD TV image modification software makes pre-judgments about what the user likes which may or may not correspond to what specific uses want to see.
LCD/HDTV
Regarding static images, as I have noted in other missives, LCD TV firmware has facial recognition processing that images faces differently than the rest of the picture. This software can have two negative effects, each tied to the two things that the facial recognition processing does.
First there is the ideal color correction. As I mentioned in another article, what is ideal for one culture may be anathema for another. The ideal color correction for a Japanese person (northwest on the CIE X-Y plane, toward the white point) is almost 180 degrees opposite for a western Caucasian (toward red). For a mid-range Black American, zone 7 for those of us with photography backgrounds, no color correction at all may be closer to the ideal. Inevitably, making the color correction will leave some unhappy with the facial images, forcing them to globally compensate across the image. This can result in the facial images of other races being severely discolored. The racial impact of facial recognition software was recently brought home to camera makers with their implementation of facial software to do “red eye” correction. It seems that some cameras were noticeably unable to recognize the eyes of oriental people, bringing charges of racism upon the makers even though the makers were all from oriental countries. Obviously the problem was not racism but myopia in their development organizations.
Along with facial color correction, LCD TV firmware often includes a soft focus. This gives an overall, more pleasing image of faces when the software is working well, taking out detail such as small wrinkles and other skin imperfections or even 5 O’clock shadow. However, when a face is in motion or the software is otherwise having trouble recognizing the face as a face, the soft focus is applied intermittently. This can result in things such as moles and facial stubble appearing and disappearing as the person moves or turns their head. As with many of the dynamic impacts of image processing, it is not so much the effect, as the effect turning itself on and of as the image changes that causes irritation to the viewer. These disappearing and reappearing facial features are most common on non-HD content leading me to further believe in the myopia of the development organizations, that there is not the thorough testing on standard resolution content (even though most content is still standard resolution) that there should be. Additionally, with standard resolution content showing up as soft focus on an HDTV anyway due to the smearing out of the pixels, adding a further soft focus to faces can give them a cartoonish appearance, devoid of normal skin texture.
Beyond faces, there is the impact of color management in general. Due to the digitization of the image, most TVs process at least 36 bit color, if not higher, to avoid the impact of digitization on displayed color levels. However, most LCD TVs still exhibit considerable posterization in images with subtle color shifts. Thirty-six bits should be more than enough to cover any loss of information in the digitization process. My suspicion is that the posterization is a result of the “blue stretch”, “green stretch”, or other parts of the color management software; but I do not know this.
Returning to the disappearing and reappearing detail, the LCD has traditionally struggled with loss of detail with motion. A large part of this has been due to LCD gray to gray response times. However, it is important to remember that the HDTV signal itself looses detail with high speed motion as a consequence of its data compression. LCD TV makers have addressed the response time issue by increasing the refresh rate from 60 Hz, the same as the input image, to 120, 240, and now 408 Hz. Though the higher refresh rates do improve the detail in fast motion images, they create some detail as well, as does the edge enhancement firmware.
Further, all of the visual processing, particularly waiting for subsequent images to draw artificial intermediate images, creates considerable time delay. In higher end sets, the user is able to turn off some or all of the visual processing to improve gaming response of the TV set. The need for a “gaming mode” is, perhaps, a sign that digital image process is not a sinecure for visual shortcomings of the technology as it now stands. In general, the image processing firmware on many LCD TVs is like any other piece of bad software, it guarantees the negative outcomes, like system crashes, that it was designed to prevent.
Beyond the time delay, disappearing and reappearing image details, and color distortion, there is also gross image distortion of non-HD content. Being short and stubby myself, it is somewhat pleasing to see everyone on TV become horizontally expanded… but not really.
Summary
In short, it is important to remember that every digital manipulation creates its own set of digital artifacts. While improvements in image processing firmware may offer improvements in ideal images, the overall improvement they make in image quality is limited and may actually be negative in some categories. In the CRT era, improving the image meant improving the display; the industry no longer seems to be emphasizing this.
There are at least two intermediate solutions to these problems. At Intel, the company developed a display technology improvement to advance the energy efficiency of notebook LCDs, one that traded temporal resolution for spatial resolution in static images. Coupled with this the team developed throttling technology, the ability to slow down or turn off the effect based on what the image was doing. For the TV set industry, it would be beneficial to recognize the type of content being displayed, the aspect ratio, and have some concept of object consistency longer than two consecutive frames and use this to restrain some of the image modification firmware.
A simpler, and perhaps more effective solution would be to enable the viewer to easily select or turn off (as is the case with gaming mode) some of the image processing. Such an implementation would be simple, costless, and direct. Beyond this, the display device makers need to reduce the emphasis on firmware revision and focus on native display performance… genuine innovation. The active matrix LCD was invented in the US and many of the subsequent fundamental improvements, such as IPS, were pioneered here. There are a few US based display startups, such as Unipixel, that are developing fundamentally newer and better displays, but they are not attached to large ongoing display production operations as was the case with Westinghouse (developer of active matrix LCDs) and RCA’s Sarnoff Labs. While the electronics companies that populate the industry today are excellent at driving down costs and improving quality, they are largely yet to prove themselves in fundamental innovation. Without this fundamental innovation we get slightly better HD LCD TV pictures at the cost of increasing visual artifacts and sometimes poorer performance with standard definition content.
normh@alum.mit.edu
“Is this the real life? Is this just fantasy?
Caught in a landslide, No escape from reality
Open your eyes, look up to the skies and see….”
Queen “Bohemian Rhapsody” Lyrics
Introduction
Visual artifacts have always plagued the display industry. In the CRT era there were two; each one tied to the digitization of the image. Moiré resulted from the “beating” of the pixilation of the displayed image with any close periodicity in the captured image, e.g. plaid suits. Temporally, the time slicing digitization of the image frequently resulted in backward spinning wheels on autos or other anomalous imaging resulting from temporally resonant activity. Interestingly, even though the advertising industry is so obsessive that it demands that the ends of any length of spaghetti be buried in any image, it permits the backward spinning wheels on million dollar auto commercials because there is nothing they can do.
In the modern era, as digitization has progressed, more importantly as digital image manipulation progresses, the number and extent of digital artifacts has grown. This article discusses digital artifacts and their proliferation as the display industry has migrated from being analog/visual (human factors centered), to being computing intensive. Instead of building a better display, the industry now looks to software modifications to improve the image.
In the history of the color CRT, there was the invention of the Trinitron, Black Matrix, high voltage phosphors, and high black level glass; improvements to the native display. Early in the development of LCD technology there was the development of IPS (by RCA) and its resurrection (by Hitachi); also fundamental improvements to the basic display device. Although other technologies have subsequently been developed that match IPS, it was IPS that showed the potential of LCD technology to equal or exceed the CRT. Interestingly, at the time Hitachi LCD was a part of Hitachi Electron Devices (HED), the old Hitachi CRT organization, rather than a part of the Hitachi semiconductor or any other fundamentally digital organization. Although there have been subsequent improvements in image quality, and certainly drastic improvements in cost and product quality, this was the last quantum leap in LCD visual quality.
The issue with LCD display technology development has been an increasing reliance on software modifications and further digitization to improve the image. The industry increasingly relies on “driver firmware upgrade 9.991” rather than looking at how to improve the base technology. The result has been incremental improvements in the image when displaying unchallenging HDTV content, but increasing visual artifacts when displaying challenging images. Further, the LCD TV image modification software makes pre-judgments about what the user likes which may or may not correspond to what specific uses want to see.
LCD/HDTV
Regarding static images, as I have noted in other missives, LCD TV firmware has facial recognition processing that images faces differently than the rest of the picture. This software can have two negative effects, each tied to the two things that the facial recognition processing does.
First there is the ideal color correction. As I mentioned in another article, what is ideal for one culture may be anathema for another. The ideal color correction for a Japanese person (northwest on the CIE X-Y plane, toward the white point) is almost 180 degrees opposite for a western Caucasian (toward red). For a mid-range Black American, zone 7 for those of us with photography backgrounds, no color correction at all may be closer to the ideal. Inevitably, making the color correction will leave some unhappy with the facial images, forcing them to globally compensate across the image. This can result in the facial images of other races being severely discolored. The racial impact of facial recognition software was recently brought home to camera makers with their implementation of facial software to do “red eye” correction. It seems that some cameras were noticeably unable to recognize the eyes of oriental people, bringing charges of racism upon the makers even though the makers were all from oriental countries. Obviously the problem was not racism but myopia in their development organizations.
Along with facial color correction, LCD TV firmware often includes a soft focus. This gives an overall, more pleasing image of faces when the software is working well, taking out detail such as small wrinkles and other skin imperfections or even 5 O’clock shadow. However, when a face is in motion or the software is otherwise having trouble recognizing the face as a face, the soft focus is applied intermittently. This can result in things such as moles and facial stubble appearing and disappearing as the person moves or turns their head. As with many of the dynamic impacts of image processing, it is not so much the effect, as the effect turning itself on and of as the image changes that causes irritation to the viewer. These disappearing and reappearing facial features are most common on non-HD content leading me to further believe in the myopia of the development organizations, that there is not the thorough testing on standard resolution content (even though most content is still standard resolution) that there should be. Additionally, with standard resolution content showing up as soft focus on an HDTV anyway due to the smearing out of the pixels, adding a further soft focus to faces can give them a cartoonish appearance, devoid of normal skin texture.
Beyond faces, there is the impact of color management in general. Due to the digitization of the image, most TVs process at least 36 bit color, if not higher, to avoid the impact of digitization on displayed color levels. However, most LCD TVs still exhibit considerable posterization in images with subtle color shifts. Thirty-six bits should be more than enough to cover any loss of information in the digitization process. My suspicion is that the posterization is a result of the “blue stretch”, “green stretch”, or other parts of the color management software; but I do not know this.
Returning to the disappearing and reappearing detail, the LCD has traditionally struggled with loss of detail with motion. A large part of this has been due to LCD gray to gray response times. However, it is important to remember that the HDTV signal itself looses detail with high speed motion as a consequence of its data compression. LCD TV makers have addressed the response time issue by increasing the refresh rate from 60 Hz, the same as the input image, to 120, 240, and now 408 Hz. Though the higher refresh rates do improve the detail in fast motion images, they create some detail as well, as does the edge enhancement firmware.
Further, all of the visual processing, particularly waiting for subsequent images to draw artificial intermediate images, creates considerable time delay. In higher end sets, the user is able to turn off some or all of the visual processing to improve gaming response of the TV set. The need for a “gaming mode” is, perhaps, a sign that digital image process is not a sinecure for visual shortcomings of the technology as it now stands. In general, the image processing firmware on many LCD TVs is like any other piece of bad software, it guarantees the negative outcomes, like system crashes, that it was designed to prevent.
Beyond the time delay, disappearing and reappearing image details, and color distortion, there is also gross image distortion of non-HD content. Being short and stubby myself, it is somewhat pleasing to see everyone on TV become horizontally expanded… but not really.
Summary
In short, it is important to remember that every digital manipulation creates its own set of digital artifacts. While improvements in image processing firmware may offer improvements in ideal images, the overall improvement they make in image quality is limited and may actually be negative in some categories. In the CRT era, improving the image meant improving the display; the industry no longer seems to be emphasizing this.
There are at least two intermediate solutions to these problems. At Intel, the company developed a display technology improvement to advance the energy efficiency of notebook LCDs, one that traded temporal resolution for spatial resolution in static images. Coupled with this the team developed throttling technology, the ability to slow down or turn off the effect based on what the image was doing. For the TV set industry, it would be beneficial to recognize the type of content being displayed, the aspect ratio, and have some concept of object consistency longer than two consecutive frames and use this to restrain some of the image modification firmware.
A simpler, and perhaps more effective solution would be to enable the viewer to easily select or turn off (as is the case with gaming mode) some of the image processing. Such an implementation would be simple, costless, and direct. Beyond this, the display device makers need to reduce the emphasis on firmware revision and focus on native display performance… genuine innovation. The active matrix LCD was invented in the US and many of the subsequent fundamental improvements, such as IPS, were pioneered here. There are a few US based display startups, such as Unipixel, that are developing fundamentally newer and better displays, but they are not attached to large ongoing display production operations as was the case with Westinghouse (developer of active matrix LCDs) and RCA’s Sarnoff Labs. While the electronics companies that populate the industry today are excellent at driving down costs and improving quality, they are largely yet to prove themselves in fundamental innovation. Without this fundamental innovation we get slightly better HD LCD TV pictures at the cost of increasing visual artifacts and sometimes poorer performance with standard definition content.
Subscribe to:
Posts (Atom)