Why is it More Difficult to Create an Under-Screen Camera for the iPhone?

Since the launch of the iPhone 15, the under-screen camera has become the focus of attention in many news media outlets. In this post, we will analyze the challenges of iPhone under-screen camera development.

Some time ago, Macrumor revealed that Apple’s suppliers in South Korea had begun researching and developing under-screen smartphone cameras, and netizens immediately pointed out that similar rumors had already been circulating five years ago.

We don’t know when to see the iPhone pictured here, but the debate will continue.
Compared to Android’s increasingly smaller front-facing cameras, and even its attempts at “true full screen,” Apple is more like a spectator sitting on the sidelines, in the same space and time, but not participating in the game, standing aside and sticking to the black spot above the screen.

To understand why Apple has been slow to launch the much-anticipated under-screen camera, we must first clarify one issue:

Why is The Front Module/Notch so important to Apple?

The more furniture, the bigger the house. Do you know that a secret is hidden in Apple’s camera notch?

The front module includes 8 sensing components, including an “infrared lens,” a “floodlight sensor element,” a “speaker,” and others. Looking at it from this angle again, we can understand the existence of the bangs.

The infrared lens, dot projector, and front camera are the fundamental reasons Apple insists on retaining the large module.
Although the iPhone 15 series uses a shorter “Dynamic Island” and many sensors have been rearranged, the three brothers still stand in the center of the “island.”

iPhone has always had its unique understanding of photography. The review website DXOMARK ranked the iPhone 15 Pro Max’s “selfie effect” score at the top of the list in the past two years. Even last year’s 14 achieved an excellent second place among the new phones in 2023.

The iPhone’s paper data has never been the best, but Apple’s shadow is always present in all horizontal comparisons of photography reviews; its photo style is not the most pleasing, but its natural imaging effect is quite a highlight on major lists.

In this era of increasingly frequent online interactions, selfie live broadcasts, video calls, and online meetings all require the participation of the front camera, so the iPhone has placed great importance on front-facing photography.

The twin brothers, the “infrared lens” and the FaceID “dot projector,” have merged into the gatekeeper of Apple’s ecosystem: Face ID.

Apple users often complain about the iPhone’s battery life and signal quality, but we rarely hear rumors about Apple’s poor security. This is thanks to Apple’s deep involvement in security verification.

As early as 2017, Face ID joined the Apple ecosystem. At the same time, the iPhone completely abandoned Touch ID, which it had pioneered.

The security of face recognition is incomparable to fingerprint unlocking. The unlocking process of a “3D structured light dual camera” is a bit awkward to explain in detail, so we will only briefly explain the key points.

“Payment-level” facial unlocking is the biggest compliment to Face ID. Even the vault can confidently rely on the technology. Of course, there are some insurmountable thresholds, and the “dot projector” is one of the main contributors.

The invisible IR Dot is projected onto the human face through the lens, forming a three-dimensional model with 30,000 dot groups on the face. The virtual mask is scanned with an infrared lens and matched with the dot matrix information in the mobile phone to confirm whether the person unlocking the phone is the owner.

How Does The Security of The Face ID Compare with Fingerprint Unlocking?

The security of facial information is actually 20 times higher than that of fingerprint solutions. The probability of unlocking a phone with the same fingerprint is one in 50,000, while in Face ID, this probability is reduced to one in a million.

After years of upgrades to facial recognition, users can use it while wearing masks, glasses, and hats, but it cannot be unlocked using photos of fake faces or even 1:1 face models.

To prevent others from forcibly unlocking the phone while the owner is sleeping, Face ID has also added “attention detection” technology. Only when the owner’s attention is on the phone screen will the iPhone identify it as an unlocking command, and the above unlocking process can be carried out.

When Will Apple Launch Its First Under-Screen Camera Phone?

Three ways to plug the gap

Since OLED luminous materials are not transparent, to allow both the screen and the lens to see us under the screen, it is necessary to increase the screen’s light transmittance in the area of ​​the front lens, whether by making the gaps between pixels larger or removing some pixels. The ultimate effect is that light can pass through the gaps smoothly and enter the lens.

Two Main Challenges of iPhone Under-screen Camera Development

Apple also uses the same principle to design the under-screen camera, but compared with previous manufacturers, the iPhone under-screen camera has two inherent problems to deal with:

• iPhones with FaceID have larger front modules that need to be hidden
• Larger modules require larger light transmission areas.

The Red Magic 9 Pro is the market device with the best control over the resolution difference between the “under-screen area” and the “main screen area.” After experiencing it for some time, we can’t feel the abruptness of the lack of resolution in most scenarios, but this effect can only be achieved with only one camera.

In response, Apple used the “integration + hiding” method in its patent to minimize the impact on screen resolution.

In the patent document code-named 20240077648, a technology called “integration of IR dot projector and flood illuminator” is mentioned.

The flood illuminator is the infrared lens mentioned above, which emits infrared light to help the phone recognize faces and determine whether the user is looking directly at the screen to ensure that only the user can unlock the screen.

How Does FaceID Impact Apple’s Under-Screen Camera Development

Apple can combine the flood illuminator and dot projector into a single compact projection module, reducing the pair of sensors’ area to nearly half their original size, thereby reducing the need for the under-screen area.

The “planar optical waveguide” in another patent extends the above-mentioned integration technology.

Modules such as ambient light sensing, image capture, gesture recognition, and proximity sensing in traditional front-end modules need to emit or receive light radiation in different spectral regions through various physical apertures in order to complete corresponding operating instructions and perceive the environment and users.

Therefore, most of the full-screen mobile phones currently leave a black area on the screen, not because they like it, but because they have no choice. 

However, planar light waveguides seem to have solved this problem. This technology can use planar light waveguides on the mobile phone display to guide radiation through the display surface without interfering with the display function.

Get rid of obscure professional terms. With it, most sensors (except the camera) can be placed under the screen without affecting usage, just like fingerprint unlocking on a mobile phone.

In theory, Apple’s combination of measures reduces the size of the front module and the area required to reduce screen resolution. The only thing that needs to compromise the main screen is the front optical lens.

Therefore, this rendering of the iPhone 16 does make sense.

 However, to be truly comprehensive, the lens is a topic that cannot be avoided. After all, the resolution above the module differs from the main screen. Even on the Red Magic 9 Pro, you can still see the “screen-like” area in the light-colored and highlighted interface under the screen. 

Apple’s third weapon was designed specifically for this purpose.

First, the special “W” arrangement of pixels increases light transmittance, reduces diffraction, and avoids the appearance of “halo” and “fogging” in selfies.

Then, components in this area (such as touch sensor metal) can be moved toward or away from their neighboring pixels in different usage scenarios to reduce the display difference between the “pixel removal area” and the “full pixel density area.” 

Judging from the description in the patent document alone, Apple has achieved variable resolution in the physical sense. While reading the patent documents, you can discover a little trick of Apple.

All major manufacturers have slowed their exploration of full-screen displays in the past two years. 

On the one hand, after years of attempts, ideas such as front and back double-sided screens, flip lenses, and sliding screens have all become fleeting historical products. The ideas are fresh, but the market does not buy them. Under the current level of technology, the ideal mobile phone of “lightweight,” “high resolution,” and “complete screen” can only meet two of the requirements.

The item that is discarded in this impossible triangle also forms the characteristics of different brands and series. 

On the other hand, the incomplete screen is not entirely a disadvantage. Returning to the top of the screen, I suddenly realized that the existence of the hollowing out, bangs, or water drops not only gives the front module room to play but also provides a natural habitat for the information bar.

Impact of Under-Screen Cameras on Mobile Phone Status Bars

What will happen to the status bar information when the under-screen solution becomes popular? What tricks will manufacturers devise to retain and remove these icons? It is worth looking forward to.

However, people’s pursuit of the perfect screen has kept this protracted “space tug-of-war” going. What will the mobile phone screen look like in the future? In fact, the answer has long been known.

Former Apple chief designer Jony Ive believes that an “infinity pool” will be a form of mobile phone screen in the future. From Samsung Galaxy Note Edge to Huawei Mate 30 Pro, this concept has been inadvertently running through them. However, the only one that is close to achieving infinity visually is the Xiaomi Mix α placed in the display cabinet. However, this machine is still far from being mass-produced.

These patent documents give us hope for a “full-screen iPhone,” but Apple is a little slow in finding the perfect screen. 

Just like Steve Jobs, who would rather have an empty house than make do with furniture he didn’t like, the design of the iPhone seems to have always maintained this philosophy. It is always unique and one step behind others, and of course, it attracts the world’s attention when changes are made. Apple always places a premium on quality rather than quantity.