A Headless browser is a browser without a graphical user interface (GUI). It can run in the background, allowing programmers to control and manipulate the browser through a command line interface or scripts.
Now it's time to start learning about detecting and anti-detecting headlesschrome with Nstbrowser!
Headlesschrome is a headless mode of Google Chrome. It allows developers to control and manipulate the Chrome browser through the command line or scripts without displaying the browser's user interface.
Headlesschrome combines the power of Chrome with the efficiency of headless mode, making it an indispensable tool for modern web development and testing.
Often blocked when scraping the web?
Nstbrowser fully unblocks websites to avoid detection
Try Free Nstbrowser Now!
Headless browsers are also often used for anti-crawler purposes due to their highly automated nature. Therefore, it becomes crucial to detect its presence.
Below are some common ways to detect headless browsers using Javascript code as an example and some anti-detection methods:
The user agent is an identifier that the browser sends to the server. By checking whether the user agent contains the word “Headless” or not, we can initially determine whether the request comes from a headlesschrome. This method is often unreliable because the user agent can be forged or modified.
const isHeadlessUserAgent = navigator.userAgent.toLowerCase().includes('headless');
// custom userAgent without 'headless'
Object.defineProperty(navigator, 'userAgent', {
value: 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36'
});
Headless Chromes usually do not execute JavaScript or their JavaScript execution capability is weak, so we can determine whether the browser supports JavaScript by executing some simple JavaScript code.This can be done by checking whether certain JavaScript APIs or functions are available.
const isJavaScriptEnabled = typeof window !== 'undefined';
Certain browser features may be disabled or behave abnormally in headlesschromes. Then, we can use these features for detection. For example, the Canvas API can be used to detect it.
const isCanvasSupported = () => {
const canvas = document.createElement('canvas');
const context = canvas.getContext('2d');
return !!context;
}
Headlesschrome detection can be done by checking for the presence of the navigator.webdriver
property. Headless browsers usually set this property, while normal browsers do not. Related source: Chromium can be determined if it is a Chrome headless.
const isWebDriverSupported = () => {
return navigator.webdriver;
}
Object.defineProperty(navigator, 'webdriver', {
get: () => undefined
});
window.chrome
code detectionDetecting the presence of window.chrome
can help determine if the browser is in headless mode. In headless mode, this attribute is usually not present.
const isWindowChromeSupported = () => {
return window.chrome !== undefined;
}
WebRTC is a web standard used for real-time communication in browsers, and more details will be the Webrtc official website. In other words, Headless Chromes usually disable WebRTC, so you can determine whether the browser is headless or not by checking the availability of WebRTC.
const isWebRTCSupported = () => {
return !!window.RTCPeerConnection;
}
Playing audio or video is usually disabled in headless mode. By trying to play audio or video, we can obviously detect whether the browser is headless or not.
const isAudioVideoSupported = () => {
const audio = document.createElement('audio');
const video = document.createElement('video');
return !!(audio.canPlayType && video.canPlayType);
}
Currently, headlesschromes are not able to manipulate browser permissions, so they don't support Notification.permission
. They don't handle user interface events, only page loading and rendering, so in headlesschrome, Notification.permission
and navigator. permissions.query
are inconsistent.
const isHeadlessPermissionMatched = () => {
return navigator.permissions.query({name: 'notifications'})
.then(permissionStatus => {
return Notification.permission === 'denied' && permissionStatus.state === 'prompt';
})
.catch(() => {
return false;
});
};
navigator.plugins
detectionnavigator.plugins
can figure out the list of plugins present in the browser. Normal browsers will install some plug-ins by default such as: Chrome PDF viewer or Google Native Client, etc., but in the headless mode generally will not have any plug-ins. So you can determine whether it is a headless browser by checking the plug-ins list.
const pluginInstalled = () => {
return navigator.plugins.length > 0
}
const originalPlugins = navigator.plugins;
Object.defineProperty(navigator, 'plugins', {
get: () => originalPlugins
});
In Chrome, the language used by the user can be obtained through navigator.language
and navigator.languages
, the previous one being the language of the browser UI and the second being a list of the user's preferred languages. In headerlesschrome, navigator.languages
will return an empty string.
const isHeadlessLanguage = () => {
return !navigator.languages || navigator.languages.length === 0
}
Object.defineProperty(navigator, 'languages', {
value: ['en-US', 'en']
});
Detection Methods:
anti-detection methods:
It is important to note that because headless browsers can mimic the behavior of almost all normal browsers, and will use a variety of anti-detection methods to make the headless Chrome look more like the behavior of a real user, making it more difficult to counter-crawlers. So no matter what methods are used to anti-detect headless browsers, they are not absolutely reliable, at least not until now.
We can also use fingerprint browser to bypass headless Chrome detection. Let's take Nstbrowser anti-detect browser as an example. Nstbrowser API solution is one of the best choices for avoiding bot detection nowadays. You can use it and get the API key for free.
Next, we'll compare the effectiveness of headless browser detection by visiting the CreepJS and Areyouheadless headlesschrome detection sites via the Nstbrowser anti-fingerprint browser and my local GoogleChrome browser, respectively.
We'll be using the Puppeteer call API to launch the headless fingerprint browser, specifically we'll be using the LaunchExistBrowser
interface, but of course you can choose any other API you need based on the documentation.
Before you start you need to:
Once everything is ready, it's time to code. The following code shows starting the Headless Mode Browser and visiting the Headless Browser Inspection site and taking a screenshot:
import puppeteer from 'puppeteer-core';
// wiat for millseconds
function sleep(millseconds) {
return new Promise(resolve => setTimeout(resolve, millseconds));
}
// visit headless detection site and take screenshots
async function execPuppeteer(browserWSEndpoint) {
try {
const browser = await puppeteer.connect({
browserWSEndpoint: browserWSEndpoint,
defaultViewport: null,
});
const page = await browser.newPage();
// detect headless on creepjs site
await page.goto('https://abrahamjuliot.github.io/creepjs');
await sleep(5 * 1000);
await page.screenshot({ fullPage: true, path: 'detect_headless_creepjs.png' });
// detect headless on areyouheadless site
await page.goto('https://arh.antoinevastel.com/bots/areyouheadless');
await sleep(2 * 1000);
await page.screenshot({ path: 'detect_headless_areyouheadless.png' });
await page.close();
await browser.disconnect();
} catch (err) {
console.error(err);
}
}
// LaunchExistBrowser: Connect to or start an existing browser
// You need to create the corresponding profile in advance
// Support custom config
async function launchAndConnectToBrowser(profileId) {
const host = 'localhost:8848';
const apiKey = 'you api key;
const config = {
headless: true, // support: true, 'new'
autoClose: true,
};
const query = new URLSearchParams({
'x-api-key': apiKey, // required
config: encodeURIComponent(JSON.stringify((config))),
});
const browserWSEndpoint = `ws://${host}/devtool/launch/${profileId}?${query.toString()}`;
console.log('browserWSEndpoint: ', browserWSEndpoint);
await execPuppeteer(browserWSEndpoint);
}
launchAndConnectToBrowser('your profile id').then();
Comparing the screenshots of the results to my local Google Chrome results, we found that the headless detection on both sites did not recognize that we were using a headless browser, which is what we wanted.
Through the discussion and sample code in this article, we hope to provide developers with effective reference and guidance to improve the security of the site and data protection capabilities.
Detection and anti-detection of headlesschrome is an evolving process of technological confrontation. The risk of detection and blocking can be countered with the help of effective anti-detection tools. Nstbrowser can help you ensure absolutly undetectable and use advanced technology to unlock network blocking.
In addition, developers need to keep an eye on the latest technological developments and update their detection strategies to meet new challenges. Whatever method is used to counter-detect headlesschrome, it needs to be continuously optimized and improved to ensure its effectiveness and reliability.