A Headless browser is a browser without a graphical user interface (GUI). It can run in the background, allowing programmers to control and manipulate the browser through a command line interface or scripts.

Now it's time to start learning about detecting and anti-detecting bots with Nstbrowser!

What Is Headless Chrome?

Headless Chrome is a headless mode of Google Chrome. It allows developers to control and manipulate the Chrome browser through the command line or scripts without displaying the browser's user interface. Headlesschrome combines the power of Chrome with the efficiency of headless mode, making it an indispensable tool for modern web development and testing.

What Can Headless Chrome Do?

1. Efficient automated testing: Headlesschrome can quickly run browser test scripts to verify the functionality, performance, and compatibility of web pages without loading and rendering the user interface, dramatically increasing the speed and efficiency of testing.
2. Web scraping: Headless Browser can be used to scrape web content and simulate user actions to access and extract data from dynamic web pages. This is especially useful for dealing with web pages that require logging in, loading dynamic content, or relying on JavaScript-generated content.
3. Performance Monitoring: Developers can use headless Chromes to monitor and analyze web page performance, such as load times, resource usage, etc.

Often blocked when scraping the web?
Nstbrowser fully unblocks websites to avoid detection
Try Free Nstbrowser Now!

How to Detect and Anti-detect Headless Chrome?

Headless browsers are also often used for anti-crawler purposes due to their highly automated nature. Therefore, it becomes crucial to detect its presence.

Below are some common ways to detect headless browsers using Javascript code as an example and some anti-detection methods:

1. User Agent Detection

The user agent is an identifier that the browser sends to the server. By checking whether the user agent contains the word “Headless” or not, we can initially determine whether the request comes from a headlesschrome. This method is often unreliable because the user agent can be forged or modified.

• Detection Code

const isHeadlessUserAgent = navigator.userAgent.toLowerCase().includes('headless');

• Anti-detection code

// custom userAgent without 'headless'
Object.defineProperty(navigator, 'userAgent', {
    value: 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36'
});

2. JavaScript Execution Detection

Headless Chromes usually do not execute JavaScript or their JavaScript execution capability is weak, so we can determine whether the browser supports JavaScript by executing some simple JavaScript code.This can be done by checking whether certain JavaScript APIs or functions are available.

• Detection code

const isJavaScriptEnabled = typeof window !== 'undefined';

3. Browser Feature Detection

Certain browser features may be disabled or behave abnormally in headlesschromes. Then, we can use these features for detection. For example, the Canvas API can be used to detect it.

• Detection code

const isCanvasSupported = () => {
    const canvas = document.createElement('canvas');
    const context = canvas.getContext('2d');
    return !!context;
}

4. Webdriver Detection

Headless Chrome detection can be done by checking for the presence of the navigator.webdriver property. Headless browsers usually set this property, while normal browsers do not. Related source: Chromium can be determined if it is a Chrome headless.

• Detection code

const isWebDriverSupported = () => {
    return navigator.webdriver;
}

• Anti-detection code

Object.defineProperty(navigator, 'webdriver', {
    get: () => undefined
});

5. The window.chrome code detection

Detecting the presence of window.chrome can help determine if the browser is in headless mode. In headless mode, this attribute is usually not present.

• Detection code

const isWindowChromeSupported = () => {
    return window.chrome !== undefined;
}

6. WebRTC detection

WebRTC is a web standard used for real-time communication in browsers, and more details will be the Webrtc official website. In other words, Headless Chromes usually disable WebRTC, so you can determine whether the browser is headless or not by checking the availability of WebRTC.

• Detection code

const isWebRTCSupported = () => {
    return !!window.RTCPeerConnection;
}

7. Audio/Video Detection

Playing audio or video is usually disabled in headless mode. By trying to play audio or video, we can obviously detect whether the browser is headless or not.

• Detection code

const isAudioVideoSupported = () => {
    const audio = document.createElement('audio');
    const video = document.createElement('video');
    return !!(audio.canPlayType && video.canPlayType);
}

8. Headless Permission Detection

Currently, headlesschromes are not able to manipulate browser permissions, so they don't support Notification.permission. They don't handle user interface events, only page loading and rendering, so in headlesschrome, Notification.permission and navigator. permissions.query are inconsistent.

• Detection code

const isHeadlessPermissionMatched = () => {
    return navigator.permissions.query({name: 'notifications'})
        .then(permissionStatus => {
            return Notification.permission === 'denied' && permissionStatus.state === 'prompt';
        })
        .catch(() => {
            return false;
        });
};

9. The navigator.plugins detection

navigator.plugins can figure out the list of plugins present in the browser. Normal browsers will install some plug-ins by default such as: Chrome PDF viewer or Google Native Client, etc., but in the headless mode generally will not have any plug-ins. So you can determine whether it is a headless browser by checking the plug-ins list.

• Detection code

const pluginInstalled = () => {
    return navigator.plugins.length > 0
}

• Anti-detection code

const originalPlugins = navigator.plugins;
Object.defineProperty(navigator, 'plugins', {
    get: () => originalPlugins
});

10. Language detection

In Chrome, the language used by the user can be obtained through navigator.language and navigator.languages, the previous one being the language of the browser UI and the second being a list of the user's preferred languages. In headerlesschrome, navigator.languages will return an empty string.

• Detection code

const isHeadlessLanguage = () => {
    return !navigator.languages || navigator.languages.length === 0
}

• Anti-checking code

Object.defineProperty(navigator, 'languages', {
    value: ['en-US', 'en']
});

11. Other common methods

Detection Methods:

• Page load speed: Headless Chromes usually load pages faster because they do not need to render pages or execute JavaScript.
• Keyboard Event: Headless Chromes usually do not trigger keyboard events because they generally do not need to interact with the user. Headless browsers can be determined by listening for keyboard events and checking to see if they are triggered.
• Mouse event: Similarly, headlesschromes do not activit mouse events because they do not usually need to simulate user actions.
• CSS detection: CSS styles may behave differently in headless browsers. That is, certain CSS properties may not be supported or may render differently. You can determine a headless by loading a specific CSS style and checking the rendering result.
• LocalStorage detection: A headlesschrome typically does not support LocalStorage because they are not required to maintain user sessions or state.
• Embedding JavaScript interaction: Headless browsers usually do not execute a lot of JavaScript code or interact with pages. You can determine whether a browser is headless by embedding some JavaScript code in the page that requires user interaction and detecting its execution result.
• Detecting Specific Environment Variables: Headless browsers usually set some specific environment variables or attributes, which can be used to determine headless chromes. For example, you can detect the presence of a specific identifier in an environment variable, or detect the presence of a specific global variable in JavaScript.

anti-detection methods：

• Simulate Human Behavior: Headless browsers usually perform tasks such as page loading, form submission, etc. in a highly automated manner, which is different from the behavioral patterns of human users. Human behavior can be simulated, e.g. adding random mouse movements, keyboard inputs, etc., to make the detector think that it is a real user action instead of a headlesschrome.
• Randomize request intervals: Headless Chromes usually send requests at a very regular frequency. It is possible to simulate real user behavior by adding random intervals between requests, which makes it difficult for the detector to determine whether it is a headless browser by the frequency of requests.
• Simulate browser environment: Modify some properties or functions in the headlesschrome to make it more like a real browser. For example, the browser window size, display resolution, User-Agent, etc. can be modified to simulate the environment of a real browser.

It is important to note that because headless browsers can mimic the behavior of almost all normal browsers, and will use a variety of anti-detection methods to make the headless Chrome look more like the behavior of a real user, making it more difficult to counter-crawlers. So no matter what methods are used to anti-detect headless browsers, they are not absolutely reliable, at least not until now.

Headless Browser Anti-Detection with Fingerprint Browser

We can also use fingerprint browser to bypass headless Chrome detection. Let's take Nstbrowser anti-detect browser as an example. Nstbrowser API solution is one of the best choices for avoiding bot detection nowadays. You can use it and get the API key for free.

Why use Nstbrowser for detection?

• Nstbrowser is completely free to use!
• Nstbrowserless is compatible with the popular Puppeteer and Playwright, providing more comprehensive technical support
• Nstbrowser supports launching the browser in headless mode via the API.

Next, we'll compare the effectiveness of headless browser detection by visiting the CreepJS and Areyouheadless headlesschrome detection sites via the Nstbrowser anti-fingerprint browser and my local GoogleChrome browser, respectively.

We'll be using the Puppeteer call API to launch the headless fingerprint browser, specifically we'll be using the LaunchExistBrowser interface, but of course you can choose any other API you need based on the documentation.

Before you start you need to:

• Step 1. download and install Nstbrowser in advance
• Step 2. Generate an API Key
• Step 3. Create a Profile and download the kernel. You can also launch your Profile on the client side and it will trigger an automatic kernel download.

Once everything is ready, it's time to code. The following code shows starting the Headless Mode Browser and visiting the Headless Browser Inspection site and taking a screenshot:

import puppeteer from 'puppeteer-core';

// wiat for millseconds
function sleep(millseconds) {
  return new Promise(resolve => setTimeout(resolve, millseconds));
}

// visit headless detection site and take screenshots 
async function execPuppeteer(browserWSEndpoint) {
  try {
    const browser = await puppeteer.connect({
      browserWSEndpoint: browserWSEndpoint,
      defaultViewport: null,
    });

    const page = await browser.newPage();
    // detect headless on creepjs site
    await page.goto('https://abrahamjuliot.github.io/creepjs');
    await sleep(5 * 1000);
    await page.screenshot({ fullPage: true, path: 'detect_headless_creepjs.png' });

    // detect headless on areyouheadless site
    await page.goto('https://arh.antoinevastel.com/bots/areyouheadless');
    await sleep(2 * 1000);
    await page.screenshot({ path: 'detect_headless_areyouheadless.png' });

    await page.close();
    await browser.disconnect();
  } catch (err) {
    console.error(err);
  }
}

// LaunchExistBrowser: Connect to or start an existing browser
// You need to create the corresponding profile in advance
// Support custom config
async function launchAndConnectToBrowser(profileId) {
  const host = 'localhost:8848';
  const apiKey = 'you api key;
  const config = {
    headless: true, // support: true, 'new'
    autoClose: true,
  };
  const query = new URLSearchParams({
    'x-api-key': apiKey, // required
    config: encodeURIComponent(JSON.stringify((config))),
  });
  const browserWSEndpoint = `ws://${host}/devtool/launch/${profileId}?${query.toString()}`;
  console.log('browserWSEndpoint: ', browserWSEndpoint);
  await execPuppeteer(browserWSEndpoint);
}

launchAndConnectToBrowser('your profile id').then();

Running results:

• CreepJS test results

• Areyouheadless test results

Comparing the screenshots of the results to my local Google Chrome results, we found that the headless detection on both sites did not recognize that we were using a headless browser, which is what we wanted.

The Bottom Lines

Through the discussion and sample code in this article, we hope to provide developers with effective reference and guidance to improve the security of the site and data protection capabilities.

Detection and anti-detection of headless browser is an evolving process of technological confrontation. The risk of detection and blocking can be countered with the help of effective anti-detection tools. Nstbrowser can help you ensure absolutly undetectable and use advanced technology to unlock network blocking.

In addition, developers need to keep an eye on the latest technological developments and update their detection strategies to meet new challenges. Whatever method is used to counter-detect headlesschrome, it needs to be continuously optimized and improved to ensure its effectiveness and reliability.