Bright Data Web Unlocker (How-to and Best Tips) 2022

bright-data-icon-small
Verified
Staff Pick

Get a $250 bonus when you add $250 *

For a limited time only. Top up your account with any amount, and Bright data will match an equal amount of credits.
$250
$250
Rating
5
On-Going Offer

Who wants the unwanted barriers while scraping publicly available data on the internet? Usually, when you make a scraping script to scrape and save data from websites like Amazon and Alibaba, it works fine, as long as you run it very often. But when this process is the basis of your business, and you need to scale scraping and collect massive data, your scraper may get blocked due to rate-limiting. Rate-limiting is the feature that blocks a user’s IP when the user sends requests to a website aggressively (too many within a short time).

That’s why businesses use residential proxies to scrape data. But sometimes, you do not want to worry about the blocking and need an advanced API that automatically rotates the IP address when it gets blocked. You need Bright data web unlocker proxy. 🏆

What is a bright data web unlocker?

Some websites on the internet are tough to reach, especially when you aim to extract data (at scale) for whatever purpose. 

Do you know what can help you to get that? The answer is pretty obvious now, and that is Bright Data Web Unlocker. 

Web Unlocker is one technology that can handle all site unblocking problems and deliver the data just as you expect. When you scrape data from a website like amazon.com or alibaba.com, you need to ensure that you’re not blocked due to rate-limiting. To avoid blocking with the rate-limiting rule, you can use web unlocker. Web unlocker proxy will automatically rotate the IP address if the website blocks your scraper’s IP address.

This unblocking, extracting info, and structuring is all done via a bunch of automated protocols applied while attempting data collection. You can use a tool like Octoparse or Phantombuster. Bright Data Web Unlocker is excellent. But how does it work? Keep reading. 

How does Bright Data Web Unlocker work?

It’s pretty clear that web unlocking happens because of a series of automated protocols, but how does Bright Data make it possible? 

Why not look at the steps to understand its job more deeply? Here we go.

  1. As a user, you first signup to Bright Data and request the team to unlock a particular website or page. 
  2. This request is sent to the web unlocker algorithm that takes care of the rest. 
  3. What Web Unlocker Algorithm does is edit the request as required, mainly the header and protocol. This modified request is then sent to the Super Proxy. 
  4. Super Proxy is where all the proxy-related queries are shared with the main data infrastructure. You know how important this station is in the whole web unlocking journey. 
  5. Super Proxy shares the web unlocking request with the Bright Data Proxy Infrastructure. This infrastructure then used the user fingerprint on your behalf for visitor identification and goes to the target website. (Don’t worry, this has nothing to do with your personal information or identification.)
  6. The target website then shares the information to the Data Infrastructure, which goes to the Super Proxy, then Web Unclocker Algorithm, and straight to you, the user. 

This may look like an extensive, time-consuming process, but it happens so fast that you won’t find any delay in accessing your target website. All you need to do is make a simple request; the rest is Bright Data’s responsibility. 

You can integrate web unlocker proxy with your script in any language or software. 

No matter what you use for scraping (Shell script, Node.js app, PHP scripting, or Ruby software), you can generate the command/code using this tool.

How to use Web Unlocker?

Well, it does a pretty good job of giving you insights into any target website you have been seeking. But the question is, how does it work for you? Here are a bunch of easy steps that take you to that locked-up website. 

Create an account (Skip if you already have one)

  • First, you visit the site of Bright Data, and from the top menu, select the proxy. 
  • You will find a drop-down menu from where you would need to select the web unlocker option. 
  • On the Web Unlocker page, you will find 2 options. If you wish, you can look at this service’s pricing. Otherwise, follow the steps as we go. 
  • For starters, click on the ‘Start Now button. 
  • This button will take you to a form asking you for the details to help you create an account with Bright Data. 
  • All you need to do is fill in the information, tick the ‘Agreement and Policy’ option, and click on the Create Account button. 
  • You can choose a strong password for your Bright Data Account on the next page. 
  • You know how it works. Right? Follow a bunch of steps until you are ready with your Bright Data account. 
  • Once your account is ready, you will find yourself on the Bright Data Dashboard with multiple options as shown in the image. 

Add the zone (The Important Steps)

On the dashboard, you will see multiple options but don’t be overwhelmed, we are here to guide you.

  • From these options, choose the Web Unlocker and click on the ‘Start Now’ Button right under the tiny description. 
  • Now you can simply type the website address you want to scrape. The Smart wizard of Bright Data will recommend you Web Unlocker for sites it fits best. 
  • Now you can configure your proxy zone. 
  1. A zone name. 
  2. Choose how narrow targeting you want for IP geolocation.
  3. Linkedin Unblocking permissions (only if you want to scrape linkedin).
  • You can see how much you will be charged for 1000 requests on the right side. Pricing for the pay-as-you-go plan is $3/CPM.
  • You can reduce this cost by upgrading to a higher plan. The cost comes around $2.10/CPM on Advanced plan. But in this plan, you need a commitment of $1000/month budget(paid annually).
  • This is best if you need to make around 476k requests in a month.
  • Getting back to the proxy, click on “Add Zone” button.
  • Confirm the creation of the zone. 
  • You will see the success message. By default, the zone has No limit on the number of requests it can make. You can set the limit by clicking the pencil icon here.
  • Now you’re ready to send requests to your target website and scrape data in HTML or JSON format.

Generate the Command

On your left sidebar, inside the API & Integration menu, go to Proxy and Web Unlocker API.

Here, you can check the wizard where you can generate the command for your automation script. 

If you’re using a VPS to scrape the data using a bash script, you can simply generate a CURL command (this is the first step for scraping a website).

Choose the Zone, Country and other options for the command. You can leave the Destination site black in the wizard and make it dynamic. 

Simply put your dynamic destination URL tag in place of “http://lumtest.com/myip.json

Now, you’re good to go ahead and execute this code using crontab. Also, if you want to save the output of the scraped data, you can add the following code to the script.

> /root/output.html

Cheers.

Testing in Terminal on your Mac

To test the command in your Mac Terminal App, you can simply choose “Getting Started” as the Example. 

Copy the command.

Open Terminal and paste the command there. 

You can run this command and the whole Response of target URL (http://lumtest.com/myip.json) will be printed in your Terminal.

It works.

Best tips to use Web Unlocker

You know how Bright Data Web Unlocker works using the shell and PHP script, but some tips will help you make the best of this service. 

You never know when one of these or maybe all these tips will come in handy once you start unlocking your web world. 

1. Be specific about target website while scraping

Web Unlocker is an advanced premium proxy. That integrates a super proxy that automatically changes when it receives an adverse HTTP response. But in some cases, you may not need Web Unlocker, but a simple residential proxy can do the task.

2. Use only when you are getting blocked

Some websites do not implement rate-limiting. As a result, you can scrape those website freely. Just a simple rotating proxy can work.
After running a several scraping tasks, if you see frequent IP blocks, you can make use of Web Unlocker Proxy.

3. Little knowledge would be the best

Bright Data’s services are renowned for their amateur usability. Someone with minimal technical knowledge can perform technical tasks like a pro. However, you might need a little kick when it comes to Web Unlocker. 
Wondering what I mean? If you are a beginner with no technical knowledge, you can unlock any website but if you are aware of things like Python, you can take this service miles further. You know what I mean now. 

Bright Data Web Unlocker Pros and Cons

Pros

  • An automated tool saves your time and resources.
  • Untraceable with real-time adjustment.
  • Resolves the captcha problem with machine learning.
  • Bypasses all the bot detection systems.
  • Worldwide geo coverage.
  • Unlimited concurrent connections.

Cons

  • Costlier than self-managed proxies

More to check

Conclusion 

Facing restrictions, permission denial, and going through utmost security formalities, we all have been there while scraping. Credit goes to all the creators who have come up with the most satisfactory solution to escape these problems. 

Web Unlocker is the right way to beat anything that stops you from exploring the boundaries of the internet world. Try it out once. You never know what treasure you might extract. 

Leave a Comment