Using Hyperbrowser Session

Using Hyperbrowser's session

In this guide, we will see how to use Hyperbrowser and Puppeteer to create a new session, connect to it, and scrape current weather data.

Setup

First, lets create a new Node.js project.

mkdir weather-scraper && cd weather-scraper
npm init -y

Installation

Next, let's install the necessary dependencies to run our script.

npm install @hyperbrowser/sdk puppeteer-core dotenv

Setup your Environment

To use Hyperbrowser with your code, you will need an API Key. You can get one easily from the dashboard. Once you have your API Key, add it to your .env file as HYPERBROWSER_API_KEY.

Code

Next, create a new file index.js and add the following code:

import { Hyperbrowser } from "@hyperbrowser/sdk";
import { config } from "dotenv";
import { connect } from "puppeteer-core";

config();

const client = new Hyperbrowser({
  apiKey: process.env.HYPERBROWSER_API_KEY,
});

const main = async () => {
  const location = process.argv[2];
  if (!location) {
    console.error("Please provide a location as a command line argument");
    process.exit(1);
  }

  console.log("Starting session");
  const session = await client.sessions.create();
  console.log("Session created:", session.id);

  try {
    const browser = await connect({ browserWSEndpoint: session.wsEndpoint });

    const [page] = await browser.pages();

    await page.goto("https://openweathermap.org/city", {
      waitUntil: "load",
      timeout: 20_000,
    });
    await page.waitForSelector(".search-container", {
      visible: true,
      timeout: 10_000,
    });
    await page.type(".search-container input", location);
    await page.click(".search button");
    await page.waitForSelector(".search-dropdown-menu", {
      visible: true,
      timeout: 10_000,
    });

    const [response] = await Promise.all([
      page.waitForNavigation(),
      page.click(".search-dropdown-menu li:first-child"),
    ]);

    await page.waitForSelector(".current-container", {
      visible: true,
      timeout: 10_000,
    });
    const locationName = await page.$eval(
      ".current-container h2",
      (el) => el.textContent
    );
    const currentTemp = await page.$eval(
      ".current-container .current-temp",
      (el) => el.textContent
    );
    const description = await page.$eval(
      ".current-container .bold",
      (el) => el.textContent
    );

    const windInfo = await page.$eval(".weather-items .wind-line", (el) =>
      el.textContent.trim()
    );
    const pressureInfo = await page.$eval(
      ".weather-items li:nth-child(2)",
      (el) => el.textContent.trim()
    );
    const humidityInfo = await page.$eval(
      ".weather-items li:nth-child(3)",
      (el) => el.textContent.trim()?.split(":")[1]
    );
    const dewpoint = await page.$eval(
      ".weather-items li:nth-child(4)",
      (el) => el.textContent.trim()?.split(":")[1]
    );
    const visibility = await page.$eval(
      ".weather-items li:nth-child(5)",
      (el) => el.textContent.trim()?.split(":")[1]
    );

    console.log("\nWeather Information:");
    console.log("------------------");
    console.log(`Location: ${locationName}`);
    console.log(`Temperature: ${currentTemp}`);
    console.log(`Conditions: ${description}`);
    console.log(`Wind: ${windInfo}`);
    console.log(`Pressure: ${pressureInfo}`);
    console.log(`Humidity: ${humidityInfo}`);
    console.log(`Dew Point: ${dewpoint}`);
    console.log(`Visibility: ${visibility}`);
    console.log("------------------\n");

    await page.screenshot({ path: "screenshot.png" });
  } catch (error) {
    console.error(`Encountered an error: ${error}`);
  } finally {
    await client.sessions.stop(session.id);
    console.log("Session stopped:", session.id);
  }
};

main().catch((error) => {
  console.error(`Encountered an error: ${error}`);
});

Run the Scraper

To run the weather scraper:

1. Open a terminal and navigate to your project directory

2. Run the script with a location argument:

node index.js "New York"

Replace "New York" with the location you want weather data for.

The script will:

1. Create a new Hyperbrowser session

2. Launch a Puppeteer browser and connect to the session

3. Navigate to the OpenWeatherMap city page

4. Search for the specified location and hit the Search button

5. Select the first option from a list in a dropdown menu and navigate to that page

6. Scrape the current weather data from the page

7. Print the weather information to the console

8. Save a screenshot of the page

9. Close the browser and stop the Hyperbrowser session

You should see output like:

Weather Information:  
------------------ 
Location: New York City, US
Temperature: 9°C
Conditions: overcast clouds
Wind: Gentle breeze, 3.6 m/s, west-southwest  
Pressure: 1013 hPa
Humidity: 81%
Dew Point: 6°C 
Visibility: 10 km
------------------

And a screenshot.png file saved in your project directory.

How it Works

Let's break down the key steps:

1. We import the required libraries and load the environment variables

2. We create a new Hyperbrowser client with the API key

3. We start a new Hyperbrowser session with client.sessions.create()

4. We launch a Puppeteer browser and connect it to the Hyperbrowser session

5. We navigate to the OpenWeatherMap city page

6. We search for the location provided as a command line argument

7. We wait for the search results and click the first result

8. We scrape the weather data from the page using Puppeteer's page.$eval method

9. We print the scraped data, take a screenshot, and save it to disk

10. Finally, we close the browser and stop the Hyperbrowser session

Next Steps

This example demonstrates a basic weather scraping workflow using a Hyperbrowser session. You can expand on it to:

• Accept multiple locations and fetch weather data for each

• Get the 8-day forecast for the location

• Schedule the script to run periodically and save historical weather data