OpenAI CUA
OpenAI's Computer-Using Agent (CUA) is an advanced AI model designed to interact with computer interfaces much like a human user. It can navigate graphical user interfaces (GUIs), perform tasks such as clicking buttons, typing text, and managing multi-step processes. This capability allows CUA to automate complex workflows without relying on specialized APIs. Actually, this is the model that powers Operator, OpenAI's AI agent capable of executing various web-based tasks like filling out forms, ordering products, and scheduling appointments. Operator utilizes CUA to mimic human interactions within a browser, allowing it to handle repetitive or intricate tasks on behalf of users.
Hyperbrowser's CUA agent allows you to easily execute agent tasks on the web utilizing OpenAI's Computer-Using Agent with just a simple call. Hyperbrowser exposes endpoints for starting/stopping a CUA task and for getting it's status and results.
By default, CUA tasks are handled in an asynchronous manner of first starting the task and then checking it's status until it is completed. However, if you don't want to handle the monitoring yourself, our SDKs provide a simple function that handles the whole flow and returns the data once the task is completed.
Installation
npm install @hyperbrowser/sdk
or
yarn add @hyperbrowser/sdk
Usage
import { Hyperbrowser } from "@hyperbrowser/sdk";
import { config } from "dotenv";
config();
const hbClient = new Hyperbrowser({
apiKey: process.env.HYPERBROWSER_API_KEY,
});
const main = async () => {
const result = await hbClient.agents.cua.startAndWait({
task: "what are the top 5 posts on Hacker News",
});
console.log(`Output:\n\n${result.data?.finalResult}`);
};
main().catch((err) => {
console.error(`Error: ${err.message}`);
});
CUA tasks can be configured with a number of parameters. Some of them are described briefly here, but a list can be found in our OpenAI CUA API Reference.
Reuse Browser Session
You can pass in an existing sessionId
to the OpenAI CUA task so that it can execute the task on an existing session. Also, if you want to keep the session open after executing the task, you can supply the keepBrowserOpen
param.
import { Hyperbrowser } from "@hyperbrowser/sdk";
import { config } from "dotenv";
config();
const hbClient = new Hyperbrowser({
apiKey: process.env.HYPERBROWSER_API_KEY,
});
const main = async () => {
const session = await hbClient.sessions.create();
try {
const result = await hbClient.agents.cua.startAndWait({
task: "What is the title of the first post on Hacker News today?",
sessionId: session.id,
keepBrowserOpen: true,
});
console.log(`Output:\n${result.data?.finalResult}`);
const result2 = await hbClient.agents.cua.startAndWait({
task: "Tell me how many upvotes the first post has.",
sessionId: session.id,
});
console.log(`\nOutput:\n${result2.data?.finalResult}`);
} catch (err) {
console.error(`Error: ${err}`);
} finally {
await hbClient.sessions.stop(session.id);
}
};
main().catch((err) => {
console.error(`Error: ${err.message}`);
});
Session Configurations
You can also provide configurations for the session that will be used to execute the CUA task just as you would when creating a new session itself. These could include using a proxy or solving CAPTCHAs. To see the full list of session configurations, checkout the Session API Reference.
import { Hyperbrowser } from "@hyperbrowser/sdk";
import { config } from "dotenv";
config();
const hbClient = new Hyperbrowser({
apiKey: process.env.HYPERBROWSER_API_KEY,
});
const main = async () => {
const result = await hbClient.agents.cua.startAndWait({
task: "what are the top 5 posts on Hacker News",
sessionOptions: {
acceptCookies: true,
}
});
console.log(`Output:\n\n${result.data?.finalResult}`);
};
main().catch((err) => {
console.error(`Error: ${err.message}`);
});
Last updated