Screen Scraping: What It Is, How It Works and How It Differs from Web Scraping

Explainer · Last reviewed 28 September 2026 · 11 min read

Screen scraping is the practice of collecting data from what a program displays on screen, rather than from a database, a file or an API behind it. It began with software that read text from mainframe terminals, and today covers everything from reading desktop applications with automation tools to capturing rendered web pages. This guide explains what screen scraping is, how it works, how it differs from web scraping, where it is still used, and its technical and legal limits.

What Is Screen Scraping?

Every application stores its data somewhere and then displays it to a user: as characters on a terminal, as buttons and fields in a desktop window, or as a rendered web page. This technique works at that last step. A program looks at the display, or at the structure the operating system uses to draw it, and extracts the values a human would read.

The term comes from the mainframe era. Organisations with decades-old systems, reachable only through text terminals, wrote programs that pretended to be a terminal user, sent keystrokes, and read the characters that came back on the “screen”. That let new applications use old systems without rewriting them. The idea now appears in robotic process automation, accessibility tools and testing software.

How Screen Scraping Works

Terminal emulation

A program connects to a legacy system as if it were a terminal, sends commands and reads fixed positions on the returned screen, for example “row 5, columns 20 to 35 hold the account balance”. Fast and reliable as long as the screen layout never changes.

User interface automation

Desktop operating systems expose the structure of windows, fields and buttons to assistive technology such as screen readers. Microsoft UI Automation, for example, provides programmatic access to most user interface elements on the Windows desktop. Automation and robotic process automation (RPA) tools use these interfaces to read values from fields and to click through applications.

Optical character recognition (OCR)

When nothing structured is available, the program takes a screenshot and recognises the text in the image. Open-source engines such as Tesseract include a neural network (LSTM) engine focused on line recognition. OCR works on anything visible, but accuracy depends on fonts, resolution and layout.

Rendered web pages

On the web, a headless browser loads the page, runs its JavaScript and then either reads the rendered document or captures a screenshot. This sits on the boundary between the two approaches, and is common for sites that build their content in the browser. See our guide to headless browsers.

from PIL import Image
import pytesseract          # requires the Tesseract engine installed

img = Image.open("screen.png")
text = pytesseract.image_to_string(img, lang="eng")
for line in text.splitlines():
    if "Balance" in line:
        print(line.strip())

The tools in use today

Modern tools rarely use one technique alone. RPA platforms combine UI automation with OCR as a fallback for elements they cannot read, and they record a person’s clicks to build a bot. Testing frameworks drive browsers and desktop apps and compare what they display with expected values. Document-processing tools apply OCR to scans and then use machine learning to find fields such as dates, totals and names. Whatever the tool, the same rule holds: the more structure the tool can read, the more reliable the result. Reading an element’s value through an accessibility interface is exact; recognising the same value from pixels is an estimate.

Choosing between them is mostly a question of what the source exposes. A green-screen system exposes characters at fixed positions, a Windows application exposes an accessibility tree, a scanned invoice exposes only pixels, and a website exposes HTML and JSON. Start from the richest layer available.

That is why teams that start with OCR often move to UI automation, and teams that start with UI automation move to an API as soon as one becomes available. Each step up removes a layer of guessing, cuts maintenance and makes the data easier to trust.

Screen Scraping vs Web Scraping

Web scraping downloads a web page, or the data a page loads, and extracts values from its HTML or JSON. It works with the structure behind the page, not with its appearance. Its older cousin works with the appearance: characters on a terminal, elements in a window, pixels in a screenshot.

In practice, web scraping is usually the better choice for anything on the web. Most screen capture of websites today is really web scraping with a headless browser, used only for pages that cannot be read without running JavaScript.

Where Screen Scraping Is Used Today

• Legacy system integration. Connecting modern applications to mainframe or green-screen systems that have no API.
• Robotic process automation. Bots that copy data between desktop applications, fill in forms and process documents.
• Data migration. Extracting records from an old system being retired when direct database access is not possible.
• Testing and monitoring. Checking that an application or web page shows the right values, including visual regression tests with screenshots.
• Accessibility. Screen readers use the same interfaces to read applications aloud.
• Documents and images. OCR of scanned forms, receipts and PDFs that only exist as images.

Screen Scraping in Banking

The best-known use of the technique outside IT departments was in personal finance. Budgeting apps and payment services asked customers for their online banking username and password, logged in on their behalf and read balances and transactions from the pages. It worked, but it meant sharing banking credentials with third parties and gave banks no way to tell a customer’s app from an attacker.

Open banking rules changed that picture. In the European Union, the revised Payment Services Directive (PSD2) created regulated access for account information and payment initiation services with the customer’s consent, and further reforms of the framework are under way. Similar open banking and open finance rules exist or are being developed in other countries. Where these regulated routes exist, they are the right way to access account data.

Never scrape banking or payment sites with other people’s credentials. ProxyEmpire blocks banking, payment and other financial websites on all products by default.

Limits and Risks of Screen Scraping

• Fragility. A new font, a moved field or a different screen resolution can break extraction without any error, producing wrong values rather than none.
• Speed and cost. Rendering screens and running OCR takes far more computing than fetching data directly.
• Accuracy. OCR misreads characters such as 0 and O or 1 and l; financial or medical values need verification.
• Security. Automations often need stored credentials and broad access to desktops; secure them like any privileged account.
• Legal and contractual limits. Software licences, website terms, copyright and privacy law apply here just as they do to other data collection.

Choosing the Right Method

1. Official API or export. If the system or website offers an API, a data export or a bulk download, use it.
2. Web scraping of the HTML or JSON for public web data without an API.
3. Headless browser scraping when the page only works with JavaScript.
4. UI automation for desktop applications without an API.
5. OCR only when the data exists as images or pixels and nothing else.

Where Proxies Fit

Proxies do not help with terminal emulation or desktop automation, which run inside your own network. They matter when the data comes from public websites: web scraping and headless-browser capture send many requests to the same sites, which will slow down or block a single IP. Rotating residential proxies spread the requests over millions of household IPs, with country, city and ISP targeting at no extra charge so you capture the page each local visitor sees. Sticky sessions keep one IP for multi-step flows, such as moving through search results.

If you capture screenshots for visual monitoring, for example to check how a page, an ad or a price appears in different countries, mobile proxies show the mobile version as phone users on 4G and 5G networks see it. Our Chrome proxy settings guide shows how to route a browser through a proxy.

Screen Scraping FAQ

References

1. Microsoft Learn: UI Automation OverviewProgrammatic access to user interface elements on the Windows desktop
2. Tesseract OCROpen-source OCR engine with an LSTM-based recogniser
3. European Commission: Payment servicesPSD2 and its review