Proxies for Academic Research: Uses, Ethics, Setup and Free Access for Researchers

Research guide · Last reviewed 1 October 2026 · 13 min read

Proxies for academic research let a study see the web the way people in a chosen country, city or network see it, and collect public web data at a scale one campus connection can’t. Researchers use them for web measurement, corpus building, cross-country comparisons of search results, ads and prices, and misinformation studies. This guide covers why researchers use proxy servers, which type suits which study, the ethics and data-protection steps a methods section should show, a Python setup on rotating residential proxies, and how to apply for free proxies for verified academic research.

What “proxy” means in research

The word turns up in research in three unrelated senses, and it helps to separate them before going further.

• A proxy variable or proxy measure. In research methods, a proxy is a measurable stand-in for something you can’t observe directly, such as household income standing in for socioeconomic status. That is a question of study design, not networking.
• A library proxy. Many university libraries run an authentication proxy so staff and students can reach the journals and databases the institution subscribes to from off campus. It works through your library login and your library’s licences. Use your library’s own service for subscribed content; a commercial proxy doesn’t give you subscription access.
• A proxy server for data collection. A server that forwards your requests so websites see the proxy’s IP address, location and network instead of your university’s. This is what this guide is about.

The rest of this page covers the third sense: proxy servers as a research instrument for studies that collect or observe public web content.

Why researchers use proxy servers

Much of the web is no longer the same for everyone. What a page shows can depend on the visitor’s country, city, language, network and device. A study run from one university network sees one version of it. A proxy lets the study choose where it looks from, and repeat that choice later.

Computational social science and web measurement

Web measurement studies ask how the web behaves: which trackers load on news sites, how cookie banners differ by country, which content is blocked where, how fast pages load on different networks. The answer depends on where the request comes from. A single campus address also tends to be a data centre or education network, which some sites treat differently from ordinary home connections. Residential and mobile proxies let a measurement run from the kinds of networks the study’s population actually uses.

Public web data for corpora and NLP datasets

Language resources, from domain-specific corpora to evaluation sets for NLP models, are often built from public web pages: news articles, forum posts, product descriptions, government notices, public reviews. Collection jobs that fetch many thousands of pages need to spread requests over time and addresses so that no single site is hit hard, and so the job isn’t stopped halfway through by per-IP limits. Before you crawl, check whether an existing open crawl archive such as Common Crawl already covers what you need; collecting only what an existing dataset lacks is faster, cheaper and easier to justify to an ethics board.

Search results, ads and prices across countries

Geo-differentiated content is one of the main reasons researchers use proxies. Search engines localise results, ad platforms target by location, and online shops and travel sites can show different prices or availability in different markets. To study that, you need requests that genuinely originate in each market, ideally from ordinary consumer networks, with everything else held constant. The same technique powers commercial ad verification and price monitoring, so the tooling is mature.

Misinformation and platform research

Researchers studying misinformation and online harms need to see what users in a given country are actually served: which search results appear for a claim, whether a video or post is geo-restricted in some countries, whether a warning label shows up in one market and not another, and how quickly a takedown takes effect in each. Observing that from the target country, rather than inferring it from the researcher’s own location, is the only way to measure it directly.

Reproducibility

If results depend on the vantage point, the vantage point is a variable, and an unrecorded variable makes a study hard to reproduce. A proxy with explicit targeting turns “whatever network the lab was on” into a stated, repeatable setting: country, city, ISP or ASN, connection type, and whether the IP was held for a session or rotated. Another team can use the same settings, and you can rerun your own study a year later from the same kind of network.

Which proxy type suits which study

The three main proxy types differ in where their IP addresses come from, and that decides what a website sees and how it treats your requests.

Residential proxies with city and ASN targeting

Residential proxies route requests through IP addresses that internet providers assign to ordinary households. To a website, the request looks like a home visitor in that place, which is what location-sensitive studies need. ProxyEmpire’s rotating residential network has 30M+ IPs in 179 countries and territories, with targeting by country, region or state, city, ZIP code, ISP, ASN and OS fingerprint, all included at no extra charge. ASN targeting matters for network-level questions: it lets you say that a set of requests came from one specific provider’s network, not just from one city.

Mobile carrier IPs

Mobile proxies send traffic out through real mobile carrier connections, so sites see a phone on a mobile network. Use them when the object of study is the mobile experience: mobile search, mobile ad delivery, content served to apps, or how sites treat carrier traffic. Carrier addresses are usually shared by many subscribers through carrier-grade NAT, so they behave differently from home addresses. ProxyEmpire’s rotating mobile proxies cover 4M+ mobile IPs in 174 countries, with carrier targeting included.

Datacenter proxies

Datacenter proxies use addresses hosted in data centres. They are fast and cheap, but websites can tell they aren’t consumer connections, and some treat them differently. That makes them a good fit for collecting from sites that don’t vary content by visitor type, such as many public-sector, archive and documentation sites, and a poor fit for studies about what ordinary users see. ProxyEmpire’s rotating datacenter proxies start from $0.35/GB with country-level targeting.

Research ethics and ethics review

Collecting public web data is still research, and where the data is about people, it may need ethics review before you start. The trigger isn’t the proxy; it is what you collect and what you do with it.

In the United States, the Common Rule (45 CFR 46) defines a human subject as a living individual about whom an investigator obtains information through intervention or interaction, or obtains, uses, studies, analyses or generates identifiable private information. “Private information” there covers behaviour in a context where a person can reasonably expect not to be observed or recorded, and information given for a specific purpose that a person can reasonably expect won’t be made public. Posts, profiles and comments can fall on either side of that line, which is why institutional review boards (IRBs) want to see the protocol. In the UK, the EU and elsewhere, a university research ethics committee plays the same role.

The Association of Internet Researchers’ Internet Research: Ethical Guidelines 3.0, approved by its membership in October 2019, is a widely used reference for this kind of work. It is written as a starting point for reflection rather than a checklist, and asks researchers to reason through the context of each project: whose data it is, what they would reasonably expect, and what harm publication could cause them.

For a proxy-based study, an ethics application typically needs to explain:

• what you will collect, from which sites, and why each field is needed;
• whether the data identifies people, and how you will pseudonymise, aggregate or discard it;
• how you will store it, who can access it, and when you will delete it;
• how you will limit load on the sites you collect from;
• which proxy service you use and how its IPs are sourced. ProxyEmpire’s residential network is ethically sourced.

GDPR and data minimisation

If your collection includes personal data about people in the UK or the EU, such as names, usernames, photos or anything that can identify someone, data protection law applies, whether or not the page was public. Using a proxy doesn’t change that: it hides your IP address from the site, not your obligations for the data you take away.

The principle that matters most for scraping is data minimisation, in Article 5(1)(c) of the GDPR: personal data must be “adequate, relevant and limited to what is necessary” for the purpose. The UK Information Commissioner’s Office (ICO) explains that as holding enough to do the job, with a rational link to the purpose, and no more; and it advises reviewing periodically and deleting what you no longer need. In practice:

• Write the parser to extract only the fields your research question needs, and drop the rest before storage.
• Pseudonymise user identifiers at collection time if you only need to link posts by the same account.
• Keep raw HTML only if you need it for validation, and set a deletion date.

The GDPR also has research provisions (Article 89). The ICO’s guidance on them explains that research with appropriate safeguards can be treated as compatible with the purpose the data was first collected for, and personal data processed only for research can be kept longer than usual, provided those safeguards are in place. It also stresses that processing should be targeted and proportionate. Your data protection officer is the right person to confirm the lawful basis for your project.

robots.txt, terms of service and rate limiting

robots.txt

The Robots Exclusion Protocol, standardised as RFC 9309 in 2022, is how site owners tell crawlers which paths they would prefer not to be crawled. The RFC is explicit that these rules are not a form of access authorization; they are a request from the site. It also sets out how a crawler should read the file: if robots.txt is unavailable (a 4xx status), a crawler may access anything; if it is unreachable because of a server error (5xx), the crawler must assume everything is disallowed; and a cached copy shouldn’t be used for more than 24 hours. Many research teams follow robots.txt as a matter of policy. Whatever you decide, state it in your protocol and your methods section. For the difference between discovering pages and extracting data from them, see web crawler vs web scraper.

Terms of service

Check the terms of each site you study, record what they say about automated access, and follow your institution’s guidance on how to weigh them against the research purpose; your ethics board or legal office can advise on your case.

Rate limiting

A proxy pool spreads requests across many addresses, but the target site still receives every one of them. Keep the load you put on each site low, whatever the pool size:

• cap concurrent requests per domain, and add a delay between requests to the same site;
• back off when a server answers 429 Too Many Requests (RFC 6585) or 503, and honour the Retry-After header if it sends one (RFC 9110), which can be a number of seconds or a date;
• schedule large jobs for the target’s off-peak hours;
• cache what you have already fetched, so reruns don’t fetch it again.

On identification, there is a genuine design choice. If your study doesn’t depend on looking like an ordinary visitor, a User-Agent string that names your project and links to a page about it lets site operators contact you. If the study is about what ordinary visitors see, a browser-like client may be part of the method; say so in your protocol.

Documenting vantage points in the methods section

A web study should say where its data was collected from, not just when. Record these for every collection run:

Also check where each exit IP geolocates, rather than assuming the targeting worked. IP geolocation databases don’t always agree with one another, so name the one you used. A short methods paragraph can then state the vantage point in a few sentences, for example:

Python setup example with requests

ProxyEmpire’s rotating residential and mobile proxies use one gateway: host v2.proxyempire.io, port 5000, with username and password authentication, so there is no IP allowlist to set up and the same code runs on a laptop, a lab server or an HPC node. Your targeting and session choices live in the proxy username: build them in the dashboard’s Connection Builder and copy the full string. The example below uses the shape of the sample on ProxyEmpire’s product pages, where country-us selects the United States and sid-123456 holds a sticky session.

The Requests library takes a proxies dictionary with the proxy URL in http://user:password@host:port form. Its documentation warns that proxies set on a session can be overridden by environment proxy variables, so the example passes proxies on each request, sets a timeout (Requests has none by default), and keeps credentials in environment variables rather than in code.

import json, os, time
from datetime import datetime, timezone
from urllib.parse import quote
import requests

USER = os.environ["PE_USER"]                # from the Connection Builder
PASS = quote(os.environ["PE_PASS"], safe="")  # percent-encode special characters
GATEWAY = "v2.proxyempire.io:5000"
HEADERS = {"User-Agent": "ExampleUni-WebStudy/1.0 (+https://example.edu/web-study)"}

def proxies_for(country, session_id):
    url = f"http://{USER}-country-{country}-sid-{session_id}:{PASS}@{GATEWAY}"
    return {"http": url, "https": url}

def fetch(session, url, proxies, tries=5):
    for attempt in range(tries):
        r = session.get(url, proxies=proxies, headers=HEADERS, timeout=(10, 30))
        if r.status_code in (429, 503):
            wait = r.headers.get("Retry-After", "")
            time.sleep(int(wait) if wait.isdigit() else 30 * (attempt + 1))
            continue
        return r
    return None

with requests.Session() as s, open("collection-log.jsonl", "a") as log:
    p = proxies_for("us", "123456")         # one sticky session for this task
    exit_ip = s.get("https://api.ipify.org", proxies=p, timeout=(10, 30)).text
    for url in ["https://example.com/page-1", "https://example.com/page-2"]:
        r = fetch(s, url, p)
        log.write(json.dumps({
            "utc": datetime.now(timezone.utc).isoformat(),
            "url": url, "country": "us", "session": "123456",
            "exit_ip": exit_ip, "status": r.status_code if r is not None else None,
        }) + "\n")
        time.sleep(5)                       # polite delay per domain

Keep the session ID the same and the exit IP stays the same, with no fixed time limit, until you change the ID or the device behind it goes offline; change the ID to get a new address. For a study that needs a new IP on every request, pick per-request rotation in the Connection Builder instead. The same gateway also speaks SOCKS5: install requests[socks] and use a socks5h:// URL if you want DNS resolved on the proxy side, which the Requests documentation describes.

For quick manual checks during a pilot, such as confirming what a page looks like from a target city, ProxyEmpire’s Chrome extension “ProxyEmpire Proxy Manager” switches the browser to a proxy without code. For mobile studies on Android, the free “ProxyEmpire Proxy Manager” app on Google Play routes a test device through the proxy.

What to know before you start

• Ports 80 and 443 only. By default the proxies carry web traffic on ports 80 and 443. Targets on other ports won’t connect.
• No UDP. UDP isn’t supported, so measurements that depend on it, such as direct DNS queries or QUIC and HTTP/3, can’t run through the proxy. Plan TCP-based alternatives.
• Mostly IPv4. Exit IPs are mostly IPv4, with some IPv6. There is no dedicated IPv6 product, so IPv6-only measurements need another setup.
• Some site categories are blocked. Financial, government, crypto (exchanges and DEXs), bank and payment sites are blocked on all proxy types. If your study needs them, unblocking requires KYC and a review of the use case; see are there any blocked websites.
• Estimate bandwidth with a pilot. Fetch a sample of your targets, measure the average bytes per page, and multiply by the pages you plan to collect. That figure is also what the free access application asks for.

Free proxies for academic research

ProxyEmpire can provide free proxies for academic research purposes once your academic credentials are verified. If you are working on a research project at a university or research institute and your study needs proxies, you can apply by email instead of buying a plan.

A clear description of the project and a realistic usage estimate make the request easier to review. If you want to test the setup first, the $1.97 trial includes 100 MB of residential and 50 MB of mobile traffic and doesn’t expire, and datacenter proxies start at a single GB for $2.

FAQ about research proxies

References

1. Association of Internet Researchers — “Internet Research: Ethical Guidelines 3.0” (2019). aoir.org/reports/ethics3.pdf
2. US Code of Federal Regulations — 45 CFR 46.102, definitions for the protection of human subjects (Common Rule). law.cornell.edu/cfr/text/45/46.102
3. UK Information Commissioner’s Office — “Principle (c): Data minimisation”. ico.org.uk — data minimisation
4. UK Information Commissioner’s Office — “The research provisions: principles and grounds for processing”. ico.org.uk — research provisions
5. IETF — RFC 9309, “Robots Exclusion Protocol” (2022). rfc-editor.org/rfc/rfc9309
6. IETF — RFC 6585, “Additional HTTP Status Codes” (429 Too Many Requests). rfc-editor.org/rfc/rfc6585
7. IETF — RFC 9110, “HTTP Semantics”, section 10.2.3 (Retry-After). rfc-editor.org/rfc/rfc9110
8. Requests documentation — Advanced Usage: proxies, SOCKS and timeouts. requests.readthedocs.io — advanced usage