Getting Book Prices from AbeBooks.com

Reverse Engineering a REST API

Often, we encounter the need to extract data from websites that lack publicly documented REST APIs. In this post, I demonstrate how to analyze POST and GET requests from AbeBooks.com using a browser and build a Python API wrapper to programmatically retrieve book prices and recommendations.

Python
API
books
Published

April 10, 2020

Modified

August 3, 2026

Note

I wrote this blog post about five years ago, and it remains one of my most viewed posts. If this topic is of interest to you, there was a recent blog posted on Hacker News which details even more cool ways to find undocumented REST APIs: All the data can be yours

Motivation

I have a large collection of electronic books, which I manage using Calibre. Using Calibre’s “Extract ISBN” plugin, I am able to parse the ISBN identifier from most of my files, which then makes fetching the rest of the metadata very easy. (Below is an example of my library’s metadata.)

Thus, I have access to a very convenient and ever-growing virtual library of books, which I like to use on the go, and for exploratory research. Nevertheless, whenever I find a particularly good book, the thing that I want most, is to own a physical copy.

A screenshot of my calibre e-book library

Enter here AbeBooks.com. Next to Amazon, and occasionally Ebay, it is my go-to site for buying cheap used textbooks. Given that I have stored the ISBN data for most of my electronic books, I would like to be able to automatically fetch pricing information for any book in my virtual library, perhaps even keeping track of changes in price over time.

However, until now, the main problem stopping me from writing a script to do this was that AbeBooks does not have a publicly available API… or at the very least, none that is explicitly documented.

AbeBooks search results page

REST APIs

REST, or Representational State Transfer, is a method used by the HTTP protocol to provide interoperability between servers. It is based on a request/response system, where a request is a “payload”, normally formatted as HTML, XML, or JSON, and the response can be a link to a resource, a data payload in any of the aforementioned formats, or a confirmation that some data was modified in the server.

Several common REST methods exist: GET, HEAD, POST, PUT, PATCH, DELETE, CONNECT, OPTIONS and TRACE. Among these, the two most common are GET and POST:

GET

  • Used to request data from a server.
  • Parameter data is stored in the URL of the query as string parameters
  • Number of parameters is limited to the length that can fit in the URL
  • Not secure for sensitive information. (Passwords can be easily seen)

POST

  • Used to submit data to a server, and can modify server contents.
  • Parameters are passed in the message body, rather than the URL.
  • It has no restrictions on the number of parameters.
  • Is more secure for sending sensitive information.

Exploring Network Packets

I found that inspecting the network packets for an AbeBooks search results page is simple, and yields promising results. If we open Firefox’s developer tools, under the Network tab, we can see a list of all the packets that are loaded. In particular we are interested in those that have a JSON response, highlighted in red below:

Network packet inspection in Firefox

We can see that there are four POST requests, to a service called “pricingservice”, and one GET request to a “RecomendationsApi”.

If we look more closely at one of the POST requests, we can see which parameters it takes in:

POST request parameters

ISBN! Just what we needed! Furthermore, looking at the response tab, we can see that this request returns the prices for new and used books, among other things:

Example POST response

Wrapping the API in Python

Now that we know a bit more about how AbeBooks works under the hood, we can start implementing our API wrapper in Python. We will need the requests module:

import requests

Sending POST requests

The first REST method that we will implement is the POST method that fetches prices for a given book. From inspecting the page elements, we know that the URL for this service is:

url = "https://www.abebooks.com/servlet/DWRestService/pricingservice"

There seem to be three main parameter groups, and we can infer their purpose. (Parameters shown in bold below are to be replaced by user values)

Searching prices by ISBN
Parameter Value
action getPricingDataByISBN
isbn isbn
container pricingService-isbn
Searching prices by title and author
Parameter Value
action getPricingDataForAuthorTitleStandardAddToBasket
an author
tn title
container oe-search-all
Searching prices by title, author, and hardcover/softcover binding
Parameter Value
action getPricingDataForAuthorTitleBindingRefinements
isbn 9781250297662
an author
tn title
container priced-from-soft OR priced-from-hard

The parameters can be stored as keys in a dictionary, and sent to the request’s post method. For example:

# Search prices by ISBN
payload1 = {'action': 'getPricingDataByISBN',
           'isbn': 9781250297662,
           'container': 'pricingService-9781250297662'}

# Search prices by author and title
payload2 = {'action': 'getPricingDataForAuthorTitleStandardAddToBasket',
            'an': 'liu ken',
            'tn': 'broken stars',
            'container': 'oe-search-all'}

# Sending a request
resp = requests.post(url, data=payload1)
print(resp.status_code, resp.reason)
resp.json()

The response is:

200 OK
{'errorTexts': [None],
 'errorCodes': [None],
 'success': True,
 'newExists': True,
 'usedExists': True,
 'pricingInfoForBestNew': {'bestListingid': 32081713842,
  'totalResults': 6,
  'bestPriceInPurchaseCurrencyWithCurrencySymbol': 'US$ 28.07',
  'bestPriceInSurferCurrencyWithCurrencySymbol': 'US$ 28.07',
  'shippingToDestinationPriceInPurchaseCurrencyWithCurrencySymbol': 'US$ 0.00',
  'shippingToDestinationPriceInSurferCurrencyWithCurrencySymbol': 'US$ 0.00',
  'shippingDestinationNameInSurferLanguage': 'U.S.A.',
  'vendorCountryNameInSurferLanguage': 'U.S.A.',
  'vendorId': 8304062,
  'bestPriceInPurchaseCurrencyValueOnly': '28.07',
  'bestShippingToDestinationPriceInPurchaseCurrencyValueOnly': '0.00',
  'purchaseCurrencySymbol': 'US$',
  'refinementList': None,
  'internationalEdition': False,
  'bookCondition': 'New',
  'bookDescription': 'LOCUS AWARD FINALIST FOR BEST ANTHOLOGYSixteen...',
  'freeShipping': True},
 'pricingInfoForBestUsed': {'bestListingid': 31422279270,
  'totalResults': 12,
  'bestPriceInPurchaseCurrencyWithCurrencySymbol': 'US$ 8.05',
  'bestPriceInSurferCurrencyWithCurrencySymbol': 'US$ 8.05',
  'shippingToDestinationPriceInPurchaseCurrencyWithCurrencySymbol': 'US$ 0.00',
  'shippingToDestinationPriceInSurferCurrencyWithCurrencySymbol': 'US$ 0.00',
  'shippingDestinationNameInSurferLanguage': 'U.S.A.',
  'vendorCountryNameInSurferLanguage': 'U.S.A.',
  'vendorId': 51315977,
  'bestPriceInPurchaseCurrencyValueOnly': '8.05',
  'bestShippingToDestinationPriceInPurchaseCurrencyValueOnly': '0.00',
  'purchaseCurrencySymbol': 'US$',
  'refinementList': None,
  'internationalEdition': False,
  'bookCondition': 'Good',
  'bookDescription': 'Former library book; may include...',
  'freeShipping': True},
 'pricingInfoForBestAllConditions': None,
 'isbn': '9781250297662',
 'totalResults': 18,
 'containerId': 'pricingService-9781250297662',
 'refinementList': [{'name': 'collectibleSigned',
   'label': 'Signed',
   'count': 1,
   'url': 'attrs=sgnd&ds=5&isbn=9781250297662&sgnd=on&sortby=17',
   'soutaRequest': {'debug': None,
    'rollup': None,
    'deep': None,
    'facets': True,
    'bsaExpandedFacets': None,
    'strict': None,
    'shippingDestination': 'USA',
    'shippingMode': None,
    'currency': 'USD',
    'clientId': None,
    'sort': 'TOTAL_PRICE_ASC',
    'pageSize': 5,
    'pageDescriptor': None,
    'featureFlags': None,
    'query': {'domain': 'COM',
     'booleanQuery': None,
     'isbns': [{'isbn13': True, 'isbn10': False}],
     'contributor': None,
     'title': None,
     'publisher': None,
     'keyword': None,
     'minYearPublished': None,
     'maxYearPublished': None,
     'minPrice': None,
     'maxPrice': None,
     'createdWithin': None,
     'minCreatedTimestamp': None,
     'maxCreatedTimestamp': None,
     'includedLanguages': None,
     'includedProductTypes': None,
     'requiredAttributes': ['SIGNED'],
     'includedConditions': None,
     'requiredBsaCodes': None,
     'requiredFreeShippingCountries': None,
     'minSellerRating': None,
     'sellerProvState': None,
     'sellerCountry': None,
     'includedSellerRegions': None,
     'sellerCatalog': None,
     'requiredSellerGroups': None,
     'includedSellers': None,
     'excludedSellers': None,
     'includedListingIds': None,
     'excludeSellerPictures': None}}},
  {'name': 'collectibleJacket',
   'label': 'Dust Jacket',
   'count': 1,
   'url': 'attrs=dj&dj=on&ds=5&isbn=9781250297662&sortby=17',
   'soutaRequest': {'debug': None,
    'rollup': None,
    'deep': None,
    'facets': True,
    'bsaExpandedFacets': None,
    'strict': None,
    'shippingDestination': 'USA',
    'shippingMode': None,
    'currency': 'USD',
    'clientId': None,
    'sort': 'TOTAL_PRICE_ASC',
    'pageSize': 5,
    'pageDescriptor': None,
    'featureFlags': None,
    'query': {'domain': 'COM',
     'booleanQuery': None,
     'isbns': [{'isbn13': True, 'isbn10': False}],
     'contributor': None,
     'title': None,
     'publisher': None,
     'keyword': None,
     'minYearPublished': None,
     'maxYearPublished': None,
     'minPrice': None,
     'maxPrice': None,
     'createdWithin': None,
     'minCreatedTimestamp': None,
     'maxCreatedTimestamp': None,
     'includedLanguages': None,
     'includedProductTypes': None,
     'requiredAttributes': ['JACKET'],
     'includedConditions': None,
     'requiredBsaCodes': None,
     'requiredFreeShippingCountries': None,
     'minSellerRating': None,
     'sellerProvState': None,
     'sellerCountry': None,
     'includedSellerRegions': None,
     'sellerCatalog': None,
     'requiredSellerGroups': None,
     'includedSellers': None,
     'excludedSellers': None,
     'includedListingIds': None,
     'excludeSellerPictures': None}}},
  {'name': 'bindingHard',
   'label': 'Hardcover',
   'count': 18,
   'url': 'attrs=hc&bi=h&ds=5&isbn=9781250297662&sortby=17',
   'soutaRequest': {'debug': None,
    'rollup': None,
    'deep': None,
    'facets': True,
    'bsaExpandedFacets': None,
    'strict': None,
    'shippingDestination': 'USA',
    'shippingMode': None,
    'currency': 'USD',
    'clientId': None,
    'sort': 'TOTAL_PRICE_ASC',
    'pageSize': 5,
    'pageDescriptor': None,
    'featureFlags': None,
    'query': {'domain': 'COM',
     'booleanQuery': None,
     'isbns': [{'isbn13': True, 'isbn10': False}],
     'contributor': None,
     'title': None,
     'publisher': None,
     'keyword': None,
     'minYearPublished': None,
     'maxYearPublished': None,
     'minPrice': None,
     'maxPrice': None,
     'createdWithin': None,
     'minCreatedTimestamp': None,
     'maxCreatedTimestamp': None,
     'includedLanguages': None,
     'includedProductTypes': None,
     'requiredAttributes': ['HARDCOVER'],
     'includedConditions': None,
     'requiredBsaCodes': None,
     'requiredFreeShippingCountries': None,
     'minSellerRating': None,
     'sellerProvState': None,
     'sellerCountry': None,
     'includedSellerRegions': None,
     'sellerCatalog': None,
     'requiredSellerGroups': None,
     'includedSellers': None,
     'excludedSellers': None,
     'includedListingIds': None,
     'excludeSellerPictures': None}}},
  {'name': 'collectibleFirstEdition',
   'label': 'First Edition',
   'count': 1,
   'url': 'attrs=fe&ds=5&fe=on&isbn=9781250297662&sortby=17',
   'soutaRequest': {'debug': None,
    'rollup': None,
    'deep': None,
    'facets': True,
    'bsaExpandedFacets': None,
    'strict': None,
    'shippingDestination': 'USA',
    'shippingMode': None,
    'currency': 'USD',
    'clientId': None,
    'sort': 'TOTAL_PRICE_ASC',
    'pageSize': 5,
    'pageDescriptor': None,
    'featureFlags': None,
    'query': {'domain': 'COM',
     'booleanQuery': None,
     'isbns': [{'isbn13': True, 'isbn10': False}],
     'contributor': None,
     'title': None,
     'publisher': None,
     'keyword': None,
     'minYearPublished': None,
     'maxYearPublished': None,
     'minPrice': None,
     'maxPrice': None,
     'createdWithin': None,
     'minCreatedTimestamp': None,
     'maxCreatedTimestamp': None,
     'includedLanguages': None,
     'includedProductTypes': None,
     'requiredAttributes': ['FIRST_EDITION'],
     'includedConditions': None,
     'requiredBsaCodes': None,
     'requiredFreeShippingCountries': None,
     'minSellerRating': None,
     'sellerProvState': None,
     'sellerCountry': None,
     'includedSellerRegions': None,
     'sellerCatalog': None,
     'requiredSellerGroups': None,
     'includedSellers': None,
     'excludedSellers': None,
     'includedListingIds': None,
     'excludeSellerPictures': None}}}],
 'bibliographicDetail': None}

In our example, we have 6 results and the best price for a new book bestPriceInPurchaseCurrencyWithCurrencySymbol is $28.07, and the best price for a used book is $8.05.

Sending a GET request

The API also has a GET method for obtaining book recommendations given an ISBN. The url and parameter names are different, but the way we send the request is very similar:

url = "https://www.abebooks.com/servlet/RecommendationsApi"
Get recommended books from an ISBN
Parameter Value
pageId plp
itemIsbn13 isbn 
# Get book recommendations by ISBN
payload = {'pageId': 'plp',
           'itemIsbn13': 9781250297662}

resp = requests.get(url, params=payload)
print(resp.status_code, resp.reason)
resp.json()

Response:

200 OK
{'widgetResponses': [{'slotName': 'detail-1',
    'title': 'Customers who bought this item also bought',
    'algoName': 'abeBooksBlendedPurchaseSims',
    'ref': 'pd_b_p_1',
    'recommendationItems': [{'attributes': [],
        'thumbNailImgUrl': 'https://pictures.abebooks.com/isbn/9780765384201-us-300.jpg',
        'itemLink': '/products/isbn/9780765384201?cm_sp=rec-_-pd_b_p_1-_-plp&reftag=pd_b_p_1',
        'subTitle': None,
        'isbn13': '9780765384201',
        'title': 'Invisible Planets: Contemporary Chinese Science Fiction...',
        'author': 'Liu, Ken'},
    {'attributes': [],
        'thumbNailImgUrl': 'https://pictures.abebooks.com/isbn/9781250306029-us-300.jpg',
        'itemLink': '/products/isbn/9781250306029?cm_sp=rec-_-pd_b_p_1-_-plp&reftag=pd_b_p_1',
        'subTitle': None,
        'isbn13': '9781250306029',
        'title': 'The Redemption of Time: A Three-Body Problem Novel...',
        'author': 'Baoshu'},
    {'attributes': [],
        'thumbNailImgUrl': 'https://pictures.abebooks.com/isbn/9780765389312-us-300.jpg',
        'itemLink': '/products/isbn/9780765389312?cm_sp=rec-_-pd_b_p_1-_-plp&reftag=pd_b_p_1',
        'subTitle': None,
        'isbn13': '9780765389312',
        'title': 'Waste Tide',
        'author': 'Qiufan, Chen'},
    {'attributes': [],
        'thumbNailImgUrl': 'https://pictures.abebooks.com/isbn/9780765384195-us-300.jpg',
        'itemLink': '/products/isbn/9780765384195?cm_sp=rec-_-pd_b_p_1-_-plp&reftag=pd_b_p_1',
        'subTitle': None,
        'isbn13': '9780765384195',
        'title': 'Invisible Planets: Contemporary Chinese Science Fiction...',
        'author': 'Liu, Ken'},
    {'attributes': [],
        'thumbNailImgUrl': 'https://pictures.abebooks.com/isbn/9781784978518-us-300.jpg',
        'itemLink': '/products/isbn/9781784978518?cm_sp=rec-_-pd_b_p_1-_-plp&reftag=pd_b_p_1',
        'subTitle': None,
        'isbn13': '9781784978518',
        'title': 'The Wandering Earth',
        'author': 'Liu, Cixin'}]},
    {'slotName': 'ext-search-detail-1',
    'title': None,
    'algoName': 'heroWidgetIsbnSims',
    'ref': 'pd_hw_i_1',
    'recommendationItems': [{'attributes': [],
        'thumbNailImgUrl': 'https://pictures.abebooks.com/isbn/9780804172448-us-300.jpg',
        'itemLink': '/products/isbn/9780804172448?cm_sp=rec-_-pd_hw_i_1-_-plp&reftag=pd_hw_i_1',
        'subTitle': 'Best Selling',
        'isbn13': '9780804172448',
        'title': 'Station Eleven',
        'author': 'Mandel, Emily St. John'},
    {'attributes': [],
        'thumbNailImgUrl': 'https://pictures.abebooks.com/isbn/9781786073495-us-300.jpg',
        'itemLink': '/products/isbn/9781786073495?cm_sp=rec-_-pd_hw_i_1-_-plp&reftag=pd_hw_i_1',
        'subTitle': 'Top Rated',
        'isbn13': '9781786073495',
        'title': 'Zuleikha',
        'author': 'Yakhina, Guzel'}]}]}

An Object-Oriented Module

I created a small Python module abebooks.py to encapsulate the requests. The full code is below:

Show the code 
import requests


class AbeBooks:

    def __get_price(self, payload):
        url = "https://www.abebooks.com/servlet/DWRestService/pricingservice"
        resp = requests.post(url, data=payload)
        resp.raise_for_status()
        return resp.json()

    def __get_recomendations(self, payload):
        url = "https://www.abebooks.com/servlet/RecommendationsApi"
        resp = requests.get(url, params=payload)
        resp.raise_for_status()
        return resp.json()

    def getPriceByISBN(self, isbn):
        """
        Parameters
        ----------
        isbn (int) - a book's ISBN code
        """
        payload = {'action': 'getPricingDataByISBN',
                   'isbn': isbn,
                   'container': 'pricingService-{}'.format(isbn)}
        return self.__get_price(payload)

    def getPriceByAuthorTitle(self, author, title):
        """
        Parameters
        ----------
        author (str) - book author
        title (str) - book title
        """
        payload = {'action': 'getPricingDataForAuthorTitleStandardAddToBasket',
                   'an': author,
                   'tn': title,
                   'container': 'oe-search-all'}
        return self.__get_price(payload)

    def getPriceByAuthorTitleBinding(self, author, title, binding):
        """
        Parameters
        ----------
        author (str) - book author
        title (str) - book title
        binding(str) - one of 'hard', or 'soft'
        """
        if binding == "hard":
            container = "priced-from-hard"
        elif binding == "soft":
            container = "priced-from-soft"
        else:
            raise ValueError(
                    'Invalid parameter. Binding must be "hard" or "soft"')
        payload = {'action': 'getPricingDataForAuthorTitleBindingRefinements',
                   'an': author,
                   'tn': title,
                   'container': container}
        return self.__get_price(payload)

    def getRecommendationsByISBN(self, isbn):
        """
        Parameters
        ----------
        isbn (int) - a book's ISBN code
        """
        payload = {'pageId': 'plp',
                   'itemIsbn13': isbn}
        return self.__get_recomendations(payload)

Using the AbeBooks Module

from abebooks import AbeBooks

ab = AbeBooks()
results = ab.getPriceByISBN(9780062941503)
if results['success']:
    best_new = results['pricingInfoForBestNew']
    best_used = results['pricingInfoForBestUsed']
# Best New Price
print(best_new['bestPriceInPurchaseCurrencyWithCurrencySymbol'])
US$ 21.49
# Best Used Price
print(best_used['bestPriceInPurchaseCurrencyWithCurrencySymbol'])
US$ 24.42