Webpage Text API Documentation

Overview

All calls to the Webpage Text API are authenticated using OAuth 2.0. Before making a call to the API, request an access token from the authentication servers. Use that access token until it expires, and then request another one from the authentication servers. Send that access token to Webpage Text API calls with an Authentication: Bearer access_token header.

I recommend that bodies of POST requests be in JSON format. You will need a Content-Type: application/json HTTP header with every request.

Response bodies for all requests are in JSON format.

If an API call is successful the response will have a status code of 200 (OK). If not it will have a 4xx or 5xx status code. A 4xx status code indicates a client-side error. A 5xx status code indicates a server-side error. These are the most likely 4xx and 5xx status codes:

400 (Bad Request): The request body was missing values or is somehow not in the expected format. The authentication endpoint will return this if the Client ID or Client Secret is incorrect.

401 (Unauthorized): The Webpage Text API will return this if the access token is not found, is expired, or is not valid.

404 (Not Found): The request URL is not valid.

429 (Too Many Requests): The monthly request limit or URL limit has been exceeded.

500 (Internal Server Error): A server-side error occurred. This could be the result of a temporary issue with our system. The Single Webpage Text Retrieval endpoint will return this if it is unable to retrieve or generate webpage text for the specified URL.

API Calls

Authentication

You will receive a Client ID and Client Secret as part of the provisioning process. Before making a request for webpage text, a client needs to retrieve an access token. You can request one with this endpoint:

POST https://auth.goldenhillsoftware.com/1.0/tokens

That request must include the following parameters:

client_id: The Client ID you received as part of the provisioning process.

client_secret: The Client Secret you received as part of the provisioning process.

scope: The value https://webpagetextapi.goldenhillsoftware.com/.

grant_type: The value client_credentials.

The response will be a JSON object with the following properties:

access_token: The access token to use for API calls.

expires_in: The number of seconds for which the access token is valid.

token_type: The value bearer.

scope: The value https://webpagetextapi.goldenhillsoftware.com/.

The client should use that access token until it expires, as specified by the expires_in response property.

Sample

curl \
 --header "Content-Type: application/json" \
 --request POST \
 --data '{ "client_id": "Your Client ID", "client_secret": "Your Client Secret", "scope": "https://webpagetextapi.goldenhillsoftware.com/", "grant_type": "client_credentials" }' \
 https://auth.goldenhillsoftware.com/1.0/tokens

{
     "access_token": "Your access token",
     "expires_in": 86400,
     "token_type": "bearer",
     "scope": "https://webpagetextapi.goldenhillsoftware.com/"
 }

Single Webpage Text Retrieval

Use this API call to retrieve webpage text for a single webpage URL:

POST https://webpagetextapi.goldenhillsoftware.com/1.0/retrievals

Include an Authorization header with the string Bearer access_token, replacing access_token with the access token obtained from the authentication endpoint.

This endpoint accepts the following request parameters:

url: The webpage URL. This is required.

include_linked_article: If true and if the URL is a Hacker News or Lobsters comments page about an external article, webpage text for that external article will also be included. Details are available in the Link Articles section below. This is optional. If specified, the value must be either true or false.

ignore_cached: If true, the Webpage Text API will ignore any cached information it has for the URL. It will retrieve the webpage again and regenerate the resulting webpage text content. Please only use this when there is reason to believe that the webpage text content has changed. For example, it is appropriate to offer a reload function and send ignore_cached: true when that function is performed. This will update the Webpage Text API’s caches if the webpage text content has changed. This is optional. If specified, the value must be either true or false.

The response will be a JSON object with the following properties:

url: The URL specified in the request.

response_url: The URL from which the webpage was retrieved. This may be different from the url property if the server issued a redirect.

title: The title of the webpage. This property will not be present if no title is found.

author: The author of the webpage. This property will not be present if no author is found.

html: The HTML of the main webpage content.

link_to_url: If the webpage is a Hacker News or Lobsters comments page about an external article, this will be the URL of that external article.

linked_article: If there is a link_to_url and the include_link_articles input parameter was set to true, this will be content about the linked article. Details are available in the Link Articles section below.

Sample

curl \
 --header "Content-Type: application/json" \
 --header "Authorization: Bearer Your Temporary Access Token" \
 --request POST \
 --data '{ "url": "https://www.goldenhillsoftware.com/2021/08/unread-27-improves-the-experience-of-reading-linked-list-articles/" }' \
 https://webpagetextapi.goldenhillsoftware.com/1.0/retrievals

 {
     "url": "https://www.goldenhillsoftware.com/2021/08/unread-27-improves-the-experience-of-reading-linked-list-articles/",
     "response_url": "https://www.goldenhillsoftware.com/2021/08/unread-27-improves-the-experience-of-reading-linked-list-articles/",
     "title": "Unread 2.7 Improves the Experience of Reading Linked List Articles",
     "author": "John Brayton",
     "html": "\u003cp\u003eUnread 2.7 is \u003ca href=\"https://apps.apple.com/us/app/unread-2/id1363637349\"\u003eavailable from the App Store\u003c/a\u003e. This update improves the experience of reading linked list articles and adds other improvements. Additional HTML removed."
 }

Bulk Webpage Text Retrieval

Use this API call to retrieve webpage text for up to 100 webpage URLs:

POST https://webpagetextapi.goldenhillsoftware.com/1.0/bulk-retrievals

Include an Authorization header with the string Bearer access_token, replacing access_token with the access token obtained from the authentication endpoint.

This accepts the following request parameters:

urls: Between 1 and 100 unique URLs. This is required.

include_linked_article: If true, then for each article that is a Hacker News or Lobsters comments page about an external article, webpage text for that external article will also be included. Details are available in the Link Articles section below. This is optional. If specified, the value must be either true or false.

The response will be a JSON array. Each element will have the following properties:

url: The URL of the requested webpage.

status: The status of the retrieval, specified as an HTTP status code. A value of 200 indicates success. A 4xx value indicates a client-side error. A 5xx value indicates that the server was unable to fulfill the request.

result: A JSON object with the retrieved webpage text result. This property will not be present if status is not 200.

Each result object will have the following properties:

url: The URL specified in the request.

response_url: The URL from which the webpage was retrieved. This may be different from the url property if the server issued a redirect.

title: The title of the webpage. This property will not be present if no title is found.

author: The author of the webpage. This property will not be present if no author is found.

html: The HTML of the main webpage content.

link_to_url: If the webpage is a Hacker News or Lobsters comments page about an external article, this will be the URL of that external article.

Sample

curl \
 --header "Content-Type: application/json" \
 --header "Authorization: Bearer Your Temporary Access Token" \
 --request POST \
 --data '{ "urls":["https://www.goldenhillsoftware.com/2021/08/unread-27-improves-the-experience-of-reading-linked-list-articles/","https://www.goldenhillsoftware.com/2021/06/unreads-compact-and-expansive-article-list-formats/"] }' \
 https://webpagetextapi.goldenhillsoftware.com/1.0/bulk-retrievals

 [
     {
         "url": "https://www.goldenhillsoftware.com/2021/08/unread-27-improves-the-experience-of-reading-linked-list-articles/",
         "status": 200,
         "result": {
             "url": "https://www.goldenhillsoftware.com/2021/08/unread-27-improves-the-experience-of-reading-linked-list-articles/",
             "response_url": "https://www.goldenhillsoftware.com/2021/08/unread-27-improves-the-experience-of-reading-linked-list-articles/",
             "title": "Unread 2.7 Improves the Experience of Reading Linked List Articles",
             "author": "John Brayton",
             "html": "<p>Unread 2.7 is <a href=\"https://apps.apple.com/us/app/unread-2/id1363637349\">available from the App Store</a>. This update improves the experience of reading linked list articles and adds other improvements. Additional HTML removed."
         }
     },
     {
         "url": "https://www.goldenhillsoftware.com/2021/06/unreads-compact-and-expansive-article-list-formats/",
         "status": 200,
         "result": {
             "url": "https://www.goldenhillsoftware.com/2021/06/unreads-compact-and-expansive-article-list-formats/",
             "response_url": "https://www.goldenhillsoftware.com/2021/06/unreads-compact-and-expansive-article-list-formats/",
             "title": "Unread\u2019s Compact and Expansive Article List Formats",
             "author": "John Brayton",
             "html": "<figure><img src=\"https://www.goldenhillsoftware.com/unread26rsrcs/articlelistpost/hero.jpeg\" width=\"2000\" height=\"1125\"></figure> <p>Last week I <a href=\"https://www.goldenhillsoftware.com/2021/06/unread-26-adds-full-text-search-a-compact-article-list-option-for-iphone-and-more/\">released Unread 2.6</a> with full-text search capabilities, a compact article list option for iPhone, and more. Additional HTML removed."
         }
     }
 ]

Link Articles

The Webpage Text API identifies comments pages on Hacker News and Lobsters that discuss external webpages as Link articles.

For each such article, the result has a link_to_url attribute that identifies the external article. If the include_link_article input parameter is true, webpage text results will include that external article.

The article result will include a linked_article attribute with these properties:

url: The URL of the linked webpage.

status: The status of the linked webpage, specified as an HTTP status code. A value of 200 indicates success. A 4xx value indicates a client-side error. A 5xx value indicates that the server was unable to fulfill the request.

result: A JSON object with the retrieved webpage text result. This property will not be present if status is not 200.

That result will have these properties:

url: The URL specified in the request.

response_url: The URL from which the webpage was retrieved. This may be different from the url property if the server issued a redirect.

title: The title of the webpage. This property will not be present if no title is found.

author: The author of the webpage. This property will not be present if no author is found.

html: The HTML of the main webpage content.

link_to_url: If the webpage is a Hacker News or Lobsters comments page about an external article, this will be the URL of that external article.

The Webpage Text API does not support nesting of these results. If one Hacker News comments page is a link to another, there will be no linked_article attribute for the latter page.