Yesterday I released Unread 3.0 with Unread Cloud. Unread Cloud is a new syncing and article retrieval system for Unread.
Unread Cloud has five subsystems: the Subscription Manager, the Feed Retriever, the Article Repository, the Article Status Manager, and the API Gateway. Each subsystem has its own PostgreSQL database, a Rails app with a REST API, a Sidekiq-based background processor, and a Redis cache.
External Dependencies
Unread Cloud also has two external dependencies: an Authentication Service and the Webpage Text API.
The Authentication Service handles the server-side component of creating or logging in to Unread Cloud accounts via Sign in with Apple, and periodically refreshing authentication tokens. Since Local accounts also use Unread Cloud for efficient article retrieval, the Authentication service also allows the app to create and delete Local accounts. I will publish more detail about how Local accounts work differently from Unread Cloud accounts in a subsequent post.
Unread Cloud uses the Webpage Text API to incorporate author names and hero images into articles from feeds that omit them. Unread Cloud also uses the Webpage Text API to determine which feeds are summary-only and to set appropriate default webpage text options on imported subscriptions. Caching of webpage text from summary-only feeds is done outside the context of Unread Cloud. When using an Unread Cloud account or a Local account, Unread retrieves webpage text directly from the Webpage Text API just as it does when using an external feed service account.
Subsystems
The Subscription Manager stores the set of feeds to which each customer subscribes. This includes customer-specific feed titles, tags, and webpage text settings. Its responsibilities include:
- Being the authority for the current set of subscriptions when the app refreshes an account.
- Processing requests to subscribe to a feed, to update a subscription, to delete a subscription, or to bulk import a set of subscriptions.
- Keeping the Feed Retriever updated with the set of feeds it needs to poll.
- Keeping the Article Repository updated with the set of feeds to which each customer subscribes.
- Using the Webpage Text API, determining which feeds are summary-only feeds and setting default webpage text options for imported subscriptions.
The Feed Retriever checks feeds for new articles and changes to articles. It does not store article content, but it stores enough metadata for each article to know when an article has been seen and to detect when an article has changed. When it sees a new or changed article it sends that article to the Article Repository. The Feed Retriever does not know who is subscribed to what feed, but it has an active subscriber count for each feed. Feed Retriever responsibilities include:
- Sending new articles and changed articles to the Article Repository.
- Using the Webpage Text API, supplementing article content with hero images and author names when a feed omits them.
- Determining how often each feed should be polled, taking into account the number of active subscribers, how often the feed publishes new articles, and whether recent attempts to poll the feed have been successful.
- Sending polling status changes to the Subscription Manager. Feed polling status is synced with Unread, and Unread reports polling failures on its Feed Errors screen.
The Article Repository stores articles. It knows what users are subscribed to what feeds, but does not know anything about an individual user or feed other than their unique identifiers. The Article Repository also stores what users have saved what articles. Its responsibilities include:
- Receiving new articles and article updates from the Feed Retriever.
- Receiving subscription additions and removals from the Subscription Manager.
- Processing requests from the app to save or unsave an article.
- Responding to requests from the app for lists of article IDs for articles that are new, have been updated, have been saved, and have been unsaved since the last refresh.
- Responding to requests from the app for article content for specific articles.
- Deleting old articles. An old article is deleted if it is more than 90 days old, there are at least 10 newer articles from the same feed, and no user has saved that article.
- Responding to requests from the Article Status Manager for a list of current article identifiers for a specific user. The Article Status Manager uses this to delete information for articles that have been aged out.
The Article Status Manager stores which articles have been read or marked read for each account. Its responsibilities include:
- Being the authority for the set of articles marked read and for article-specific webpage text settings when the app refreshes an account.
- Accepting requests from the app to mark articles as read or unread.
- Accepting requests from the app to mark a specific article as one where the user prefers webpage text, feed text, or both, when that setting differs from the default for the associated feed.
- Deleting information pertaining to articles that have aged out of an account.
The API Gateway accepts requests from the app and forwards those request to an appropriate subsystem. This is the only subsystem of the five that accepts requests directly from the app. Its responsibilities include:
- Authenticating requests, and routing them to the appropriate subsystem: the Subscription Manager, the Article Status Manager, or the Article Repository.
- Accepting and processing notifications from the Authentication Service that an account has been deleted. When an account has been deleted, it is up to the API gateway to tell each of the other subsystems that they should delete their data associated with that account. The API Gateway is also responsible for rejecting otherwise-valid requests associated with a deleted account.
If you have not yet tried Unread 3, you can download it from the App Store.