Inquisitor

Inquisitor is a content aggregator for personal use. Unlike RSS aggregators, it is not restricted to content feeds of a particular format. Instead, feed items are generated by user-provided arbitrary Python snippets.

Producing items

Items are created and added to the Inquisitor feed with the update command.

inquisitor update provider1:argument provider2 ...

Source API for Inquisitor

The update command searches the source directory for source providers. The source directory is specified by --srcdir and defaults to ./sources. A source provider is any file in the source directory that ends in .py, validly imports as a Python 3 module, and provides a get_sources function.

The get_sources function implemented by a source provider should take one argument, a string. When a provider is specified as provider:argument, Inquisitor passes the string after the colon to that provider's get_sources when invoking it. Otherwise, the value is None. get_sources should return an iterable of sources. Inquisitor will execute get_sources on each source provider specified in the update invocation.

A source is a dictionary with two keyvalues. 'name' should map to a string containing the name of the source. 'fetch_new' should map to a function. The fetch_new function takes one argument, a state dictionary. This is used for caching a persistent state between invocations. It should return a list of items. Inquisitor will execute fetch_new for each source returned by a get_sources call, add the new items to the source's folder, and then clean out old, inactive items from that source.

Items are distinguished by their id field, which becomes the item's filename in the source's item folder. If an item already exists from a previous fetch_new call from that source, the new values for any fields will overwrite the old ones, except for the active and created fields.

Inquisitor API for sources

When a source provider is imported or a function from a provider or source is executed, the current working directory is the source provider directory.

The builtin Item() is provided for creating items. id and title are required parameters. Item() sets the item's source, created, and active fields automatically. The other item fields (link, ts, author, body, and tags) can be specified as keyword arguments to the builtin or set by the corresponding field on the returned Item object.

Consuming items

inquisitor run launches the built-in Flask server. The Flask development server is insecure and shouldn't be publicly exposed, but Inquisitor is for personal use, so it probably shouldn't be publicly exposed in the first place. You should either run the Inquisitor server on the same computer you plan to use for reading the feed or put it behind a reverse proxy only accessible to your local network.

The Inquisitor feed consists of items. Each item has a title. Items with a url field will have a link to open the URL in a new tab. All items have a button to mark them as read. Items marked as read will fade out. Items with a body field will expand to display the body when the title is clicked.

Feed logic

A source produces a list of items when called. Each of these items is written to a file and displayed in the feed. When an item is marked as read, its active field is marked as false. This is to prevent the item from reappearing in the feed if the source produces it the next time it is called. Conversely, an item continues to appear in the feed if it is still marked as active, even if its source no longer returns it. This ensures that all items from a source are shown to the user. An item is only removed from the source's folder once it is both inactive and no longer appears in the source

Some feeds may return an item, not return it later, then return it again. Normally, this would result in the same item appearing in the feed after having been deactivated. To prevent this, set the ttl field on an item with a timestamp. The item will not be cleaned up until the current time postdates the ttl value.