-*- coding: utf-8 -*-

Calibre OPDS client

What’s this?

This is a calibre plugin that is an OPDS client intended to read the contents of another calibre installation, find the differences to the current calibre and offer to copy books from the other calibre into the current calibre

How do I install it?

Requires git and calibre installed:

• Clone the repository:

  git clone https://github.com/steinarb/opds-reader.git
      

• Install the plugin in calibre

  cd opds-reader/calibre_plugin/
  calibre-customize -b .
      

• Start calibre (if calibre was already running, stop calibre and start it again)
• Click the button “Preferences”
• In the dialog “calibre - Preferences”:
  • Under “Interface”, click on the button “Toolbar”
  • In the dialog “calibre - Preferences - Toolbar”:
    • In the dropdown, select “The main toolbar”
    • In “Available actions” scroll down to find “OPDS Client” and select it
    • Click the top arrow button (arrow pointing right)
    • Click the “Apply” button
  • Click the “Close” button

How do I use it?

I made this tool to backup my book collection between two PCs in my home LAN, and that is the procedure I will document here:

1. In the calibre you wish to copy from (in this example called calibre1.home.lan):
   1. Click Preferences
   2. In the “calibre - Preferences” dialog:
      1. Click “Sharing over the net”
      2. In the “calibre - Preferences - Sharing over the net” dialog:
         1. Click the “Start Server” button
         2. Select the checkbox “Run server automatically when calibre starts”
         3. Click the “Apply” button
      3. Click the “close” button
2. In the calibre you wish to copy to
   1. Install this plugin (see the “How do I install it?” section)
   2. Click the “OPDS client” button
   3. In the “OPDS client” dialog
      1. Edit the “OPDS URL” value, change

         http://localhost:8080/opds
                     

         to

         http://calibre1.home.lan:8080/opds
                     

         and then press the RETURN key on the keyboard

      2. Click the “Download OPDS” button
      3. Wait until the OPDS feed has finished loading (this may take some time if there is a large number of books to load)
         • Note: if no books appear, try unchecking the “Hide books already in the library” checkbox. If that makes a lot of books appear, it means that the two calibre instances have the same books
      4. select the books you wish to copy into the current calibre and click the “Download selected books”
         • calibre will start downloading and installing the books:
           • The Jobs counter in calibre’s lower right corner, will show a decrementing number and the icon will spin
           • The book list will be updated as the books are downloaded
      5. The downloaded books will be in approximately the same order as in the original, but the time stamp will be the download time. To fix the time stamp, click on the “Fix timestamps of the selection” button
         • The updated timestamps may not show up immediatly, but they will show up after the first update of the display, and the books will be ordered according to the timestamp after stopping and starting calibre

License

This calibre plugin is copyright Steinar Bang, 2015, and licensed Under GPL version 3.

See the LICENSE file for more detail.

List of TODO items [27/41]

Create an icon

• The plugin API seems to mandate one for a plugin that is to add an action to the GUI
• Need an icon that doesn’t have any copyright limitations, it doesn’t have to be pretty

Put a list of books into the dialog

• <2015-08-23 søn 08:50> Will use a QTableView
  • import:

    from PyQt5.Qt import QTableView
            

  • Tried using the derived class BookView, but this failed because the parent (OpdsDialog) was missing the field iactions
  • Will try filling the QtTableView with BooksModel which derives from QAbstractTableModel

    from calibre.gui2.library.models import BooksModel
            

  • The argument to BooksModel.setData will be a list of SearchResult instances

    from calibre.gui2.store.search_result import SearchResult
            

  • SearchResult contains the following fields:
    • store_name
    • cover_url
    • cover_data
    • title
    • author
    • price
    • detail_item
    • drm
    • formats
    • downloads
      • dictionary
    • affilate
      • boolean
    • plugin_author
    • create_browser
  • SearchResult equality is determined in the following way:

    return self.title == other.title and self.author == other.author and self.store_name == other.store_name and self.formats == other.formats
            

    • The “formats” part of the comparison may be an issue when comparing with the current database? Could be that a comparison that excludes formats may be needed? E.g. I may want to keep ORIGINAL_EPUB in the calibre where I did the conversion, but not bother copying it to other calibres
  • An example usage of creating a list of SearchResult is in calibre in MobileReadStore.deserialize_books

Read RSS from the plugin and put the resulting data into a datastructure compatible with the calibre db API

• <2015-08-23 søn 10:17> Will try putting the resulting data into SearchResult initially (as mentioned in the previous TODO item)
• <2015-08-23 søn 11:05> The feedparser is already present in calibre
  • import statement:

    from calibre.web.feeds.feedparser import parse
            

• <2015-08-23 søn 12:02> The BooksModel used in view.py is directly connected to the database, i.e. can’t use that BooksModel
  • Instead created a new BooksModel patterned on the one in the mobileread store
  • Use a new data structure OpdsBook instead of SearchResult

Populate the list in the GUI with data read from the RSS

• <2015-08-23 søn 10:18> Hopefully this will be as simple as calling BooksModel.setData

Make the list of books look a little better (resize the dialog to make room for everything)

Add a checkbox to filter out newspapers

Add a checkbox to filter out books already present in the library

Make the book datamodel be Metadata (add a field to hold the parsed OPDS structure), and parse all available metadata info

Add a download button to download the selected books

• <2015-09-04 fre 21:44> Makes a list of all book download links with EPUB first if found and download the first URL in the list

Fix line height after updates

Move OPDS reading to the model and use the model refresh instead of setting a new model

Restore the authors to the OPDS book list

Make sure all of the books in the library are listed

• <2015-09-05 lør 10:34> Calibre is set up by default to only deliver 30 items
• <2015-09-05 lør 23:20> Following the “next” links of the feed until there are no more “next”

Give feedback on the number of OPDS books downloaded

• <2015-09-06 søn 12:43> Perhaps update the list after each 30 book chunk has been added?

Get a display value back for “updated”

• <2015-09-06 søn 12:28> The value is back, but now there is a value for the initial empty lines

Reverse the order of requested downloads

• <2015-09-06 søn 08:12> The idea is that what’s started first will finish first and that this will give the same book order in the two calibres

Set the date/time of the copied book to the date of the original

• <2015-09-05 lør 23:08> The date returned by “Updated” was the same for all books and the date/time of the last change to the db of the remote calibre
  • Opened an issue to replace this with the Metadata.timestamp attribute of the book: https://bugs.launchpad.net/bugs/1492651
• <2015-09-06 søn 09:25> Use the ajax.py REST API to download the metadata for the remote book
  • Endpoints are:
    • /ajax/book/{book_id}/{library_id=None}
      • Return the metadata of the book as a JSON dictionary.
      • Query parameters: ?category_urls=true&id_is_uuid=false&device_for_template=None
    • /ajax/books/{library_id=None}
      • Return the metadata for the books as a JSON dictionary.
      • Query parameters: ?ids=all&category_urls=true&id_is_uuid=false&device_for_template=None
    • /ajax/categories/{library_id=None}
      • Return the list of top-level categories as a list of dictionaries
      • Each category has the form:

        {
          "name": "Display Name",
          "url": "URL that gives the JSON object corresponding to all entries in this category",
          "icon": "URL to icon of this category",
          "is_category": "False for the All Books and Newest categories, True for everything else"
        }
                        

    • /ajax/category/{encoded_name}/{library_id=None}
      • Return a dictionary describing the category specified by name
      • Query parameters: ?num=100&offset=0&sort=name&sort_order=asc
      • Response:

        {
            "category_name": "Category display name",
            "base_url": "Base URL for this category",
            "total_num": "Total numberof items in this category",
            "offset": "The offset for the items returned in this result",
            "num": "The number of items returned in this result",
            "sort": "How the returned items are sorted",
            "sort_order": "asc or desc",
            "subcategories": "List of sub categories of this category.",
            "items": "List of items in this category"
        }
                        

    • /ajax/books_in/{encoded_category}/{encoded_item}/{library_id=None}
      • Return the books (as list of ids) present in the specified category
      • Query parameters: ?num=100&offset=0&sort=title&sort_order=asc&get_additional_fields=
    • /ajax/search/{library_id=None}
      • Return the books (as list of ids) matching the specified search query.
      • Query parameters: ?num=100&offset=0&sort=title&sort_order=asc&query=
• <2015-09-06 søn 10:30> Tried using the API
  • http://edwards.hjemme.lan:8080/ajax/books/ returned a 404
  • Added an

    Accept: application/json
            

    header to the GET, but still got a 404

• <2015-09-14 ma 23:56> Timestamps are now received and placed correctly on the book Metadata objects from the OPDS feed, the downloaded metadata hasn’t been updated yet
• <2015-09-15 ti 22:47> Added a “Fix timestamps of selection” button that can be clicked after download
  • I would have preferred to do this automatically, but that would require a callback that could be called after download, and no such callback system exists
  • The “Fix timestamps of selection” button only works for books that have a single author, the db.find_identical_books() method returns nothing when the metadata has more than one author

Make the “Fix timestamps of selection button” work for books with more than one author

Format the book list with different color for alternate lines

Add licensing information (GPLv3) on the project level

Add licensing and copyright information to all files

Add recently used dropdown to the opds_url configuration

Make the OPDS parsing more robust (hardcoded to the default structure of calibre right now)

Put the OPDS feed combobox on the main dialog

Add a catalog selection combobox

• <2015-09-20 sø 12:13> Non-editable, populated from the catalogs in the root catalog when downloading a new feed

Add search in downloaded results

• <2015-09-26 lø 19:55> Solved, but not entirely satsified with the results:
  • Only the currently visible items are searchable
  • Only the search matches are shown, instead of scrolling to the matches while showing all
    • Probably the best that can be done, using just built-in Qt functionality?

Read OPDS feeds other than calibre

• <2015-09-18 fr 19:01> Some examples
  • feedbooks:

    http://www.feedbooks.com/books/top.atom?category=FBFIC028000&lang=en
            

    or

    http://www.feedbooks.com/catalog.atom
            

  • Internet archive:

    http://bookserver.archive.org/catalog/
            

  • Pragmatic bookshelf:

    http://pragprog.com/magazines.opds
            

  • ManyBooks:

    http://www.manybooks.net/opds/index.php
            

  • Project Gutenberg:

    http://m.gutenberg.org/ebooks/?format=opds
            

  • O’Reilly:

    http://opds.oreilly.com/opds/
            

  • Baen ebooks:

    http://www.baenebooks.com/stanza.aspx
            

Add auto discovery of calibre instances in the LAN

• <2015-09-06 søn 11:49> Perhaps use the Bonjour protocol? (is this what FBReaderJ uses?)

Find out why the OPDS reader dialog sometimes disappear after downloading the OPDS

Add username/password information to saved opds_url values

Migrate own code from underscore separation to camelCase (Python has a camelCase modula/pascal feel to it)

Find out why some books (in PDF…?) aren’t downloaded

Explore the documentation format to see if it is relevant to this plugin

Try to keep the line length correct during intermediate model updates

• <2015-09-06 søn 16:10> When updating the book list model after each OPDS chunk, the line heights are wrong
  • They are corrected after the final read but they look a bit silly during the intermediate chunks

Get better matching with existing books (the “Maven cookbook” was already present, but it still showed up)

Add configuration options for defaults for the “hide” checkboxes

Remove all leftover debug trace

Copy read marks in calibre’s reader from the remote

Refresh the list as books are downloaded (suppress downloaded books from the list)

Add cover thumbnails to the list of books

Add an exclusion list (a list of books that should be permanently hidden from the comparison)