new post: rsyncing my RSS feed database
This commit is contained in:
parent
6e65c5aa90
commit
b166cdfbe5
129
_posts/2024-10-28-rsyncing-my-rss-feed-database.md
Normal file
129
_posts/2024-10-28-rsyncing-my-rss-feed-database.md
Normal file
|
|
@ -0,0 +1,129 @@
|
|||
---
|
||||
permalink: "/{{ year }}/{{ month }}/{{ day }}/rsyncing-my-rss-feed-database"
|
||||
title: "rsyncing my RSS feed database"
|
||||
published_date: "2024-10-28 20:45:00 +0100"
|
||||
layout: post.liquid
|
||||
data:
|
||||
route: blog
|
||||
excerpt: |
|
||||
At the moment I'm using a web-based RSS feed reader.
|
||||
That works, is reliable and accessible from anywhere.
|
||||
|
||||
However I will soon spend some time with less connectivity
|
||||
and thus were considering my options to have a local feed reader app that works offline.
|
||||
A long while ago I used Newsbeuter and then I recently found its successor Newsboat.
|
||||
That solves the local & offline.
|
||||
But it still requires connectivity to fetch feeds,
|
||||
and for fetching some 100 feeds that's quite a bit of traffic.
|
||||
---
|
||||
|
||||
At the moment I'm using a web-based RSS feed reader[^1].
|
||||
That works, is reliable and accessible from anywhere.
|
||||
|
||||
However I will soon spend some time with less connectivity
|
||||
and thus were considering my options to have a local feed reader app that works offline.
|
||||
A long while ago I used [Newsbeuter] and then I recently found its successor [Newsboat].
|
||||
|
||||
It can import feeds from an OPML file, so that's what I did.
|
||||
|
||||
|
||||
|
||||
That solves the local & offline.
|
||||
But it still requires connectivity to fetch feeds,
|
||||
and for fetching some 100 feeds that's quite a bit of traffic.
|
||||
|
||||
So it would be better to fetch that on a server.
|
||||
That's doable for example by sticking `newsboat -x reload` into a cronjob on the server.
|
||||
But then I still need to fetch down the database when I want to read new stuff.
|
||||
|
||||
Doable, but the database quickly grows large enough that this is prohibitive on slow and spotty connections.
|
||||
Oh, and also Newsboat stores whether an article was read and I don't want to lose that.
|
||||
So I need to occasionally sync back the local database to the server,
|
||||
so the unread status is not overwritten.
|
||||
|
||||
[rsync] could handle this.
|
||||
Just push up the local database, reload feeds, sync back the updated database[^2].
|
||||
|
||||
That's when I stumbled upon the new [sqlite-rsync].
|
||||
It's labeled as a "Database Remote-Copy Tool For SQLite"
|
||||
and essentially compares the SQLite pages remotely and locally
|
||||
and only transfers those that are different on the replica.
|
||||
|
||||
To get the tool[^3]:
|
||||
|
||||
```
|
||||
cd /tmp
|
||||
git clone https://github.com/sqlite/sqlite.git
|
||||
cd sqlite
|
||||
./configure
|
||||
make sqlite3_rsync
|
||||
```
|
||||
|
||||
Put the `slite3_rsync` binary both locally and on the server in a folder in the `$PATH`.
|
||||
Then saves this script as `newsboat-sync`:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
|
||||
# CHANGE THIS:
|
||||
HOST=user@host
|
||||
LOCAL=~/.newsboat/cache.db
|
||||
REMOTE=.newsboat/cache.db
|
||||
|
||||
echo "Sync up READ status"
|
||||
sqlite3_rsync -v $LOCAL $HOST:$REMOTE
|
||||
|
||||
if [[ "$1" = "-u" ]]; then
|
||||
echo "READ status sync only. Exiting."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo "Updating feeds"
|
||||
ssh $HOST ' \
|
||||
[[ $(($(date +%s) - $(cat ~/.newsboat/lastfetch.time 2>/dev/null || echo 0))) -ge 900 ]] && \
|
||||
{ echo "Updating feeds (remote)"; newsboat -x reload; date +%s > ~/.newsboat/lastfetch.time; } || echo "No update" \
|
||||
'
|
||||
|
||||
echo "Sync back new items"
|
||||
sqlite3_rsync -v $HOST:$REMOTE $LOCAL
|
||||
```
|
||||
|
||||
Now when invoked, it
|
||||
|
||||
1. syncs up the current state,
|
||||
2. if at least 15 minutes (900 seconds) passed since the last fetch, fetches all feeds,
|
||||
3. syncs down the database to the local machine.
|
||||
|
||||
```
|
||||
$ newsboat-sync
|
||||
Sync up READ status
|
||||
sent 4,114 bytes, received 174,994 bytes, 320,407.87 bytes/sec
|
||||
total size 34,131,968 speedup is 190.57
|
||||
Updating feeds
|
||||
Updating feeds (remote)
|
||||
Sync back new items
|
||||
sent 174,994 bytes, received 53,326 bytes, 253,971.08 bytes/sec
|
||||
total size 34,131,968 speedup is 149.49
|
||||
```
|
||||
|
||||
That's quite a speedup for a database that is 34 MB in size.
|
||||
|
||||
```
|
||||
$ du -h ~/.newsboat/cache.db
|
||||
34M /Users/jer/.newsboat/cache.db
|
||||
```
|
||||
|
||||
Now I can hopefully read new blog posts offline with minimal data transfer overhead.
|
||||
|
||||
[newsbeuter]: https://en.wikipedia.org/wiki/Newsbeuter
|
||||
[newsboat]: https://newsboat.org/
|
||||
[rsync]: https://rsync.samba.org/
|
||||
[sqlite-rsync]: https://sqlite.org/rsync.html
|
||||
|
||||
---
|
||||
|
||||
_Footnotes:_
|
||||
|
||||
[^1]: A self-hosted [Stringer](https://github.com/stringer-rss/stringer) instance.
|
||||
[^2]: `rsync` would totally work just fine for this use case, but then i wouldn't get to try new tools.
|
||||
[^3]: via [simonw on lobsters](https://lobste.rs/s/2ngsl1/database_remote_copy_tool_for_sqlite#c_3hlhlz)
|
||||
Loading…
Reference in a new issue