Wiki-to-print/Installation log/Servpub
Jump to navigation
Jump to search
Install day
13 Dec 2023
https://git.vvvvvvaria.org/varia/wiki-to-print
https://cc.vvvvvvaria.org/wiki/Running_wiki-to-print_on_the_server
https://github.com/hackersanddesigners/wiki2print
requirements
- mediawiki
- localsettings: https://git.vvvvvvaria.org/varia/wiki-to-print/src/branch/master/wiki-to-print.LocalSettings.php.example
- common.css: https://git.vvvvvvaria.org/varia/wiki-to-print/src/branch/master/wiki-to-print.Common.css.example
- common.js: https://git.vvvvvvaria.org/varia/wiki-to-print/src/branch/master/wiki-to-print.Common.js.example
- php-fpm
- mysql/mariadb
- nginx
- config: https://git.vvvvvvaria.org/varia/wiki-to-print/src/branch/master/wiki-to-print.nginx.example
- extensions: https://cc.vvvvvvaria.org/wiki/Special:Version
- web app
- flask
- nginx config: https://git.vvvvvvaria.org/varia/wiki-to-print/src/branch/master/wiki-to-print.nginx.example
- beautifulsoup
- (gunicorn)
- pandoc
- service file: https://git.vvvvvvaria.org/varia/wiki-to-print/src/branch/master/wiki-to-print.service.example
Servpub
for computational publishing
why 2 pi's? wanting to have many nodes, flexibility to grow, but also not enough space on the servpub pi
might even be 3 pis in the end
---> aarhus
---> wiki2print
---> servpub
maybe doing this with other communities, e.g. NeON
becky:
initially exciting initially that pis could be mobile
bringing wiki2print into the workshop space
geoff:
expanded network is the idea
manetta:
will wikitoprint be used in the future?
geoff:
yes, possibly for TM again
and maybe for a publication about all of this endeavour
we would invite all contributors so far to this project
batool:
it's nice to talk about different parts of the project
we also don't have the full picture
-----
to join tmux session:
$ sudo su -
# tmux attach
OS is armbian
check your OS
$ cat /etc/os-release
we'll see if Armbian software supports wiki2print
geanolgy of documentation :: hackers and designers -- > martino (flask) -->> based on book of manetta,& jara rocha
flask allowed from 1 wiki to make multiple books
preview pdf / link mediawiki with pandoc
Newspaper "Toward a Minor Tech", made with CC and participants at Transmediale workshop, 2022
https://cc.vvvvvvaria.org/wiki/Pdf:Toward_a_Minor_Tech
https://wiki2print.hackersanddesigners.nl/ (returns an error atm)
https://titipi.org/wiki-to-pdf/
paged.js --- a javascript library that renders HTML into PDF preview in the browser
https://pagedjs.org/
paged.js tries to stay close to W3C, thinking is based on single text flow - https://www.w3.org/
flask --> { content (wiki page) + css (wiki page) --> pandoc --> HTML --> HTML + paged.js (pdf) }
wiki-to-pdf
wiki2print
wiki-to-print
wiki4print
wikitooooooooprint
https://wiki2print.servpub.net/
MediaWiki documentation: https://www.mediawiki.org/wiki/Manual:Installing_MediaWiki
Download mediawiki:
$ wget https://releases.wikimedia.org/mediawiki/1.40/mediawiki-1.40.1.tar.gz
Unzip the tar:
$ tar -xzvf mediawiki-*.tar.gz
MediaWiki requires:
- PHP 7.4.3+
on CC we installed php7.4-fpm
- a webserver software
- and either MariaDB 10.3.0+, MySQL 5.7.0+, SQLite 3.8.0+ or PostgreSQL 10.0+. Using MariaDB or MySQL is recommended as Wikipedia uses MariaDB.
tmux scroll //
Ctrl-b then [ then you can use your normal navigation keys to scroll around (eg. Up Arrow or PgDn). Press q to quit scroll mode.
Install PHP:
$ sudo apt-get install php-fpm
Check the version:
$ php -v
We now have PHP 8.1.2-1ubuntu2.14
tmux note:
ctrl b (then let go) then use square brackets to scroll through printed history
q (to get back to command line)
Checking the version of nginx:
$ nginx -v
Install database software (we are going to use mariadb server):
$ sudo apt-get install mariadb-server
Where to install the wiki on the server?
we copied the wiki2print to the /var/www directory and put mediawiki inside wiki2print
Set up nginx configuration to serve the wiki:
An example nginx configuration from the CC server wiki-to-print: https://git.vvvvvvaria.org/varia/wiki-to-print/src/branch/master/wiki-to-print.nginx.example
in wiki2print.conf in /etc/nginx/sites_available:
return 301 https://wiki2print.servpub.net$request_uri
change servername to wiki2print.local
change root path to /var/www/wiki2print
(full conf file is now there to reference)
Make a symbolic link (symlink) between the configurations in /sites-available and /sites-enabled:
$ sudo ln -s /etc/nginx/sites-available/wiki2print.conf /etc/nginx/sites-enabled/wiki2print.conf
"ln" is link "-s" is symbolic.
check to see the symbolic link is made:
$ ls -la sites-enabled
However... we did not use ln -s in the end, as it did not work, we don't know why. In the end we used:
$ ln sites-available/wiki2print.conf sites-enabled/wiki2print.conf
Reload nginx
First, check if the nginx configuration works, before reloading
$ nginx -t
"-t" is test
If it works, hold your breath and reload nginx
$ sudo systemctl reload nginx
or
$ service nginx reload
Edit the wiki2print.conf file, refernce this example: https://git.vvvvvvaria.org/varia/wiki-to-print/src/branch/master/wiki-to-print.nginx.example
then enable PHP
ERROR on https:/wiki2print.servpub.net
MediaWiki 1.40 internal error
Installing some PHP extensions is required.
The following PHP libraries are missing:
- mbstring
- xml
- intl
$ sudo apt install php-mbstring
$ sudo apt install php-xml
$ sudo apt install php-intl
If you're running tinc, you can open:
http://10.10.12.4/
---
https://wiki2print.servpub.net/ (PHP request)
--> browser
--> jean server
--> wiki2print server (index.php)
works!
But,
https://wiki2print.servpub.net/resources/assets/mediawiki.png (does not work)
http://10.10.12.4/resources/assets/mediawiki.png (works)
--> browser
--> jean server
--> wiki2print server
---
maybe usefull thread - https://www.reddit.com/r/selfhosted/comments/kphfd3/nginx_reverse_proxy_via_vpn/
--- for reasons of vp/reverse proxy settings - we decide to install by accessing the vpns by it's IP ADDRESS
Continuing from this URL from now: http://10.10.12.4
missing dependency:
php-mysql
sudo apt install php mysql
ENVIRONMENT CHECKS --
Warning: Could not find APCu or WinCache.
Object caching is not enabled.
installing image magick:
sudo apt install imagemagick
On the MariaDB/Mysql settings:
create user
not advised to create root user (the password on LocalSettings.php is stored in plain text)
create user
$ mariadb
CREATE USER print@localhost IDENTIFIED BY 'XXX';
create database called wiki:
CREATE DATABASE wiki;
https://mariadb.com/kb/en/create-database/
grant access to the user (called print) for this database (called wiki). the "*" is to give acces to all table
GRANT ALL PRIVILEGES ON wiki.* TO 'print'@'localhost' WITH GRANT OPTION;
check that access is granted:
...
User rights profile:
Accoutn creation (can disble button)
For later:
- Need to discuss what license we want to have, for now we went with:
CC license: ‘Attribution-NonCommercial-ShareAlike’.
- maybe adding email settings list
- check if this works: deleted files location on server:
opt/wiki-deleted-files/
- set logo
- switch to another skin, we wanted minervaneue, but it didn't work
Copy the LocalSettings.php from the database set up into server put in www site root folder
cd /var/www/
ls -lha .
What does -h do?
Ah it makes the filesizes human readable.
https://www.mediawiki.org/wiki/Category:Stable_skins
https://www.mediawiki.org/wiki/Special:SkinDistributor/MinervaNeue
$ wget https://extdist.wmflabs.org/dist/skins/MinervaNeue-REL1_40-edf9394.tar.gz
$ tar -xzf MinervaNeue-REL1_40-edf9394.tar.gz -C /var/www/wiki2print/skins/
Still didn't work...
We added it to the list for later.
Then we change the wgserver address in Localsettings.php... to wiki2print.servpub.net
And it still does not work...
<?php
phpinfo();
?>
https://wiki2print.servpub.net/info.php
service nginx restart
service php-fpm restart
service --status-all
service php-fpm status
sudo apt search php8.1-common
= installed
php-fpm install
https://www.digitalocean.com/community/tutorials/php-fpm-nginx
service php8.1-fpm status
media wiki install nginx - https://www.howtoforge.com/how-to-install-mediawiki-with-nginx-and-lets-encrypt-ssl-on-ubuntu-20-04/
chown -R www-data:www-data /var/www/wiki2print
chmod -R 755 /var/www/wiki2print
To continue in the next 20 min:
install flask
http://10.10.12.4
We will install flask in /opt/
$ cd /opt/
$ git clone https://git.vvvvvvaria.org/varia/wiki-to-print
$ cd wiki-to-print
and again
$ cd wiki-to-print
Don't read the requirements.txt !
Do we want to do it in a virtual environment ?
Make a virtual environment (venv)= the name
$ python3 -m venv venv
You now have a folder called "venv".
Activate the venv
$ source venv/bin/activate
$ pip3 install flask
$ pip3 install bs4
$ pip3 install gunicorn
- nginx config: https://git.vvvvvvaria.org/varia/wiki-to-print/src/branch/master/wiki-to-print.nginx.example
The flask application programme is in web-interface.py with routes to different parts of wiki-to-print
edit config:
nano config.json
change:
"project_name"
"base_url"
http://10.10.12.4/wiki/ --> wiki
http://10.10.12.4/wiki4print/ --> flask
file ? / config.json
Port 5522 needs to be opened on the wiki2print server?
(?)
Maybe we proxy pass it to nginx through a URL, then this is not needed.
The subject_ns + styles_ns settings need to match with the settings in the Localsettings of the wiki, see an example here, line 176-182:
https://git.vvvvvvaria.org/varia/wiki-to-print/src/branch/master/wiki-to-print.LocalSettings.php.example
Next steps for the flask app:
- install pandoc
- configure the namespaces in the wiki's Localsetting
- set up a service file for wiki4print
- configure nginx, add a proxy pass for the wiki4print flask app
Next steps for the wiki:
- configure it in nginx to /wiki/, not / (not sure if still needed...)
- change hostname (and all other mentions) of the pi to wiki4print
- add the common.css + common.js pages to the wiki
- backup!
Maybe there are docs here for installing the flask app + configure the wiki further?
https://github.com/hackersanddesigners/wiki2print
https://github.com/hackersanddesigners/wiki2print/blob/master/wiki/README.md#namespaces
And all the example conf files here:
- localsettings: https://git.vvvvvvaria.org/varia/wiki-to-print/src/branch/master/wiki-to-print.LocalSettings.php.example
- common.css: https://git.vvvvvvaria.org/varia/wiki-to-print/src/branch/master/wiki-to-print.Common.css.example
- common.js: https://git.vvvvvvaria.org/varia/wiki-to-print/src/branch/master/wiki-to-print.Common.js.example
- nginx conf: https://git.vvvvvvaria.org/varia/wiki-to-print/src/branch/master/wiki-to-print.nginx.example
- service file: https://git.vvvvvvaria.org/varia/wiki-to-print/src/branch/master/wiki-to-print.service.example
We will continue on the 17th of January, 10:00 (UK time) 11:00 (NL time).
We left the installation with a working wiki on http://10.10.12.4, but didn't get the wiki to work on https://wiki2print.servpub.net. Maybe this is a proxy pass issue with php? We're not sure. ooooo is planning to dive into this i think? In conversation with Mara?
We half installed the flask application, to do list is above on line 390.
----------------/ jean config file nginx
##
# You should look at the following URL's in order to grasp a solid understanding
# of Nginx configuration files in order to fully unleash the power of Nginx.
# https://www.nginx.com/resources/wiki/start/
# https://www.nginx.com/resources/wiki/start/topics/tutorials/config_pitfalls/
# https://wiki.debian.org/Nginx/DirectoryStructure
#
# Inst cases, administrators will remove this file from sites-enabled/ and
# leave it as reference inside of sites-available where it will continue to be
# updated by the nginx packaging team.
#
# This file will automatically load configuration files provided by other
# applications, such as Drupal or Wordpress. These applications will be made
# available underneath a path with that package name, such as /drupal8.
#
# Please see /usr/share/doc/nginx-doc/examples/ for more detailed examples.
##
# Default server configuration
#
server {
listen 80;
listen [::]:80;
server_name wiki2print.servpub.net;
return 301 https://wiki2print.servpub.net$request_uri;
}
server{
# SSL configuration
#
listen 443 ssl;
listen [::]:443 ssl;
ssl_certificate /etc/letsencrypt/live/servpub.net/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/servpub.net/privkey.pem;
#
# Note: You should disable gzip for SSL traffic.
# See: https://bugs.debian.org/773332
#
# Read up on ssl_ciphers to ensure a secure configuration.
# See: https://bugs.debian.org/765782
#
# Self signed certs generated by the ssl-cert package
# Don't use them in a production server!
#
# include snippets/snakeoil.conf;
root /var/www/html;
# this ^ seems weird to me...
# Add index.php to the list if you are using PHP
index index.html index.htm index.nginx-debian.html;
server_name wiki2print.servpub.net;
location / {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_pass http://10.10.12.4;
proxy_read_timeout 90;
# First attempt to serve request as file, then
# as directory, then fall back to displaying a 404.
try_files $uri $uri/ =404;
}
location /media/ {
proxy_pass http://10.10.12.4/media/;
try_files $uri /media/ =404;
}
#location / {
#This is need with https if we have a certificate
#rewrite ^ https://$host$request_uri? permanent;
#}
# pass PHP scripts to FastCGI server
#
#location ~ \.php$ {
# include snippets/fastcgi-php.conf;
#
# # With php-fpm (or other unix sockets):
# fastcgi_pass unix:/run/php/php7.4-fpm.sock;
# # With php-cgi (or other tcp sockets):
# fastcgi_pass 127.0.0.1:9000;
#}
# deny access to .htaccess files, if Apache's document root
# concurs with nginx's one
#
#location ~ /\.ht {
# deny all;
#}
}
}
Wednesday 17 January
- Installing Flask (the flask part of wiki-to-print)
- cronjobs for backups (on external USB)
- If we have time or maybe later, setting up email through 3rd party.
Flask part
what we need to set up:
- install flask + requirements
- configure nginx: connect URL to flask process
- make a service file: to let the flask run in the background
- make changes to the wiki:
- add css/js to common.js/css for wiki-to-print buttons
- create wiki namespace for wiki-to-print
example configuration files:
- localsettings: https://git.vvvvvvaria.org/varia/wiki-to-print/src/branch/master/wiki-to-print.LocalSettings.php.example
- common.css: https://git.vvvvvvaria.org/varia/wiki-to-print/src/branch/master/wiki-to-print.Common.css.example
- common.js: https://git.vvvvvvaria.org/varia/wiki-to-print/src/branch/master/wiki-to-print.Common.js.example
- nginx conf: https://git.vvvvvvaria.org/varia/wiki-to-print/src/branch/master/wiki-to-print.nginx.example
- service file: https://git.vvvvvvaria.org/varia/wiki-to-print/src/branch/master/wiki-to-print.service.example
ssh
wiki4print = 10.10.12.4
$ ssh username@10.10.12.4
shared user account: servpub01@10.10.12.4
for the workshop
- no limit of users on the wifi
- bring ethernet cable with the Pi
- possibly bring router?
where are the files on wiki4print?
- wiki: /var/www/wiki2print
- wiki-to-print flask: /opt/wiki-to-print
TO DO
- install flask + requirements
check - where we left our installation
/opt
/opt/wiki
virtual environment
# check install
pip
# check base url
$ nano config.json
"base_url": "http://10.10.12.4/wiki4print/"
needs to be changed
semantically : flask( we opt for technical name - for basic understanding)
"base_url": "https://wiki4print.servpub.net/flask/",
-------- wondering - references section in pages didn't show up in the /print is it here?
test if it is running
local test ( we remove . as we run our virtual environment /env/
local:
venv/bin/python3 web-interface.py
run it through gunicorn - later
server:
@SCRIPT_NAME=/wiki-to-print .venv/bin/gunicorn -b localhost:5522 --reload web-interface:APP
we run: make local
it runs! on port 5522
we try to check http://10.10.12.4:5522/, but we cannot reach that port on the tinc ip address
is this related to the firewall on jean?
maybe not, as we try to connect directly to wiki4print over the tinc network
on jean - we opened 5522 port we leave it open but technically we think it is not necessary
Lets close it if it runs also through proxy pass configuration
ufw allow 5522
ufw status
- configure nginx: connect URL to flask process
cd /etc/nginx/sites-available/wiki2print.conf
nano wiki2print.conf
example config for wiki-to-print:
https://git.vvvvvvaria.org/varia/wiki-to-print/src/branch/master/wiki-to-print.nginx.example
we copied the following part:
(...)
and then run the flask application again in /opt/wiki-to-print/wiki-to-print/ with:
$ make local
CHECK IN BROWSER
http://10.10.12.4/flask/
https://wiki4print.servpub.net/flask/
does not work
trying to run the flask with gunicorn
$ make server
we realised that need to change the base_url to the base url of the wiki
$ nano config.json
"base_url": "https://wiki4print.servpub.net"
$ make server
500 internal error
maybe the wiki api does not work?
let's try it in the browser:
https://wiki4print.servpub.net/api.php?action=query&format=json&list=allpages&apnamespace=3000
it cannot find the namespace 3000
we need to create these in the wiki localsettings:
"subject_ns": { "name": "Pdf", "id": 3000 },
"styles_ns": { "name": "PdfCSS", "id": 3001 }
$ nano /var/www/wiki2print/LocalSettings.php
we can follow this example conf: https://git.vvvvvvaria.org/varia/wiki-to-print/src/branch/master/wiki-to-print.LocalSettings.php.example
We copy this part into the localsettings:
# Custom publishing namespace
define("NS_Pdf", 3000);
define("NS_PdfCSS", 3001);
$wgExtraNamespaces[NS_Pdf] = "Pdf";
$wgExtraNamespaces[NS_PdfCSS] = "PdfCSS";
$wgNamespacesWithSubpages[NS_Pdf] = true;
$wgNamespacesWithSubpages[NS_PdfCSS] = true;
# We change the content model of the CSS namespace from the default WIKITEXT.
# The forces the CodeEditor to replace the VisualEditor for better legibility.
$wgNamespaceContentModels[NS_PdfCSS] = CONTENT_MODEL_CSS;
$ cd /opt/wiki-to-print/wiki-to-print/
$ make server
works!!!
- make a service file: to let the flask run in the background
$ cd /etc/systemd/system/
$ nano wiki4print.service
we can use this example conf: https://git.vvvvvvaria.org/varia/wiki-to-print/src/branch/master/wiki-to-print.service.example
We edit it to:
[Unit]
Description=wiki-to-print
After=network.target
[Service]
User=servpub01
WorkingDirectory=/opt/wiki-to-print/wiki-to-print/
ExecStart=/usr/bin/make server
Restart=always
[Install]
WantedBy=multi-user.target
and change the permissions of the flask folder to:
$ chown -R root:servpub01 /opt/wiki-to-print/opt-wiki-to-print/
$ ls -la /opt/wiki-to-print/wiki-to-print
worked
to initialize the changes made in the service files:
$ systemctl daemon-reload
$ systemctl enable wiki4print
$ systemctl start wiki4print
$ systemctl status wiki4print
works
- make changes to the wiki:
- add css/js to common.js/css for wiki-to-print buttons
- create wiki namespace for wiki-to-print --> done already
COMMON.JS
to let the wiki4print buttons appear, there is some javascript sitting in the common.js page on the wiki:
https://wiki4print.servpub.net/index.php?title=MediaWiki:Common.js
To be able to edit this page, we need to be a bureaucrat.... or have the correct edit rights on the wiki:
$wgGroupPermissions['user']['editinterface'] = true;
We add this to the LocalSettings
Does not work..
This php script should be able to change the user groups: https://www.mediawiki.org/wiki/Manual:CreateAndPromote.php, it's in the maintenance folder.
Meanwhile, the admin access arrives..... if that works, we can change user groups through the web interface.
We're in!
So back to https://wiki4print.servpub.net/index.php?title=MediaWiki:Common.js
We can use this example config: https://git.vvvvvvaria.org/varia/wiki-to-print/src/branch/master/wiki-to-print.Common.js.example
We find and replace:
- the flask URL from: https://cc.vvvvvvaria.org/wiki-to-print/ into: https://wiki4print.servpub.net/flask/
- wiki2print into wiki4print
Now we can try to open a page in the Pdf namespace, and see if the buttons appear:
https://wiki4print.servpub.net/index.php?title=Pdf:Test
they do
but the CSS button is not changed, as the js is written for the vector skin
we go back to common.js: https://wiki4print.servpub.net/index.php?title=MediaWiki:Common.js
COMMON.CSS
We will first change the common.css and then come back to this.
So to: https://wiki4print.servpub.net/index.php?title=MediaWiki:Common.css
We can use this example conf: https://git.vvvvvvaria.org/varia/wiki-to-print/src/branch/master/wiki-to-print.Common.css.example
we find/replace wiki2print first into wiki4print in a code editor
and then paste it into the page
Checking Pdf:Test again if the CSS is applied: https://wiki4print.servpub.net/index.php?title=Pdf:Test
it works, we will fine tune the css later
---
WIKI4PRINT WIKI ENVIRONMENT
We start with testing the wiki4print environment first.
A page in the Pdf namespace for Content-Form
https://wiki4print.servpub.net/index.php?title=Pdf:Content-Form
works
but when clicking "view pdf" it gives a 500 error
where to find the logs?
this shows the logs of the wiki4print service:
$ journalctl -u wiki4print
https://wiki4print.servpub.net/flask/update/Content-Form
= update(pagename)
we were missing the library requests
$ pip3 install requests
https://wiki4print.servpub.net/flask/pdf/Content-Form
works now!
we're updating paged.js to the latest version, just in case:
https://pagedjs.org/documentation/2-getting-started-with-paged.js/#using-paged.js-as-a-polyfill-in-web-browsers
and the pagedjs.css (= interface.css):
https://gitlab.coko.foundation/pagedjs/interface-polyfill
issue with paged.js, we are asking the developers about it... and will come back to it later.
to test wiki4print more, we're moving the dummy over from CC to wiki4print...
but they look very different...
https://wiki4print.servpub.net/flask/pdf/Content-Form
https://cc.vvvvvvaria.org/wiki-to-print/pdf/Content-Form_(dummy)
mistake in the placeholder page: https://wiki4print.servpub.net/index.php?title=Placeholder_500_words&action=edit§ion=T-1
fixed now
PHP UPLOAD LIMIT
https://www.mediawiki.org/wiki/Manual:Configuring_file_uploads
change upload size to 5 MB
$ nano /etc/php/8.1/cli/php.ini
$ nano LocalSettings.php
Still not working, also not after reboot of pi..
Backups
Formatting the USB to ExFAT
Finding the USB..
$ lsblk
it should be: /dev/sda1
mounted at: /media/backup
$ sudo su
$ cd /media/servpub01/Transcend
what filesystem type is it?
$ df -T
/dev/sda1 vfat 15833096 17320 15815776 1% /media/servpub01/Transcend
it's a vfat filesystem
> There are several variants of the FAT file system (e.g. FAT12, FAT16 and FAT32). FAT16 refers to both the original group of FAT file systems with 16-bit wide cluster entries and also to later variants. "VFAT" is an optional extension for long file names, which can work on top of any FAT file system. Volumes using VFAT long-filenames can be read also by operating systems not supporting the VFAT extension.
https://en.wikipedia.org/wiki/File_Allocation_Table
what to use for an usb that needs to be able to connect to different computers? (just in case?)
> The most common file systems are FAT32, exFAT, and NTFS on Windows, APFS and HFS+ on macOS, and EXT on Linux ---though you may run into others on occasion.
> You might think that modern operating systems would natively support each other's file systems, but they largely do not.
> FAT32 was developed many years ago, and was based on older FAT filesystems meant for DOS computers. The large disk sizes of today were only theoretical in those days, so it probably seemed ridiculous to the engineers that anyone would ever need a file larger than 4 GB. However, with today's large file sizes of uncompressed and high-def video, many users are faced with that very challenge.
> The conclusion to draw from all this is that while FAT32 has its issues, it's the best file system to use for most portable drives. FAT32 finds support on the most devices, allows volumes up to 8 TB, and file sizes up to 4 GB. If you need to transport files greater than 4 GB, you'll need to take a closer look at your needs. (...) And if you need support for more devices and bigger files, exFAT may fit the bill. exFAT is not supported on quite as many different devices as FAT32 is --- support is often lacking on older gadgets --- but it comes close.
https://www.howtogeek.com/73178/what-file-system-should-i-use-for-my-usb-drive/
> File System Individual File Size Limit Single Volume Size Limit
> FAT32 Less than 4 GB Less than 8 TB
> exFAT Greater than commercially available drives 64 ZB
> EXT 2, 3 16 GB (or up to 2 TB) 32 TB
> EXT 4 16 TiB 1 EiB
[i don't like the "how to geek" website so much, but it helped me to get further... it would be so great to grow feminist server docs collectively somehow]
so i think exfat seems the fs to go for, because:
- it allows big files, which we need for full disk copies (FAT32/vFAT support up to 4GB files)
- it is compatible on different operating systems, the EXT2/3/4 filesystems are only supported on Linux (maybe some of us use macs?)
- it seems to be good for flask drives such as USBs
- but, it does weird things to capitals in filesnames (';076ocal;083ettings.php')
how to reformat the disk?
following this page: https://itsfoss.com/format-exfat-linux/
to list the disks:
$ fdisk -l
Disk /dev/sda: 15.12 GiB, 16231956480 bytes, 31703040 sectors
Disk model: Transcend 16GB
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xc3072e18
Device Boot Start End Sectors Size Id Type
/dev/sda1 5888 31703039 31697152 15.1G c W95 FAT32 (LBA)
to format it into an exfat disk:
$ mkfs.exfat -n LABEL /dev/sda
but do i use /dev/sda or /dev/sda1?
> Identify the partition that you want to format in your external hard disk. For me, the USB disk was labelled as /dev/sdb1.
this suggests partition, which would be sda1...
hmm, in the $ fdisk -l command it outputs:
- "disk": /dev/sda
- "device": /dev/sda1
the example uses the partition/device, not the disk, so i will follow that, but i don't understand why
formatting the usb with the name "backup":
$ mkfs.exfat -n backup /dev/sda1
zsh: command not found: mkfs.exfat
> Starting Linux kernel 5.4, exFAT filesystem support is enabled in the Linux kernel itself. Check which Linux kernel version you are running. If it is kernel 5.4 or higher, you should be fine (mostly). Otherwise, you can enable exFAT support explicitly.
which kernal is wiki4print on?
$ hostnamectl
Static hostname: wiki2print
Icon name: computer
Machine ID: 525bb04e1ca94e33a4564419ceb9b986
Boot ID: 22cd24a2edf947509913bed6687c64f2
Operating System: Armbian 23.11.1 jammy
Kernel: Linux 6.1.50-current-bcm2711
Architecture: arm64
oke, that's higher than 5.4
maybe i need to install mkfs?
> For Ubuntu 22.04 and higher, you should use this command instead:
> sudo apt install exfat-fuse exfatprogs
ah! we are on ubuntu of course...
$ apt install exfat-fuse exfatprogs
$ mkfs.exfat -n backup /dev/sda1
exfatprogs version : 1.1.3
open failed : /dev/sda1, Device or resource busy
exFAT format fail!
$ cd /
$ umount /media/servpub01/Transcend
$ mkfs.exfat -n backup /dev/sda1
exfatprogs version : 1.1.3
Creating exFAT filesystem(/dev/sda1, cluster size=32768)
Writing volume boot record: done
Writing backup volume boot record: done
Fat table creation: done
Allocation bitmap creation: done
Upcase table creation: done
Writing root directory entry: done
Synchronizing...
exFAT format complete!
oke that worked
$ fdisk -l
oke it appears again as /dev/sda
$ df -T
but not in this list, i need to mount it first i think
$ mount /dev/sda /media/backup
mount: /media/servpub01: wrong fs type, bad option, bad superblock on /dev/sda, missing codepage or helper program, or other error.
hmm..
following this: https://askubuntu.com/questions/703183/cannot-mount-usb-drive-with-wrong-fs-type-bad-option-bad-superblock-error-me
> The USB drive seems to be partitioned. Do not mount the device directly, but the partition /dev/sdb1 instead: $ mount /dev/sdb1 /mnt
$ mount /dev/sda1 /media/backup
works!
$ df -hT
/dev/sda1 exfat 16G 128K 16G 1% /media/servpub01
oke the USB is ready to go
but still needs to be automounted after a reboot
need to go now!
making backup scripts
for: the wiki, nginx configs, wiki4print flask application, and a few other system files perhaps
in: /opt/backup-scripts/
Starting with the wiki
There are a few things to include in the backup of the wiki:
- the mysql database
- LocalSettings.php
- images folder
- skins folder
- extentions folder
- resources folder (for the wiki4print logo?)
With these files, we could reinstall the wiki, by downloading the latest version of mediawiki again and copying all these files back.
$ cd /opt/backup-scripts
$ nano backup-wiki.sh
First defining two variables with the paths to the folders that we need in this script:
`wikipath=/var/www/wiki`
(this is where the wiki is installed)
`backuppath=/media/backups/wiki`
(this is where the backups USB is mounted)
Then, to "dump" the mysql into a file:
`mysqldump --user=print --password=INSERTPASSWORDHERE wiki > $wikipath/wiki_database_backup.sql`
The user, password and database name are configured in the LocalSettings.php file, you can copy them from there.
The program `mysqldump` saves the database into a sql file, which is what is proposed in the documentation: <https://dev.mysql.com/doc/refman/8.0/en/mysqldump.html>.
Let's try if it works in the commandline first:
$ mysqldump --user=print --password=INSERTPASSWORDHERE wiki > /var/www/wiki/wiki_database_backup.sql
$ ls /var/www/wiki
Yes it is there.
Then, we can use `rdiff-backup` to copy this file plus the other wiki files to the USB stick. From their website (<https://rdiff-backup.net/>):
> Rdiff-backup backs up one directory to another, possibly over a network. The target directory ends up a copy of the source directory, but extra reverse diffs are stored in a special subdirectory of that target directory, so you can still recover files lost some time ago. The idea is to combine the best features of a mirror and an incremental backup. Rdiff-backup also preserves subdirectories, hard links, dev files, permissions, uid/gid ownership (if it is running as root), modification times, acls, eas, resource forks, etc. Finally, rdiff-backup can operate in a bandwidth efficient manner over a pipe, like rsync. Thus you can use rdiff-backup and ssh to securely back a hard drive up to a remote location, and only the differences will be transmitted.
I learned aboout `rdiff-backup` through the work on another server, which i took over from someone else who used it. It's nice that backups are made incrementally, which means that files are only backed up when they are changed in since the previous backup was being made. This makes backing up quick and lighter to do. Also, and this is what was used on that other server i worked on, we can use `rdiff-backup` to not store all the versions of the wiki infinitely. The current script deletes backups older than 2 weeks, which is nice i think, data does not need to be endlessly available, that does not seem to make sense in the case of wiki4print.
First `rdiff-backup` needs to be installed.
I'm following the documentation on their github: <https://github.com/rdiff-backup/rdiff-backup#1-installation>
$ sudo apt install rdiff-backup
Now pasting the example from that other server into backup-wiki.sh:
rdiff-backup \
backup \
--include $wikipath/wiki_database_backup.sql \
--include $wikipath/LocalSettings.php \
--include $wikipath/images \
--include $wikipath/extentions \
--include $wikipath/skins \
--include $wikipath/resources \
--exclude '**' \
$wikipath \
$backuppath
From the documentation (<https://rdiff-backup.net/examples.html>):
> Sometimes you don’t want to back up all files. The various --include and --exclude options can be used to select exactly which files to back up.
> Multiple include and exclude options take precedence in the order they are given.
> ** matches any path
> * matches any path without a / in it
But, also mentioned on this page, it seems that you need to first run:
$ rdiff-backup source-folder backup-folder
and then:
$ rdiff-backup backup source-folder backup-folder
But it seems that the first command only need to be ran once.
So to put it in one line:
$ rdiff-backup --include /var/www/wiki/wiki_database_backup.sql --include /var/www/wiki/LocalSettings.php --include /var/www/wiki/images --include /var/www/wiki/extentions --include /var/www/wiki/skins --include /var/www/wiki/resources --exclude '**' /var/www/wiki /opt/backup-test/wiki
(I'm using "/opt/backup-test" as the USB is not mounted yet.)
This takes a bit of time.
Oke it worked.
Now trying to run the backup-wiki.sh script, to see if that also works.
$ sh backup-wiki.sh
> Fatal Error: Switches missing or wrong number of arguments
> See the rdiff-backup manual page for more information.
Hmm...
Let's maybe try it in one line?
$ rdiff-backup backup --include /var/www/wiki/wiki_database_backup.sql --include /var/www/wiki/LocalSettings.php --include /var/www/wiki/images --include /var/www/wiki/extentions --include /var/www/wiki/skins --include /var/www/wiki/resources --exclude '**' /var/www/wiki /opt/backup-test/wiki
Same error..
Ah! There seems to be an old and new version of rdiff-backup:
> Starting with rdiff-backup version 2.1, there have been two separate syntaxes for the command line interface, differentiated by the --new flag. Legacy CLI commands follow the syntax rdiff-backup [options…] [source…] [destination…], while the new CLI uses the syntax rdiff-backup [global options…] action [action-specific options…] [locations…]. The functionality of the commands are the same, but for completeness the examples in this document will be given in both the old and new syntax side-by-side.
Which version is installed?
$ rdiff-backup --version
> rdiff-backup 2.0.5
Oke, this is the old one.
So we don't use the global action "backup", taking it out of the backup-wiki.sh.
And running the script:
$ sh backup-wiki.sh
Oke works.
Now, one last thing for the wiki backups: it would be nice to make the size of the backups managable and delete backups older then 2 weeks.
See in the documentation: https://rdiff-backup.net/examples.html#deleting_older_files
Adding to the backup-wiki.sh script:
rdiff-backup --remove-older-than 2W $backuppath
And running the script again to see if it works:
$ sh backup-wiki.sh
> No increments older than Thu Jan 11 16:23:55 2024 found, exiting.
Oke works.
backup of nginx config files
Using rdiff-backup again.
$ nano backup-nginx.sh
#! /bin/sh
nginxpath=/etc/nginx
backuppath=/opt/backup-test/nginx
rdiff-backup \
--include $nginxpath/sites-available/wiki2print.conf \
--include $nginxpath/snippets/fastcgi-php.conf \
--exclude '**' \
$nginxpath \
$backuppath
rdiff-backup --remove-older-than 2W $backuppath
backup of wiki4print flask part
Using rdiff-backup again.
$ nano backup-wiki4print.sh
#! /bin/sh
w4ppath=/opt/wiki-to-print/wiki-to-print
backuppath=/opt/backup-test/wiki4print
rdiff-backup \
--exclude $w4ppath/__pycache__/ \
--exclude $w4ppath/venv/ \
$w4ppath \
$backuppath
rdiff-backup --remove-older-than 2W $backuppath
backup of other system files
Using rdiff-backup again.
$ nano backup-misc.sh
#! /bin/sh
backuppath=/opt/backup-test/misc
rdiff-backup \
--include /etc/hostname \
--include /etc/hosts \
--include /etc/fstab \
--include /opt/backup-scripts \
--include /opt/backup-full-disk.sh \
--exclude '**' \
/ \
$backuppath
rdiff-backup --remove-older-than 2W $backuppath
testing the backup scripts
The USB is plugged in now and mounted at /media/backup.
Running the scripts one by one to test them.
The scripts give this warning:
> Warning: hard linking not supported by filesystem at /media/backup/wiki4print/rdiff-backup-data
Found this https://ubuntuforums.org/showthread.php?t=1297114:
> I've just found out that with FAT32 formated USB sticks, although the message :
> "Warning: hard linking not supported by filesystem at /media/16GB_USB/USB_backup/rdiff-backup-dat"
> appears, the cursor justifies left and flashes oin and off. ie: it does not go back to the user prompt. This is because the backup is taking place. Knowing this I waited the required amount of time for the backup to complete, and then the cursor gave the user prompt. After that I checked the destination USB stick and it had the complete backup on it. So apparently, it would seem we dont have to worry about the warning message, unless you want to do a bit of "HARD LINKING" whatever that may be.
So the warning seems to be innocent?
It feels related to the exfast filesystem format.
Unfortunately don't have time to dive into it now...
This is how much disk space the backups take in on the USB:
$ cd /media/backup
$ du -h -d 1 -c
2.9M ./misc
1.9M ./nginx
9.8M ./wiki4print
1.4G ./wiki
1.4G .
1.4G total
1.4GB is a lot?
$ du -h -d 1 -c /var/www/wiki
35M /var/www/wiki/includes
160M /var/www/wiki/extensions
22M /var/www/wiki/tests
41M /var/www/wiki/skins
79M /var/www/wiki/languages
912K /var/www/wiki/docs
55M /var/www/wiki/images
96K /var/www/wiki/mw-config
34M /var/www/wiki/vendor
8.0K /var/www/wiki/cache
3.4M /var/www/wiki/maintenance
19M /var/www/wiki/resources
455M /var/www/wiki/
455M total
Which is much less..
No time unfortunately to dive into this now...
after thoughts
Hmm, after doing all this, i wonder if it would not be useful, maybe even more useful, to make a copy of the full disk with dd?
In that case, if something happens, we can just copy that image back on a new sd card, and everything is back.
Checking the size of the sd card..
$ fdisk -l
> Disk /dev/mmcblk0: 29.12 GiB, 31267487744 bytes, 61069312 sectors
It seems to be a 32GB sd card, recognized as a 29.12GB disk here.
The USB is 64GB, so it would fit to store even both a full disk copy, plus the individual files of the rdiff backups from above?
Hmm...
Checking which command would need to be used for a full disk copy...
$ df -h
Filesystem Size Used Avail Use% Mounted on
tmpfs 362M 38M 325M 11% /run
/dev/mmcblk0p2 29G 8.4G 20G 31% /
tmpfs 1.8G 0 1.8G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 1.8G 12K 1.8G 1% /tmp
/dev/mmcblk0p1 253M 130M 123M 52% /boot/firmware
/dev/zram1 47M 28M 16M 64% /var/log
tmpfs 362M 64K 362M 1% /run/user/1000
overlaid 362M 64K 362M 1% /run/user/1000/servpub01-chromium
/dev/sda1 16G 128K 16G 1% /media/servpub01
tmpfs 362M 40K 362M 1% /run/user/1001
Oke, funny how the os is spread over different filesystems and drives?
What is /dev/zram1?
And the tmpfs filesystems??
The /dev/sda1 is the current smaller USB..., 16GB...
In other situations, i would run something like:
$ dd if=/dev/mmcblk0 of=/media/USBNAME/wiki4print-backup.img status=progress
Not sure what input file to use here...
- /dev/mmcblk0
- /dev/mmcblk0p1
- /dev/mmcblk0p2
Also curious how big this file would become.
Maybe some compression could be used?
Also, how to back up these image files incrementally?
I guess borg can be used then? https://borgbackup.readthedocs.io/en/stable/
Varia uses this, and also XPUB.
This could be a tutorial to follow: https://borgbackup.readthedocs.io/en/stable/deployment/image-backup.html
On another note, it's nice to find https://git.systerserver.net/systerserver/doc-website/-/wikis/Container-Backups-with-Restic and see the approach.
Hmm, what to do now?
Proposal:
- we make manual backups during the workshops with a simple dd command, to at least have full disk copies just in case anything happens next week
- we also run the backup scripts that are in /opt/backup-scripts/, because we have enough space on the usb
- the full disk backups can be quite good to have for the long term, but using the dd command is not really a durable solution, as it creates huge files which cannot be made incrementally using a version control system, because dd does not support that. This means that backups will need attention again at a later point i think, to make it future proof
cronjob
Adding the backup scripts to a cronjob, so they run every hour.
.---------------- minute (0 - 59)
| .------------- hour (0 - 23)
| | .---------- day of month (1 - 31)
| | | .------- month (1 - 12) OR jan,feb,mar,apr ...
| | | | .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
| | | | |
* * * * * command to be executed
0 * * * * /bin/sh /opt/backup-scripts/backup-nginx.sh
1 * * * * /bin/sh /opt/backup-scripts/backup-wiki4print.sh
5 * * * * /bin/sh /opt/backup-scripts/backup-misc.sh
10 * * * * /bin/sh /opt/backup-scripts/backup-wiki.sh
I usually test that syntax with this "guru" website: https://crontab.guru/
This means, that the next backups are made at:
- 11:00 backup-nginx.sh
- 11:01 backup-wiki4print.sh
- 11:05 backup-misc.sh
- 11:10 backup-wiki.sh
And again at 12:00, 12:01, 12:05, 12:10, etc.
The wiki one is the last one, as it takes the most time.
$ sudo su
$ crontab -e
---
automounting the USB for backups
There is a 64GB USB external drive attached to the Pi. We can use this for backups. The system needs to be configured so that the USB drive mounts automatically when there is a reboot of the Pi. I am following documentation here: https://www.raspberrypi-spy.co.uk/2014/05/how-to-mount-a-usb-flash-disk-on-the-raspberry-pi/ and here https://docs.s3it.uzh.ch/how-to_articles/how_to_automatically_mount_a_volume_at_instance_startup/.
Here is what I did to configure the system:
Run lsblk to show which drives are physically connected to the Pi. The -o flag is for options that follow, which specify the NAME, FSTYPE (filesystem type), UUID (Universally Unique Identifier), SIZE, and LABEL
$ sudo lsblk -o NAME,FSTYPE,UUID,SIZE,LABEL
This is the output:
NAME FSTYPE UUID SIZE LABEL
sdb 57.8G
└─sdb1 vfat 7147-2941 57.7G KINGSTON
mmcblk0 29.1G
├─mmcblk0p1 vfat 9398-4898 256M RPICFG
└─mmcblk0p2 ext4 0a5bc2ea-7eb8-4ceb-87f6-74b915db2e3a 28.9G armbi_root
zram0 1.8G
zram1 50M
zram2 0B
We can see that the 64GB drive is named "sdb1" (the effective size given here is 57.7G).
* The FSTYPE is vfat (should it be exfat?)
We also see that the UUID of the drive named sdb1 is 7147-2941. We will need to use this UUID later, so I note it down.
Next, we create a mount point, which is a directory that points to the contents of the drive. You can all this directory whatever you like, I am calling this one "backup":
$ sudo mkdir /media/backup
Next I run this command to check which user owns this directory:
* which user should own this directory? Is root ok?
$ ls -la /media/backup
The output shows that the root user owns this folder:
total 8
drwxr-xr-x 2 root root 4096 Jan 26 07:00 .
drwxr-xr-x 4 root root 4096 Jan 26 07:00 .
Next, so that when the Pi is restarted the USB drive mounts automatically, we edit the /etc/fstab file:
$ sudo nano /etc/fstab
Each line of this file matches the following format:
<filesystem UUID> <mount point> <type> <options> <dump> <pass> (each element is separated by whitespace)
Where:
<filesystem UUID> describes the volume or remote filesystem UUID to be mounted, which can be obtained with the lsblk command.
<mount point> describes the mount point for the filesystem; in other words, the path to access the files stored on volume.
<type> describes the type of filesystem to be used (usually ext4).
<options> can be used to add metadata to the associated filesystem.
I add these options:
auto: makes the drive automatically mount after reboot
nofail: allows the boot process to proceed if the drive is not plugged in
noatime: stops the file access from being updated every time a file is read from the USB, which helps improve performance
user: allows any user to mount the filesystem, ovveriding the default to permit only root
rw: mounts the drive as read/write
uid/gid: sets the owner and group of the files in the filesystem
UUID=7147-2941 /media/backup vfat auto,nofail,noatime,user,rw,uid=root,gid=root
I save the file (ctrl+x, follwed by Y to save and exit nano), and reboot the Pi
$ sudo reboot
The drive is now automatically mounted at /media/backup whenever the Pi reboots or restarts.
I formatted the USB to exfat, and changed the fstab file accordingly.
$ sudo reboot
Hmm, the USB does not show up in the df -h list after reboot...
TODO list:
wiki related things
- test wiki4print: test the environment for the content/form newspaper (TO FINISH LATER, waiting for reply Paged.js developers) [manetta/simon]
note on the speed: "update text" is very slow on wiki4print, also, the page takes a very long time to load because of livereload.js
- update paged.js to the latest version, and found the mistake we made when trying to update yesterday [manetta]
- OPEN QUESTION: is the plan to switch to another pi to deal with speed issues?
- edit wiki4print related pages to remove any old CC traces (FOR LATER) [simon]
- add a link to the wiki4print Content-Form environment somewhere [manetta]
- create & prepare pages on https://wiki4print.servpub.net/index.php?title=Content-Form for the newspaper: colophon, contributors, license?, cellular contributions?, servpub/infra?, wiki4print/publishing workflow? [simon/manetta]
- change the CSS icon in PDF: namespace [manetta/simon]
- wiki time is at UK time, maybe update to Berlin time for the workshop?
- test uploading fonts to the wiki [manetta/simon]
- test uploading a stylesheet [manetta/simon]
- make these two template pages, wrapping them into a class that we can use to style them: [manetta/simon]
Template:Content form - wiki-page
Template:Content form - wiki-spam
- renamed the wiki folder from "wiki2print" to "wiki" to avoid confusion [manetta]
- add a toc extention to the wiki, to decrease the amount of scrolling in the Pdf: page [manetta] --> extention did not exist, customized the css
wiki4print things
- pick some fonts already
- transclude contributions already (optional) --> made such a mess that i reverted it
php upload limit
- increase upload limit to 5mb (TODO)
--> this has been changed in the configurations below (TODO FOR LATER: reboot the pi to see if this restarts php properly)
it should change from 2MB to 5MB on this page: https://wiki4print.servpub.net/index.php?title=Special:Upload
backups
- connect USB to wiki4print pi [Becky] DONE
- reformat usb to exfat [manetta] --> change to other format? no let's stay with this
- automount the USB [simon DONE] --> /media/backup --> not sure if it still works
- copy/collect/prepare backup scripts [manetta] --> /opt/backup-scripts/ + /opt/backup-full-disk.sh
- make cronjob [manetta]
- test back up scripts, see if they write to the usb [manetta]
other things
- set up email for wiki (LATER)
- check ethernet configuration of the pi [becky] DONE Up and running on ethernet
- is the ethernet hardware turned off? Ethernet worked no problem, so it was probably a dodgey cable or physcial infrastructure of the building at SET (which is likely). Super!
- check network infrastructure at workshop venue
- make sure there is ethernet access for the rasberry pi during workshop (HKW?)
- ask for contact details of person responsible for the network at the venue (HKW?)
- contact person responsible for the network at the venue to already ask for ethernet access (HKW?)
- think of a backup plan (if tinc is down/internet connection is too slow)
- bring wifi router to connect locally to the raspberry pi (for debugging or if vpn is down :/)
based on Varia/Constant experience, we noticed that the built-in wifi hardware of the pi has not been strong enough to work with 10+ participants at the same time
- strong and stable wifi access during workshop (min 25 users: 15 participants + organisers + in-grid + systerserver + cc)
- check tinc service auto-enabled
Maintainance Upate 2025:
Issue:
On a more technical guidance note, on wiki4print we are currently getting a 500 error on the update text and update media functionality / urls:
https://wiki4print.servpub.net/flask/update/ServPub
We tried a reboot of the whole pi to just see if that would kick it back into life. I have checked the wiki4print.service systemctl status and it seems to be running:
The other “static” endpoints are still working:https://wiki4print.servpub.net/flask/html/ServPub https://wiki4print.servpub.net/flask/pdf/ServPub
So I’m assuming that the flask application is running, but that there’s probably an issue with the update functionality in particular (my assumption could be wrong!). This is the nginx error log which is not particularly illuminating, but as mentioned the other flask routes / endpoints seem to be working:
Nginx logs:
Failed (111: Unknown error) while connecting to upstream, client: 10.10.12.2, server: wiki4print.local, request: "GET /flask/update/ServPub HTTP/1.0", upstream: "http://[::1]:5522/flask/update/ServPub"
Fixes:
Code base has moved to here on CC:
- https://git.vvvvvvaria.org/CC/wiki-to-print
To get more insight in what is happening, i would stop the wiki4print service and run it manually from the terminal
To run it manually, you can use this makefile: https://git.vvvvvvaria.org/CC/wiki-to-print/src/branch/master/wiki-to-print/Makefile and run $ make server
where to find the logs?
this shows the logs of the wiki4print service:
$ journalctl -u wiki4print
error trace from the logs:
Jul 25 11:17:31 wiki2print make[2198]: [2025-07-25 11:17:31,032] ERROR in app: Exception on /update/ServPub [GET]
Jul 25 11:17:31 wiki2print make[2198]: Traceback (most recent call last):
Jul 25 11:17:31 wiki2print make[2198]: File "/opt/wiki-to-print/wiki-to-print/venv/lib/python3.10/site-packages/flask/app.py", line 1455, in wsgi_app
Jul 25 11:17:31 wiki2print make[2198]: response = self.full_dispatch_request()
Jul 25 11:17:31 wiki2print make[2198]: File "/opt/wiki-to-print/wiki-to-print/venv/lib/python3.10/site-packages/flask/app.py", line 869, in full_dispatch_request
Jul 25 11:17:31 wiki2print make[2198]: rv = self.handle_user_exception(e)
Jul 25 11:17:31 wiki2print make[2198]: File "/opt/wiki-to-print/wiki-to-print/venv/lib/python3.10/site-packages/flask/app.py", line 867, in full_dispatch_request
Jul 25 11:17:31 wiki2print make[2198]: rv = self.dispatch_request()
Jul 25 11:17:31 wiki2print make[2198]: File "/opt/wiki-to-print/wiki-to-print/venv/lib/python3.10/site-packages/flask/app.py", line 852, in dispatch_request
Jul 25 11:17:31 wiki2print make[2198]: return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
Jul 25 11:17:31 wiki2print make[2198]: File "/opt/wiki-to-print/wiki-to-print/web-interface.py", line 109, in update
Jul 25 11:17:31 wiki2print make[2198]: create_publication(
Jul 25 11:17:31 wiki2print make[2198]: File "/opt/wiki-to-print/wiki-to-print/api.py", line 119, in create_publication
Jul 25 11:17:31 wiki2print make[2198]: 'html': create_html(wiki, subject_ns, pagename, full_update),
Jul 25 11:17:31 wiki2print make[2198]: File "/opt/wiki-to-print/wiki-to-print/api.py", line 163, in create_html
Jul 25 11:17:31 wiki2print make[2198]: soup = inlineCiteRefs(soup)
Jul 25 11:17:31 wiki2print make[2198]: File "/opt/wiki-to-print/wiki-to-print/api.py", line 416, in inlineCiteRefs
Jul 25 11:17:31 wiki2print make[2198]: text = cite[0].find(class_="reference-text")
Jul 25 11:17:31 wiki2print make[2198]: IndexError: list index out of range
Problem was that there was a malformed ref tag (or two) resulting in an array out of bounds error when attempting to get item at index 0 in the cite array, copying the api.py file here:
406 def inlineCiteRefs(soup):
407 """
408 soup = BeautifSoup (HTML)
409 """
410 refs = soup.find_all("sup", class_="reference")
411 for ref in refs:
412 href = ref.a['href']
413 res = re.findall('[0-9]+', href)
414 if(res):
415 cite = soup.find_all(id="cite_note-"+res[0]) <--- !!!!! THE PROBLEM
416 text = cite[0].find(class_="reference-text") <--- !!!!! Error list index out of range meaning there is no item at index 0 in this array
417 text['class'] = 'footnote'
418 ref.replace_with(text)
419 # remove the reference from the bottom of the document
420 for item in soup.find_all(class_="references"):
421 item.decompose()
422 return soup