Posts tagged with "Linux"

converting JPEG and PNG images to WebP format

People tell me the 'WebP' format is the recommended format for images displayed on a Web site as WebP files are smaller than PNG and JPEG formats. My site uses JPEG and PNG image formats. The advent of WebP is hardly news but I'm always late to the party.

There is a WebP converter available in the Arch Linux AUR repository.

This looks promising so I immediately go to install it.

$ yay -S webp-converter-git
Sync Make Dependency (7): gendesk-1.0.10-1, gn-0.2324.304bbef6-2,
    rust-bindgen-0.72.1-2, gperf-3.3-2, python-httplib2-0.31.2-1,
    python-pyparsing-3.3.2-1, patchutils-0.4.5-1
Sync Dependency (1): libvips-8.18.2-1
AUR Explicit (1): webp-converter-git-1.0.0.r2.gce5d96c-1
AUR Dependency (1): electron33-33.4.11-2
:: PKGBUILD up to date, skipping download: webp-converter-git
:: PKGBUILD up to date, skipping download: electron33
  2 webp-converter-git                       (Build Files Exist)
  1 electron33                               (Build Files Exist)
==> Packages to cleanBuild?
==> [N]one [A]ll [Ab]ort [I]nstalled [No]tInstalled or (1 2 3, 1-3, ^4)

Oh no, hang on, wait. it's a bloated Electron application with a lot of dependent Rust and Python packages. Forget that.

However, I recall that Emacs uses the 'imagemagick' package which is already installed and includes a utility (mogrify) that can perform the same conversion (from JPEG/PNG to WebP).

$ mogrify -format webp -quality 80 2026-04-14-BlogMore-DraftPost.png
$ ls -lh 2026*
100k     2026-04-14-BlogMore-DraftPost.png
44k      2026-04-14-BlogMore-DraftPost.webp

Half the size. Compare the PNG versus the WebP image. Copy the WebP file to the images directory. Update the image link in the blog post. Job done (apart from the existing 135 images).

in praise of Silver Searcher

Occasionally, I have to search lots of files for a pattern. It was only recently I discovered the wonderful silver searcher utility which saves me a lot of time.

To install 'ag' on Fedora, use the following (which isn't entirely obvious or intuitive if you're used to typing 'ag').

$ sudo dnf install the_silver_searcher

I believe there is an Emacs interface which would save me even more time.

$ time ag 'sql statement execute time' *.log
real    0m0.125s
user    0m0.128s
sys     0m0.257s

$ time find ~ -type f -print0 | xargs -0 grep -i 'sql statement execute time'
real    0m23.725s
user    0m7.965s
sys     0m1.618s

extending Bash history

I have used the Unix bash shell for many years. As I am incredibly lazy and forgetful, I have become accustomed to using ctrl-r find to find and scroll though the latest find commands I have issued.

Occasionally, I noticed that a lengthy, complex, useful 'find' command' (which I mercilessly plagiarised from a clever person via Google) was no longer in my shell history.

Investigations revealed the default bash history is a paltry 1000 commands so I decided to increase this to 10000 by adding the following line to '~/.bash_profile'.

HISTFILESIZE=10000

After all, disk space is much cheaper than my time.

how mu4e changed my life

Getting email

No mail. In three whole days. Weird. I wonder if it's Thanksgiving over in the States. Not even any football related banter. Is this thing even on ?

Then I realised precisely why I was sitting alone in an island of blissful isolation, devoid of all email communications and staring at an Inbox in a perpetual state of 'Zero'.

I had forgotten to configure inbound email.

When I was testing, I used mbsync to synchronise emails from my ISP which worked well (fast, reliable, well documented) with bi-directional sync between IMAP and my local Maildir.

Note: For Linux types, the 'mbsync' utility is contained within the 'isync' package.

My needs at work are slightly different though. We use IMAP but are encouraged to download and work locally. This corporate edict is implicitly enforced by configuring users' IMAP folders with a measly 1GB quota.

Using mu4e, I needed to fetch new messages from the IMAP server, transfer them to ~/Maildir and then delete them from the IMAP server. Although mbsync has 'push', 'pull' and 'expunge' options, it wasn't entirely clear (to me) if removal of messages from the server was possible. However, the getmail utility written in Python met the requirements exactly and was easy to configure.

Previously, I used Thunderbird to get new messages from IMAP, filter them at source and transfer them directly to various 'Local Folders'. The filtering was pretty basic - typically mailing lists and routing corporate communications to /dev/null.

I tended to configure mailing list folders with a retention period of 30 days and retain 'Starred' messages for ever. Essentially, this meant mailing list traffic didn't clutter my precious Inbox and would automatically expire. I then had a full month to scan the mailing lists and mark any interesting messages for further reading or more unlikely in my case, action.

Anything left over simply came to Inbox. This was normally email directed (To: or CC:'ed to me) so this system worked pretty well.

Sending email

Sending email is easy - mu4e sensibly uses Gnus message mode to send messages. I also built the latest version of msmtp from source for future flexibility (multiple account support) and it seemed marginally quicker than talking directly to the SMTP server.

Reading email

Initially, I didn't like the mu4e initial screen. Where's my Inbox ? Where's my new messages ? I need to see my new emails and start working, dammit !

Then, to my horror, I realised mu4e doesn't have an Inbox per se - just a list of email messages that sit in ~/Maildir. Unread messages reside in '/new' and read messages live in '/cur'.

I was immediately annoyed at a plethora of irrelevant mailing list messages and corporate communications littering my Inbox - sorry littering my 'List of unread messages'.

God - this is terrible. I can see that the only solution here is for me to shave that yak again and configure procmail or use Sieve and Dovecot merely to mimic what I had working fine in Thunderbird out of the box.

Then I saw a post from Dirk (mu4e's author) on the mu4e mailing list.

'mu4e doesn't really have folders - instead "All Mail", "Inbox", "Important", "Sent Mail" etc. are queries - so the same messages can be present in more than one of those'.

Then it struck me like a bolt of lightning. He was absolutely right. I don't need a 'Corporate' folder. I don't need a stupid 'Oracle/MailingLists/dba-wizards' folder with a 3 level hierarchy. I don't even need an 'Inbox'.

What I need is a set of queries to mine the database. Yes - a set of structured text files is in fact a database. mu4e calls these queries 'bookmarks' and provides some useful ones out of the box.

  • bu (unread messages)
  • bw (messages sent in the last week)
  • bp (message with attachments)

Then I would need some ad-hoc or stored queries (e.g. large messages). For example, if I wanted to find that excellent email from Frank about table partitioning he sent out last year:

  • contact:oracle-dba-wizards from:frank date:2015 partitioning

The 'Inbox' processing now changes slightly. I needed to read each unread message and quickly decide what to do with it:

  • Delete it. 'Blue Nova with lights on in car park 710 East'
  • Act on it. Do something. Or reply. Then (optionally) delete it.
  • Mark it for (later) 'Action'. Absolute last resort. Obviously.
  • Archive it. Something potentially interesting but not now.

Then I remembered that this tied in nicely with an post by Ben Maughan on his excellent Pragmatic Emacs blog which made me question the need for a 'ToDo' folder. Or any folders, in fact.

'I also had folders for putting emails in and I would occasionally have a painful cleanout of my towering inbox, agonising over which folder to put an email in, or whether I should create a new folder for it'.

Then something else struck me. Joking aside, it was quite pleasant and liberating when I had no incoming email. The fact I didn't have a cluttered Inbox presented by default, staring me in the face was great. I had to make a conscious effort to get new mail ('U') and I then had to make a conscious effort to read it 'b u').

Initially, I explicitly disabled periodic automatic fetching of new email (as I had configured in Thunderbird) so I could verify getmail was fetching (and deleting) the correct number of messages from the server and attachments were being handled correctly.

However, I decided to stick with manual email processing initiated by me when I was ready to process email. Notifications of 'new email' are incredibly hard to ignore and a needless context switch if you are busy concentrating on something else (watching Shetland ponies dancing on YouTube).

Further thoughts

Now I am wondering what the point of all my historic folders are; archives by year, mailing lists, personal, sent etc. Previously, in Thunderbird, they were logical groupings and I viewed annual archives as 'read-only' but now in the new scheme, they are merely entries in a database and strictly speaking every single message belongs in '~/Maildir' for simplicity. With one file per message, there is no longer any advantage in logical folders.

So yes, I must be the only person in the world who intentionally went from 'Inbox Zero' to 'Inbox 47,339' and didn't care.

Also, inevitably, I am now being increasingly tempted by the lure of org-mode. In particular, Ben's quote struck a chord with me.

'your inbox is not a todo list'

This is so true and something I have been abusing for years. An email message doesn't have a start date, an end date, a category, a priority or a current status whereas org-mode supports all of those elements.

In addition, org-mode capture takes this further. You can capture anything from any source; an email message, a Web site, a phone call, an instant message, a news article, a blog post, anything.

Configuration

A lot of people conclude and help the reader by including their 'gnus', 'mu4e', 'msmtp' and 'getmail' configuration files but mine are simply variants on the many excellent, annotated examples out there and I'd only repeat an idiotic mistake and post something crucial and security related in clear text on the Internet.

why Linux will never succeed in the mainstream

I have been running Linux Mint for 8 weeks now and I've been delighted with it. My desktop PC is fast and responsive and I am hugely impressed by the sheer amount and quality of software available for Linux. Printing, scanning, wireless networking, audio, DVD writing and all my USB devices just work.

I don't have a virus scanner consuming memory and chewing clock cycles. I am no longer considering a memory upgrade as Linux works fine with my paltry 512MB.

I have all my favourite applications available (Picasa, Chrome, FileZilla, Emacs). Linux is brilliant as a development platform and installing software is easy. The Mint desktop looks great and with the addition of Microsoft TrueType fonts, my display is razor sharp and crystal clear.

Finally, and perhaps, most importantly, my wife has also embraced the change. She now uses Thunderbird instead of Outlook Express, Firefox instead of Internet Explorer, OpenOffice instead of Microsoft Word and Excel and Nautilus instead of Windows Explorer. All of this was fairly transparent and painless.

This fulsome praise all sounds like an advert for the wonders of freedom loving, a precursor for some open source software evangelism and a concerted attempt to convert the great unwashed to Linux. However, there is an elephant sitting in the room. Right there - in the corner.

The Ubuntu and Linux Mint (which is based on Ubuntu) distributions have a 6 month development cycle. This means that a new release will appear twice a year which is great because users know when the next major release is due. In addition, minor fixes, security patches and improvements are continually being pushed out via automatic updates. What is not so good is the actual process of upgrading to a major release which, in my opinion, is relatively complicated and risky for an inexperienced, new user.

To be fair, Linux Mint are upfront and honest and describe the upgrade process fully, the options available and the pros and cons of each approach.

'There is no guarantee that it will work for you. In fact, this [dist upgrade] is quite a risky process. If you're experienced and if you know how to troubleshoot and solve common Linux problems then you're probably OK. If you're a novice user we recommend you perform a fresh installation of Linux Mint 8 instead.'

David Marsden is an experienced Linux user and comments that he is comfortable performing Ubuntu upgrades, quickly and reliably without losing his data. He claims that Ubuntu upgrades are quicker and easier than applying a Windows Service Pack.

Of course, David's absolutely right. Even I managed to upgrade to Linux Mint 8 at the first attempt without losing any of my user data and even managed to preserve the configuration settings for all my favourite applications. In fact, apart from the modified login screen and wallpaper, the four people who use the Linux computer would have struggled to notice the change, it was that transparent.

In fact, all I needed to do was:

  • When originally installing Linux, create dedicated, separate partitions for user home directories and data. I use '/home' (user directories) and '/data' (music, photos, documents).
  • Try to stick to the default Mint (and Ubuntu) software repositories.
  • Note down the additional applications and software packages you have installed.
  • Jot down user and group id's (copy '/etc/passwd' and '/etc/group').
  • Backup the home and data file systems (twice). Check the numbers of files. Check the size of the directories. Check the checksums. Check the backups are readable. Check the hidden directories. Check the backup disk isn't full. Check everything.
  • Burn the Install CD and install the 'upgrade'.
  • Preserve the '/home' and '/data' file systems, leaving all existing data intact. You did remember to jot down that '/home' is '/dev/sda6' and '/data' is '/dev/sda8', didn't you ?
  • Move '/home/user' to '/home/user.backup'. Repeat for each user account. This ensures that Gnome and desktop related settings are re-created.
  • Re-create the necessary user accounts and ensure the user and group identifiers are the same as before.
  • Selectively, copy the various, 'hidden' dot directories for applications (Rhythmbox, Picasa, Pidgin) back into the user directory to preserve the application settings.
  • Reconfigure wireless networking.
  • Reconfigure the printer.
  • Remove the irritating fortune cookie from 'Terminal' (Mint only).

Now I am fairly technical and understand most of this. I have no problem whatsoever doing all of this. David is correct - all of this is common sense, quick to do and the whole process takes less than 2 hours. I don't even mind repeating this process every six months because, as David points out, I have a new, shiny operating system with new features, additional applications, bug fixes and improvements.

What I have a problem with is trying to explain this whole, convoluted process to my father. Or rather, rescuing his system after he has failed to follow this process. Remotely.

Of course, my father has a few options available. If it ain't broke, don't fix it - if Mint Gloria works fine than stick rather than twist. Alternatively, he could use a distribution that automatically performs rolling upgrades so his software is always the latest and greatest.

how I ditched Windows and embraced Gloria

After the Yak was shaved, we suddenly realised we had missed Football Focus and proceeded directly to the casting couch.

I fully appreciate that evaluating a Linux distribution is just 18 minutes and 32 seconds isn't probably sufficient to probably gain a full picture of the capabilities (or otherwise) of any software let alone a full blown operating system.

However, this superficial, high level, cursory review of all the distros below is useful as it solely focuses on the installation process, ease of use - both of which are key for prospective Linux users; particularly those people who might be looking to migrate, as in my case, from Windows.

Ubuntu (Hardy Heron)

As I have installed Ubuntu before, this was fairly straightforward. As I now had the full 80GB disk to play with, I allocated four 20GB partitions (root, home, apps and data).

The installation prompted for my timezone, keyboard type and was pretty quick. Once I rebooted, I was able to configure my Linksys wireless adapter and access the network. As I was using a Ubuntu version that was 12 months old, the system update utility soon notified me that I had over 800MB of available updates which I downloaded.

I was mortified to see the file transfer speeds was variable. Very variable - it ranged between 35-50 KB/sec and I also noticed the strength of the wireless signal was incredibly sporadic. On Windows, I would consistently get a signal strength between 75 and 95%. This computer was in the same room as the router with clear line of sight. I tried browsing the Web - the experience was painful, It was like the bad old days of 56KB dialup modems. I killed the system update which only improved the state of affairs slightly.

I then discovered that there was a Ubuntu mirror hosted by my ISP (Virgin Media) so reconfigured system update appropriately but still - no improvement. A ping to ubuntu.virginmedia.com took 4-5 seconds - yes seconds - not milliseconds. A ping from another Windows PC in the house took 20 ms. This was a major concern and might kill this experiment dead in its tracks with yet another Windows install beckoning.

Another of my concerns about adopting Linux full-time was printer support so I was delighted to see that drivers for my Canon IP4000 inkjet printer were included in a comprehensive list and printed a test page to prove it.

I imported photos, music and documents from my external USB Windows (NTFS) drive (which just needed to be mounted as 'root'). OpenOffice was able to read all my Word 2003 (and 2007) documents, Excel spreadsheets (and PDF's were also accessible using Document Viewer. I could view photos fine and I intended to use F-Spot to manage my photo library.

I was puzzled to see Evolution installed as the default email client so I installed Thunderbird and configured it to access Virgin Media email servers to send and receive email which worked fine.

Mint 5

Next, I installed another year old distribution - Mint 5 with the XFCE desktop interface (as opposed to the more common Gnome desktop which comes with Ubuntu).

Again, I simply reformatted the root partition and left the user data well alone. Mint looked impressive and has a slightly better utility to manage the wireless network. I was offered the chance to install 'Restricted' packages for proprietary software (e.g. the nVidia drivers, Flash). Open source purists may object but I accepted the invitation. I was rewarded by being able to play YouTube videos straight away with no fuss. I recall hunting around on Ubuntu for ages until I discovered the 'Restricted-Extras' package.

My main concern, the performance of the networking seemed unchanged (not wholly surprising as I believe Mint 5 is also based on Hardy Heron).

OpenSUSE

Next up was OpenSUSE which offered a slightly different administration interface and a green theme. This was another old CD and I was gratified and hugely relieved to see that OpenSUSE started loading system updates at a decent speed (500-705KB/sec). If all else failed, I could also revert to this distro although I wasn't entirely comfortable with the Nautilus File Manager after becoming accustomed to Gnome.

CrunchBang

CrunchBang was an interesting distro with a very different interface - Black background, very minimalist with a handy system dashboard (CPU, memory, disk utilisation) with all applications were available from a right-click menu.

I really liked CrunchBang as it appeared to have a small set of carefully selected and useful packages without being overly cluttered with megabytes of additional software. I prefer distros with a compact base where I can then install additional software as required.

If I was the sole user of the computer, I probably would have gone for CrunchBang as the monochrome, minimalist beauty of the desktop was appealing. However I doubted that Norma would be as comfortable with the cut-down interface and funereal colour scheme.

Fedora

I embarked on the now familiar installation process but once Fedora booted, I got a couple of disconcerting kernel diagnostic error messages when using Firefox and the automatic system update failed. There was also a worrying amount of blue - both during the installation process and on the default theme.

Debian

I dearly wanted to love Debian as I get the impression it is the distribution of choice for Linux aficionados and the technically minded crowd. Also, it had lots of red during the installation process which was very promising.

By now, I was getting bored of these time consuming full blown installations onto hard disk so thought I would evaluate the remaining distros simply using the Live CD approach and make allowances for any sluggish performance.

Unfortunately, Debian assume that Debian is the one and only and the best Linux distribution for you so you will obviously be installing the software on to the hard disk. For some reason, (as far as I could see) Debian don't see fit to provide a Live CD so people can choose to evaluate their software so, once again, I found myself performing a full blown installation.

Thus far, every single Linux distro had correctly auto-detected, configured my Linksys wireless adapter and managed to connect to my Linksys Router (running Tomato firmware) which was a pleasant surprise.

But not Debian. Oh no - although the WiFi card was recognised and I could enable wireless networking, see the available networks, enter my WPA credentials, Debian stubbornly refused to connect. There seemed to be a worrying assumption that all wireless networks used WEP and I felt I would have to do some serious digging to fix this and that somehow because I wasn't prepared to do so, I was unworthy to have the privilege of running Debian. Maybe it's some sort of Debian initiation ceremony. In any case, life's too short.

Arch

Arch is an interesting distro - it uses a character based installer. It reminded me of installing SUSE Linux way back in November 1999 when I had to recompile the kernel in order to get the mouse to work. Having been spoiled by all the GUI installers, I was tempted to give up on Arch there and then but proceeded until the partitioning utility decided that I had overlapping partitions (which was odd as every other distribution disagreed) and refused to go any further.

PCLinuxOS

Another short lived experiment - some obtuse failure meant I wasn't able to evaluate this distro.

Mandriva

Finally, I looked at Mandriva which was a great looking distro and very user friendly. I particularly liked the WiFi management utility. It also performed well (even booting off the Live CD).

Ubuntu 9.04

After all this effort, I still had a nagging doubt, that I was using dated versions of Ubuntu and Mint that were more than 12 months old. As there will have been a lot of updates and two full releases during this period, I decided to burned two more CD's for Ubuntu 9.04 (Jaunty) and Mint 7 (Gloria). My main reason was to see if the problems with my wireless Linksys card was resolved in the current stable version of Ubuntu and I also wanted to try the Gnome version of Mint.

Sure enough, support for the wireless USB adapter in Jaunty seemed much, much better and I was able to use the network normally and large downloads were now as quick as with Windows and Open SUSE.

Mint 7 (Gloria)

The Linux Mint distribution is based on Ubuntu which means you can also access the Ubuntu repositories which have a wealth of software packages.

I liked the look and feel of Mint 5 (using the smaller, lightweight XFCE desktop) and was similarly impressed by Mint 7 (using Gnome).

The software update process is slightly more refined on Mint than Ubuntu - updates are classified based on priority.

The clinching factor in Mint's favour was the new Mint Menu introduced in Mint 7 which offered a (I hesitate to say it), a start menu that would be reassuringly familiar to Windows users.

The desktop interface was simple, stylish and uncluttered and I liked the Mint management tools. Also, support for an Ubuntu based distro would be easily available.

So that was it - after a day of evaluating various Linux distributions, I decided to go for Mint 7. Now the boring bit was over with, it was time to sit back and enjoy my new computer.

Yak shaving

How I ditched Windows and embraced Linux - the prequel.

On Saturday morning, I thought it might be fun to install Linux on my home computer. A year ago, I had experimented with a few Live CDs and actually installed Ubuntu (not officially supported) followed by Oracle Enterprise Linux (officially supported) on my work laptop (Thinkpad T61). For various reasons, I subsequently had to reverse that change so I thought it might be worthwhile and interesting to install Ubuntu on my own computer (an ageing but reliable Dell 4500).

There was no problem with the computer, there was no problem with Windows XP. I am not a rabid, freedom loving, Microsoft hating individual. I just wanted to play around with Linux before 'Football Focus'.

As Norma also uses this computer (email, Web, Word documents), I decided to install Ubuntu as a dual-boot option alongside the existing stable Windows environment. This was firstly to evaluate whether Linux supported all my various peripherals and software requirements and secondly, let people continue to use Windows.

I dug out an old Ubuntu (8.04 - Hardy Heron) Live CD and started the installation process. When I came to partition the hard disk, I chose the 'Advanced' option so I could preserve all my existing data with 60GB allocated to Windows with another 20GB for Linux.

The re-partitioning failed; the 'ntfsresize' program helpfully told me with a typically cryptic Unix style error that '1028 is greater than 1024'. No matter - just get a Windows recovery disk and use the 'fdisk' utility to re-partition.

I booted from the Windows recovery CD. After a while I saw 'Files being copied for installation...' and wondered if I had missed the prompt to enter a DOS prompt enabling me to run fdisk. So, I hit 'Control-C' and rebooted. As I expected, my computer was now completely screwed as I had aborted during a full installation.

I sighed inwardly at my impetuousness and stupidity. I also blamed David Marsden who had helped sow this original tiny, little seed in my brain which was now turning into a long running saga and a complete waste of time and effort. Just to get back to where we had started. Almost.

This whole exercise was a classic example of Yak shaving as its finest. I was justifiably proud of myself.

I decided to postpone the Linux experiment to another day and simply re-installed Windows. One hour later, the machine was usable again. All my data was preserved so I took a full backup to an external USB drive. However, although none of my programs were installed and the Windows registry was pure and virginal, the 'Program Files' directory was horrendously cluttered with lots of unused folders.

Now I had a choice; dig out all those drivers and install disks for my Linksys Wireless Adapter, Canon printer, scanner, reinstall and reconfigure all the important software (Picasa, Firefox, Office, Outlook Express) so the machine would actually be usable again. Preferably before Norma got home and asked 'Can I just quickly check my email ?'

Alternatively, now I had a full backup, I could do the right thing, rebuild the computer completely and install Linux. So that's what I did. Only I didn't because I wasn't entirely sure Ubuntu was the best Linux distribution for me. Using the excellent, open source program InfraRecorder. I embarked on burning ISO images for all those distributions I was aware of, have seen recommended or reviewed recently.

I also perused DistroWatch to look for the most popular and used Dan Lynch's excellent, comprehensive, honest, balanced reviews of various Linux distributions.

The cast list, in no particular order: