Frustratingly, I can't locate it now but I recently saw a post on Mastodon from someone who had great difficulty creating a new file in his Documents folder.

This struck me as very amusing but I think the point he was trying to express was that his 'Documents' directory is a disorganised, unstructured, cluttered mess of files so he is incredibly reluctant and can barely bring himself to compound the issue further by typing vi linux-notes.txt while in the 'Documents' directory.

I have a similar issue. I use Emacs and orgmode - primarily for note taking. Well it's sort of a bastard combination of notes and a journal - but not a proper orgmode journal. Maybe it should be. Come to think of it, every notes file contains a 'ToDo' section but not a proper orgmode list of ToDo items. Maybe it should. Sue me.

To clarify a little, when I'm working on a technical project, I am almost always required to produce some project documentation later on. Typically, this might be an installation guide or a handover document for a client. Obviously, this needs to be a polished, structured, coherent document that is technically accurate.

Clearly, it is very difficult to create such a document after the fact when the memory is hazy and the precise, important details are almost forgotten. As an input to that formal document, and for my own benefit, I tend to keep notes as I go along. If I encounter a problem, I write it down together with the error codes and the various avenues I explored and the final resolution. Often, my notes include links to articles or technical blogs describing the same or similar issues. Occasionally, you won't be surprised to know the notes may even contain an outburst or a mini-rant.

Bug first noted and ack'ed in September 2017. Still fecking present in May 2020. God help us.

Currently, I create a new directory for each project I work on. For the ACME project, I will work in the $HOME/work/acme directory. Often there will be additional sub-directories 'apex', 'scripts', 'documents' etc.

My notes file for each project used to be named notes.org but this was sub-optimal as I would then have multiple notes.org files in the Emacs buffer list and 'Recent Files' list so it was hard to distinguish them.

To address this, I then prefixed the filenames thus; 'customer-notes.org'. This worked a little better as typing 'acme' offered the correct completion almost immediately.

However, this isn't ideal either as sometimes I am hunting for the details of an APEX issue that I encountered last year for that customer but was it 'ACME' or was it 'Vodafone' or was it 'ABC Corp' ?

I suspect the orgmode experts would say the solution to this is a single, larger orgmode file ('work.org') containing all the projects I have worked on which is searchable and only the currently active customer is expanded and all the others are hidden.

Funnily enough, years ago, I did experiment with the 'single large text file' idea and I have also used TiddlyWiki when I needed to share notes between a laptop and a desktop machine and later DokuWiki when, for some reason,I thought my notes, jottings and ill-considered rants should be available on the Internet.

Using a single orgmode file for work is attractive but that file would obviously have to reside in the ~/orgmode directory whereas when I am taking notes and want to open source code, M-x find-file works neatly as I am already in the context of the project directory. If I move my notes to the orgmode directory, I sense more context switching, more typing and more work.

Maybe I could investigate the Emacs projectile package to see if that would help resolve this minor issue. That would represent an ideal opportunity for more procrastination.

Connor McDonald posts an excellent series of articles about tuning a database application.

This analysis from a end user perspective reminded me of my own experiences when I was a technical consultant helping customers running a large CRM application, typically in call centres scattered across Europe.

I was often summoned onsite and told to solve the problem that 'The application is slow'. Usually, different people were eager to give me their view on the issue:-

• Oracle DBA's often would be pouring over AWR reports or a monitoring tool and examining wait events in minute detail. 'We can see multiple ITL waits over 700 ms. This means we need to increase the FREELISTS for the ORDERS table but the business won't let us have an outage'.
• The application developers would also proffer their own diagnosis - 'Oh yes, we already know what causes that. It a custom workflow written by the previous integrator. It needs refactoring but it will take 3 months'.
• The CEO brusquely told me - 'This CRM application isn't fit for purpose. If this isn't resolved by Thursday, we're going to evaluate SAP and Safra will be hearing about this'.

Now, this is all detailed, technical analysis and background providing useful information to be considered but I would often ask to see the problem at first hand by talking to an individual who was using this application all day, every day to see the perspective from his point of view.

This simple request was often met with puzzlement and resistance by the technical team - 'Why do you want to watch a user ? We've already told you what the problem is. This will just waste time'.

Sometimes, this resistance was born out of a concern that the user feedback would unearth different, unrelated functional issues and distract me from the performance problems under investigation. Alternatively, a floor supervisor would air the valid concern that my conversation with an agent would distract him from dealing with the customer call. This was easily overcome by letting the agent handle the call with me simply watching and taking notes. Then, after the call was finished, we'd have our chat.

On one occasion, this approach of listening to the users proved particularly beneficial. The client was a utility company but could have easily been a bank or a telco. The business scenario in the busy call centre was typical. he, Essentially, the customer calls in with a query or complaint which is resolved by the agent.

Some call centres use CTI technology where the application looks up the customer from the inbound telephone number and then presents the customer details to the agent on the screen so he can start the dialogue, typically security checks.

However, this call center didn't use CTI so the agent had to manually search for the customer before the call could commence.

I watched the agent process an entire customer enquiry from start to finish and took notes.

The call started and after the initial exchange, the agent asked for customer's surname and started a search. In this example, the customer was Mr. Johnson. I watched with interest as the agent typed in 'J' into the customer tab and his 'Search'. This operation took a long time. There are 66 million people in the UK and 38 million of them appear in this client's database. Searching a table for all customers with a surname starting with 'J' is expensive performance wise.

The agent didn't seems phased or perturbed or even irritated as the hour glass popped up. He merely continued to clear security with the customer. By the time, this exchange was complete, the search had finally returned.

My eyes widened as the agent then proceeded to sort all these thousands of customers by surname and scrolled down page by page searching for 'Johnson'. Again, sorting a large data set like this is sub-optimal performance wise. This is an online application where users are expecting each button click to return within 3 seconds - not 3 minutes. The solution isn't for the DBA to increase the PGA to allow larger temporary segments to accommodate the massive sort operation. The solution is not to issue the request to sort thousands of records in the first place.

It would have been marvellous if the agent had uttered the immortal words 'Sorry, Mr. Johnson but the system is really slow today'. Unfortunately, he didn't but you can certainly envisage similar scenarios where this excuse is proffered.

When the agent finally identified 'Mr. David Johnson' of '23 New Street, Canterbury, CT2 6AD', the rest of the customer call went pretty quickly. It was either taking a payment, changing a tariff, lodging a complaint, a billing enquiry or a change of personal details and common to most agents working on that floor.

After the call ended, I asked the agent why he used that sequence of searches and scrolling to identify that specific customer. The answer, inevitably, was 'We always do it that way and when I joined, that's what Barry showed me...'

Then we revisited the call using a different technique. This time, I recommended he searched for the complete surname (he has that available as soon as the customer starts talking). When he searched for 'Johnson', the query ran much quicker but there are still probably thousands of people called 'Johnson' in the UK.

Instead of sorting and endlessly scrolling to locate the customer in question, I suggested he simply entered the postcode into the 'Address' section. The postcode is now known after the customer completed the security questions. He could have used customer number but that's a long 12 digit number with scope for error when entering it.

[ Ironically, one of his reasons for typing in 'J' instead of 'Johnson' was that 'Hey - I'm pretty lazy and that's a lot of typing' which resonated with me as that's normally my attitude. ]

The agent just needed to type the first element of the postcode ('CT2') to refine the search further and now we have the customer details on the screen in a fraction of the time it used to take him.

I thanked him for his time and told him it had been a very interesting exercise for me to see the application actually in use. He reciprocated and thanked me. As he went to put his headset back on, he smiled and said:-

'You're not going to tell my Supervisor about what we've just done, are you ?'

'Well, yes I am. Trying to solve these performance issues is why I've been asked to come in. Why do you say that ?'

'If Barry gets to hear about this, our call targets will probably be doubled !'.

He smiled and nodded at the electronic rolling ticker display detailing how many calls have been handled, how many are waiting, average call duration etc.

I tend to switch Linux distributions quite often. Consequently, I tend to have this process down to a fine art and it doesn't take me that long. The most time consuming element is ensuring the necessary backups are in place.

However, you normally find some package or configuration option you forgot about and my recent switch from Arch Linux to Fedora 29 and back again unearthed a strange problem with the Dovecot IMAP server I hadn't encountered before.

When I accessed my email, my automatic message filtering (using sieve) wasn't working so all messages ended up in INBOX. Worse, on every transfer, the messages were duplicated.

The logging revealed a strange error related to file permissions

$ journalctl | grep dovecot
Nov 29 08:45:32 <host> CROND[3356]: (andy) CMDOUT (msg 53/60 (12176 bytes),
delivery error (command dovecot-lda 3415 wrote to stderr: lda(andy,)
Error: net_connect_unix(/var/run/dovecot/stats-writer) failed:
Permission denied))

The permissions on this socket file were as follows:

$ ls -l /var/run/dovecot/stats-writer
srw-rw---- 1 root dovecot 0 Nov 29 08:52 /var/run/dovecot/stats-writer

Google revealed a couple of fixes. One was to simply change the permissions on the socket file to mode 777 which works for the duration of that session but the problem simply reappears after the next reboot.

Another, more promising avenue was to add a new section to '/etc/dovecot.conf'.

service stats {
   ...
}

I tried a couple of combinations which didn't work but then I went back to basics. I am processing my email as 'andy' but the dovecot processes are running as 'root' and' 'dovecot' (although they could be SETUID executables).

root      8276     1  0 11:45 ?        00:00:00 /usr/bin/dovecot -F
dovecot   8278  8276  0 11:45 ?        00:00:00 dovecot/anvil
root      8279  8276  0 11:45 ?        00:00:00 dovecot/log
root      8280  8276  0 11:45 ?        00:00:00 dovecot/config

However, I wondered if Local Delivery Agent (LDA) was somehow running as 'andy' (the error log implied this) which explained the permissions issue as the Linux user 'andy' isn't able to read that socket file.

Sure enough, the fix, inevitably, was a simple one-liner to add user 'andy' to the existing 'dovecot' group.

$ sudo usermod -a -G dovecot andy

Occasionally, I have to search lots of files for a pattern. It was only recently I discovered the wonderful silver searcher utility which saves me a lot of time.

To install 'ag' on Fedora, use the following (which isn't entirely obvious or intuitive if you're used to typing 'ag').

$ sudo dnf install the_silver_searcher

I believe there is an Emacs interface which would save me even more time.

$ time ag 'sql statement execute time' *.log
real    0m0.125s
user    0m0.128s
sys     0m0.257s

$ time find ~ -type f -print0 | xargs -0 grep -i 'sql statement execute time'
real    0m23.725s
user    0m7.965s
sys     0m1.618s

When I initially revisited Emacs, I used mu4e (instead of Thunderbird) for my email.

I used the wonderful Gmane service to read mailing lists in Gnus and Elfeed to read blogs and RSS feeds within Emacs.

This worked fine but after a while it became a little tiresome having to remember different key bindings to essentially perform the same repetitive tasks; reading messages, navigating (next/previous) messages, moving messages, saving messages, marking messages, deleting messages, searching messages, forwarding messages, replying to messages and occasionally composing brand new messages.

The solution was blindingly obvious. Just use Gnus for everything involving a 'message' instead of three separate packages. One set of key bindings to learn and master and Gnus has comprehensive functionality. Less is more.

Converting all my email processing to Gnus was easy enough to address as I previously used Gnus to handle my email and Gnus natively supports the maildir format.

Accessing mail was fast as I had already invested in establishing a Dovecot mail server and transferred messages from the corporate IMAP server to local Maildir directories (using getmail).

To replace Elfeed for reading RSS subscriptions, Gnus offers a nnrss back end but it was so slow and sequential, it was virtually unusable. Investigations revealed another option. Lars doesn't like anything that is slow and sequential so years ago, he created the Gwene (Gnus Web to Newsgroups) service which took any RSS or Atom feed and converted it to a pseudo newsgroup on the Gwene news server.

Gwene already carries a lot of popular blogs and feeds and if your esoteric favourite blog isn't present, you can simply add it and the content appears immediately.

I was about to celebrate and put the kettle on when, suddenly, unexpectedly and rather inconveniently, Lars decided to shutdown the freely available Gmane (and Gwene) services for understandable reasons (idiots launching DDOS attacks on the servers, threat of legal action).

I then researched alternative methods and experimented with a number of RSS to mail gateways. The best one was rss2leafnode which takes a list of blog subscriptions and periodically fed the content into a local NNTP news server - leafnode.

This solution worked well as I was now using Gnus to read email, mailing lists, newsgroups, blogs and RSS feeds.

Perfect - well almost. One of the advantages of mu4e (and notmuch) is the lightning fast search and powerful abilities. In Gnus, you are able to limit lists of articles by author, subject and marks and use IMAP search functionality.

However, I regularly used combinations of search terms and full text search on the message body. For example, display all messages from Peter to the 'Footy' mailing list in 2016 where he mentions 'Basingstoke'. In mu4e, this search is done using a query

'from:peter to:footy date:20160101..20161231 basingstoke'

I have always tended to use hierarchical directories and mail folders so my mail is archived by year. Gnus could also limit a search by author and full text search can be done using the 'nnir' engine so this type of search is possible in Gnus but unwieldly.

Combinations of multiple search terms were slightly more problematic but as I was already running a local mail server (Dovecot), I decided to use Solr to implement more flexible searching in my mail folders.

Occasionally, I want to search for a message and I have no clue who sent it or when so I need to search all email messages in all folders for 'cobain conspiracy'. This is easy in Thunderbird and I found the best way to achieve this was to create Dovecot virtual folders for 'All Mail' and 'Sent Mail' that spanned all messages regardless of date. Virtual folders can also be used to implement mu4e's built-in queries for 'Last 90 days'.

Why is Gnus unbelievably speedy ? Everything is local.

• No need for a direct IMAP connection.
• All email periodically delivered to local folders accessed via Dovecot.
• Message filtering and spam handling performed by Dovecot/Sieve prior to delivery.
• Solr maintains search indexes of email transparently.
• Mailing lists and RSS feeds updated automatically by Gmane/Gwene.
• Virtual folders automatically maintained by Dovecot.
• Gnus is simply the most fantastic piece of software.

My only reservations were that the number of Linux packages to be installed, configured and updated but thankfully, in September 2016, new owners stepped forward to resurrect the invaluable Gmane and Gwene services so I was able to dispense with Leafnode.

The only prolonged, expensive network access involved here is fetching articles from the Gmane and Gwene servers but this has proved so fast and reliable that I didn't need to pursue the option of mirroring this small number of feeds locally in leafnode (although, undoubtedly, that would have been an interesting and fun exercise).

birth

At the tantalising climax of the last episode, I was invited by Steve for a whistle-stop tour of Emacs.

Steve explained that the main reason he used Emacs was pure laziness. Naturally, this immediately got my attention. He explained:

'I'm lazy. It's not a fault. It's a fact. Most decent programmers are lazy. You're lazy'.

'Hang on, just a minute ! What do you mean - I'm lazy ?'

'Andy - you alias cd .. to 'up' and 'l' to ls -ltr. Just to save five characters typing. So don't tell me you're not lazy. Anyway, it's not a criticism'.

'I used to work just like you. xterm open. Edit some C code. Compile the code. Fix the compilation errors. Repeat. Or search for a string a set of files using grep. Identify the file of interest. Go to line 723. Edit the file. That's what most programmers do. But when you think about it, it's a very inefficient process. All that typing, all that mouse movement, all that context switching. You may think having multiple xterm windows open solves it but it doesn't'.

'In Emacs, we can do all of that processing within Emacs. Look.'

Steve then showed me the edit-compile-fix cycle in Emacs. This was fantastic. Edit the source code. Invoke make using M-x compile. Review the errors in a separate buffer. Hit the compilation error and you're immediately positioned on the offending line in the source file.

Similarly, for grep. Invoke search for 'parser' within all C and header files within Emacs. Then select the occurrence of interest and you're immediately taken to the source file at the right place. No more 'grep -n' and jotting down the line number.

The other biggest advantage Steve showed me was perhaps more basic and fundamental; simply the use of multiple buffers and 'C-x C-b' to manage the buffer list.

Previously, I would use multiple 'vi' sessions or ':n' to edit the next file in the list or toggle between two files using :e #. Worse, I thought that was advanced wizardry.

Steve also showed me the Emacs directory editor (dired) whch was impressive and a very quick way of navigating around the Unix file system.

childhood

And thus, my life using Emacs started in 1992. Initially, I was delighted enough with just this basic Emacs functionality as a great improvement over my previous approach.

However, I wasn't converted into an evangelist preacher from day one. I still used the 'elm' email client in a terminal window and 'rn' to read Usenet.

Also, I can admit this now. I would frequently fire up 'vi'; specifically for any global search and replace operations as it was just simpler and quicker than me trying to learn and master the Emacs equivalent.

Then I got moved to an OS/2 project but, Emacs was available for that now dead IBM platform so I copied my .emacs file and was happy.

Once, I got sent to Boston (America not Lincolnshire) to fix a tricky, long-standing problem. I was instructed to take the source code with me, given an open return ticket and was told to stay on-site until it was resolved. Fortunately, it transpired to be a one-liner involving a subtle race condition. Unfortunately, I fixed it on a Saturday and it was a public holiday weekend which meant I now had a lot of free time on my hands.

I remember sitting in the client's empty offices, revisiting the Emacs tutorial and really taking the time to read and understand it fully. I spent a lot of time experimenting and trying out all the features including stuff I never used, failed to understand or had simply forgotten about.

Even now, I still enjoy reading or watching Emacs tutorials to see how other people use it. Or maybe I'm just weird.

teenage years

In 1996, I took a job at Sequent who produced high end, expensive, fast, parallel processing, NUMA based Unix servers. Although I was now working in a Unix environment again, the DYNIX/ptx operating system wasn't the ideal development environment as a lot of my preferred tools, packages and utilities either weren't available or hopelessly out-dated and buggy.

However, the GNU C compiler was available so I embarked on building a lot of the GNU packages. This was a useful and instructive exercise as, occasionally, changes to the source code were needed to build cleanly - normally the addition of a #ifdef ptx macro.

Of course, inevitably there was a modicum of yak shaving involved here. For example, I had successfully built Emacs from source but it didn't interact properly with the brain-dead ptx version of 'grep'. So then I had to go and build grep which needed flex which needed...

honeymoon period

During this time, I was being slightly more inquisitive about extending my use of Emacs. I converted to ViewMail (VM) within Emacs to read my email and Gnus for reading Usenet news groups. I remember thinking that VM wasn't too far removed from 'elm' on an xterm.

I liked Gnus. A lot. I simply loved the Gnus manual which is undoubtedly the best software manual ever written. I was using adaptive scoring, threading and marks but still, I suspect I was (and am) merely scratching the surface of what Gnus is capable of.

I used the Insidious Big Brother Database (BBDB) as an address book for both VM and Gnus and I started to realise that 'ange-ftp' was much better than remotely logging into various Unix servers where 'vi' was the only option for editing files.

I discovered XEmacs looked prettier and had more packages available so I turned to that. Plus I liked Jamie Zawinski's style,.

Then I was forced to leave the comfort of my office based job, was given a Windows laptop and told to go and work in London on an Oracle data warehouse project. I might have done something awful or insulted my manager at the Christmas office party. I can't quite recall.

mid-life crisis

Naturally, Emacs on Windows was available but it wasn't like the real thing. Without a Unix environment, even Emacs couldn't mask the many deficiencies and limitations of using a Windows laptop as a, err, development environment although I later discovered Cygwin which eased the pain (just a little).

To compound the agony, I was now reading and writing a lot of documents in Microsoft Word and I needed to accept meeting invitations in Microsoft Outlook.

It was now 1999 and the dot com boom was well underway so I decided to join a small startup as an Oracle DBA/developer. This was a slightly unusual environment; well, in fact, there was no development environment. It was my job to create one without spending any money (if possible). That was great so we soon had Oracle databases running on Linux servers which doubled as our development machines.

This happy state of affairs lasted for four years until I moved on to a technical consultancy where I enjoyed some challenging work, troubleshooting tricky (and some trivial) problems at call centres located across Europe. My modus operandi now was (don't laugh here) was to make copious jottings in Notepad during meetings, phone calls and while I worked. When I returned home, I collated the notes into a lengthy Word document.

During this period, I talked to a lot of taxi drivers, visited a lot of sports bars and sat in countless anonymous airport lounges but I didn't even install Emacs, let alone celebrate of the arrival of a long-awaited package manager.

rejuvenation

In 2012, after some enforced sick leave, I returned to work and transferred to a role in product development. This was great because my enlightened employer allows me to use Linux on my work computer. OK - it's Oracle Linux and you may scoff - but it's Linux and I can run the latest version of Gnome, the email client of my choice and am able to install all my favourite packages. And still get support from IT.

Why - now I've got WebEx working and my enlightened colleagues use Wikis, I don't even have to fire up my contingency Windows Virtual Machine any more.

Bizarrely, history repeated itself and I found myself firing up a lot of terminal windows and multiple 'vi' sessions which was where this story originated.

Funnily enough, when I realised I was spending an increasing amount of time editing source code, I looked at what editors were out there. I looked at Sublime Text (commercial) and then at Atom (GitHub's excellent offering) which included many (over 5,000) plugins to support language formatting, colouring, compilation, search, ssh...

But wait, hang on a minute. I had invested a lot of time and acquired a lot of knowledge about a powerful text editor that could do all of this and more.

That's when I revisited Emacs. This time, I endeavoured to go even further and attempted to use Emacs for everything. Well, everything apart from a Web browser, SQL developer and LibreOffice.

Now I use the Prelude starter kit which is excellent coupled with the stylish monokai theme. I use Gnus and BBDB for email, news and RSS feeds. I use Python and the SQL modes extensively.

However, the biggest and most interesting addition to Emacs since my departure is the much lauded (and deservedly so) org-mode package. After a few false starts, I am now using org-mode for note taking, blogging, managing my calendar and ToDo's (my work and Google calendars are sync'ed using vdirsyncer). I recently ditched Pidgin in favour of the Emacs Jabber package.

Nirvana

Finally, I configured $EDITOR and alias'ed 'vi' as 'emacsclient'.

the early years

1962¶

Entered the world as I intend to leave it. Kicking, screaming, naked, held upside down by a nurse slapping me on the backside.

a night at the Lesser Free Trade Hall

1977¶

Wrote my first basic program in BASIC on a Tandy TRS-80. Editing facilities were fairly limited. I think to modify Line 10, you had to simply re-enter Line 10. In its entirety. This was rather time consuming, tiresome and almost put me off computers for life.

10 PRINT "DR. HANDS IS A RUBBISH TEACHER"
20 GOTO 10

These lunchtime sessions also taught me how to interrupt a BASIC program rapidly. Useful, particularly when the physics teacher, Dr. Hands, made an unexpected return to the lab.

losing my religion

1981¶

At Warwick University, the Computer Science lab was equipped with green VT-100 terminals hooked up to a PDP-11. Warwick was where I was introduced to C programming and forcibly indoctrinated into the 'vi' text editor. I remember thinking it was handy that Rogue used the same key bindings for 'up', 'down', 'left' and right'.

1984¶

I left the confines of my room and went on the milk round. Instead of coming back with a pint of silver-top, I somehow managed to land a job with a software house as a 'Junior Programmer' and started work on a Unix project using 'vi' and state of the art amber VT-100's.

a short stay in purgatory

1986¶

Reallocated to a project on VMS where I was forced to use a bewildering command syntax and a limited, primeval text editor called EDT that made extensive use of the numeric keypad.

Later, I was made aware of an alternative editor called EVE. Better still, EVE was actually built using an extensible TPU (Text Processing Utility) library for VMS. Joyously, I discovered some clever person had built a 'vi' clone using TPU. It took me 3 days and cost me £79 to download the software over a 2400 baud modem but it was worth it.

New Dawn Fades

1988¶

Left permanent employment and became a freelancer. My nightmares about the DIRECTORY/SIZE command subsided as did my repetitive strain injury. I now found myself porting an Ada compiler from a mainframe system to Unix, reunited with a decent editor, short commands and the joys of pipelining.

the road to salvation and true enlightenment

1992¶

Started work for a relational database company. On day one, after coffee and introductions, I was offered the choice of a Sun workstation or 'as you've used VMS a lot, you'd probably prefer this DEC Workstation'. I forcefully elected the sensible option, held up a crucifix and threw some garlic cloves in the direction of the VAXStation. I claimed I'd used a Sun workstation for many years (a lie) and had to surreptitiously watch my neighbour to learn how to manage X Windows.

I sensed my colleague was slightly suspicious but he had a beard and a pony-tail and was quiet but helpful. I noticed that while I had my screen divided equally into 6/8/10 equal sections each running an xterm, his large monitor simply displayed had a background of the Monty Python 'dead parrot' sketch and a single application taking up 90% of the monochrome screen estate.

'What is that program you're using to edit files, Steve ?'

The man with the beard and the pony tail swivelled in his chair and uttered the immortal words

'It's called Emacs. Do you want to see how I use it ?'

I have used the Unix bash shell for many years. As I am incredibly lazy and forgetful, I have become accustomed to using ctrl-r find to find and scroll though the latest find commands I have issued.

Occasionally, I noticed that a lengthy, complex, useful 'find' command' (which I mercilessly plagiarised from a clever person via Google) was no longer in my shell history.

Investigations revealed the default bash history is a paltry 1000 commands so I decided to increase this to 10000 by adding the following line to '~/.bash_profile'.

HISTFILESIZE=10000

After all, disk space is much cheaper than my time.

I had been contemplating and researching the purchase of a dedicated Network Attached Storage (NAS) for a long time. Initially, I considered a few different options; an entry level unit like a Synology DiskStation, a small server like the HP Gen 8 Microserver or Dell T20 and installing the disks or even buying the individual components and building the unit myself.

However, I'm pretty useless with hardware and as a NAS should be high quality, reliable and solid, I decided to purchase a ready made unit.

I decided I wanted something that could run FreeNAS with ZFS rather than some proprietary GUI and I found a lot of helpful advice on the FreeNAS forums.

My main requirements were basic and standard for a home user:

• Automated reliable backup of photographs, music, DVD's, home videos and documents.
• Ability to run Plex Media Server (2-4 concurrent users with transcoding). This requirement eliminates a lot of the cheaper NAS units with slow processors.
• Potential to perform backups to cloud storage (Amazon Glacier).

IXsystems sell ready made FreeNAS units but they are relatively expensive and I would have to pay shipping to the UK so I started to look for a UK supplier.

After procrastinating, delaying, repeatedly shelving then resurrecting the idea and trying to justify the cost, not for the first time, my wife tipped me over the edge. She dropped her laptop. Nothing unusual about that. She'd dropped it before and I paid a man with a soldering iron to replace the power supply jack.

This time, she'd broken it completely so it was (almost) cheaper to replace the laptop than fix it. Like many personal users, my backup strategy wasn't exactly non-existent but certainly sub-optimal. I had photos burned to miscellaneous CD's, I had various photo albums uploaded to Flickr and Google. Obviously, I had all my work data backed up (CrashPlan at work, external USB drive at home). My entire music collection (ripped to lossless FLAC) was also backed up three times. As for the wife's laptop, well, I had an three month old backup from the last time I restored the laptop coupled with her important documents safely backed up on a USB stick. Which was missing.

I could have probably re-assembled and recovered 98% of everything that was crucial and written off the gaps but for one, important, irritating factor. My wife is studying for a course and she had very recent notes, jottings and drafts of essays on her laptop. Inevitably, none of these were backed up anywhere.

I gave her her 5 year old workhorse laptop back, got her email working, restored her Firefox bookmarks and reassured her I would retrieve last night's draft of her essay, err, shortly.

I had already had to pay computer repairman £50 just to tell me the laptop was beyond economic repair and now my misery was compounded as I had to shell out another £99 for 'data recovery'. I knew this simply meant hooking up the disk drive to another computer and pressing 'Copy' but that's the price you pay for being a idiotic cobbler with holes in his shoes.

Thankfully, he managed to retrieve everything - the media library I already had as well as the important documents for the wife.

This episode gave me the impetus and justification to go ahead and purchase a NAS and get a proper backup strategy in place.

I decided to purchase a Mini FreeNAS from a UK company called Server Case who offered pre-built NAS units meeting all the recommended hardware specifications together with the latest version of FreeNAS (9.10) installed. The basic model came with 4 disk bays and 8GB of ECC RAM so I configured a system with 4 x 3TB WD Red disks and upgraded the memory to 16GB.

The technical support from Server Case was excellent and helped me customise the system and answered all my newbie questions fully and promptly. Each unit is assembled to order and the hardware stress tested but even so, the package was delivered within three days.

I unpacked the large, well packed, cardboard box and although I had measured the dimensions, I was immediately impressed with size and appearance of the unit, the modern, stylish black case and the build quality which seemed very professional and solid.

I powered the unit up and heard the disks spin up and blue lights come on. You are supposed to hook up a monitor during installation but I didn't have a VGA cable handy so I just plugged it into the router and determined the IP address so I could login to the FreeNAS Administration interface and configure the system.

I had already decided to configure the four disks in a RAID-Z2 configuration which meant two redundant disks giving me usable space of 6.1TB. As the sum total of everything I currently own is just over 1TB, this was more than adequate and hopefully future proof.

I found the FreeNAS Web admin interface easy to use and I quickly setup NFS shares and got clients working on my various Linux laptops. I also enabled ssh so I could login remotely to access the FreeBSD command line.

FreeNAS includes a number of plugins for popular packages including Plex but I decided to create a jail manually and install it which worked fine.

Then I configured periodic SMART disk tests to check the integrity of the disks and ZFS filesystem as well as a regular backup of the FreeNAS configuration database.

I migrated all my data simply by rsync'ing to the NFS filesystems over my wireless network. With hindsight, this was not the quickest way to do it - a wired connection directly into the router would have been much quicker but it chugged away, was resumable and did the job.

I then degraded the FreeNAS performance still further by moving the NAS out of the bedroom, away from the router and into a spare bedroom using TP-Link Powerline adapters to simulate a wired connection. [ I relocated the NAS as it was quiet but hummed slightly in our bedroom and the air ventilation wasn't great as it was rather cramped sitting adjacent to the router. ]

Inevitably, I then spent a lot of time playing with my new toy. I experimented with all the available plugins, installed a load of software and I learned a little about FreeBSD and jail management.

For the backups, I decided on a pretty simple strategy. I created hourly cron jobs which executed on the FreeNAS server and pulled files to the FreeNAS using rsync (but didn't sync deleted files). If a client was unavailable, the rsync was skipped.

Then I created ZFS snapshots (hourly snapshots retained for 24 hours, daily snapshots retained for a week, weekly snaps retained for a month, monthly snaps retained for a year).

This was adequate but the spectre of my wife losing 59 minutes of work on her essay still haunted me so I installed DropBox to perform real-time backups to the cloud (aka someone else's server) and after looking at many options installed a wonderful open source backup utility called Syncthing in another dedicated jail.

Syncthing is similar to dropbox but uses a notifier utility to detect changes on the filesystem which trigger an efficient, incremental backup when changes are made rather than polling regularly. Syncthing also supports multiple clients (N-way replication) and can also perform one-way sync (master-slave) which is what I required.

In conclusion, I am very pleased with my purchase and FreeNAS setup now. There's something pleasing about having a home server hidden, out of sight, always on, with an uptime of 67 days 17 hours and 3 minutes.

I recently had to re-install my work laptop with Oracle Linux 7. With backups, it didn't take too long to reinstall. The most time consuming task was compiling Emacs 24.5 from source. Emacs 24.5 is required for the excellent Prelude starter kit I have recently adopted. There are a lot of pre-requisite packages for Emacs that are available (but not included) in Oracle Linux 7.

As part of the 'Emacs for Everything' experiment, I have also started to use an Emacs package called 'elfeed' to read RSS feeds and while it worked in my new, shiny environment, I noticed it ran much slower then previously. I tracked this degradation to the fact that OL7 ships with a dated version of 'GnuTLS' (3.3.8 released in September 2014) whereas the latest version is 3.4.9 (released in February 2016).

As GnuTLS is a secure transport communications layer, I decided to upgrade it and recompile Emacs including the updated libraries as I'd done before.

Usually, I remove the bundled software package using yum and install the new version from source in '/usr/local'. However, in this case, lots of packages depend on GnuTLS including fundamental ones like Gnome 3 so I had to leave the existing package alone.

Inevitably, since I last built it, the GnuTLS package had been updated and now depended on a crypto package called Nettle that was installed but version 2.x. More yak shaving ensued involving 'pkgconfig' and LD_LIBRARY_PATH but finally I was able to build Emacs using the latest version of GnuTLS.

I was rewarded by 'elfeed' performance reverting to its previous stellar levels. elfeed fetches feeds in parallel and is very fast. It takes less than 20 seconds to update my 150 feeds and the keyboard interface enables me to process feeds very quickly compared with a Web interface like Feedly although the latter does sync between multiple platforms.