Software Section Menu

Collection of Tech/Software Tips from My Blogs

Introduction

I've been blogging about many technical/software tips, but they've been scattered in many individual posts. As a result, I decided to concentrate most of them here in reverse chronological order, and was encouraged to do so by some online friends.

2023-07-13: Tips for eliminating whitespace in Template Toolkit (TT2/TT3) outputs

See Template Toolkit.

Get rid of trailing whitespace: /[\t ]+$/ms.
Get rid of multiple consecutive empty lines, or ones at the start/end of files: /\n\n\n/ms , /\A\n\n/ms , /\n\n\z/ms.
Converting [% to [%- and %] to -%] helps.
I use "canaries", unique pieces of text placed between template directives, to debug.

Licence

You can reuse this entry under the Creative Commons Attribution 4.0 Unported licence, or at your option any later version of it. See the instructions on how to comply with it.

Whatever code given here is provided under the MIT / Expat licence.

2020-10-06: Tech Tips: Sound on Fedora; neovim build; add committers on src.fedoraproject.org; nginx warning on startup

This is a collection of recent tech tips, which I decided to share.

The default CMAKE_BUILD_TYPE in neovim's wrapper makefile is Debug which generates slower code than Release. So make sure you use a command similar to this one to build it: make CMAKE_BUILD_TYPE=Release CMAKE_EXTRA_FLAGS="-DCMAKE_INSTALL_PREFIX:PATH=$HOME/apps/neovim".
It may be a testament to neovim's efficiency that I found its speed acceptable even when it was built in “Debug” mode.
When trying to use nginx with a custom configuration file, by executing for example: nginx -c ~/conf/trunk/shlomif-settings/nginx/httpd/minimal-nginx.conf using the -c flag, nginx may emit this warning:
```
[shlomif@localhost ~]$ nginx -c ~/conf/trunk/shlomif-settings/nginx/httpd/minimal-nginx.conf
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied)
```
This warning should be ignored, and is output even if the configuration file sets the log file's path to a different value. Instead, one should handle the errors which follow it. ( I was told that eliminating this misfeature will be problematic for nginx, given its architecture, so I wanted to document that. )
Sometimes the sound gets muted on Fedora Linux when using a desktop. What has been working for me to unmute it is to execute systemctl --user stop pulseaudio.service as your desktop (non-root) user in the command line. systemctl --user restart pulseaudio.service may be even more preferable but on my system pulseaudio starts again automatically after being stopped.
In order to add new contributors with commit or admin privileges to Fedora Linux packages, one should go to the package's page on src.fedoraproject.org and then to Settings → Users & Groups.
When testing web sites and web applications, it is advisable to test staging versions on a localhost web service (e.g: http://127.0.0.1:8000/) because caching may interfere with the version served to you on remote Internet hosts.

Happy Sukkoth everyone!

Licence

You can reuse this entry under the Creative Commons Attribution 4.0 Unported licence, or at your option any later version of it. See the instructions on how to comply with it.

Whatever code given here is provided under the MIT / Expat licence.

2020-08-14: Tech Tip: Making pip, twine, and other python tools faster on Fedora Linux

Recently I noticed that using pip, and twine upload and other Python tools stalled for many seconds trying to poll dbus (at least on Fedora).

I was able to fix it by issuing sudo dnf remove python3-keyring, but there are other ways to disable keyring as referenced on this keyring ticket, which may also work for you.

Licence

You can reuse this entry under the Creative Commons Attribution 4.0 Unported licence, or at your option any later version of it. See the instructions on how to comply with it.

Whatever code given here is provided under the MIT / Expat licence.

2020-04-25: Tech Tip: Search for file paths by pattern using DNF (on Fedora / CentOS / etc)

Using DNF one can search for file paths which are contained in uninstalled packages (similar to apt-file on Debian / Ubuntu / etc. or urpmf on Mageia) using dnf whatprovides '/usr/bin/start*'.

Licence

You can reuse this entry under the Creative Commons Attribution 4.0 Unported licence, or at your option any later version of it. See the instructions on how to comply with it.

Whatever code given here is provided under the MIT / Expat licence.

2020-04-25: Tech Tip: GFile type in python3 / gi / Gio / Gtk+

The GFile type in python can be retrieved using the following idiom:

import gi
from gi.repository import Gio

FILE_TYPE = Gio.File.__gtype__

# Use FILE_TYPE

Licence

You can reuse this entry under the Creative Commons Attribution 4.0 Unported licence, or at your option any later version of it. See the instructions on how to comply with it.

Whatever code given here is provided under the MIT / Expat licence.

2020-04-25: Tech Tip: Vim: Setting a different X.org/X11 Connection

If you are using Vim or neovim and wish to set a different X.org connection (e.g: for copy+paste), you can use :let $DISPLAY = 'myremotehost.tld:0.0'. Other environment variables can be changed or set in a similar way using the dollar-sign notation.

Licence

You can reuse this entry under the Creative Commons Attribution 4.0 Unported licence, or at your option any later version of it. See the instructions on how to comply with it.

Whatever code given here is provided under the MIT / Expat licence.

2020-04-06: Tech Tip: Releasing / Uncapturing the keyboard input focus when using a KVM / virt-manager VM

In order to release the keyboard input focus when using KVM (Kernel-based Virtual Machine) and virt-manager one can press "left control" and "left alt" together (not "left shift"). This was described in this post from StackExchange, where it shows how to set it to a different key combination.

This was another post emulating Lisha Sterling's old "know where the ledge is" blog (which seems to have disappeared).

Licence

You can reuse this entry under the Creative Commons Attribution 4.0 Unported licence, or at your option any later version of it. See the instructions on how to comply with it.

Whatever code given here is provided under the MIT / Expat licence.

2020-02-21: Tech Tip: Setting up remote X11 over TCP to Xephyr

Warning! The instructions given here may decrease your systems' security and may make it more vulnerable. Use at your own risk.

I recently wanted to set up a remote X.org / X11 connection to a Xephyr server, which is an X server that runs inside a window. I only found partial information in web searches, so I wanted to provide a top-to-bottom guide for accomplishing this task.

The Xephyr man page omits the info in Xnest's man page and that in the "Xserver" man page. Xephyr supports most of the flags in all of them.

To facilitate customisation, we will declare and use the following shell constants:

xephyr_port="1"
remote_xephyr_ip_host="192.168.1.100"

You may need to stop the firewalls on both client and server using something like:

sudo systemctl stop firewalld.service

Or the stop-iptables script.

Now start Xephyr on the local desktop:

Xephyr -listen tcp -listen unix -listen local -screen 1024x768 :"$xephyr_port"

Then connect to it and use xhost:

# On the X server
export DISPLAY=":$xephyr_port"
xterm &
xhost +

Now you should be able to connect to the Xephyr X server from the remote box:

# On the remote machine
export DISPLAY="$remote_xephyr_ip_host":"$xephyr_port"
xterm &
# Or a different window manager.
starticewm

Licence

You can reuse this entry under the Creative Commons Attribution 4.0 Unported licence, or at your option any later version of it. See the instructions on how to comply with it.

Whatever code given here is provided under the MIT / Expat licence.

2020-01-02: Tech Tip: Implementing assert() in PARI/GP

One can implement assert() in PARI-GP, a GPLed computer algebra system, using the error() function:

assert(cond,err)=if(!cond,error(err))
myfunc()=4
assert(myfunc()==5,"myfunc should return 5")

Licence

You can reuse this entry under the Creative Commons Attribution 4.0 Unported licence, or at your option any later version of it. See the instructions on how to comply with it.

Whatever code given here is provided under the MIT / Expat licence.

2019-05-20: Tech Tip: How to set VLC's default audio output to Mono

In order to permanently set the audio output of the VLC media player to "Mono" by default, edit ~/.config/vlc/vlcrc (or the equivalent) and set the [core] → stereo-mode line to stereo-mode=7. Why 7? Because! Thanks to #videolan on Freenode for the tip.

Note that the “Audio” menu setting does not persist by design.

Cheers!

Licence

You can reuse this entry under the Creative Commons Attribution 4.0 Unported licence, or at your option any later version of it. See the instructions on how to comply with it.

2019-02-13: Tech Tip: Some TypeScript Tips

If you get an error TS2307: Cannot find module 'web-fc-solve'. error , try prefixing a "./" to the import name.
See "typescript - How do you produce a .d.ts "typings" definition file from an existing JavaScript library? - Stack Overflow" question and especially the tip about using dts-gen.
Don't compile more than one module into an amd.
Convert your JS dependenciess to TypeScript bottom up.
- tslint
- syntastic as a vim interface and similar.
- Code::TidyAll::Plugin::TSLint - by me - provides caching and stuff.
For prettyprinting/formatting/indenting, use prettier.
- tslint-config-radiosilence: LML tslint config
- My tslint.json based on it

Licence

You can reuse this entry under the Creative Commons Attribution 4.0 Unported licence, or at your option any later version of it. See the instructions on how to comply with it.

2018-06-26: Tech Tip: Opening a new kate buffer in a new window

Kate is a text editor that I sometimes prefer to use. Normally, saying kate [optional file path here] from the command line opens a new buffer as a new tab in an existing window. However, one can instruct it to open in a new window using the -n command line flag.

Cheers!

Licence

You can reuse this entry under the Creative Commons Attribution 4.0 Unported licence, or at your option any later version of it. See the instructions on how to comply with it.

2018-06-21: Tech Tip: Nested Tmux Session

One can use the following command-line to start a new tmux session with a different hotkey than the default one (e.g.: for use as a nested session after su/sudo or in a nested ssh session, so both sessions can be controlled):

tmux new-session \; set-option prefix C-a

Replace "C-a" with the hotkey of your choice.

Licence

You can reuse this entry under the Creative Commons Attribution 3.0 Unported licence, or at your option any later version of it. See the instructions on how to comply with it.

2018-02-17: Tech Tip: Several Tips

Starting libreoffice under X Window

Sometimes it will behave erratically (not displaying a menubar/etc.) until you do unset DBUS_SESSION_BUS_ADDRESS on the command line which also often helps with other apps.

htop -C

This starts htop with no colours in case they are hard to read.

git stash

I recommend reading about git stash - it is useful.

VirtualBox Release Keyboard

When using VirtualBox one can use the right Ctrl key to release the keyboard and allow one to enter window manager key combinations such as switching to a different host workspace.

Testing executables using python

You can find a working example for testing executables using pytest / subprocess.py / python in this repository. Thanks in part to this post by Ynon.

Licence

You can reuse this entry under the Creative Commons Attribution 3.0 Unported licence, or at your option any later version of it. See the instructions on how to comply with it.

2017-06-04: Cython: Report and Tips

Cython allows one to compile code written using a superset of the Python programming language into C code for improved performance. I recently tried it in an attempt to speed up my solution to a Project Euler problem, which was ran slowly, even by pypy. After some amount of work, the cython-based solution outperformed the pure-Python code, without me having to reimplement it all in C or C++ or similar. So you too should consider using it, if you need the extra performance.

The rest of this post contains some Cython tips for using it well that I discovered in my (so far relatively limited) experimentation with it.

I noticed that cdef int i; for in xrange(0, mymax) does not get optimised while cdef int i; for i in range(mymax); does.
cdef int arr[100][100] declares a local array of 100*100 elements , and keeps copying it, which is likely not what you want. On the other hand, cdef int [100] * ptr generates a pointer.
“cdef” cannot be used inside loops.
It is a good idea to declare the C types as late as possible - only right before the time-critical code when you absolutely need to use them. One can initialize them from dynamically-typed python data structures.
One can copy+paste the compiler's invocation command on the command line and edit it to apply better optimisation flags. There may be a way to customise the compilation command by editing the “setup.py” file, but I didn't bother to figure it out yet.
One can and should inspect the generated .c file to see the resultant C code to see if anything is done sub-optimally.

Licence

You can reuse this entry under the Creative Commons Attribution 3.0 Unported licence, or at your option any later version of it. See the instructions on how to comply with it.

2017-02-22: Tech Tip: Saving hard disk space on Amazon AWS Linux Instances

By default, AWS' EC2 machines are given a hard disk of only 8 GB of space, and each extra gigabyte costs extra, even if you rent a machine with much more RAM than that. Some options you can try if you don't want to enlarge the hard disk are:

Remove unnecessary system packages (using the “apt remove” command on Ubuntu/Debian VMs or equivalents. From my experience, the default Ubuntu VMs ship with a lot of junk preinstalled.
Remove packages that you installed and no longer need (e.g: gcc).
One can also try to do “rm -fr /usr/share/man /usr/share/doc” and other unnecessary directories that are cheaper to use on local VMs or host systems.
Consider using S3 or similar instead of the local hard disk, which should be much cheaper.

Good luck!

Licence

You can reuse this entry under the Creative Commons Attribution 3.0 Unported licence, or at your option any later version of it. See the instructions on how to comply with it.

2016-12-15: Tech Tip: How to Make Pidgin’s Incoming Messages Icon Animated

Pidgin is an open source chat client. If you want to make its incoming messages system tray icon animated, simply right click on the icon, and select “Blink on new messages”. This way it will be harder to miss and that is what I was used to having.

Licence

You can reuse this entry under the Creative Commons Attribution 3.0 Unported licence, or at your option any later version of it. See the instructions on how to comply with it.

2016-07-06: Tech Tip: Nullifying/Resetting/Overriding a Remote Git Repository's Branch History

If you have a branch in a remote repository of the Git version control system which you need to reset/nullify/override its history, you can do it by setting up a brand new branch with the same name (say starting in an empty repository) and then pushing it from there to the remote while specifying the --force flag. Thanks to the people from #git on Freenode for enlightening me.

Licence

You can reuse this entry under the Creative Commons Attribution 3.0 Unported licence, or at your option any later version of it. See the instructions on how to comply with it.

2016-07-06: Tech Tip: Disabling the Touchpad/Trackpad on Linux

If your touchpad/trackpad (= the built-in mouse control found in some laptops) is a Synaptic one, then you can disable it on Linux systems by issuing the command synclient TouchpadOff=1 (you may need to install synclient first. I originally learned of it from this link on kde-apps.org.

Licence

You can reuse this entry under the Creative Commons Attribution 3.0 Unported licence, or at your option any later version of it. See the instructions on how to comply with it.

2016-05-27: Tech Tip: Checking the Latest Build in Travis CI

If you are using Travis CI, you can check the latest build after a failed build by going to the “Build History” tab in your project’s main page (= https://travis-ci.org/shlomif/fc-solve or equivalent), and selecting the new commit.

Licence

You can reuse this entry under the Creative Commons Attribution 3.0 Unported licence, or at your option any later version of it. See the instructions on how to comply with it.

2016-03-10: Tech Tip: How to properly sync the clock using NTP on Linux

In order to sync the clock using NTP, the network time protocol so your computer clock won't be skewed, one should run the following command as root: “ntpd -q -g -x”. This has been a constant problem for me in part because SSL/HTTPS sites may not work if your clock is out-of-date.

You can find the meaning of the individual flags in the ntpd man page.

Cheers and happy upcoming Purim in case you celebrate.

Licence

You can reuse this entry under the Creative Commons Attribution 3.0 Unported licence, or at your option any later version of it. See the instructions on how to comply with it.

2015-12-03: Tech Tip: Speed up building perl 5 by not installing the man pages

When building perl 5 from source, one can speed up the "make" and "make install" stages significantly by not installing the man pages. To do so, pass the options “-Dman1dir=none -Dman3dir=none” to “sh Configure”.

Cheers, and happy upcoming Chanukkah, and happy Christmas and/or Yulth or whatever winter holiday you celebrate.

Call for action

Creative Commons has set up a campaign to stop TPP - the Trans-Pacific Partnership. Please consider signing it and spreading the word.

Licence

You can reuse this entry under the Creative Commons Attribution 3.0 Unported licence, or at your option any later version of it. See the instructions on how to comply with it.

2015-11-06: Tech Tip: Make Panes Equally Heighted in Tmux

When pressing «Ctrl+B;"» in tmux to split a pane vertically there, one will notice that tmux splits the existing pane into two which can result in smaller and smaller panes. In order to fix this, one can type «Ctrl+B;M-2» (where “M” is the Meta key and is normally “Alt” on most keyboards.) which will make all the panes equally heighted.

I found this feature after wondering whether it existed, and then searching for it in the tmux man page.

Licence

You can reuse this entry under the Creative Commons Attribution 3.0 Unported licence, or at your option any later version of it. See the instructions on how to comply with it.

2015-09-15: Tech Tip: Locally Host the Firefox’s Personas Favourites File

A constant annoyance when trying to use the Firefox extension “Personas Rotator” was that downloading the favourites file from the Firefox addons site failed, which made it stop rotating. I recently found a solution for that which I'd like to share.

First of all, login to the Mozilla addons site and access and bookmark the JSON data file’s URL. Save it to a file and host it using a local HTTP service (such as Apache, the one line Python HTTP service command line, or sthttpd’s “thttpd -p $PORT” invocation).

Then, go to Firefox’s “about:config”, access the “extensions.personas.favorites-feed.url” key and change it to the locally served one. It should work fine after restarting Firefox, and one upside to it would be that you no longer need to be logged in to addons.mozilla.org for it to work.

Remember to save the file again every once-in-a-while, after you have added new favourites. A final note is that a “file://” URL may work fine as well, but I have not tried it yet.

Happy rotating, and Shanah Tovah if appropriate!

Licence

You can reuse this entry under the Creative Commons Attribution 3.0 Unported licence, or at your option any later version of it. See the instructions on how to comply with it.

2015-09-01: Tech Tip: Don’t pass the -j$N flag to make when building GNU guile2

In order to build GNU Guile (or at least its version 2.x), it is important not to pass the “-j$N” flag to “make”. Without the flag, the compilation will be faster (counterintuitively I admit), because some of the build tasks spawn as many processes or threads as the CPU cores on the machine, and with the flag, they get congested and the build is slower. Cheers!

Licence

You can reuse this entry under the Creative Commons Attribution 3.0 Unported licence, or at your option any later version of it. See the instructions of how to comply with it.

2015-07-20: Tech Tip: Trimming Old Releases from minicpan Directories

minicpan is a tool to create or update a minimal local mirror of CPAN (= the “Comprehensive Perl Archive Network”), that contains only the latest releases of CPAN distributions. Now, one problem I ran into when using it is that “minicpan” did not seem to delete the older downloaded releases after they were superseded by newer ones. A few days ago I found a solution to this problem.

What you can do is:

Move away the minicpan’s mirror’s top directory (do not delete it!)
Set up a localhost HTTP service on a high TCP port which will serve the moved minicpan directory. For that you can use the one line Python HTTP service command line or sthttpd’s “thttpd -p $PORT” invocation or whatever.
Run the “minicpan” command while pointing the remote to the localhost mirror of the old minicpan mirror, using the “-r” flag or by temporarily editing the “~/.minicpanrc” file.

Voila! Now the mirror will only contain the latest releases, and you can remove the old, moved-away, minicpan directory, and cancel the HTTP service.

Licence

You can reuse this entry under the Creative Commons Attribution 3.0 Unported licence, or at your option any later version of it. See the instructions of how to comply with it.

2015-07-14: Tech Tip: Skip Tests in RPM .spec Files conditionally-to-version

Let’s suppose you are packaging a software application using RPM or a similar packaging system, and this application contains some automated tests that can be run using “make test”, “make check” or similar. You may opt to put them inside a “%check” section, but the problem now is that they may fail.

The solution I used in this case (assuming fixing the tests was out of scope of my work as a packager) was to comment out the command that ran the tests, but this is not ideal because that way the tests will also be disabled in subsequent versions of the application, unless the packager remembers to reenable them.

Recently, I discovered a better way to do that, after inspecting the source of a Mageia Linux package. One needs do something like that:

%define known_to_fail_version 1.82

%check

if test "%{version}" = "%{known_to_fail_version}" ; then
    true
else
    # Run the tests.
    make check
fi

This way, the tests will get run again for subsequent versions of the packaged application, and if they fail again one can set “known_to_fail_version” to a new value.

I hope you also find it a cool trick. Cheers!

Licence

You can reuse this entry under the Creative Commons Attribution 3.0 Unported licence, or at your option any later version of it. See the instructions of how to comply with it.

2015-07-03: Tech Tip: How to Debug Firefox’s UI Code Using the Debugging Tools

In order to debug Mozilla Firefox’s User Interface code (so-called “chrome” code), you should toggle the “Enable chrome and add-on debugging” checkbox in the Developer Toolbar’s setting and then go to “Tools → Web Developer → Browser Toolbox”. Thanks to jdm on irc.mozilla.org for part of this tip. Cheers and happy United Stated Independence Day (4th of July) in case you are an American.

Licence

You can reuse this entry under the Creative Commons Attribution 3.0 Unported licence, or at your option any later version of it. See the instructions of how to comply with it.

2015-06-25: Tech Tip: Fix Poor Internet Connectivity by Restarting the Router

I noticed that one can often fix poor home Internet connectivity (I have an ADSL connection but it may apply to other types of Internet connection), by powering off the router and starting it again (“restart/reboot”). So I’ve made a mental note to keep it in mind and now it’s in this blog as a tip. Cheers!

Licence

You can reuse this entry under the Creative Commons Attribution 3.0 Unported licence, or at your option any later version of it. See the instructions of how to comply with it.

2015-06-06: git Tip: How to Call “git bisect start”

“git” is an open-source distributed version control system that has recently gained a lot of popularity. The “git bisect start” synopsis in “git help bisect” is somewhat misleading:

git bisect start [--no-checkout] [<bad> [<good>...]] [--] [<paths>...]

One has to specify both the “bad” and the “good” revisions explicitly or else “git bisect” will wait for the missing ones to be specified, which will be confusing. So specify them immediately at the “git bisect start” invocation.

Incidentally, perl-5.22.0 was released a few days ago. Grab it while it’s hot!

Licence

You can reuse this entry under the Creative Commons Attribution 3.0 Unported licence, or at your option any later version of it. See the instructions of how to comply with it.

2015-06-02: Linux Tip: How to Determine Which Program is Emitting Sounds

Here's a tip you may find useful about how to determine which program is emitting sounds on a modern Linux system. This tip requires PulseAudio to be enabled , and if that's the case, just start up KDE’s KMix and see which applications spring up in the “Playback Streams” tab. Hope it helps, and good luck.

Licence

You can reuse this entry under the Creative Commons Attribution 3.0 Unported licence, or at your option any later version of it. See the instructions of how to comply with it.

2015-05-29: Vim Tip: Going to the top or bottommost window/viewport

In the Vim and GVim text editors, one can use Ctrl+W;t (or Ctrl+w;Ctrl+t) to go to the topmost/most top window (= a section of the screen in Vim terminology) and Ctrl+W;b to go to the bottommost/most bottom one.

Cheers!

2015-02-28: Tech Tip: How to Configure Qt 5 Behaviour When Running on KDE4

Recently, I noticed that when running the VLC-2.2.0 prerelease, which is based on Qt 5 for its GUI, on my Mageia Linux 5 system on top of KDE 4, then in the playlist a single-click immediately played a file instead of selecting it, while reserving a double click for activation. After a long amount of research and thought, I figured out a way to configure Qt 5 on top of KDE.

To do so:

Install lxqt-config and the “lxqt-qtplugin”.
Add the line “export QT_QPA_PLATFORMTHEME=lxqt” somewhere before the desktop startup in your “.Xclients” or “.xinitrc” file (or in your “.bashrc”).
Restart the X/KDE environment.
Run “lxqt-config” to configure the appropriate behaviour.

This way one can use the Qt5 customisations of lxqt in KDE 4. Enjoy!

Licence

You can reuse this entry under the Creative Commons Attribution 3.0 Unported licence, or at your option any later version. See the instructions of how to comply with it.

2015-02-22: Optimisation Tip: Avoid Excessive Forks+EXECVEs to Processes

This will come as a surprise to few, but performing a separate fork() and EXECVE system calls (e.g: for `factor "$n"` in Perl or popen in C), on each iteration of a loop can really kill performance. One option to reduce that would be to read individual lines from a seq 2 999999 | xargs factor loop, or avoid forking altogether.

I discovered this issue while working on Project Euler problem #141, and I should have realised that the sub-100% CPU utilisation of the program was caused by the excessive spawning of new processes.

Licence

You can reuse this entry under the Creative Commons Attribution 3.0 Unported licence, or at your option any later version. See the instructions of how to comply with it.

2015-02-05: Tech Tip: Make Jamendo Playback Work in Firefox on Mageia Linux

If you’re having a problem playing Jamendo tracks after pressing the “Listen” button (such as on this page) on Firefox running on Linux, then try to install the packages «gstreamer0.10-plugins-bad gstreamer0.10-plugins-good gstreamer0.10-plugins-ugly gstreamer0.10-mpeg gstreamer0.10-ffmpeg» (relevant to Mageia; try their equivalent in other distributions), restart Firefox and try again. The problem is that Firefox needs extra gstreamer plugins to play proprietary formats in HTML audio and video elements. Cheers!

Licence

You can reuse this entry under the Creative Commons Attribution 3.0 Unported licence, or at your option any later version. See the instructions of how to comply with it.

2014-03-08: Tech Tip: add a notification (or any other command) after a command-line task has ended

I discovered that one can add a notification after a command you've already started running has ended on a UNIX system by suspending the command using “Ctrl + Z” and then typing something like “fg ; n --msg 'Command finished'” (where “n” is the command I use for notifications). “fg” brings a job to the foreground, and for more information about it see its wikipedia entry and the Unix Background Jobs entry on thegeekstuff.com.

Licence

You can reuse this entry under the Creative Commons Attribution 3.0 Unported licence, or at your option any later version. See the instructions of how to comply with it.

2014-03-08: Tech Tip: Resetting the GLib/Gtk+ Default Apps on Linux (e.g: the Default Browser)

In this tip, I would like to describe how to reset the default apps, and especially the default browser, for GLib and GTK+. These are used by desktops like GNOME and XFCE, and by GTK+ applications such as HexChat or claws-mail. My problem originally was that I accidentally pressed the "Yes" button when Firefox Nightly (installed under a prefix under /opt) asked me whether I should set it as the default browser, which caused HexChat and Claws-Mail to open links using it by default.

Anyway, to resolve this problem do this:

Go to ~/.local/share/applications.
Backup this directory somewhere safe.
The mimeapps.list file contains the reference to use the offending apps under the *.desktop files there which you can search for the reference to the apps.
Remove these line or their portions from mimeapps.list.

Now you may need to restart the desktop apps or the desktop environment (not sure).

Hope it helps.

Here’s how I found it: after some false leads of either trying to recursively search my home directory for occurrences of /opt/firefox and reading the source code of HexChat, GLib and gtk+ to see where this happens (there were too many levels of indirection there), I ended up doing “strace -f -o hexchat.strace hexchat” and inspecting the strace file for hints.

Licence

You can reuse this entry under the Creative Commons Attribution 3.0 Unported licence, or at your option any later version. See the instructions of how to comply with it.

2014-01-02: Tech Tip: Empty <div> tag in XHTML served as text/html

If you serve XHTML (= HTML written using XML grammar and conventions) as Content-Type: text/html, you may be tempted to write an empty <div> tag as a standalone XML tag with a trailing slash using <div id="my_anchor" />. Don’t do that, because with text/html (a.k.a “tag-soup”) the browser may think you just have an opening tag and will look for a closing tag elsewhere. Instead, write a pair of opening and closing tags, e.g: <div id="my_anchor"></div>.

Using the first form confused both Firefox (24.2.x in my case) and Google Chromium (whatever shipped with Mageia at the time), and made them misrender my page, despite the fact that it validated as valid XHTML. Perhaps I should have considered putting the id=".." inside a meaningful sub-section of the document, but I implemented something for skipping a section navigation menu.

2014-01-02: Tech Tip: Mitigating “git clone”’s inability to be resumed using rsync

Happy new civil year, everyone. As you may know Git is a distributed version control system, but its often time-consuming (if the repository’s history is large) “git clone” operation cannot be resumed, which is a problem with bad Internet connections. There was a service that did “git clone” and then allowed people to download using HTTPS Called “Git bundler” but it has been down for sometime now. However, I found a different solution to the problem.

What can be done is use ssh to log in to a remote host, where the “git clone” is performed (preferably, but not absolutely necessarily, when running on top of a session of tmux, GNU Screen or similar). After that, you can use rsync over ssh to download the .git directory to the local workstation (I like to use the invocation rsync -a --progress -v --inplace for that).

Following that all you have to do is run git clone to a different directory to the one where you put the .git and set “git remote” appropriately.

Hope that helps, and the same can be done with other distributed version control systems such as Mercurial, or Bzr.

2013-12-31: Tech Tip: Combining/Merging Bookmarks From a Different Firefox Profile

Happy New Civil Year! Let’s suppose you collected some bookmarks in a different Firefox profile (possibly on a different computer or system, or in a different profile), and wish to place them somewhere under your current or main profile’s bookmark tree - how can it be done? I had a need for that and in this entry, I describe a solution that worked for me.

First of all export your bookmarks from the source profile to a JSON file (“Bookmarks → Show All Bookmarks → Import and Backup → Backup…”)
Create a new Firefox profile in the machine of the target profile. (see this page for more information about Firefox profiles.).
Import the bookmarks from the JSON file to it (while overriding the default bookmarks).
Copy all the folders you want to merge to the system clipboard (Ctrl+C or equivalent).
Paste them into the default profile’s bookmark manager.

Then all you need to do is sort them. Cheers!

2012-12-10: Tech Tip: Disabling Audio or Video in mplayer and VLC

In order to completely disable video in VLC, from the command line, use the --novideo command line flag - this way only the Audio will be heard. To do the same in mplayer, use the -novideo flag. You can do the same for audio using --noaudio and -noaudio respectively.

A different propsoed solution is using -vo none in mplayer or --vo none in VLC, but this causes the video to still be decoded (just not presented) - at least in VLC.

Cheers! You can reuse this entry under the Creative Commons Attribution 3.0 Unported licence (see how to comply with it).

2012-12-10: Tech Tip: Overriding the Audio Track of a Video using ffmpeg

In this tip I will cover how to use ffmpeg to override the audio track of a video from a different audio track (such as the one in a WAV, an OGG or an MP3 file). To do that use the following recipe (based on this out-of-date blog post and some help from ubitux on #ffmpeg, with a lot of trial and error):

ffmpeg -i in_audio.wav -i in_video.ogv -map 0:0 -map 1:1 -shortest \
    -c:a libvorbis -q:a 7 -c:v copy output.ogv

Some of these files are not needed if you're not dealing with Ogg Video files. -shortest is useful for making sure that the length of the output is trimmed to that of the shortest input.

I ended up using this command for the latest screencast I prepared. Cheers, and happy holidays.

2012-08-18: Bash Function for Caching Results

In this blog I’d like to blog about the newest edition to my Bash (the UNIX shell) aliases file: the cache() function. What it does is cache the result of a lengthy command in a file in a common place, and then simply output this file if it already exists. I found it of use when I compiled a report which involved some lengthy recursive grep operations which were time consuming, and required a lot of waiting. Another upside to caching is the fact that one can monitor its progress by following the file on disk.

Here is the cache function, available under the MIT/X11 licence:

# What this function does is cache the result of a command in a file, and
# use the file to output the results in case it exists.
# Format is: cache "$basename_to_cache_in" $cmd $arg1 $arg2 $arg3...
cache()
{
    local cache_fn="$1"
    shift

    local dir="${CACHE_DIR:-.}"

    if ! test -d "$dir"; then
        mkdir -p "$dir"
    fi

    local fn="$dir/$cache_fn"
    if ! test -f "$fn" ; then
        "$@" > "$fn"
    fi
    cat "$fn"
}

Hope you find it useful.

(Note: part of the reason why I'm writing this post is to see if it shifts away the spam comments from my previous post, which attracted a lot of spam in the past days. It's an experiment to see how spam behaves.).

2012-08-18: Tech Tip: debugging Ruby’s rspec scripts

This post will cover how to debug test scripts (and their equivlent production code) of the Ruby programming language’s RSpec framework using an interactive debugger, which was a problem I've ran into lately when working on some Ruby code. Most of the other posts I found about the topic were either incomplete or out-of-date, but I got some help from #ruby-lang on Freenode, so I was able to figure it out.

What you should do is:

install the ruby-debug gem: gem install ruby-debug.
Add require 'ruby-debug' to the top of the test script.
Add a debugger statement to lines where you want to place a breakpoint. This is important because otherwise the debugger will never stop. (I am unhappy with it.)
Use the --debug flag in rspec: rspec --debug t/parse-board.rb.

After doing all that it worked for me, so now I'm happy.

Copyright and License

The text is copyrighted by Shlomi Fish, and is made available under the Creative Commons Attribution License 3.0 Unported (or at your option, any later version). In order to attribute the text to me (= Shlomi Fish), please link to this page and to my homepage.

2012-06-27: Tech Tip: How to Build Firefox with Debugging Symbols on Linux

I spent a large part of today building Firefox on one of my Linux machines and trying to figure out why gdb did not display debugging symbols, and instead only displayed question marks in the backtrace (?? ). Eeventually, I found a solution, so I'd like to document the process for other people in the future.

The solution is documented in this bug report that I filed and involves the following steps:

Checkout the firefox source from the mozilla-central repository

Put something like the following in the .mozconfig file in the source's root (where client.mk can be found):

. $topsrcdir/browser/config/mozconfig
export LDFLAGS="-Wl,--no-keep-memory"
mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/ff-dbg
ac_add_options --disable-optimize
ac_add_options --enable-debug
ac_add_options --enable-tests
ac_add_options --enable-debug-symbols
ac_add_options --prefix="$HOME/opt/firefox-from-hg"

The important lines are the --enable-debug, --enable-debug-symbols and the LDFLAGS one.

Type make -f client.mk. Now you'll need to wait.
Type make -f client.mk install PKG_SKIP_STRIP=1 and make sure you do not forget the PKG_SKIP_STRIP=1 parameter.
Now you can use gdb ~/opt/firefox-from-hg/bin/firefox to debug Firefox with all the debugging symbols.

Enjoy!

2012-06-27: How to Listen to Jamendo Radio and Tracks from the Browser

In case you do not know, Jamendo is a show case for music licensed under Creative Commons licences, with a lot of great tracks to listen to. However, after the recent upgrade, I found that I was no longer able to listen to their radio stations or press the listen button of their tracks using Firefox, my browser of choice.

Here is what I discovered on how to re-enable it after some investigation: first of all, you need to enable and/or whitelist JavaScript for jamendo.com. If you are using NoScript, you can use the “S” button on the Add-on bar for that.

Furthermore, the new Jamendo requires Adobe Flash to be whitelisted as well. If you are using FlashBlock, then go to “Tools → Add-Ons → Extensions → Flashblock → Preferences → Whitelist” and add http://www.jamendo.com/ there. I don’t know how to properly enable the listening on browsers and platforms which do not have Adobe Flash installed, so blame Jamendo.com - not me. (And, yes, I know that Flash is evil.)

There may be some other obstacles on the way, but now everything is working fine for me. I am going to complain to Jamendo that in this HTML 5 day and age, one needs to use Flash to listen to their streams, which, as I recall, was not necessary in the old Jamendo.

Happy listening.

2012-05-27: Tech Tip: GNU tar’s “-a” Flag

On this post to the Mageia development mailing list by Thierry Vignaud, I discovered that GNU tar (at least in recent versions) has an “-a” flag which is useful in conjunction with its “-c” (create new archive) mode. This is because it detects the suitable compression based on the extension and uses the appropriate flag.

So: “tar -cavf myarchive.tar.gz ./mydir/” is equivalent to “tar -czvf myarchive.tar.gz ./mydir/”, “tar -cavf myarchive.tar.bz2 ./mydir/” does the same thing as “tar -cjvf myarchive.tar.bz2 ./mydir/” and so forth. When unpacking archives, you can omit the “-a/-z/-j/-J” flags, because GNU tar will detect the compression of the archive based on the file magic of the compressed formats.

Another useful (and open-source) tool for manipulating tarballs and other archives is patool, but I've been meaning to suggest they do a short-circuiting when converting tarballs from .tar.gz to .tar.xz to .tar.bz2 / etc.

Anyway, enjoy.

2012-03-09: Tech Tip: Getting hgsubversion to Work with subversion-1.7.x

Apache Subversion version 1.7.x broke some backwards compatible and some stuff got broken, including hgsubversion, which provides a way to use subversion remotes with Mercurial.

In order to fix it, just install subvertpy, which provides alternative python bindings for subversion, which hgsubversion prefers by default, and with which it works fine under subversion-1.7.x.

I discovered all that after I attempted to fix it the hard way by installing subversion-1.6.x under /opt/svn-1.6.x, which required building an old version of SWIG, and then setting PATH, LD_LIBRARY_PATH, and PYTHONPATH and then having to remove the global python-svn bindings from under /usr/lib (as root!) because hgsubversion did not like them, a process which took me an entire evening and was frustrating and was ridden with a lot of trial and error, so I would not recommend it.

Hope it helps. Also see this hgsubversion bug report.

2012-02-18: Tech Tips: Vim, Facebook and Firefox’s Personas Rotator Extension

Here are three tech tips, that I felt did not necessitate their own entry.

In the Vim text editor(homepage), one can use the keybinding Ctrl+W ; O to close all windows except the current one, which will be the only one left open (hence the "O").

In Facebook (Wikipedia page) one can use Shift+Enter to start newlines in the messages you compose in the site’s composer. Press it twice to start a new paragraph.

If you are using Firefox’s Personas Rotator Extension ( Addons.mozilla.org page), then you can switch the currently shown persona ( in case you don't want it at the moment) using Ctrl+Alt+P or an alternate configurable keybinding. I discovered this latest tip by accident and can no longer imagine my life without it.

Cheers!

2012-02-18: Vim Tips: scp URLs, "set tabpagemax" and fixing C indentation

Here are some Vim tips I ran into recently. First of all, when opening scp:// URLs, one should use two slashes after the hostname instead of 1, like scp://hostname//home/myuser/foo.txt instead of scp://hostname/home/myuser/foo.txt. I don’t know why that is the case, but it does not work properly without it. It also seems that netrw is buggy as it displays an irritating grey line on the cursor, the syntax highlighting tends to be off and saving a file displays several lines at the bottom.

Another tip is that gvim limits the number of tabs it opens when doing gvim -p [file1] [file2] [file3]. As a result, it is possible that not all files will be opened. If you want to change it you can set set tabpagemax in your .vimrc.

Finally, I noticed that Vim c-indentation tends to indent parameters to functions on subsequent lines using 8 spaces instead of 4 by default. I was able to change it to 4, which is my preference by adding set cinoptions+='(0,W4' to my .vimrc. There is plenty of other nifty stuff available in the cinoptions parameter.

Enjoy!

2012-01-22: git tip: adding remotes to .git/config

When working with the git version control system and editing .git/config to add a new remote, some people may be tempted to copy and change the origin remote that reads something like:

[remote "origin"]
	fetch = +refs/heads/*:refs/remotes/origin/*
	url = git@github.com:shlomif/perl.git

However, note that origin also appears at the fetch = and needs to be changed there as well, or else all the branches will be placed in remotes/origin. Maybe there's a better way to add a new remote using the git config commands.

Other than that, I should note that there doesn't seem to be a consensus among git users whether git pull --rebase is better than a simple git pull: the perl people told me to use --rebase and the Amarok people and someone on Freenode's ##programming told me not to use it. Now I'm just confused.

2011-12-13: Tech Tip: Removing Bash’s Command Completions

The normal way to remove a completion for a Bash command (say “mv”) is to do “complete -r mv”. However, with the bash-completion package installed on Mageia Linux Cauldron, this is not enough because it also adds a default completion for every invoked command. So in order to override this behaviour, type “complete -r -D” and then you can remove the commands’ completions permanently, using “complete -r mv” or whatever.

2011-09-05: Recommendation and Tip: The QUnit JavaScript Automated Tests Library

I've begun working on a JavaScript port of some algorithmic perl code, to allow it to run inside a browser. As a result, I had to find a good JavaScript automated tests library to allow me to write automated tests for the code. I first looked at Test.More and Test.Harness from JSAN (the JavaScript Archive Network) but as it turned out, JSAN was defunct, and no one had time to mark it as such. Then I asked the “Test.Run” developer (another thing I had found on jsan) to instruct me how to get it up and running, and he gave me a link to its archive, which was 1.5 mb compressed (!), and so was not acceptable.

I looked at the Wikipedia list of unit testing frameworks, but there were too many. at least I understood that jsUnit was no longer actively maintained. I looked at Jasmine, which is a bdd framework for JS, but its syntax seemed too horrid and unnatural.

Eventually, I decided to ask for recommendation on Stack Overflow and, as after I wrote my title, I found a a previous question, where there was a recommendation of QUnit, which I noticed was developed by the jQuery people, and as I'm fond of jQuery, I decided to look deeper into it.

I wasn't disappointed by QUnit - it does what it does well, and I was able to write my test suite using it, so I can recommend it as well. It has primitives that are very similar to perl's Test::More, and it can even assert that the number of assertions ran within a test are right (like Test::More can).

So here's a tip for it: if you're writing your testing code in a different file, and you wish to check that no compile-time or run-time exception was thrown (which will cause qunit to report a success with zero assertions) you should wrap the testing code in a try { .. } catch block and in the catch block, run ok(false...).

Here's an example from my code:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
                    "http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<title>ABCPath Test</title>
<script src="jquery-latest.js"></script>
<link rel="stylesheet" href="qunit.css" type="text/css" media="screen" />
<script type="text/javascript" src="qunit.js"></script>
<script type="text/javascript" src="joose.mini.js"></script>
<script type="text/javascript" src="abc-path.js"></script>
<script type="text/javascript" src="abc-path-test.js"></script>

  <script type="text/javascript">
      $(document).ready(function () {
          try {
              test_abc_path();
          }
          catch (err) {
            module("Meta");
            test("global_failure", function() {
                ok (false, 'test_abc_path() has thrown an exception or is invalid.');
            });
          }
          // Hide passed tests by default.
          $('#qunit-tests').addClass('hidepass');
      });
  </script>

</head>
<body>
  <h1 id="qunit-header">QUnit example</h1>
 <h2 id="qunit-banner"></h2>
 <div id="qunit-testrunner-toolbar"></div>
 <h2 id="qunit-userAgent"></h2>
 <ol id="qunit-tests"></ol>
 <div id="qunit-fixture">test markup, will be hidden</div>
</body>
</html>

Enjoy.

2011-07-22: Tech tip: killing all processes of a certain UNIX user.

In order to send a signal (such as kill) to all process of a certain UNIX user, one can do:

pkill -U "$username"

To kill them forcibly using the 9 signal ("SIGKILL"), you can do:

pkill -9 -U "$username"

To just list their process IDs, you can do:

pgrep -U "$username"

Personal Log

svn.berlios.de has not been available in the past day , which really drives me mad, so I decided to:

Do stuff that don't require it, like blog or work on non-svn.berlios.de code.
Transition most of my code away from svn.berlios.de to bitbucket.org and Mercurial once it is up.

Cheers.

2011-06-16: More Vim Tips

Here are some more Vim tips I collected recently:

One can repeat the last t, f, T or F command which moves to the next specified character in a line, or right before that, using ;, and repeat in the opposite direction using ,. I discovered those after starting to read through the vim usr_* manuals.
One can get the Unicode value of the character under the cursor using the :ascii command-line command or the ga normal mode command.
If you want to find how many lines/words/etc. are in a visual block, you can select it using "v" and friends and then type g-Ctrl+G.
Finally, you can pipe the contents of the buffer or a range to an external program and view its output temporarily. So you can say :w !grep 'FOO' | wc -l to find the number of lines containing 'FOO', or :'a,'bw !grep 'FOO' | wc -l within a range.

I should note that one thing that annoyed me lately, is the fact that after I have a visual selection (v or V), I sometimes press u instead of y (because they are so nearby) and so cause the selection to become lowercase instead of yanking it.

Happy Vimming!

2011-06-16: Tech Tip: KDE 4: Changing the Names of the Virtual Desktops from the Command Line

I've written a small command-line utility to change the names of the KDE 4 virtual desktops (also known as "virtual workspaces").

To use it, do something like this:

#!/bin/bash
./set_kde4_virtual_desktop_names \
    "Main" "Devel" "Chat" "Donkey" \
    "Music" "Config" "Workspace 7" "Games"

I've originally thought I could do it by setting the value of _NET_DESKTOP_NAMES in xprop, but that turned out to be futile. Then I decided that imitation was the sincerest form of flattery and ripped the relevant code out of kdebase-workspace, and after some modifications, it worked. I'd like to thank pinotree from #kde-devel for some assistance.

Here is the code, but the repository also contains a CMakeLists.txt file:

/*
 * A program to set the names of KDE 4' virtual desktops (a.k.a virtual
 * workspaces).
 *
 * Adapted from kdebase-workspace-4.6.4 's
 * kwin/kcmkwin/kwindesktop/main.cpp , which is:
********************************************************************
KWin - the KDE window manager
This file is part of the KDE project.

Copyright (C) 2009 Martin Gräßlin <kde@martin-graesslin.com>

This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program.  If not, see <http://www.gnu.org/licenses/>.
*********************************************************************
 *
 * Skeleton of KApplication taken from:
 * http://techbase.kde.org/Development/Tutorials/First_program
 *
 * Written by Shlomi Fish ( http://www.shlomifish.org/ ) while disclaiming
 * all rights on the modifications.
 *
 */
#include <QX11Info>
#include <NETRootInfo>
#include <KApplication>
#include <KAboutData>
#include <KCmdLineArgs>
#include <KMessageBox>
#include <KLocale>

int main(int argc, char * argv[])
{
    KAboutData aboutData(
                         // The program name used internally.
                         "tutorial1",
                         // The message catalog name
                         // If null, program name is used instead.
                         0,
                         // A displayable program name string.
                         ki18n("Tutorial 1"),
                         // The program version string.
                         "1.0",
                         // Short description of what the app does.
                         ki18n("Displays a KMessageBox popup"),
                         // The license this code is released under
                         KAboutData::License_GPL,
                         // Copyright Statement
                         ki18n("(c) 2007"),
                         // Optional text shown in the About box.
                         // Can contain any information desired.
                         ki18n("Some text..."),
                         // The program homepage string.
                         "http://example.com/",
                         // The bug report email address
                         "submit@bugs.kde.org");

    KCmdLineArgs::init( argc, argv, &aboutData );
    KCmdLineOptions options;
    options.add("!+command");

    KCmdLineArgs::addCmdLineOptions( options );

    KApplication app;

    KCmdLineArgs * parsedArgs = KCmdLineArgs::parsedArgs();

    QStringList desktopNames;

    for (int i = 0; i < parsedArgs->count() ; i++)
    {
        desktopNames << parsedArgs->arg(i);
    }

    int numDesktops = desktopNames.count();

#ifdef Q_WS_X11
    NETRootInfo info( QX11Info::display(),
        NET::NumberOfDesktops | NET::DesktopNames );
    // set desktop names
    for( int i = 1; i <= numDesktops; i++ )
    {
        QString desktopName = desktopNames[ i -1 ];
        info.setDesktopName( i, desktopName.toUtf8());
        info.activate();
    }
    // set number of desktops
    info.setNumberOfDesktops( numDesktops );
    info.activate();

    XSync(QX11Info::display(), false);
#endif

    return 0;
}

Hope you like it. The reason I've written it is because the names of the virtual desktops got reset for me in KDE 4 and I didn't want to set them times and again.

2011-05-18: Tech Tip: Displaying Hebrew/Arabic/etc. Right-aligned in Kate

By default, the Kate editor of the KDE project displays Hebrew/Arabic text etc. left aligned in the window instead of aligned to the right edge of the window even after the "Enhanced support for languages written right-to-left" is checked in the qtconfig applet. To remedy this you need to enable in the settings “Appearance → General → Dynamic Word Wrap” and after doing that the mostly Hebrew lines will be properly aligned.

(I don't know why the “Dynamic Word Wrap” should affect that, and it took a long time to realise this was the problem due to the web searches and the KDE developers on IRC being ignorant, but that seems to be the solution to this.)

2011-04-16: Tech Tip: Caching the XHTML DTDs

Some days ago I needed to speed up validating an XHTML document (the module XHTML 1.1) based on the DTDs provided by the W3C, so I sought a way to cache them on the disk. Problem is I could not effectively download them from the W3C because their download speed was throttled.

Eventually, the Gentoo Wiki's “W3C Validator HOWTO” document provided some insights. Just download http://validator.w3.org/sgml-lib.tar.gz, unpack it, cd to its directory and do export XML_CATALOG_FILES="$(pwd)/catalog.xml" to set the XML catalogue appropriately. It's too bad I could not find a Mandriva Linux package for that.

Anyway, after I did that, the validation and other XML processing worked nicely.

2011-04-08: Speeding Up Bulk SQLite Inserts

This is a tip on how to speed up bulk inserts in SQLite. In case, you don't know SQLite is an SQL database implemented as a small, yet very fast library, which, accesses everything from the file-system. Furthermore, it is open-source, public domain, and reliable.

Recently, I wanted to create a server-side script to individually display my fortune cookie quotes - each on its own page based on its ID. In order to speed up the lookup, I decided to store everything in a database, be it the raw file-system, something like Berkeley DB or Tokyo Cabinet, or an SQL-based database. I ended up using SQLite, which worked pretty well.

I've written a program to populate the database by parsing the data from the XML source files and extracted the data from there. I prepared an INSERT statement:

my $insert_sth = $dbh->prepare(<<'EOF');
INSERT INTO fortune_cookies (collection_id, str_id, title, text)
VALUES(?, ?, ?, ?)
EOF

(It started a bit simpler, but that's what I have now.)

And then I executed it for each record. It took a lot of time to complete - many minutes. So I consulted #perl about it and got a few suggestions:

That I should disable the so-called “shared database” mode, which is the default SQLite mode that assumes other processes on the machine might access the database.
That I should turn off Autocommit and commit in batches with more than one record in each in every transaction.

The number two option seemed the most likely, and I tried commiting in batches of 50 records each and it made the populating program finish in a few seconds. So I was very happy.

I guess I can try the shared database mode later on, since I populate the database offline. Now the program for displaying every quote individually is working nicely, and I am very happy. Thanks to the #perl people for helping me again.

2010-12-29: C Pointer Comparison Gotcha

Some days ago I integrated a new BSD-licensed balanced binary tree, based on the red-black tree of kazlib into Freecell Solver (fc-solve) and ported two of its collections to it. I've written a comparator function for it and asked my IRC pal Amadiro to test the new fc-solve with some memory optimisations on one of his University's HPC machines which he had access to. However, it crashed pretty quickly.

I was able to reproduce a similar crash on my x86-64 laptop, and tried to investigate it. Then it hit me, because the comparator callback was this:

extern int fc_solve_compare_lru_cache_keys(
    const void * void_a, const void * void_b, void * context
)
{
#define GET_PARAM(p) ((unsigned long) \
    ((const fcs_cache_key_info_t *)(p))->val_ptr)
    return GET_PARAM(void_a) - GET_PARAM(void_b);
}

Do you see the problem? Subtracting two unsigned longs will always yield a non-negative value, which will yield the comparison void. I considered several alternatives to this including using longs instead of unsigned longs (which will still be a problems on platforms where long is less than the pointer size), or subtracting const char * (which may overflow the int return) and eventually opted for the safest version of:

typedef const char * fcs_lru_side_t;

extern int fc_solve_compare_lru_cache_keys(
    const void * void_a, const void * void_b, void * context
)
{
#define GET_PARAM(p) ((fcs_lru_side_t) \
    (((const fcs_cache_key_info_t *)(p))->val_ptr))
    fcs_lru_side_t a, b;

    a = GET_PARAM(void_a);
    b = GET_PARAM(void_b);

    return ((a > b) ? 1 : (a > b) ? (-1) : 0);
#undef GET_PARAM
}

This way I'm sure that the comparison works correctly, and indeed after that Freecell Solver ran fine on the HPC machine, and would have appeared to scale to up to 600 million positions within 64 GigaBytes instead of the previous 400 million positions.

If you're a sucker for pretty graphs, you may wish to inspect the OpenDocument spreadsheet (.ods) of the memory consumption of the solver, though the data is quite estimative.

2010-11-20: Tech Tips: Fixing KDE and GNOME problems

These are a few assorted tech tips in my tech tips column. The first one is about how to make sure KDE's Konqueror in file manager (or Doplhin for that matter) will remember the state of its "Detailed" vs. "Icons" view for particular folders. While there are many variables for this equation, my problem in a particular place, was that I tried to do it on a Samba SMB/CIFS share (a Windows-like share) that was mounted with permissions of 755 and under my UNIX user-ID/group-ID (but still read-only) which fooled konqueror and dolphin to believe it could write its settings there. Reconfiguring samba solved the problem, though it seems like I found two bugs - one in Samba and the other in KDE.

The second issue also required a lot of experimentation on my part to solve. My problem was that some X.Org applications such as KDE's Konsole, XFCE's Terminal and the IRC client XChat had larger fonts on XFCE and IceWM, than on GNOME and KDE. After a lot of trial and error with stracing the sessions, and moving away dot-directories and dot-files, I realised the GNOME problem was caused by a stray problem in ~/.gconf/ which moving that directory away solved it (one can solve it more hermetically by getting rid of the offending configuration problem, but I did not bother with it yet). The KDE problem persisted afterwards, but was easier to fix: I acked ~/.kde4/share/config for "dpi" and found share/config/kcmfonts and a line in another file that set the dpi to a different value which I fixed, and that made both the GNOME and KDE font size and DPIs (dots-per-inches) to be the same as those of IceWM and XFCE. Now the relief has made me very happy.

If you find these tips helpful, drop me a note at the comments below.

2010-10-28: Tech Tip: Making Single Click on a URL Do the Right Thing on Linux

I had a long-time problem on my Mandriva Linux system where, after I set Firefox as the default browser on KDE, it either made KDE first download the page to the hard-disk and redirect Firefox to access it from /var/tmp - or, if I set the default browser to "firefox %u", it opened the link in a new window twice.

Today I decided to go to the bottom of it, after seeing that an upgrade of KDE on Mandriva Cooker did not fix the problem (which I was told would). After consulting some people on IRC, I realised that I could not reproduce the bug on a new UNIX user account, so it was a configuration problem here. Then I realised that /usr/bin/www-browser which is what the BROWSER environment variable points to also opens it this way. After looking through the source code of www-browser, and testing some command-line invocation of capture-shells there, I suspected the problem was with a stray firefox.desktop file.

Running locate found two such files inside my home directory: /home/shlomif/.gnome2/panel2.d/default/launchers/firefox.desktop and /home/shlomif/.local/share/applications/firefox.desktop. Moving away them both resolved the problem and now if the default browser is set to "firefox", the Firefox opens a link in a single new tab after a single click.

Cheers, and hope it helps.

2010-09-12: Perl Debugger Tip: Viewing Using The Pager

Here's another Perl 5 debugger tip: if the output of a command is too long, one can prefix it with "|" (a vertical bar / pipe) to display it using the pager. So, for example |x $my_object will display $my_object using the shell's pager ( GNU less or more or whatever). Enjoy!

2010-08-25: Perl Debugger Tip: A Session Startup File

Here's a small Perl debugger tip: in order to have a file whose commands will be executed at the start of the debugging session (for example in order to get to a certain point in the code) - a session startup file, similar to gdb's --command=cmds.gdb flag - one can do the following:

First of all write a file with the debugger commands you want (let's call it cmds.perldb) and then when inside the perl debugger say:

source cmds.perldb

This will execute all the commands.

I noticed that after a while the perl debugger stores it inside the history (assuming you're using Term-ReadLine-Gnu), so you can recall it with s and then pressing the history-search-forward and history-search-backward that were set up (Mandriva assigns them to Page Up and Page Down).

Enjoy and I'm sorry for having neglected this blog for a long while!

2010-07-30: Git Tip: git status

I've had to work with the git version control system extensively in the past few weeks (due to the proliferation of projects I'm involved in that use it.). I found the standard "git status" output to be far too wordy and annoying, and sought a way to improve it and make it more subversion like. "man git-status" helped with this: one can write git status -s to get an output similar to svn status (or svn st for short) and git status -s . to restrict the output only to files below the current directory (and not get the ../… files). Enjoy!

2010-07-11: Tech Tip: Automatically Restoring the Wifi Connection on Linux

Having bought a new laptop and after installing Mandriva Linux on it, I noticed that the wifi (wireless networking) connection tends to get disconnected after a while. Eventually I wrote this script to automatically restore it. Feel free to re use it under the terms of the MIT X11 License. You may need to adjust some parameters. Run this script as root in the background (using tmux or GNU screen or whatever).

#!/bin/bash
while true ; do
	if ! ping -c 1 10.0.0.1 ; then
		/etc/init.d/network restart
	fi
	sleep 5;
done

Hope it helps.

2010-03-21: Tech Tip: Getting AAC to Play in gstreamer

If you're getting a message like the following when playing a video or an audio file with AAC sound in a gstreamer-based and no sound is played:

** Message: Missing plugin: gstreamer|0.10|totem|MPEG-4 AAC decoder|decoder-audio/mpeg, mpegversion=(int)4, framed=(boolean)true (MPEG-4 AAC decoder)

Then you need to install the 'faad' gstreamer element from gst-plugins-bad or as a separate pacakge. Thanks to "__tim" from Freenode for the insight.

2009-12-30: Vim Tip: Finding the First Non-Matching Line from the Cursor

For a long time I've been viewing and editing vim text files, for example those of the UNIX find output or of archive contents, that contained many consecutive lines of similar data. What I wanted to do is find the line where these consecutive data end where they no longer end. Among the workarounds I tried was jump to the end of the file (using G) and then search backwards for the last matching line (which may not be very reliable). But a few days ago I decided that I had enough and sought a better way, and I found one.

You can do it using a negated pattern. If for example you enter / home\/shlomi\/$Arcs\/$\@! you will find the first line that contains "home/shlomi" that is not followed by "home/shlomi/Arcs/". For information see :help \@!.

2009-12-29: KDE 4 Tip for KMail, Kopete, etc.: Right-aligning Text

For a long time, I've been bothered by the fact that on the Linux/Unix KDE 4 desktop on applications such as KMail, and Kopete, the Hebrew text was left-aligned. I expected it to be right-aligned when it encounters a leading right-to-left text, like it was on KDE 3, or now is the case in Kate (the KDE text editor). However, there is a way to align the text to the right for display.

To do so, press Right-Ctrl and Right-Shift on the keyboard (possibly while selecting the text). Like in Windows. It's much better in KMail than in Kopete, where you have to right-align the text this way after every message. But at least it makes life with KMail much more bearable. And please try to reproduce and comment on my "Hebrew Messages are Displayed with Jumpy Margins" bug in KMail.

Cheers and a happy new calendar year.

2009-08-10: Tech Tip: listing all files without "." and ".."

Today I had to deal with some pesky dot-files again, and again ran into the problem that ls -a listed them along with "." and ".." which were useless at best. A common idiom in filtering them out is using ls -a | grep -v '^..\?$' (or something similar using egrep or grep -P if it's available). But being too lazy to write it, I did a man ls and discovered a better way, right at the first page of the manual.

One can simply use ls -A or ls --almost-all to filter out the implied "." and "..". At least, this works with GNU ls - don't know about BSD or the various proprietery UNIXes.

So for example, ls -a may give you:

.  ..  FCFS_RWLock_Scheme_RLE.txt  FCFS_RWLock_Scheme.txt  .svn

While ls -A may give:

FCFS_RWLock_Scheme_RLE.txt  FCFS_RWLock_Scheme.txt  .svn

Enjoy! I hope to write about August Penguin soon, so stay tuned.

2009-07-01: Tech Tip: Finding CPAN Distributions that only have a Build.PL

A few times in the past, I wanted to find perl 5 CPAN distributions that only had a Module-Build-based Build.PL file and not a fallback Makefile.PL file. Yesterday, after some trial and error, I was able to formulate Yahoo Search query, to do just that.

Here it is - the shortened URL with a hyperlink to the full one:

http://xrl.us/bezbkx - Yahoo Search to find Build.PL only distributions

It works by looking for specific phrases in the /dist pages, looking for "Build.PL" and specifically excluding "Makefile.PL". I hereby place this URL and whatever associated techniques under CC0 / Public Domain. Enjoy!

2009-05-31: Tech Tip: Perl Debugger: Preserving the History Between Invocations

One thing that always bothered me about the Perl debugger is the fact that it didn't rememeber / preserve the GNU readline command history from previous sessions like a shell does for example. Today I discovered how to enable the preservation of the command history from previous sessions of the debugger. To do so, add something like the following the the file "~/.perldb" (under your home directory):

&parse_options("HistFile=$ENV{HOME}/.perldb-history HistSize=5000");

I explictly set the history size to a large value, but this is optional. You can also set the path to the history file to a different value.

Now when you start perl -d it will recall all the latest values from previous invocations.

2009-05-28: Tech Tip: Kopete: Recalling Previous Messages

When using Kopete, a unified instant-messaging client for the KDE desktop environment, one can recall previous messages using Ctrl+Up and Ctrl+Down. A friend, who is a KMess developer, told me it was also a KMess feature.

This wasn't very discoverable, and I discovered it only by accident. But it seems useful.

2009-04-30: Several Vim Tips

I have collected several vim tips in the past weeks, so here they are:

One can use the :sp(lit) command to split a window into two. :new splits and starts editing a new buffer. Finally, both of these commands followed by a filename stats editing the specified file in the new viewport.
This tip is long overdue. You can make Vim behave more like a Windows-like editor by adding source $VIMRUNTIME/mswin.vim to your .vimrc. Reportedly, it causes many problems and is not recommended, but I still like to use it, afer all these years.
One can match any char including newline using \_.. The "\_" construct works for other character classes, and allows you to match all of them as well as newline.
It took me a long time to find that, back when I did, so I'm documenting it here.
After one indents or unindents several lines using "<", ">", "<<", ">>", etc. and the indentation level needs to be promoted or demoted further, one can use the "." (= repeat) command to repeat the shifting again (and again) until it is right.
I discovered this trick by accident, but immediately found it useful.

Hope all Israelis had a nice Independence Day. Happy Vimming!

2009-04-25: Tech Tip: using iotop to monitor the disk activity.

Today I discovered iotop ( on Freshmeat). It's a (Linux-only) tool to find which processes are doing the most hard-disk activity at present. Useful for the cases where your hard-disks' LED is active and you don't know why.

The 0.2.1 version (which is the latest) is available in the Mandriva Cooker repositories, and it was a single urpmi command away from installation. I assume installing it on Debian, Ubuntu, etc. should be just as easy.

I learned about iotop in this blog post (in Hebrew) by Ilan Shavit, which I read today. Thanks Ilan, for the tip!

2009-04-05: Escaping Special Characters in MySQL Regular Expressions

MySQL sports the REGEXP to enable search using a regular expression pattern. However, it does not contain a built-in function (such as Perl's quotemeta) to escape the special characters in such patterns in order to prevent injecting malicious regular expression code.

Some time ago, I needed to search for a string within word-boundaries in MySQL in a PHP site I help maintain. Here's the solution I came up with for constructing this search:

function enbackslash($matches)
{
    return '\\' . $matches[1];
}

$safe_word = $mysqli->real_escape_string(
    preg_replace_callback('/([^\\w\\s])/', "enbackslash", $word)
    );

$sql .= "text regexp '[[:<:]]${safe_word}[[:>:]]'";

Basically what happens is that I prefix a backslash to any non-alphanumeric and non-whitespace character, and then escape the string again using the standard RDBMS-specific escaping function.

I would appreciate any ideas for improving upon this code, but it seems to work for me.

2009-03-23: Tech Tip: Installing CPAN Distributions without .pm Files

I recently needed to install cpan_bot from the CPAN (the Comprehensive Perl Archive Network). However, it does not contain any .pm's (Perl Module files) - only a script. All my attempts to install it failed until I tried this:

perl -MCPAN -e 'CPAN::install("ZOFFIX/cpan_bot-0.11.tar.gz")'

So you need to use the author's ID followed by a slash and the tarball's filename.

2009-02-21: How I Recovered Data from a Hosed SVK Repository

Many months ago, I noticed that SVK, a version control system based on Subversion, stopped propagating changesets to the remote Subversion repositories. Furthermore, I was unable to mirror other remote repositories. I wrote about it, asking for help:

All of this has been left unsolved until yesterday when I decided that I had enough and wanted to finally resolve this problem. My first idea was to try to write a script that will propagate the changesets using the Subversion APIs, so I looked at the SVK code trying to see where exactly it happens. I found the SVK code hard-to-follow, so I first contemplated fixing what caused it to report an error, but was also unable to.

Then I joined the Freenode IRC network and went to the #svk channel. There was only one person there who tried to help me, and he was intermittent in helping. Then I joined #perl and casually mentioned that I thought SVK sucked (just to lose some steam), which prompted mst to try to help me. After interrogating me he suggested that in order to recover the changesets from SVK, I should follow the following steps:

Find out what is the last SVK revision which still propagated changesets to the Subversion repository. This can be found using the commit message in the svn log.
Revert (using a selective svnadmin dump and then an svnadmin create ; and svnadmin load) to the older version of ~/.svk/local that corresponds with the revision number of the commit. (While naturally keeping a copy of ~/.svk/local/ and its dump in a safe place.)
See that a simple change in the svk checkout (like svk mkdir to-delete-`date +%s`) will propagate to the remote repository.
Revert that change using the Subversion client directly.
Revert to the working-but-old copy of the SVK repository, and apply a filtered stream of the dumps (using svndumpfilter) that will exclude changes from the //mirror part of the SVK repository.
After all that is done, checkout the trunk and do svk push from there.

Of course, the devil is in the details, and it took me a lot of experimentation until I got to something working. Facts I've discovered:

svndumpfilter needs a dump that was generated without the --deltas flag (and possibly without --incremental either. Otherwise it will complain.
I couldn't seem to get svndumpfilter exclude mirror to work and instead ended up using svndumpfilter include local
The --drop-empty-revs and --renumber-revs flags to svndumpfilter which seemed like a good idea confused svnadmin load, and everything worked after I removed them.
Using tar -czf and tar -xf can really speed up restoring older versions of a Subversion repository because svnadmin load is pretty slow.
I ended up taking a range of repositories from the pristine revision to "HEAD" and then manually editing out the first revision. Maybe it was unnecessary.

After all that, "svk push" worked and propagated 800 changesets to my homepage's trunk in 8:20 hours (must be a record). Since I only have an ADSL connection with 25 KBps upstream, and the Subversion service is located in Taiwan and I'm living in Israel, it took a while and I left it overnight to run.

Today, after I woke up, I saw that the operation ended up successfully. I was able to build an up-to-date version of my homesite from the Subversion sources, and am mostly ready to stop using SVK.

Again, I'd like to thank mst for suggesting this ultra-cool idea for recovering my data from SVK. It took me the whole of one day, but now I'm finally free, and my data is out there and safe. I'm going to use plain-old-Subversion to work against it, until and assuming I'll get the hang of hg/bzr/git/whatever. But at least I no longer have to use a mostly-unusable SVK.

Hackfully and happily yours, Shlomi Fish.

2009-02-05: Tech Tip: Getting Rid of Visual Artifacts in KDE 4

I recently (or not so recently) discovered that the KDE 4 desktop on my Mandriva Linux Cooker system displayed some bad visual artifacts when loaded as the default desktop. It didn't happen on fresh UNIX accounts. As I discovered, even moving away the entire home directory (/home/shlomi) to a new place, and creating an empty directory did not resolve the problem. Nor did, copying the files from a good account.

I finally discovered that the problem was in /var/tmp/kdecache-shlomi ("shlomi" is my username), which after exiting from KDE 4 and removing solved the problem for good. Of course, all of this took many hours of trying to incrementally move away files and see if it solves the problem. I cannot guarantee it is the only cause of problems similar to that one, but it helped me.

2009-02-05: Tech Tip: Resolving JDBC+MySQL exception.

I recently tried to build a Java-based project that is using MySQL. I got the following error:

BUILD FAILED
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

Last packet sent to the server was 1 ms ago.
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at com.mysql.jdbc.Util.handleNewInstance(Util.java:406)
        at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1074)
        at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2104)
        at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:729)
        at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:46)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at com.mysql.jdbc.Util.handleNewInstance(Util.java:406)
        at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:302)
        at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:283)
        at org.apache.tools.ant.taskdefs.JDBCTask.getConnection(JDBCTask.java:319)
        at org.apache.tools.ant.taskdefs.SQLExec.execute(SQLExec.java:429)
        at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:288)
        at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
        at org.apache.tools.ant.Task.perform(Task.java:348)
        at org.apache.tools.ant.Target.execute(Target.java:357)
        at org.apache.tools.ant.Target.performTasks(Target.java:385)
        at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1337)
        at org.apache.tools.ant.Project.executeTarget(Project.java:1306)
        at org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecutor.java:41)
        at org.apache.tools.ant.Project.executeTargets(Project.java:1189)
        at org.apache.tools.ant.Main.runBuild(Main.java:758)
        at org.apache.tools.ant.Main.startAnt(Main.java:217)
        at org.apache.tools.ant.launch.Launcher.run(Launcher.java:257)
        at org.apache.tools.ant.launch.Launcher.main(Launcher.java:104)
Caused by: java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
        at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
        at java.net.Socket.connect(Socket.java:519)
        at java.net.Socket.connect(Socket.java:469)
        at java.net.Socket.<init>(Socket.java:366)
        at java.net.Socket.<init>(Socket.java:209)
        at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:256)
        at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:276)
        at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2027)
        ... 27 more

After many experimentations, I discovered the problem was that MySQL on my local machine listened only on a UNIX-domain socket and not on TCP, which is what JDBC tried to connect with. After commenting out the "skip-networking" option from /etc/my.cnf (which is the location of the MySQL configuration file on my system) and restarted mysqld, everything worked.

2008-12-24: Tech Tip: How to Copy a Remote File on rsync.net

rsync.net is a good service (and a not too pricey one) for remote storage of data (for backups/etc.) based on open protocols and open source applications. One thing that bugged me there however, was that I couldn't find a way to copy a remote file to a different name, because the SFTP client does not support a copy operation, and because I could not get to a login shell by sshing my rsync.net account. Apparently, it's doable and not very hard.

What you need to do is type ssh my_account@somewhere.rsync.net cp [source filename] [destination filename] and it will run the remote UNIX "cp" command to copy the file. You can access some other remote UNIX commands like that such as "ls".

Hope it helps.

2008-12-24: Tech Tip: Tmux: Disabling the Time Display in the Status

tmux is a usable and open source terminal multiplexer (that allows one to run several programs in the same terminal, detach them, split them into windows and viewports, etc.) which I've been happily using after switching to it from GNU screen. Some weeks ago, I noticed that if I keep tmux open in a konsole tab, then eventually that tab got highlighted as modified, even if there was no activity in the tab. I realised the problem had to do with the fact that tmux displays the current time by default.

In order to fix it, you can use a line like the following in your ~/.tmux.conf file:

set -g status-right '#H'

-g applies the assignment globally, and '#H' displays the hostname (and not the time or whatever else is there by default).

While searching for an answer, I ran into the powerline project which provides much more functionality, seems like an overkill for me, but may work for you.

Shanah Tovah to everyone!

2008-12-20: Tech Tip: Enabling Audio Notifications in KDE 3 Applications

For a long time, I didn't have any audio notifications in KDE 3 applications, including not in such applications as Kopete, where people keep sending me nudges. Today, I was finally able to fix it. Here's how I did it.

Run kcontrol, and go to "Sound & Multimedia → System Notifications". Now notice the "Player Settings" button on the bottom-right corner of the screen, and press it. Switch the option to "Use an external player" and enter "/usr/bin/mplayer", or a similar capable non-interactive audio player. Press the "OK button" and say "Apply" in the main KControl window.

Sound notifications should now work, or at least they do on Mandriva Linux Cooker, where the combination of PulseAudio, artsd, and other curses like that messed up KDE completely. Hope it works for you.

2008-11-24: Tech Tip: Setting MSIE 7 as the Default Browser on Windows

I should note that in the case of this tip, I don't know why it works whereas everything else I tried didn't, but it worked for me. Your kilometrage may vary.

At my home, we have a Windows XP computer with Microsot Internet Explorer 7. After I installed Firefox, and made it the default browser for my account, it became the default for other accounts as well. Furthermore, setting Internet Explorer as the default in its configuration tab (under "Tools → Internet Options") did not work, as Firefox remained the default browser.

The way I resolved it (eventually) was by also enabling the checbox that instructs Internet Explorer to warn on startup if it detects that it's not the default browser. Then and only then, it became the default browser again.

I have no idea why it worked for me, but it did.

2008-11-22: Tech Tip: Changing the Dialog Button Order (GNOME-style vs. Windows/KDE style) for KDE 4 Applications

On my Linux system, I had a problem that the order of the dialog buttons on the dialogs of KDE 4 applications were GNOME-style (e.g: [Cancel] [OK]) instead of Windows/KDE-style (e.g: [OK] [Cancel]). The solution I found was to change the Widget style (in "systemsettings → Appearance → Style → Style → Widget Style") from Clearlooks to Plastique. However, no one seems to know for sure what determines the actual button order.

Refer to this bugs.kde.org bug report and this Mandriva bug report for more discussion.

2008-11-05: Tech Tip: Sorting RPMs by Installation Date

In order to sort the installed RPMs (RPM Package Managers) by installation date on an RPM-based Linux distribution, use the following command:

rpm -qa --qf '%{INSTALLTIME} %{NAME}\n' | sort -n | \
    perl -MPOSIX -lpe 'my ($t,$p)=split; $_=POSIX::strftime("%Y-%m-%d",localtime($t))." $p"'

You can change the format of the strftime format to something else if you want more than the date.

2008-11-04: Tech Tip: Finding the Email of a CPAN Tester

I'm writing it here so I won't misplace it in a "Know where the ledge is..."-style. If you need to find the id of a CPAN tester based on his report, use the "Find a tester" form on cpantesters.org that allow you to enter the ID ( or the URL) and find the tester. It took me a while to find it today after I misplaced it and needed to contact a CPAN tester, and Google was no help.

2008-10-19: Three Tips: Perl with Vim, Subversion Problem and mailto: links in Firefox

First of all, my home-site's back and you're gonna be in trouble! Otherwise, this entry contains three technical tips that I collected recently:

Perl and Vim - sometimes one encounters a case where he needs to handle many errors in his Perl script. I encountered a similar case, when I had to work on a script that was written without "use strict;" and "use warnings;" and shouted at me senselessly when I added them.
To greatly help with that install perl-support, and look at its Perl → Run menu - there's "update, run script" (Ctrl+F9) and "update, check syntax) (Alt+F9) there which place the errors in the quickfix buffer and allow to quickly navigate them.
Subversion and Apache - I recently encountered this error when trying to mkdir the first directory in an empty Subversion repository, that was served using Apache:
```
shlomi:~$ svn mkdir -m "Creating TO-DEL trunk" http://localhost:8080/svn/TO-DEL/trunk
svn: OPTIONS of '/svn/TO-DEL': 200 OK (http://localhost:8080)
shlomi:~$
```
This surprised me because my client was up-to-date and the service used the same libraries, and it worked with other repositories. As it turned out the problem was that the Location entry in Apache's httpd.conf looks like this:
```
<Location /svn/TO-DEL/>
    DAV svn
    SVNPath /var/svn/TO-DEL
</Location>
```
The problem here is that /svn/TO-DEL/ in the "Location" tag has a trailing slash which confuses Subversion. Eliminating that issue solves the problem.
Firefox and mailto - if "mailto:" and other non-"http:" links stopped working in Firefox, make sure that NoScript is upgraded to the latest version. There were problems with some recent versions that broke "mailto:", "ed2k:", etc.

2008-10-10: Firefox Tip: Disabling the Drag & Drop Previews

In Firefox 3, when one drags and drops images or text, they are given as a large preview around the cursor. I had found it annoying and sought a way to disable it. This tip came to the rescue after a short Google search - just go to about:config and set the nglayout.enable_drag_images to false.

Cheers!

2008-09-27: Tip: Circular Symbolic Links in Fonts Directories Causing Slowdown

I noticed a slowdown recently on my Mandriva Linux Cooker system where X GUI applications would stall for a long time and gdb showed that it happened in fontconfig calls. It happened with all user accounts on the system, including new ones and was not fixed by upgrades. Furthermore, other people who used Cooker did not have it.

Well, today I finally found a solution. Apparently, some of my font directories, contained symbolic links to themselves, which confused fontconfig and caused it to scan them times and again.

To fix the problem, run strace -e open -o fontconfig.strace fc-cache -v, and see if it excessively scans more and more sub-directories under the directory paths. Then move away the offending symbolic links to say /usr/share/REMOVED-fonts.

Now konsole starts instantly, KDE 3 apps no longer become unresponsive when the file with the unicode character, firefox stalls less, and font zoom there is faster. I'm finally happy again.

2008-09-18: Shell Variable Injection

GuySoft writes in his blog about Unix shell commands to convert video. In the shell commands, he expands variablse like -srate 22050 $infile -o $outfile".flv", without putting them inside double-quotes. In this post, I'd like to demonstrate why this is so dangerous.

When the shell expands a regular variable outside quotes it breaks into words separated by whitespace and then evaluates every word separately. To use a variable as a single token, one must enclose it inside double-quotes: "$MY_VARIABLE".

Now, let's demonstrate what can happen when we don't do it properly. I'm in the home directory of a user called "shelltest on my machine". Let's see what we have there:

[shelltest@telaviv1 ~]$ ls
Desktop/          test-shell-injection.sh*  tmp/
Freecell-Solver/  test-shell-injection.sh~
[shelltest@telaviv1 ~]$ du -s Freecell-Solver/
86M     Freecell-Solver/
[shelltest@telaviv1 ~]$ cat test-shell-injection.sh
#!/bin/sh
mkdir -p $HOME/Backup
cp $1 $HOME/Backup/
rm $1
[shelltest@telaviv1 ~]$

We have the populated "Freecell-Solver" directory, and a shell script called "test-shell-injection.sh" that uses $1 (its first positional argument) unsafely.

Now let's abuse it:

[shelltest@telaviv1 ~]$ ./test-shell-injection.sh "-fr $HOME/Freecell-Solver"
[shelltest@telaviv1 ~]$ ls -l
total 8
drwxrwxr-x 3 shelltest shelltest 28 2008-09-18 14:14 Backup/
drwxr-xr-x 2 shelltest shelltest  6 2008-09-18 14:05 Desktop/
-rwxrwxr-x 1 shelltest shelltest 58 2008-09-18 14:11 test-shell-injection.sh*
-rw-rw-r-- 1 shelltest shelltest 60 2008-09-18 14:09 test-shell-injection.sh~
drwx------ 3 shelltest shelltest 28 2008-09-18 14:07 tmp/
[shelltest@telaviv1 ~]$

Voila! Freecell-Solver is gone. I could have also said ./test-shell-injection.sh "-fr $HOME" to delete the home directory. If the script does something like for I in * ; do ... # Something with $I .. done, then a malicious user can supply a file called "-fr $HOME" (where $HOME is the value of the home directory) in the input to delete the home directory.

In short: when writing shell scripts, always enclose variable expansion in double quotes, unless you are certain that they don't contain whitespace (which implies that you generated them yourselves), or you want to use them as a list (which also requires that you generated them yourselves).

2008-09-16: Tip: Dumping the History of Google Code's (or any Remote) Subversion Repositories

Consult this FAQ on Google Code for how to use svnsync to clone a remote Subversion repository so you can have a copy of its history. I was told on IRC that it has the same end result as using svnadmin dump on the physical repository.

This post was supposed to be a rant on why I'm not hosting my projects at Google Code because they don't have a way to retrieve the dump/history of the Subversion making it a case of vendor lock-in. While this had been true in the past, I discovered it is no longer the case today, while looking for the relevant FAQ entry.

So kudos for Google and the Subversion people for fixing it.

2008-09-10: Tip: Handling mailto: Links in Firefox 3

Resources such as this one explain how to setup Firefox 2.0.x to use kmail to send email from "mailto:" links. However, compatibility with this scheme was broken in Firefox 3, as evident from Firefox bug #428658 - "network.protocol-handler.app.mailto" users need to edit/remove mimeTypes.rdf".

The solution I found is to access "Edit → Preferences → Applications → mailto" and select the appropriate application there. This seems to work for Firefox 3.

2008-08-09: Tip: Fixing rpm hangup as root on RPM-based Linux distributions

This is more of a memo-to-self than a tip, but it may prove useful to others.

When rpm (the package manager for Linux) as root gets stuck right at the start without doing anything, try the following:

Kill all rpm-related processes.
Move away /var/lib/rpm/__db.* to a temporary location.
Run rpm --rebuilddb.

That's it! It already fixed my wedged RPM database twice, so I'm putting this advice here so I won't forget it.

2008-04-24: jQuery Tip: Adding Self-Links to Headings

On my homepage, I have many <h2>, <h3>, etc. tags with id attributes in them so one can link directly to the middle of the page. I have written a Website Meta Language API that allows one to generate a table of contents for the page based on them. However, recently I also looked for a way to have a link to their anchors somewhere close to them.

I eventually decided to try doing it using JavaScript and jQuery. It took me a bit of information lookup, trial and error and consulting people on IRC, but I ended up with:

<script type="text/javascript">
<!--
$("h1[id],h2[id],h3[id],h4[id],h5[id],h6[id]").each(
function(i){
$(this).append( ' <span class="selfl">[<a href="#' +
this.id + '">link</a>]</span>' )
})
-->
</script>

Now for some explanations:

The $("h1[id]...") construct selects all the headings with id's. There may be a shorter way to do it (comments are welcome.
The each method iterates over all of them and calls the closure inside. The closure sets the "this" variable to the current element, and accepts its index there (the "i" variable). In our case, we're not making use of the index.
$(this) constructs a jQuery object from "this".
.append() appends an expression to the inner HTML of the element. I add a little HTML there. this.id may result in an XSS attack if you have a really funky (and probably invalid) ID, but since I have control over my ID's it's ok.

You can see the result, in the headings of the presentations' page (for example), as long as you don't have JavaScript turned off. jQuery seems very nice, and I'm looking forward to making even more use of it where appropriate.

2008-04-08: Handling Repetitive Questions Once and for All

Back when I worked on Freecell Solver , I used to prepare binary distributions of the command line solver. Now it acted as a filter, and when invoked without arguments it just waited for standard input. As a result, when the Windows users double-clicked the fc-solve.exe executable, it just popped an empty console, which they couldn't understand what to do. And a lot of them sent me a "How do I use this" program email with the exact description.

Since I became annoyed of it, I decided to output the following to the standard error by default:

Reading the board from the standard input.
Type "fc-solve --help" for more usage information.
To cancel this message set the FREECELL_SOLVER_QUIET environment variable.

After the poor Winnies read it, it probably made them decide the program was not for them and to look for an alternative, but at least most of the emails I received about it stopped.

A year ago, Joel on Software published an article about having remarkable customer services, in which he said that one needs to fix everything two ways: first by helping the user, and then by making sure other users don't encounter and complain about the problem again by tweaking the behaviour. As one can see, what I did for Freecell Solver was essentially what Joel advised to do. (Albeit much later than I incorporated the fix.).

More recently, I started the Better SCM site, which compares and advocates several version control systems. However, git has been absent due to the fact a suitable maintainer did not come forward. And as a result, I received many emails asking "Where's git?". A few days ago I decided to add a Frequently Asked Questions (F.A.Q.) to the site where I answered this and other questions. However, today I received another email with "Git?" in the subject about that.

Another thing Joel had said is that "People Don't Read" or at least read very selectively. So I added a placeholder page about git with explanation why it is absent and a call for help, and added it to the comparison with the text "Unknown. Due to a lack of a maintainer for this system." with a link to the F.A.Q..

This should be enough to get rid of that particularly repetitive stream of emails, and hopefully will also yield me a contributor to add a better solution. Often designing web-sites and doing "customer service" (or feedback) requires designing the site properly based on the feedback one gets.

I didn't particularly think of what Joel said when I decided to do what I did with the F.A.Q. and the placeholders, but now that I think of it, I guess he was right.

2008-04-04: Eclipse Tip: Debugging Using a Remote GDB Session (gdbserver)

In this tip, I'll explain how to debug C/C++ programs in Eclipse using a remote GDB session (gdbserver). To do this:

Install CDT - The Eclipse C/C++ Development Tooling.
Set up a C/C++ project so it will build and compile.
Install gdb and gdbserver. The latter does not seem to be available in any Mandriva package and had to be compiled from the gdb source, where it is built by default.
Run gdbserver on the generated executable using a command like gdbserver localhost:10900 ./my-exe
In Eclipse go to Run → "Open Run Dialog...", and go to the upper debugger tab and select gdbserver Debugger in the Debugger drop-down. Afterwards, go to the "Connection" tab under Debugger Options and configure your connection. For my demonstration I've chosen "Type: TCP" "Host: localhost" and "Port number: 10900".
Select the Run → Debug in Eclipse option and start debugging the application.

The reason I needed it was that I wanted to use "remote" debugging so I can debug a C++ program that must be ran as root, and would rather not run Eclipse itself as root. I'm still not sure it can serve as a good debugger front-end, but I became very annoyed with ddd and plain gdb start to seem to be inadequate on visualising the rest of the source code. So I looked for an alternative and thought Eclipse might be OK.

2008-03-31: Vim Tip: Windows Management Commands

Windows are one of Vim's most convenient features, and learning some commands could prove to be useful. Some of the commands are available from the "Window" menu in gvim, or using the mouse, but using them from the keyboard is often more convenient.

To split a viewport in two, use Ctrl+W,S for a horizontal split or Ctrl+W,V for a vertical split. To move between windows use Ctrl+W followed by the arrow keys or h,j,k,l (but see this previous tip). To close a viewport use Ctrl+W,C.

Ctrl+W,r rotates the windows, and Ctrl+W, shift+R rotates them in the opposite direction. Finally, Ctrl+W,+ and Ctrl+W,- increase and decrease the height of the viewport by one line (in case you are mouse-deprived). If that's too little you can do something like 10,Ctrl+W,+ to increase the height by 10 lines.

2008-03-23: My Homesite and Murphy

So yesterday I decided that my homesite's markup may have had too much extraneous whitespace. So I looked at HTML Tidy to try to remove it. I searched for a way to tell it to remove whitespace on the Net, but most pages I found only said it was possible, but didn't say how. After some trial and error I found that this tidy.rc file:

indent: 0
indent-attributes: no
vertical-space: no
wrap: 0
indent-spaces: 0
break-before-br: 0
tab-size: 0

And a command line that specifies it with -config worked the best. It still left a lot of newlines, which could be removed, but I guess it's better than nothing.

While running it, I discovered that it complained about having empty files. And I checked that the files generated were indeed empty. Then I experienced with Website Meta Language (or WML) on Mandriva and found out that piping into it even simple documents resulted in empty results.

I found out that the update with the fix for the WML CVE issue broke WML. So I had to found out what the problem was generate a better patch and submit it. As it turns out the WML test suite still failed, but it didn't cause the RPM "%check" target to fail for some reason. After my fix, all tests passed again.

Then I was able to process the pages again. So back to tidy. Tidy still complained about some stuff like missing "summary" attributes in "table" tags. Sometimes, it exited with an error status without emitting any warnings or errors, so I had to tell my build system to skip these offending files.

So tidy ran, but it wasn't the end of my problems. When trying to build the homepage to run tidy with a clean copy, I found out that TTML did not run. This was caused by the upgrade to perl-5.10.0, and I looked for a way to prepare an up-to-date RPM out of it again. Both cpan2rpm and Ovid (the Perl module, not the Perl author) failed, and so I looked into cpan2dist. After experimenting with it a bit I got it running using CPANPLUS-Dist-Mdv and it gave me a working RPM. Now I'd like to prepare an equivalent package for constructing Fedora RPMs.

That was not the end of my problems. When I tried to clean-build my site, I found out that the directory structure did not get built before some of the targets. So I had to play with the make targets. Afterwards, everything went fine. The difference that tidy made was:

Before size: 4903322
After size: 4781819

What next? On the site I still have many links that start with "./". This prefix is completely redundant, and I can save some bandwidth by eliminating them. I also contemplated converting the navigation menus to JavaScript, which will read the menus' contents from the site in JSON format. Now that I think about it, the problem with this approach is that it will hinder navigating the site by web-crawlers, and may diminish the Page Rank of the various pages. It also won't work on clients which have JavaScript disabled. One thing I can do is also generate an HTML file with the current view of the navigation menu for each page and link to it directly for JavaScript-challenged clients. I still have to think about it.

2008-02-25: Vim Tip: Fixing the Left/Right Window Switching

My Vim tips column could use a more comprehensive entry about window management (a.k.a "split view") in Vim, but this is a small tip in this regard, that bothered me for a long time and that I want to record for prosperity and share.

When having a verticla split, one can use Ctrl+W,Left or Ctrl+W,H to move to the left split, and Ctrl+W,Right or Ctrl+W,L to move to the right split. However, it seems that for some reason Ctrl+W,Ctrl+Left (while holding the control) does not work. To fix it, add the following lines to your .vimrc:

" mapping to be able to move to the left and the right windows
" without needing to leave the Ctrl key.
map <C-W><C-Right> <C-W><Right>
map <C-W><C-Left> <C-W><Left>

I've been annoyed by this behaviour for a long time, and now the salvation feels sweet. I should note that other Ctrl+W,Ctrl+Someting key-combinations work perfectly, so this aliasing is not needed there.

2008-02-19: Tip: Recording TCP Sessions to a File Using tcpdump

Using tcpdump to record TCP sessions (or conversations) to a file, use the following command (as root or a privileged user):

tcpdump -w dump-file.pcap 'host 212.143.218.31'

Replace the 'hsot 212.143.218.31' with any tcpdump filter. Then you can run "wireshark" on the dump-file.pcap file. Note that I got incomplete results when I tried it now on my home machine and would love to be further enlightened.

2008-01-09: Linux Tip: Playing Sound with Different Users You've su'ed to in Mandriva

In order to play sound / audio on your Mandriva (and possibly other Linux flavours) system, after being logged in an X session with one user, and you've su'ed to a different (also possibly under-privileged) user in the shell, do the following. Just add all the users that should be able to play sound at all times to the "audio" UNIX group in the "/etc/group" file (as root, of course). Then, you may have to run "sg" or "newgrp" for these changes to take effect globally.

Then you can play sound as all the users you've su'ed to that belong to the "audio" group. Also see this thread in the Mandriva Cooker mailing list, where I asked the question originally.

And a more obscure and less useful tip: to exclude dependencies ("requires") on different RPMs (that are calculated implicitly) when editing an Mandriva SPEC use a statement like that:

%define _requires_exceptions /bin/zsh /bin/csh

There's an example of this in the Mandriva xemacs spec.

2008-01-06: Amarok Tip: Playing an Arbitrary Last.fm Tag + Hex-a-Hop

In continuation to my previous entry about Amarok and Last.fm integration, I found this bug entry regarding the main problem with it that bothers me, and I commented about it and re-opened it. A net-friend who checked it got different results, and commented as well.

One thing I noticed about the Last.fm player is that it allows one to play any arbitrary Last.fm tag, while Amarok only has a set of pre-defined tags for common music genres in its GUI. However, after experimenting with "dcop" I found out one can enqueue any arbitrary Last.fm tag using the following shell script:

#!/bin/bash
tag="$1"
shift
dcop amarok playlist addMedia "lastfm://globaltags/${tag}"

To use it save as "lasttfm-amarok-tag", make it executable (chmod +x) and do ./lastfm-amarok-tag "new age" to play the "new age" tag. You can also type a command line like dcop amarok playlist addMedia "lastfm://globaltags/reggae" directly from the command line, but that's not very convenient.

And finally, here's a recommendation for a puzzle game I saw mentioned on the Free Gamer blog: Hex-a-Hop. I was able to easily compile it on my Mandriva Cooker system, and played it, and it's very nice.

2007-12-29: ssh Tip: Customising hosts with the "Host" directive

The ~/.ssh/config accepts the Host meta-configuration that enables one to specify directives for only a specific host. With the HostName directigve, and the User directive this allows defining shortcuts and defaults. For example, this portion out of my file:

Host berlios

HostName shell.berlios.de
User shlomif

Specifies that by doing "ssh berlios" I'll login to shell.berlios.de as the user shlomif (in equivalence to ssh shlomif@shell.berlios.de).

One use I can think of for this feature is to define a common destination, with variable remote usernames, in uploading scripts. Until now, I've used something like: ${IGLU_USERNAME:-$(whoami)}@iglu.org.il:, which retrieved the value of the $IGLU_USERNAME and defaulted to the local user. However, now one can simply use iglu.org.il: or mytask.iglu.org.il: and define the appropriate user in the ssh config file.

Credit is due to a blog post by Diego Iastrubni, where he blogged about this feature. Thanks, Diego!

2007-07-07: Vim Tip: Copying Some Non-Adjacent Lines to a Register

Here's a vim tip on how to copy some non adjacent lines to a register ("r), but first the story that made me find out about how to do it exactly. I need to add full POD (Perl's documentation format) coverage to a Perl module I inherited. What it means is that every top-level function (or at least those that do not start with a "_") should have their own entries. In Perl the function definitions goes like that:

sub first_function {

}

sub second_function {

}

sub yet_another_function {

}

Etc. So I thought that I could look for all the lines that start with "sub " , collect them and append them to a register, where I'll paste them. I recalled the :g/re/$cmd that executes a command on all lines matching a certain regex, and the fact that I could append to a register instead of setting its value using a yank command.

So after a little research I tried: :g/^sub /"Fyy. :g searches for the /^sub / and invokes the command on every occurence, and "Fyy, appends the current line to the register "f". However, as it turns out :g executes an Ex command. However, fear not - after a little research, I recalled the :normal command that allows to execute normal mode commands from within the command-line (or Ex) mode.

So the final line that works is :g/^sub /normal "Fyy. You may need to do :let @f = '' before that to clear the "f" register.

Cheers.

2007-07-01: Vim Tip: Completing Lines and Other Things

One can use Ctrl+X in Vim to invoke several sub-commands for auto-completion. For example, "Ctrl+X ; Ctrl+L" completes entire line. So if you have a lot of my $self = shift; or my ($self, $args) = @_; in your Perl code you can start writing that line and press Ctrl+X;Ctrl+L to fill the rest of them.

Ctrl+X;Ctrl+F completes filenames, and there are other completions like that. I found it while reading usr_24 - inserting quickly which contains some other tips. I also realised that Ctrl+U cuts to the beginning of the line in vim editing, or in Emacs or GNU Readline.