Contents

• Experimental Setup
• Reset the Working Tree
• Reset the Index
• Reset the Working Tree and Index
• Summary

Experimental Setup

To experiment quickly, we first create an example Git repository.

mkdir foo/; cd foo/; touch a b c
git init; git add a b c; git commit -m hello

Now we make changes to the files and stage some of the changes. We then add more unstaged changes to one of the staged files.

date | tee a b c d; git add a b d; echo > b

At this point, the working tree and index look like this:

$ git status
On branch main
Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
        modified:   a
        modified:   b
        new file:   d

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   b
        modified:   c

File a has staged changes. File b has both staged and unstaged changes. File c has only unstaged changes. File d is a new staged file. In each experiment below, we will work with this setup.

All results discussed in this post were obtained using Git 2.47.3 on Debian 13.2 (Trixie).

Reset the Working Tree

As a reminder, we will always use the following command between experiments to ensure that we restore the experimental setup each time:

date | tee a b c d; git add a b d; echo > b

To discard the changes in the working tree and reset the files in the working tree from the index, I typically run:

git checkout .

However, the modern way to do this is to use the following command:

git restore .

Both commands leave the working tree and the index in the following state:

$ git status
On branch main
Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
        modified:   a
        modified:   b
        new file:   d

Both commands operate only on the working tree. They do not alter the index. Therefore the staged changes remain intact in the index.

Reset the Index

Another common situation is when we have staged some changes but want to unstage them. First, we restore the experimental setup:

date | tee a b c d; git add a b d; echo > b

I normally run the following command to do so:

git reset

The modern way to do this is:

git restore -S .

Both commands leave the working tree and the index in the following state:

$ git status
On branch main
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   a
        modified:   b
        modified:   c

Untracked files:
  (use "git add <file>..." to include in what will be committed)
        d

no changes added to commit (use "git add" and/or "git commit -a")

The -S (--staged) option tells git restore to operate on the index (not the working tree) and reset the index entries for the specified files to match the version in HEAD. The unstaged changes remain intact as modified files in the working tree. With the -S option, no changes are made to the working tree.

From the arguments we can see that the old and new commands are not exactly equivalent. Without any arguments, the git reset command resets the entire index to HEAD, so all staged changes become unstaged. Similarly, when we run git restore -S without specifying a commit, branch or tag using the -s (--source) option, it defaults to resetting the index from HEAD. The . at the end ensures that all paths under the current directory are affected. When we run the command at the top-level directory of the repository, all paths are affected and the entire index gets reset. As a result, both the old and the new commands accomplish the same result.

Reset the Working Tree and Index

Once again, we restore the experimental setup.

date | tee a b c d; git add a b d; echo > b

This time we not only want to unstage the changes but also discard the changes in the working tree. In other words, we want to reset both the working tree and the index from HEAD. This is a dangerous operation because any uncommitted changes discarded in this manner cannot be restored using Git.

git reset --hard

The modern way to do this is:

git restore -WS .

The working tree is now clean:

$ git status
On branch main
nothing to commit, working tree clean

The -W (--worktree) option makes the command operate on the working tree. The -S (--staged) option resets the index as described in the previous section. As a result, this command unstages any changes and discards any modifications in the working tree.

Note that when neither of these options is specified, -W is implied by default. That's why the bare git restore . command in the previous section discards the changes in the working tree.

Summary

The following table summarises how the three pairs of commands discussed above affect the working tree and the index, assuming the commands are run at the top-level directory of a repository.

The git restore command is meant to provide a clearer interface for resetting the working tree and the index. I still use the older commands out of habit. Perhaps I will adopt the new ones in another six years, but at least I have the mapping written down now.

Read on website | #technology | #how-to

$ sudo sojuctl user delete soju
To confirm user deletion, send "user delete soju 4664cd"
$ sudo sojuctl user delete soju 4664cd
deleted user "soju"

That confirmation token for a specific user never changes, no matter how many times we create or delete it. The confirmation token is not saved in the Soju database, as can be confirmed here:

$ sudo sqlite3 -table /var/lib/soju/main.db 'SELECT * FROM User'
+----+----------+--------------------------------------------------------------+-------+----------+------+--------------------------+---------+--------------------------+--------------+
| id | username |                           password                           | admin | realname | nick |        created_at        | enabled | downstream_interacted_at | max_networks |
+----+----------+--------------------------------------------------------------+-------+----------+------+--------------------------+---------+--------------------------+--------------+
| 1  | soju     | $2a$10$yRj/oYlR2Zwd8YQxZPuAQuNo2j7FVJWeNdIAHF2MinYkKLmBjtf0y | 0     |          |      | 2026-02-16T13:49:46.119Z | 1       |                          | -1           |
+----+----------+--------------------------------------------------------------+-------+----------+------+--------------------------+---------+--------------------------+--------------+

Surely, then, the confirmation token is derived from the user definition? Yes, indeed it is. This can be confirmed at the source code here. Quoting the most relevant part from the source code:

hashBytes := sha1.Sum([]byte(username))
hash := fmt.Sprintf("%x", hashBytes[0:3])

Indeed if we compute the same hash ourselves, we get the same token:

$ printf soju | sha1sum | head -c6
4664cd

This allows us to automate the two step Soju user deletion process in a single command:

sudo sojuctl user delete soju "$(printf soju | sha1sum | head -c6)"

But of course, the implementation of the confirmation token may change in future and Soju helpfully outputs the deletion command with the confirmation token when we first invoke it without the token, so it is perhaps more prudent to just take that output and feed it back to Soju, like so:

sudo sojuctl $(sudo sojuctl user delete soju | sed 's/.*"\(.*\)"/\1/')

Read on website | #shell | #irc | #technology | #how-to

Setup

It is quite straightforward to install and set up Soju on Debian. The following two commands install Soju:

sudo apt-get update
sudo apt-get -y install soju

Then setting up an IRC connection involves another two commands:

sudo sojuctl user create -username soju -password YOUR_SOJU_PASSWORD
sudo sojuctl user run soju network create -name bnc1 -addr irc.libera.chat -nick YOUR_NICK -pass YOUR_NICK_PASSWORD

Here, YOUR_SOJU_PASSWORD is a placeholder for a new password you must choose for your Soju user. Finally, we restart Soju as follows:

sudo systemctl restart soju

Database

What previously involved maintaining several files that had to be installed and configured on each machine running ZNC is now reduced to the two sojuctl commands above. Still, the configuration needs to live somewhere. In fact, the two sojuctl commands introduce earlier store the configuration in a SQLite database. Here is a glimpse of what the database looks like:

$ sudo sqlite3 /var/lib/soju/main.db '.tables'
Channel              MessageFTS_data      ReadReceipt
DeliveryReceipt      MessageFTS_docsize   User
Message              MessageFTS_idx       WebPushConfig
MessageFTS           MessageTarget        WebPushSubscription
MessageFTS_config    Network
$ sudo sqlite3 /var/lib/soju/main.db 'SELECT * from User'
1|soju|$2a$10$mM5Qcz8.OPMi9lyWDxPRh.bNxzq7jtLdxcoPl09AYTnqcmLmEqzSO|0|||2026-02-17T23:24:24.926Z|1||-1
$ sudo sqlite3 /var/lib/soju/main.db 'SELECT * from Network'
1|bnc1|1|irc.libera.chat|YOUR_NICK||||YOUR_NICK_PASSWORD|||||||1|1

Client Configuration

Finally, the IRC client can be configured to connect to port 6697 on the system running Soju. Here is an example of how this can be done in Irssi:

/network add -nick YOUR_NICK -user soju/bnc1 net1
/server add -tls -network bnc1 YOUR_SOJU_HOST 6697 YOUR_SOJU_PASSWORD
/connect net1

You can also set up multiple connections to IRC networks through the same Soju instance. All you need to do is repeat the sojuctl commands to create additional networks such as bnc2, bnc3 and so on, then repeat the configuration in your IRC client using new network names such as net2, net3, etc. These network names are entirely user defined, so you can choose any names you like. The names bnc2, net2 and so on are only examples.

Read on website | #irc | #technology | #how-to

issueFunction in commented("by jdoe") and comment ~ "hello"

The above example would search all comments by user jdoe that contains the string "hello". However one limitation of this solution is that the issueFunction field is provided by ScriptRunner for Jira, so you need this app to be available on your Jira for this solution to work.

Read on website | #technology | #how-to

Sorting Lines

Our first set of experiments demonstrates different ways to sort lines. Follow the steps below to perform these experiments.

1. First create a buffer that has the following text:

   Carol  200  London  LHR->SFO
   Dan    20   Tokyo   HND->LHR
   Bob    100  London  LCY->CDG
   Alice  10   Paris   CDG->LHR
   Bob    30   Paris   ORY->HND

   Let us pretend that each line is a record that represents some details about different persons. From left to right, we have each person's name, some sort of numerical ID, their current location and their upcoming travel plan. For example, the first line says that Carol from London is planning to travel from London Heathrow (LHR) to San Francisco (SFO).

2. Type C-x h to mark the whole buffer and type M-x sort-lines RET to sort lines alphabetically. The buffer looks like this now:

   Alice  10   Paris   CDG->LHR
   Bob    100  London  LCY->CDG
   Bob    30   Paris   ORY->HND
   Carol  200  London  LHR->SFO
   Dan    20   Tokyo   HND->LHR

3. Type C-x h followed by C-u M-x sort-lines RET to reverse sort lines alphabetically. The key sequence C-u specifies a prefix argument that indicates that a reverse sort must be performed. The buffer looks like this now:

   Dan    20   Tokyo   HND->LHR
   Carol  200  London  LHR->SFO
   Bob    30   Paris   ORY->HND
   Bob    100  London  LCY->CDG
   Alice  10   Paris   CDG->LHR

4. Type C-x h followed by M-x sort-fields RET to sort the lines by the first field only. Fields are separated by whitespace. Note that the result now is slightly different from the result of M-x sort-lines RET presented in point 2 earlier. Here Bob from Paris comes before Bob from London because the sorting was performed by the first field only. The sorting algorithm ignored the rest of each line. However in point 2 earlier, Bob from London came before Bob from Paris because the sorting was performed by entire lines.

   Alice  10   Paris   CDG->LHR
   Bob    30   Paris   ORY->HND
   Bob    100  London  LCY->CDG
   Carol  200  London  LHR->SFO
   Dan    20   Tokyo   HND->LHR

5. Type C-x h followed by M-2 M-x sort-fields RET to sort the lines alphabetically by the second field. The key sequence M-2 here specifies a numeric argument that identifies the field we want to sort by. Note that 100 comes before 20 because we performed an alphabetical sort, not numerical sort. The result looks like this:

   Alice  10   Paris   CDG->LHR
   Bob    100  London  LCY->CDG
   Dan    20   Tokyo   HND->LHR
   Carol  200  London  LHR->SFO
   Bob    30   Paris   ORY->HND

6. Type C-x h followed by M-2 M-x sort-numeric-fields RET to sort the lines numerically by the second field. The result looks like this:

   Alice  10   Paris   CDG->LHR
   Dan    20   Tokyo   HND->LHR
   Bob    30   Paris   ORY->HND
   Bob    100  London  LCY->CDG
   Carol  200  London  LHR->SFO

7. Type C-x h followed by M-3 M-x sort-fields RET to sort the lines alphabetically by the third field containing city names. The result looks like this:

   Bob    100  London  LCY->CDG
   Carol  200  London  LHR->SFO
   Alice  10   Paris   CDG->LHR
   Bob    30   Paris   ORY->HND
   Dan    20   Tokyo   HND->LHR

   Note that we cannot supply the prefix argument C-u to this command to perform a reverse sort by a specific field because the prefix argument here is used to identify the field we need to sort by. If we do specify the prefix argument C-u, it would be treated as the numeric argument 4 which would sort the lines by the fourth field. However, there is a little trick to reverse sort lines by a specific field. The next point shows this.

8. Type C-x h followed by M-x reverse-region RET. This reverses the order of lines in the region. Combined with the previous command, this effectively reverse sorts the lines by city names. The result looks like this:

   Dan    20   Tokyo   HND->LHR
   Bob    30   Paris   ORY->HND
   Alice  10   Paris   CDG->LHR
   Carol  200  London  LHR->SFO
   Bob    100  London  LCY->CDG

9. Type C-x h followed by M-- M-2 M-x sort-fields RET to sort the lines alphabetically by the second field from the right (third from the left). Note that the first two key combinations are meta+- and meta+2. They specify the negative argument -2 to sort the lines by the second field from the right. The result looks like this:

   Carol  200  London  LHR->SFO
   Bob    100  London  LCY->CDG
   Bob    30   Paris   ORY->HND
   Alice  10   Paris   CDG->LHR
   Dan    20   Tokyo   HND->LHR

10. Type M-< to move the point to the beginning of the buffer. Then type C-s London RET followed by M-b to move the point to the beginning of the word London on the first line. Now type C-SPC to set a mark there.

    Then type C-4 C-n C-e to move the point to the end of the last line. An active region should be visible in the buffer now.

    Finally type M-x sort-columns RET to sort the columns bounded by the column positions of mark and point (i.e. the last two columns). The result looks like this:

    Bob    100  London  LCY->CDG
    Carol  200  London  LHR->SFO
    Alice  10   Paris   CDG->LHR
    Bob    30   Paris   ORY->HND
    Dan    20   Tokyo   HND->LHR

11. Like before, type M-< to move the point to the beginning of the buffer. Then type C-s London RET followed by M-b to move the point to the beginning of the word London on the first line. Now type C-SPC to set a mark there.

    Again, like before, type C-4 C-n C-e to move the point to the end of the last line. An active region should be visible in the buffer now.

    Now type C-u M-x sort-columns RET to reverse sort the last two columns.

    Dan    20   Tokyo   HND->LHR
    Bob    30   Paris   ORY->HND
    Alice  10   Paris   CDG->LHR
    Carol  200  London  LHR->SFO
    Bob    100  London  LCY->CDG

12. Warning: This step shows how not to use the sort-regexp-fields command. In most cases you probably do not want to do this. The next point shows a typical usage of this command that is correct in most cases.

    Type C-x h followed by M-x sort-regexp-fields RET [A-Z]*->\(.*\) RET \1 RET to sort by the destination airport. This command first matches the destination aiport in each line in a regular expression capturing group (\(.*\)). Then we ask this command to sort the lines by the field matched by this capturing group (\1). The result looks like this:

    Dan    20   Tokyo   LCY->CDG
    Bob    30   Paris   ORY->HND
    Alice  10   Paris   HND->LHR
    Carol  200  London  CDG->LHR
    Bob    100  London  LHR->SFO

    Observe how all our travel records are messed up in this result. Now Dan from Tokyo is travelling from LCY to CDG instead of travelling from HND to LHR. Compare the results in this point with that of the previous point. This command has sorted the destination fields fine and it has maintained the association between the source airport and destination airport fine too. But the association between the other fields (first three columns) and the last field (source and destination airports) is broken. This happened because the regular expression matches only the last column and we sorted by only the destination field of the last column, so the association of the fields in the last column is kept intact but the rest of the association is broken. Only the part of each line that is matched by the regular expression moves around while the sorting is performed; everything else remains unchanged. This behaviour may be useful in some limited situations but in most cases, we want to keep the association between all the fields intact. The next point shows how to do this.

    Now type C-/ (or C-x u) to undo this change and revert the buffer to the previous good state. After doing this, the buffer should look like the result presented in the previous point.

13. Assuming the state of the buffer is same as that of the result in point 11, we will now see how to alter the previous step such that when we sort the lines by the destination field, entire lines move along with the destination fields. The trick is to ensure that the regular expression matches entire lines. To do so, we make a minor change in the regular expression. Type C-x h followed by M-x sort-regexp-fields RET .*->\(.*\) RET \1 RET.

    Bob    100  London  LCY->CDG
    Bob    30   Paris   ORY->HND
    Dan    20   Tokyo   HND->LHR
    Alice  10   Paris   CDG->LHR
    Carol  200  London  LHR->SFO

    Now the lines are sorted by the destination field and Dan from Tokyo is travelling from HND to LHR.

14. Type C-x h followed by M-- M-x sort-regexp-fields RET .*->\(.*\) RET \1 RET to reverse sort the lines by the destination airport. Note that the first key combination is meta+- here. This key combination specifies a negative argument that results in a reverse sort. The result looks like this:

    Carol  200  London  LHR->SFO
    Dan    20   Tokyo   HND->LHR
    Alice  10   Paris   CDG->LHR
    Bob    30   Paris   ORY->HND
    Bob    100  London  LCY->CDG

15. Finally, note that we can always invoke shell commands on a region and replace the region with the output of the shell command. To see this in action, first prepare the buffer by typing M-< followed by C-k C-k C-y C-y to duplicate the first line of the buffer.

    Then type C-x h followed by C-u M-| sort -u RET to sort the lines but remove duplicate lines during the sort operation. The M-| key sequence invokes the command shell-command-on-region which prompts for a shell command, executes it and usually displays the output in the echo area. If the output cannot fit in the echo area, then it displays the output in a separate buffer. However, if a prefix argument is supplied, say with C-u, then it replaces the region with the output. As a result, the buffer now looks like this:

    Alice  10   Paris   CDG->LHR
    Bob    100  London  LCY->CDG
    Bob    30   Paris   ORY->HND
    Carol  200  London  LHR->SFO
    Dan    20   Tokyo   HND->LHR

    This particular problem of removing duplicates while sorting can be also be accomplished by typing C-x h followed by M-x sort-lines RET and then C-x h followed by M-x delete-duplicate-lines. Nevertheless, it is useful to know that we can execute arbitrary shell commands on a region.

Sorting Paragraphs and Pages

We have covered most of the sorting commands mentioned in the Emacs manual in the previous section. Now we will switch gears and discuss a few more of the remaining ones. We will no longer sort individual lines but paragraphs and pages instead.

1. First create a buffer with the content provided below. Note that the text below contains three form feed characters. In Emacs, they are displayed as ^L. Many web browsers generally do not display them. The ^L symbols that we see in the text below have been overlayed with CSS. But there are actual form feed characters next to those overlays. If you are viewing this post with any decent web browser, you can copy the text below into your Emacs and you should be able to see the form feed characters in Emacs. In case you do not, insert them yourself by typing C-q C-l.

   Emacs is an advanced, extensible, customisable,
   self-documenting editor.
   
   Emacs editing commands operate in terms of
   characters, words, lines, sentences, paragraphs,
   pages, expressions, comments, etc.
   
   We will use the term frame to mean a graphical
   window or terminal screen occupied by Emacs.
   
   At the very bottom of the frame is an echo area.
   The main area of the frame, above the echo area,
   is called the window.
   
   The cursor in the selected window shows the
   location where most editing commands take effect,
   which is called point.
   
   If you are editing several files in Emacs, each in
   its own buffer, each buffer has its own value of
   point.
   

2. Our text has six paragraphs spread across three pages. Each form feed character represents a page break. Type C-x h followed by M-x sort-pages RET to sort the pages alphabetically. Note how the second page moves to the bottom because it begins with the letter "W". The buffer now looks like this now:

   Emacs is an advanced, extensible, customisable,
   self-documenting editor.
   
   Emacs editing commands operate in terms of
   characters, words, lines, sentences, paragraphs,
   pages, expressions, comments, etc.
   
   The cursor in the selected window shows the
   location where most editing commands take effect,
   which is called point.
   
   If you are editing several files in Emacs, each in
   its own buffer, each buffer has its own value of
   point.
   
   We will use the term frame to mean a graphical
   window or terminal screen occupied by Emacs.
   
   At the very bottom of the frame is an echo area.
   The main area of the frame, above the echo area,
   is called the window.
   

3. Finally, type C-x h followed by M-x sort-paragraphs to sort the paragraphs alphabetically. The buffer looks like this now:

   At the very bottom of the frame is an echo area.
   The main area of the frame, above the echo area,
   is called the window.
   
   Emacs editing commands operate in terms of
   characters, words, lines, sentences, paragraphs,
   pages, expressions, comments, etc.
   
   Emacs is an advanced, extensible, customisable,
   self-documenting editor.
   
   If you are editing several files in Emacs, each in
   its own buffer, each buffer has its own value of
   point.
   
   The cursor in the selected window shows the
   location where most editing commands take effect,
   which is called point.
   
   We will use the term frame to mean a graphical
   window or terminal screen occupied by Emacs.
   

References

To read and learn more about the sorting commands described above refer to the following resources:

• Emacs Manual: Sorting Text
• Elisp Manual: Sorting Text

Within Emacs, type the following commands to read these manuals:

• M-: (info "(emacs) Sorting") RET
• M-: (info "(elisp) Sorting") RET

Further, the documentation strings for these commands have useful information too. Use the key sequence C-h f to look up the documentation strings. For example, type C-h f sort-regexp-fields RET to look up the documentation string for the sort-regexp-fields command.

Read on website | #emacs | #technology | #how-to

Once in a while, I fire up one of the vintage DOS games or language interpreters in DOSBox for nostalgia's sake. DOSBox is an emulator program that emulates IBM PC compatible computers running DOS. Trying my hands on these antiquated DOS programs now evokes old memories from my childhood days days when I first came across computers as part of our primary school curriculum.

Computers were much simpler in those days. The ones in our school were IBM PC compatible computers with mostly monochrome displays. A couple of them had support for a very limited number of colours provided by CGA or EGA graphics cards. The ability to boot a computer using a 5¼-inch floppy disk containing MS-DOS, load a Logo or BASIC interpreter or a computer game from another floppy disk and then write some programs or play a few games without any distraction had its own charm that I find missing from modern day computing.

Often while using old DOS programs with DOSBox in this day and age, I want to take screenshot captures or video captures of the DOSBox sessions and share them with my friends. In this article, I will explain how I create good quality screenshot captures and video captures of DOSBox sessions in formats that I can share with others.

Contents

• Vintage DOS Programs
• Software Versions
• IBM PC Logo in DOSBox
• Digger in DOSBox
• DOSBox Screenshot Capture
• DOSBox Video Capture
• DOSBox Audio/Video Capture
• DOSBox GIF Animation
• References

Software Versions

Since this article involves several pieces of software, some of what is written here may not hold good in future if the behaviour of any of these software tools change in future. The list below contains the versions of all software tools that were used to test the commands provided in this article:

1. macOS High Sierra 10.13.6
2. DOSBox 0.74-3
3. FFmpeg 4.3.1
4. ImageMagick 7.0.10-28
5. IBM Personal Computer Logo Version 1.00
6. Digger (Original PC booter version by Windmill Software)

Note that both Logo and Digger programs in the list above are DOS programs that were released in 1983. They cannot be run directly on modern computers but they can be run with DOSBox since it emulates old IBM PC compatible computers.

IBM PC Logo in DOSBox

IBM Personal Computer Logo developed by Logo Computer Systems Inc. (LCSI) in 1983 was the first piece of software I got introduced to while learning computers as a kid. I came across it at the age of 8 when I was in Class 4 and our school had a 5¼-inch floppy disk with IBM PC Logo on it. As a result, Logo was the first programming language I learnt in my life. About 20 years later, I would realise that the first programming language I learnt is a dialect of Lisp. How wonderful!

If the Logo interpreter program LOGO.COM exists in the current directory, it can be run with DOSBox using the following command:

dosbox LOGO.COM

One of the things I enjoyed drawing with Logo was a grid of overlapping circles like this:

Here is the Logo source code for the above output:

REPEAT 20 [REPEAT 180 [FD 1 RT 2] RT 18]

Digger in DOSBox

At around the same time I learnt Logo, I also came across Digger, a computer game for IBM PC developed by Windmill Software in 1983.

If the Digger program DIGGER.COM exists in the directory, it can be run using DOSBox with the following command:

dosbox DIGGER.COM -c "config -set cpu cycles=500" -machine cga

The -machine cga option emulates a machine with Colour Graphics Adapter (CGA) because Digger requires a machine of this type to run correctly. The cycles=500 configuration option slows down the speed at which DOSBox emulates instructions in order to emulate the slow machines of olden days. Without this option, Digger runs too fast to be able to be conveniently playable.

Digger has an excellent gameplay where the player digs through underground tunnels to pick up emeralds, drop gold bags to release the gold or squash nobbins and hobbins, collect the released gold to earn more points and so on. It uses bright and attractive colours. The music is great. When Digger was released in 1983, it was quite advanced for its time.

DOSBox Screenshot Capture

The screenshots above were obtained by running IBM PC Logo and the original 1983 PC booter version of Digger on DOSBox and then resizing the screenshots such that their aspect ratio matches the aspect ratio of old CRT computer monitors.

To obtain the screenshots, we first press ctrl+f5 while DOSBox is running. The paths of the screenshots appear in the console output at the terminal where DOSBox was launched. For example:

Capturing Screenshot to /Users/susam/Library/Preferences/capture/logo_000.png
Capturing Screenshot to /Users/susam/Library/Preferences/capture/logo_001.png

Capturing Screenshot to /Users/susam/Library/Preferences/capture/digger_000.png
Capturing Screenshot to /Users/susam/Library/Preferences/capture/digger_001.png

The screenshots obtained in this manner have an aspect ratio of 8:5 which makes the output look stretched horizontally. The old CRT computer monitors for which these old DOS programs were written had an aspect ratio of 4:3 instead. This stretched look can be fixed by resizing the images to an aspect ratio of 4:3. Here are the commands used to fix the aspect ratio and produce the images:

convert logo_000.png -sample '1920x1440!' dosbox-logo-0.png
convert logo_001.png -sample '1920x1440!' dosbox-logo-1.png

convert digger_000.png -sample '1920x1440!' dosbox-digger-0.png
convert digger_001.png -sample '1920x1440!' dosbox-digger-1.png

The convert program comes with ImageMagick. There are a few things worth noting here:

• We use the -sample option here to resize the image as opposed to using -resize or -scale. The -resize or -scale option would smooth the jagged edges in the text and graphics by introducing additional colours. The -resize option is great for real world images where we do want the edges to be smooth while scaling up or down but in these screenshots we want to retain the crisp and jagged edges that is typical of DOSBox and the old CRT monitors. Therefore we use the -sample option that does not introduce any new colours. Instead it uses nearest-neighbour interpolation (point sampling) to decide the colours of the scaled image.
• The ! flag is used to ignore the aspect ratio of the original image. Without this flag, the output files would be 1920x1200 in size, that is, the largest size with an aspect ratio of 8:5 that fits in a 1920x1440 box. With this flag, the original aspect ratio of 8:5 is ignored and the output is exactly 1920x1440 in size.

DOSBox Video Capture

To start capturing video of DOSBox, we press ctrl+alt+f5. The same key combination stops capturing video. The following output appears in the console output to show where the video file is saved:

Capturing Video to /Users/susam/Library/Preferences/capture/logo_000.avi
Stopped capturing video.

Say, I want to share a video capture of DOSBox with Logo running on it with my friends who might be on devices that do not support playing AVI files. The following FFmpeg command converts the video to a format that can be distributed widely and played on a wide range of devices and players:

ffmpeg -i logo_000.avi -an -c:v libx264 -preset veryslow \
       -crf 17 -vf format=yuv420p,scale=1920:1440:flags=neighbor,fps=30 \
       dosbox-logo.mp4

Here is what the output looks like:

Let us briefly discuss the various FFmpeg options used here:

• -i logo_000.avi

  This, of course, specifies the input file.

• -an

  The audio is silent in this video, so we reduce the file size a little by disabling the audio stream with this option. For example, without this option the output file size was 317 KB but with this option it turned out to be 282 KB.

  This option should not be specified if the audio stream needs to preserved, for example, with DOS games that have audio. We will see an example of this in the next section.

• -c:v libx264

  This option selects the x264 encoder to encode the video stream into H.264 format. H.264 is also known as MPEG-4 Part 10, Advanced Video Coding (MPEG-4 AVC). Currently, it is the most popular format for recording, compression and distribution of video content.

• -crf 17

  This option provides visually lossless output, that is, high quality output without any loss in quality that can be perceived by human eyes. For completely lossless output, we need to use the -crf 0 option. However, this option sets the video profile to High 4:4:4 Predictive which prevents the video from playing in some video players. This issue is discussed in more detail in the point about yuv420p pixel format that comes later in this list. Since -crf 0 cannot be used due to this issue, the next best option is -crf 1 which while not completely lossless is much better than visually lossless. Since it trades quality for output size, the output file turns out to be 319 KB in size. The -crf 51 option produces the most lossy output, that is, the worst quality output with a file size of 159 KB.

• -preset veryslow

  This option provides better compression at the cost of encoding speed. For example, without this option it produces an output of size 355 KB in about 16 seconds on my system but with this option it produces an output of size 282 KB in about 31 seconds on the same system.

• -vf format=yuv420p

  This video filter option ensures that the output video file can be run in a wide range of devices and players.

  For example, without this video filter option, we get the output in the YUV 4:4:4 planar format. I found that QuickTime Player version 10.4 on macOS High Sierra as well as Android 9.0.0 was unable to play this format.

  $ ffmpeg -v quiet -i logo_000.avi -an -c:v libx264 dosbox-logo.mp4
  $ ffprobe -v error -show_entries stream=codec_name,profile,pix_fmt dosbox-logo.mp4
  [STREAM]
  codec_name=h264
  profile=High 4:4:4 Predictive
  pix_fmt=yuv444p
  [/STREAM]

  With this video filter option, we get the output in the YUV 4:2:0 planar format. Now both QuickTime Player version 10.4 as well as Android 9.0.0 could play this format.

  $ ffmpeg -v quiet -i logo_000.avi -an -c:v libx264 -vf format=yuv420p dosbox-logo.mp4
  $ ffprobe -v error -show_entries stream=codec_name,profile,pix_fmt dosbox-logo.mp4
  [STREAM]
  codec_name=h264
  profile=High
  pix_fmt=yuv420p
  [/STREAM]

  For maximum compatibility with very old or obsolete devices, we could add the -profile:v baseline option that setst the video profile to Constrained Baseline. This option is not recommended unless we really need to support old or obsolete devices. We also need to keep in mind that the baseline profile does not support lossless encoding with the -crf 0 option. The least lossy encoding option we can specify with this profile is -crf 1 which while not technically lossless is much better than visually lossless.

  $ ffmpeg -v quiet -i logo_000.avi -an -c:v libx264 -vf format=yuv420p -profile:v baseline dosbox-logo.mp4
  $ ffprobe -v error -show_entries stream=codec_name,profile,pix_fmt dosbox-logo.mp4
  [STREAM]
  codec_name=h264
  profile=Constrained Baseline
  pix_fmt=yuv420p
  [/STREAM]

• scale=1920:1440:flags=neighbor

  With this video filter option, we resize the video to maintain an aspect ratio of 4:3, that is, the aspect ratio of the old CRT computer monitors, so that the output looks similar to how it used to look on those monitors.

  The neighbor flag ensures that the nearest-neighbor interpolation (point sampling) is used to decide the colours of the scaled image. Without this option, the default bicubic interpolation algorithm is used. It has the effect of smoothing the edges by introducing new colours such as new shades of grey for this example video. While such smoothing of edges is good for scaling pictures of the real world, in this case, it spoils the crisp and jagged edges that is typical of output visible in DOSBox or the old CRT monitors. With the neighbor option, we retain the crisp and jagged edges visible in the original video capture.

• fps=30

  This video filter option sets the frame rate to 30 frames per second (FPS). Without this option, the output video has a frame rate of 70.09 FPS and file size of 558 KB. With this option the output frame rate is 30 FPS and the file size is 282 KB.

  The default value of machine configuration variable of DOSBox v0.74-3 is svga_s3, so by default it emulates a machine with SVGA card. While emulating a machine with SVGA card, DOSBox creates video capture files with frame rate of 70.09 FPS. When it emulates a machine with CGA card, such as when the its machine configuration variable is set to cga or when DOSBox is run with the -machine cga option, it creates video captures files with frame rate of 59.92 FPS.

  For the Logo video capture, there is no high-speed motion going on in the video, so we don't need a high frame rate. A lower frame rate of 30 FPS looks just as good.

DOSBox Audio/Video Capture

The video capture of Digger game is processed similarly, however, there are a few additional things we need to take care of. We want to include the game audio in the output file. We also want a higher frame rate because games may sometimes have high-speed motion.

Like before, we use ctrl+alt+f5 to start capturing the video. The same key combination stops capturing video. The following output appears in the console output to show where the video file is saved:

Capturing Video to /Users/susam/Library/Preferences/capture/digger_000.avi
Stopped capturing video.

Here is the command to convert the video capture of Digger to a distributable format:

ffmpeg -i digger_000.avi -c:a aac -b:a 256k -c:v libx264 -preset veryslow \
       -crf 17 -vf format=yuv420p,scale=1920:1440:flags=neighbor,fps=50 \
       dosbox-digger.mp4

Here is the output:

Most of the FFmpeg options used in the command above have been discussed in the previous section. Let us discuss the new options used here that have not been discussed earlier:

• -c:a aac

  This option selects the native FFmpeg AAC encoder to encode the audio stream to Advanced Audio Coding (AAC) format. It is a very popular format for audio streams in MP4 files.

• -b:a 256k

  This sets the audio bitrate high enough to ensure that we get good quality audio in the output. We don't need to worry about our specified bitrate being too high. If the audio can be encoded with a lower bitrate without compromising on quality, the output audio stream is encoded at a lower bitrate. For example, for this specific video, the actual audio bitrate in the output file turns out to be 245k because that is enough to encode the audio stream in the input file.

  $ ffprobe -v error -select_streams a -show_entries stream=bit_rate dosbox-digger.mp4
  [STREAM]
  bit_rate=245184
  [/STREAM]

• fps=50

  If we set the frame rate to a lower value like 30 FPS like we did in the previous section, we still get pretty good output, however, certain parts of the output video look slightly choppy. For example, at 7 seconds into the video when the player is the pushing up against the gold bag, the video becomes slightly choppy if we generate the output with a frame rate of 30 FPS. A higher frame rate such as 50 FPS prevents this problem.

  If we omit this option entirely, we get an output video that has the same frame rate as that of the input video, that is, 59.92 FPS, with an output file size of 4.6 MB. With this option, we get an output video that has a frame rate of 50 FPS and a file size of 4.2 MB.

If we look at the output video above closely enough, we see that the colours don't look as crisp as they do in the Digger game screenshot. The neighbor flag was very effective at maintaining the crisp and jagged edges in the Logo video capture but it does not produce perfect results for the Digger video capture in this section. Despite the imperfection, it is still necessary to specify the neighbor option because without this option, the output video looks even worse. We can use a different pixel format like yuv444p instead of yuv420p to work around this issue. Using the yuv444p format indeed results in perfect nearest-neighbour interpolation which helps in retaining the crisp and jagged edges in the video accurately but as explained in the previous section, many media players currently cannot play this pixel format, so we stick to using the yuv420p format in this article.

DOSBox GIF Animation

Now just for fun, let us see if we can convert the video captures into GIF animations. This can be done quite easily with FFmpeg. Here are the commands to convert the Logo video capture to GIF animation:

ffmpeg -i logo_000.avi -vf palettegen palette.png
ffmpeg -i logo_000.avi -i palette.png \
       -lavfi 'scale=1920:1440:flags=neighbor,paletteuse,fps=30' \
       dosbox-logo.gif

The first command generates a colour palette from the video capture. The second command uses this colour palette to generate a GIF animation. Like before, we use the neighbor flag to retain the crisp and jagged edges. Here is the output:

Here are the commands to convert the Digger video capture to GIF animation:

ffmpeg -i digger_000.avi -vf palettegen palette.png
ffmpeg -i digger_000.avi -i palette.png \
       -lavfi 'scale=1920:1440:flags=neighbor,paletteuse,fps=50' \
       dosbox-digger.gif

References

Here is a bunch of references that contains more details about the commands used in this article:

• DOSBox Special Keys
• ImageMagick Examples: Resize or Scaling
• ImageMagick Examples: Resampling Filters
• FFmpeg H.264 Video Encoding Guide
• FFmpeg Scaling Guide
• FFmpeg Scaler Documentation
• FFmpeg Filters Documentation

Read on website | #dos | #technology | #how-to

Fifteen years ago, writing Lisp code in Vim was an odd adventure. There were no good plugins for Vim that assisted in structured editing of Lisp s-expressions or allowed interactive programming by embedding a Lisp Read-Eval-Print-Loop (REPL) or a debugger within the editor. The situation has improved a lot since then. In the last ten years, we have seen active development of two Vim plugins named Slimv and Vlime. Slimv is over 10 years old now. Vlime is more recent and less than 3 years old right now. Both support interactive programming in Lisp.

I am going to discuss and compare both Slimv and Vlime in this article. I will show how to get started with both plugins and introduce some of their basic features. I will not cover everything though. This is not a tutorial. For tutorials, see the References section.

If you are looking only for a comparison of the two plugins or a quick recommendation, jump directly to the Comparison of Slimv and Vlime section or the Quick Recommendation section.

Contents

• Introduction
• Background
  • Lisp
  • Emacs: SLIME
  • Vim: Slimv/Vlime
• Vim Plugin Management
• Software Versions
• Get Started
  • Get Started with Slimv and SBCL
  • Get Started with Vlime and SBCL
  • Get Started with Paredit
• Use Debugger and Inspector
  • Use Debugger and Inspector with Slimv
  • Use Debugger and Inspector with Vlime
• Trace Function
  • Trace Function in Slimv
  • Trace Function in Vlime
• Nifty Features
  • Evaluate Top-Level Form
  • Rainbow Parentheses
  • Argument List
  • Omni-Completion
  • Describe Symbol
  • Expand Macro
  • Cross Reference
• Other Common Lisp Implementations
  • Use Slimv with CLISP
  • Use Slimv with ECL
  • Use Vlime with CLISP
  • Use Vlime with ECL
• Other Lisp Dialects
  • Use Slimv with MIT/GNU Scheme
  • Use Slimv with Clojure
• Comparison of Slimv and Vlime
• Quick Recommendation
• Disclosure
• References

Background

Before we get started with Slimv and Vlime, it would be nice to take a brief look at the heritage behind these plugins. These plugins provide Lisp development environments for Vim, so their story begins with Lisp.

Lisp

Lisp is a family of programming languages with a distinctive, fully parenthesised prefix notation. It is quite unlike most of the other popular programming languages today like C, Python, Ruby, etc. Its homoiconic nature and its powerful macro system that can transform arbitrary Lisp expressions make it such a flexible, versatile, extensible and introspective language that articles describing Lisp often have the word "enlightenment" in them. For example, see the following articles:

• Beating the Averages (by Paul Graham)
• The Nature of Lisp (by Slava Akhmechet)
• How Lisp Became God's Own Programming Language (by Sinclair Target)

Lisp has been described in various ways by various eminent personalities in the history of computing. Alan Kay has famously described Lisp as:

The greatest single programming language ever designed.

John Foderaro has written this about Lisp:

Lisp is a programmable programming language.

Eric S. Raymond has expressed the enlightenment one experiences by learning Lisp in his famous article titled How To Become A Hacker:

Lisp is worth learning for the profound enlightenment experience you will have when you finally get it. That experience will make you a better programmer for the rest of your days, even if you never actually use Lisp itself a lot.

Randall Munroe, the creator of the XKCD webcomic has dedicated two comic strips to Lisp:

Developed in 1958 by John McCarthy, Lisp is the second oldest programming language in use today. Only Fortran is older, by one year. Some of the popular Lisp dialects today include Common Lisp, Scheme and Clojure. Most of this article would focus on Common Lisp. Scheme and Clojure would be discussed briefly towards the end of this article.

Emacs: SLIME

Many Lisp programmers immediately think of Emacs when they think of writing Lisp code. Emacs is a family of text editors. An Emacs editor itself is typically implemented in a dialect of Lisp. There is an Emacs mode named SLIME that provides excellent support for programming in Lisp. SLIME stands for Superior Lisp Interaction Mode for Emacs. First released in August 2003, SLIME was created by Eric Marsden and then later developed further by Luke Gorrie and Helmut Eller. It offers a Read-Eval-Print-Loop (REPL), integrated debugging and interactive evaluation of expressions, all available right within the editor. There are several nifty key bindings available to compile and evaluate parts or whole of the code in the current buffer.

SLIME works by launching a Swank TCP server. Swank is a backend server program written in Common Lisp that listens on a socket to receive SLIME commands from Emacs and execute them. SLIME is so useful that it is considered to be indispensible by many Lisp programmers who write Lisp code in Emacs.

Vim: Slimv/Vlime

Is there anything similar to SLIME for Vim? Yes, there are two popular options:

• Slimv: It stands for Superior Lisp Interaction Mode for Vim. It is a Vim plugin created by Tamas Kovacs that was first released in January 2009.

• Vlime: It is a Vim plugin created by Kay Z that was first released in May 2017. It is much more recent than Slimv. Vlime is younger than Slimv by eight years.

Both plugins use a client-server architecture like SLIME does in Emacs. Both plugins rely on Swank server to be started. In fact, Slimv bundles a slightly modified version of Swank with it, so that it can launch it and connect to it to send expressions to be evaluated. Vlime does not bundle Swank server with itself but it provides a wrapper that automatically downloads Swank server when needed.

Vim Plugin Management

When I started using Vim fifteen years ago, we used to just download a Vim plugin and copy/extract it to ~/.vim directory. These days, there are a few plugin management tools for Vim such as Pathogen, Vundle, vim-plug, etc. I am not going to use any of them because I don't know which one of them you use and I don't want to write down steps for each one of them.

In fact, I have never used any Vim plugin manager myself. Until Vim 7, I used to create a ~/.vim/bundle directory, then copy each plugin to its own directory within it and add the plugin's directory path to Vim's runtimepath option.

Vim 8 has native support for packages which makes installing plugins and loading them simpler. With Vim 8, we can copy each plugin to its own directory within ~/.vim/pack/plugins/start and they are loaded automatically when Vim starts. The directory name plugins in this path is only an example. It could be any arbitrary name and Vim would still load the plugins fine.

In this article, I will use Vim 8's native support for packages to set up Vim plugins. The only exception to this would be installing Vlime. The top-level directory of Vlime is not the plugin directory. The plugin directory is contained in a subdirectory named vim. This does not conform to the directory structure of plugins in a Vim package. Therefore, in this article, I will set up Vlime in the old fashioned way by copying it to ~/.vim/bundle and then adding the path to its plugin directory to Vim's runtimepath option.

Software Versions

Since this article involves several layers of software, some of what is written here may not hold good in future as these various pieces of software change and evolve over time. Therefore, in this section, I will note down the versions of various software tools I used while writing this article. Here they are:

• Debian GNU/Linux 10.1 (buster)
• Vim 8.1
• Slimv (Git repo last updated on 30 Nov 2019 with commit 47a0070)
• Vlime (Git repo last updated on 16 Oct 2017 with commit 065b95f)
• Paredit (Git repo last updated on 30 Nov 2019 with commit d99905a)
• Rainbow Parentheses (Git repo last updated on 29 Oct 2015 with commit 27e7cd7)
• SBCL 1.4.16.debian
• GNU CLISP 2.49.92
• ECL 16.1.3
• MIT/GNU Scheme 9.1.1 on Debian GNU/Linux 9.11 (stretch)
• Clojure 1.10.1
• Quicklisp beta (libraries last updated on 30 Nov 2019)
• tmux 2.8

You will probably need only a tiny subset of the tools above depending on which sections in this article you would follow. Just pick the sections you want to try out and follow the steps written in them. They will walk you through the procedure to install the tools applicable to the sections you have picked. Regardless of which sections you pick, I recommend that you definitely go through the three "Get Started" subsections below. These subsections go into detail about some of the prerequisites such as support for tmux, Paredit, support for Python interface in Vim, etc. that are not covered in the other sections.

The choice of Debian may look like an odd one. I want the commands and steps discussed in this article to be easily reproducible in a free and open source operating system. Debian happens to be my favourite. What works on Debian is easily reproducible on Ubuntu and other derivatives, often without any changes to the steps. I believe, it will not be too difficult to translate the steps provided for Debian to the steps that would work on another operating system.

Note that Quicklisp (a prerequisite for Vlime) is still beta software at the time of writing this article. The actual steps to install Quicklisp may change in future. Check https://www.quicklisp.org/ for the most up-to-date instructions to install Quicklisp.

Get Started

Get Started with Slimv and SBCL

Here are the steps to set up Slimv and use it:

1. Install the tools required to set up a Common Lisp development environment with Slimv with this command:

   sudo apt-get install vim-nox sbcl tmux git

   The default Vim in Debian is vim.basic provided by the vim package which does not have support for Python interface. Slimv is written in Vim script, Lisp and Python 3, so it does need a Vim package that has support for Python interface. One such package is vim-nox that provides the vim.nox command. Installing it automatically updates the vim command to run vim.nox. Another such package is vim-gtk which additionally provides GUI support. The graphical Vim known as GVim can be launched with the gvim command. It runs in the desktop environment. For the purpose of this article, I will stick to vim-nox because it is lightweight. All steps meant for Slimv would run equally well on vim-gtk, MacVim and GVim.

   Installing tmux is optional. Slimv can launch Swank server automatically if Vim is running within tmux, GNU Screen or a desktop environment, so if you are using GNU Screen already, you don't need to install tmux. Also, if you are running Slimv in a desktop environment, you don't really need to install either tmux or GNU Screen, although you could if you would like to see Swank running in a separate tmux or GNU Screen window rather than a separate terminal window. In this article, I am going to assume that Vim is running within tmux.

   If you are going to run Slimv in a terminal without a desktop environment, tmux or GNU Screen, Swank server has to be run manually. Point 4 below explains how to do it.

2. Installing Slimv is pretty simple. Here is one way to do it:

   git clone https://github.com/kovisoft/slimv.git ~/.vim/pack/plugins/start/slimv
   vim +'helptags ~/.vim/pack/plugins/start/slimv/doc' +q

   That is it! Slimv is set up. It's that straightforward. The commands above show how to set up Slimv with just two shell commands. You could also use a Vim plugin manager to install Slimv for you but I am not going to cover that here.

3. This is an optional step. Slimv supports starting Swank server automatically if you are running Vim in tmux, GNU Screen or a desktop environment. To start tmux, enter this command:

   tmux

   If you use GNU screen or a desktop environment, you don't have to run tmux.

   If you do not use tmux, GNU Screen or a desktop environment, then you must start Swank server manually as explained in the next point.

4. This step is necessary only if you are not using tmux, GNU Screen or a desktop environment. The following command shows how to start Swank server manually:

   sbcl --load ~/.vim/pack/plugins/start/slimv/slime/start-swank.lisp

   If you are using tmux, GNU Screen or a desktop environment, Slimv can start Swank server automatically when needed and you don't need to perform this step.

5. Create a new Lisp source code file, say, foo.lisp with this command:

   vim foo.lisp

6. To connect to Swank server, enter the following command in normal mode:

   ,c

   If Vim is running within tmux, GNU Screen or desktop environment, Slimv would automatically launch Swank server and connect to it.

   After Slimv connects to Swank successfully, Vim window should split into two and the following prompt should appear in the new split window:

   CL-USER>

   This is the integrated REPL. It is now alive and ready for interactive programming.

   We assume here that Slimv is using the default Slimv leader key ,. If you have overridden the Vim leader key, then the Slimv leader key might be same as the Vim leader key. Enter the command :echo g:slimv_leader in Vim command-line mode to find the leader key being used by Slimv.

7. Type some code into the buffer for the new file. To do so, first type i to enter insert mode and type this code:

   (format t "hello, world~%")

   Type esc to return to normal mode.

8. To evaluate the current expression under the cursor, enter the following command in normal mode:

   ,e

   Both the current expression and its result should appear in the REPL window.

9. The REPL is interactive. Type ctrl+ww to switch to the REPL window. Then type i to enter insert mode and type this code:

   (+ 1 2)

   Type enter to evaluate the expression just like you would do in a real REPL. The result should then appear in the REPL.

   Type esc to return to normal mode again. Use the normal mode command ctrl + w w to switch between the split windows.

10. Now that you have got started with Slimv, here is a brief note on uninstallation, in case you ever need it. If Slimv is installed as described in point 2 above, enter the following command to uninstall it:

    rm -rf ~/.vim/pack/plugins/start/slimv

In steps 7 and 9, you may have noticed that as soon as you type an opening parenthesis or double quotation mark, a matching closing one is automatically inserted. That is done by the Paredit plugin which is bundled along with Slimv. Paredit ensures structured editing of Lisp s-expressions and keeps all matched characters (parentheses, brackets, braces, quotes) balanced. It also provides many new keybindings to edit s-expressions conveniently. We will look into Paredit in a little more detail in the Get Started with Paredit subsection later.

Get Started with Vlime and SBCL

Here are the steps to set up Vlime and use it:

1. Install the tools required to set up a Common Lisp development environment with Vlime with this command:

   sudo apt-get install vim sbcl git curl

   Note that unlike Slimv, Vlime can work with the default Vim in Debian, i.e. vim.basic. Vlime does not require Vim with Python interface.

2. Install Quicklisp with these commands:

   curl -O https://beta.quicklisp.org/quicklisp.lisp
   sbcl --load quicklisp.lisp --eval '(quicklisp-quickstart:install)' --eval '(exit)'
   sbcl --load ~/quicklisp/setup.lisp --eval '(ql:add-to-init-file)' --eval '(exit)'

   Type enter in the end, when prompted, to complete the installation.

3. Install Vlime and Paredit with these commands:

   git clone https://github.com/l04m33/vlime.git ~/.vim/bundle/vlime
   git clone https://github.com/kovisoft/paredit ~/.vim/pack/plugins/start/paredit
   echo 'set runtimepath^=~/.vim/bundle/vlime/vim' >> ~/.vimrc
   vim +'helptags ~/.vim/bundle/vlime/vim/doc' +'helptags ~/.vim/pack/plugins/start/paredit/doc' +q

   Unlike Slimv, Vlime does not bundle Paredit along with itself. As explained in the previous section, it helps us with structured editing of Lisp s-expressions.

   I recommend that you install Paredit but in case you choose not to, ensure that loading of filetype plugins is enabled by entering the :filetype command in command-line mode. The output should contain plugin:ON. If it is off, add the command filetype plugin on to ~/.vimrc to ensure that this is always on. Vlime won't work without this being enabled. If you install Paredit, you don't have to bother about this because Paredit takes care of enabling this by default.

4. Create a new Lisp source code file, say, foo.lisp with this command:

   vim foo.lisp

5. To start Vlime server (a wrapper around Swank server) and connect to it automatically, enter the following command in normal mode:

   \rr

   We assume here that Vim <LocalLeader> is left to its default, i.e. backslash. If it is mapped to some other key combination, then that must be used instead of backslash in the above command.

   The first time this command is run after installing Vlime, it installs Swank server using Quicklisp. Therefore, it can take a while for Vlime server to start the first time this command is run. On subsequent use of these commands, it would start faster because it would be already installed.

   The console output from Vlime server is displayed in a split window. After Vlime successfully connects to Swank, the following message is displayed at the bottom:

   Vlime Connection 1 established.

   After the above message appears, it is okay to close the split window for Vlime server by entering this command in Vim command-line mode:

   :q

   Vlime server would continue to run in background. The following command can be used in normal mode to view the console output of Vlime server anytime it is required:

   \rv

6. Type some code into the buffer for the new file. To do so, first type i to enter insert mode and type this code:

   (format t "hello, world~%")

   Type esc to return to normal mode.

7. To evaluate, the current expression under the cursor, enter the following command in normal mode:

   \ss

   Both the current expression and its result should appear in the REPL window.

   Unlike Slimv, the REPL window of Vlime is not interactive. Its nomodifiable option is set, so we cannot type code directly into the REPL window. This can be a bit of a problem if we want to type arbitrary expressions into the REPL and execute them. To mitigate this shortcoming to some extent, Vlime provides an alternative way to evaluate the current expression known as the interaction mode. This is explained in the next point.

8. Enable interaction mode by entering this command in normal mode:

   \i

   The same command disables interaction mode, i.e. this command toggles the state of interaction mode between on and off. When interaction mode is on, evaluate an expression under the cursor by simply pressing enter in normal mode.

9. Now that you have got started with Vlime, here is a brief note on uninstallation, in case you ever need it. If Quicklisp and Vlime are installed as described in the points 2 and 3 above, run these commands to uninstall them:

   rm -rf ~/quicklisp ~/.vim/bundle/vlime ~/.vim/pack/plugins/start/paredit
   sed -i.bkp '/runtimepath.*vlime/d' ~/.vimrc

   Optionally, remove ~/.sblrc or edit it to remove the code pertaining to loading quicklisp/setup.lisp.

Get Started with Paredit

You have already got started with Paredit when you wrote Lisp code while following one of the previous two subsections. The moment you typed an opening parenthesis, Paredit inserted a closing one for you automatically. Paredit keeps all matched characters such as parentheses, double quotes, etc. balanced when you edit code. Here is a very brief exercise to quickly get started with some of the very basic features of Paredit:

1. Create a new Lisp source code file, say, foo.lisp with this command:

   vim foo.lisp

2. Type i to enter insert mode and then type only this:

   (defun square (x

   At this point, Paredit should have inserted the two closing parentheses automatically. The code should look like this:

   (defun square (x))

   The cursor should be situated just after the parameter x. The block above shows where the cursor should be.

3. While you are still in insert mode, type the first closing parenthesis. Yes, type it even if the closing parenthesis is already present. The cursor should now skip over the first closing parenthesis like this:

   (defun square (x))

   Of course, there was no need to type the closing parenthesis because it was already present but typing it out to skip over it is more efficient than escaping to normal mode, then moving over it and then entering insert mode again. This is, in fact, a very nifty feature of Paredit. We can enter code with the same keystrokes as we would without Paredit.

4. You should still be in insert mode. Type enter to create a new line below. Now one of two things is going to happen. If electric return is disabled, then a newline is inserted as expected like this:

   (defun square (x))
     )

   If electric return is enabled, two newlines are inserted to create an empty line in between:

   (defun square (x)
      
     )

   In both cases, indentation of two spaces is inserted automatically. The new empty line inserted by electric return allows linewise editing of the code to be entered in this empty line.

   The electric return feature is enabled by default in both Paredit and Slimv. It works by Paredit remapping the "enter" key (<CR>) in insert mode to a function that inserts electric return. Slimv needs to remap the "enter" key to present the argument list of the current function but it takes care of performing an electric return before showing the argument list. Vlime, however, forgets to perform electric return before showing the argument list, so this feature does not work in Vlime.

   For now, we will continue with the assumption that electric return is enabled and working fine. If it is disabled or if it is not working for you, ignore the steps that discuss electric return.

5. Now, type only this:

   (* x x

   Again, Paredit would have inserted the closing parenthesis automatically. The code should look like this now:

   (defun square (x)
     (* x x)
     )

6. Now, type one more closing parenthesis to advance past the automatically inserted closing parenthesis like this:

   (defun square (x)
     (* x x) 
     )

7. Then type another closing parenthesis. Paredit would now pick the lone closing parenthesis that is present in its own line and move it at the end of the current line like this:

   (defun square (x)
     (* x x)) 

   This behaviour of consuming the extra newline inserted by an electric return on typing a closing parenthesis helps the code to conform to the popular Lisp coding convention of putting all the consecutive closing parentheses next to each other in the same line. In other words, typing closing parentheses re-gathers electric returns when applicable.

8. Let us see what happens if we try to delete the opening parenthesis around the product function (the * function). Type esc to return to normal mode. Then enter h in normal mode to move the cursor one place left so that the cursor is placed on the parenthesis just after the last x in the code like this:

   (defun square (x)
     (* x x))

   Type x to delete the closing parenthesis the cursor is on. Nothing gets deleted! Instead the cursor just skips over the parenthesis like this:

   (defun square (x)
     (* x x))

   Paredit refuses to delete the closing parenthesis because it encloses a non-empty list. It would have deleted the closing parenthesis along with the opening one if the list were empty. This is Paredit trying to ensure that the s-expressions remain valid while editing.

   Note that in this step, h is a regular Vim motion command. In Vim, by default, x deletes the character under the cursor, but when Paredit is enabled, it remaps this command to behave the way it did in this step to ensure that the parentheses remain balanced.

9. Let us now try to delete the current line. Type dd to do so. The result looks like this:

   (defun square (x)
     )

   Note how the closing parenthesis has been left intact to keep the parentheses balanced. Again, Paredit has remapped the dd command to produce this behaviour.

10. Now type da( to delete the entire defun expression. The buffer should look empty now.

11. Type i to enter insert mode and type out the following code:

    (list (* 10 20) (+ 30 40)) 

12. Type esc to return to normal mode. Type h(hh to place the cursor on the closing parenthesis of the first expression.

    (list (* 10 20) (+ 30 40))

    Now type ,>. The closing parenthesis of the first expression moves right to slurp the next expression. The buffer looks like this:

    (list (* 10 20 (+ 30 40)))

    We assume here that Paredit is using the default Paredit leader key ,. If you have overridden the Vim leader key, then the Paredit leader key might be same as the Vim leader key. Enter the command :echo g:paredit_leader in Vim command-line mode to find the leader key being used by Paredit.

13. Now type ,<. The closing parenthesis of the outer expression moves left to barf the inner expression out. The buffer looks like this again:

    (list (* 10 20) (+ 30 40))

    While the cursor is on a parenthesis, the normal mode commands ,< or ,> can be used in this manner to move the parenthesis left or right respectively, thereby slurping or barfing expressions.

That was a very brief overview of what Paredit can do. There is a lot more to Paredit than what is described above. Paredit has a rich set of keybindings to make editing s-expressions very convenient. Enter :help paredit-keys in command-line mode to see the list of the keybindings.

I think it is a good idea to read the entire Paredit documentation. Enter :help paredit to do so. It is about 500 lines long and takes about 30 to 40 minutes to read. The time spent reading this documentation is worth it because it makes editing Lisp code very pleasant and productive.

Use Debugger and Inspector

After getting started with Slimv or Vlime, the very next thing you might want to know is how to work with the debugger. The debugger window comes up whenever an error or an unhandled condition occurs. It might look quite scary to a beginner, so it is a good idea to become comfortable with it as soon as possible. Fortunately, both Slimv and Vlime provide excellent key-bindings to inspect the error or dismiss it to return to the source code buffer quickly and easily.

Use Debugger and Inspector with Slimv

The following steps trigger an error and then show how to work with the debugger in Slimv:

1. Create a file with Vim, say foo.lisp and enter the following code into it:

   (defun square (x)
     (* x x))
   
   (square "foo")

2. Enter ,b to evaluate the buffer.

3. As soon as the defective form (square "foo") gets evaluated, an error occurs. The error, possible restarts and the backtrace is displayed in a new split window for SLDB. SLDB stands for Slime Debugger. Here is an example of what may appear in the SLDB window:

   The value
     "foo"
   is not of type
     NUMBER
   when binding SB-KERNEL::X
      [Condition of type TYPE-ERROR]
   
   Restarts:
     0: [RETRY] Retry SLIME REPL evaluation request.
     1: [*ABORT] Return to SLIME's top level.
     2: [ABORT] abort thread (#<THREAD "repl-thread" RUNNING {1003274E23}>)
   
   Backtrace:
     0: (SB-KERNEL:TWO-ARG-* "foo" "foo") [external]
     1: (SQUARE "foo")
     2: (SB-DEBUG::TRACE-CALL #<SB-DEBUG::TRACE-INFO SQUARE> #<FUNCTION SQUARE> "foo")
     3: (SB-INT:SIMPLE-EVAL-IN-LEXENV (SQUARE "foo") #<NULL-LEXENV>)
     4: (EVAL (SQUARE "foo"))
     5: (SWANK::EVAL-REGION "(defun square (x) ..)
     6: ((LAMBDA NIL :IN SWANK-REPL::REPL-EVAL))
     ...

   The ellipsis in the end is added by me to denote that the actual output has been truncated in this article for the sake of brevity.

4. In the SLDB window, move the cursor to the second line of backtrace, i.e. on the following line:

     1: (SQUARE "foo")

   Then type enter. This line should now unfold to show the following details:

     1: (SQUARE "foo")
         in "(SB-INT:NAMED-LAMBDA SQUARE-----------------------------------
       Locals:
         X = "foo"

5. Move the cursor to the line that begins with in, i.e. on this line:

         in "(SB-INT:NAMED-LAMBDA SQUARE-----------------------------------

   Then type enter. Some information about its source code should appear like this:

     1: (SQUARE "foo")
         in "(SB-INT:NAMED-LAMBDA SQUARE
       (X)
     (BLOCK SQUARE (#:***HERE*** (* X X))))" byte 1
       Locals:
         X = "foo"

6. Move the cursor to the following line:

         X = "foo"

   Then type ,i to inspect this variable. A prompt would appear to confirm the variable name. Type enter to confirm. An inspector window should now appear with more details about this variable. This window should look like this:

   Inspecting #<(SIMPLE-ARRAY CHARACTER (3)) {100478AFAF}>
   --------------------
   Press <F1> for Help
   
   Dimensions: (3)
   Element type: CHARACTER
   Total size: 3
   Adjustable: NIL
   Fill pointer: NIL
   Contents:
   0: #\f
   1: #\o
   2: #\o
   
   
   [<<] Exit Inspector

7. Type enter to inspect any object under the cursor and drill down further.

8. Type backspace in normal mode to return to the previous object.

9. Enter ,q in normal mode to quit the inspector.

10. Finally, move the cursor to the following line in the SLDB window:

      1: [*ABORT] Return to SLIME's top level.

    Then type enter to execute this restart. Alternatively, enter ,a in normal mode to select the abort restart and quit to the previous level or ,q to quit to top level.

Most of the times when an error occurs, I quickly take a look at the stack trace to realise that I have made a silly mistake and enter the ,q command to abort and quit to top level. This can be quite convenient because it allows returning from debugging to coding very quickly with only two keystrokes.

Use Debugger and Inspector with Vlime

The following steps trigger an error and then show how to work with the debugger in Vlime:

1. Create a file with Vim, say foo.lisp and enter the following code into it:

   (defun square (x)
     (* x x))
   
   (square "foo")

2. Save the file, connect to Vlime server and enter \of in normal mode to compile the entire buffer.

3. As soon as the defective form (square "foo") gets evaluated, an error occurs. The error, possible restarts and the backtrace is displayed in a new split window for SLDB. SLDB stands for Slime Debugger. Here is an example of what may appear in the SLDB window:

   Thread: 1; Level: 1
   
   The value
     "foo"
   is not of type
     NUMBER
   when binding SB-KERNEL::X
      [Condition of type TYPE-ERROR]
   
   Restarts:
     0. *ABORT - Return to SLIME's top level.
     1.  ABORT - abort thread (#<THREAD "worker" RUNNING {10045D6F83}>)
   
   Frames:
     0.  (SB-KERNEL:TWO-ARG-* "foo" "foo") [external]
     1.  (SQUARE "foo")
     2.  (SB-FASL::LOAD-FASL-GROUP #S(SB-FASL::FASL-INPUT :STREAM #<SB-SYS:FD-STREAM for "file /home/susam/foo.fasl" {10045E76A3}> :TABLE #(41 #<PACKAGE "SB-IMPL"> SB-IMPL::%DEFUN #<PACKAGE "COMMON-LISP-USER">..
     3.  (SB-FASL::LOAD-AS-FASL #<SB-SYS:FD-STREAM for "file /home/susam/foo.fasl" {10045E76A3}> NIL NIL)
     4.  ((FLET SB-FASL::THUNK :IN LOAD))
     5.  (SB-FASL::CALL-WITH-LOAD-BINDINGS #<CLOSURE (FLET SB-FASL::THUNK :IN LOAD) {7F7B9B0B60BB}> #<SB-SYS:FD-STREAM for "file /home/susam/foo.fasl" {10045E76A3}>)
     ...

   The ellipsis in the end is added by me to denote that the actual output has been truncated in this article for the sake of brevity.

4. In the SLDB window, move the cursor to the second line of backtrace, i.e. on the following line:

     1.  (SQUARE "foo")

   Then type d. A new split window should appear with the following details about this frame:

   Frame: 1 (Restartable)
   
   Locals:
     X: "foo"
   
   Location:
     File: /home/susam/foo.lisp
     Position: 20
     Snippet:
       (* x x))
   
       (square "foo")

5. While the cursor is on the same line as mentioned in the previous point, type i to bring up the inspector window for this frame.

6. In the inspector window, type i to enter insert mode. Enter the following variable name in insert mode:

   x

   Then type esc to return to normal mode. Then type enter. The following details about the variable x should now appear in the inspector window:

   #<(SIMPLE-ARRAY CHARACTER (3)) {1004617ABF}>
   ============================================
   
   Dimensions: (3)
   Element type: CHARACTER
   Total size: 3
   Adjustable: NIL
   Fill pointer: NIL
   Contents:
   0: #\f
   1: #\o
   2: #\o

7. Type enter to inspect any object under the cursor and drill down further.

8. Type p to return to the previous object.

9. Enter the regular Vim command :q in command-line mode to quit the inspector window.

10. Finally, move the cursor to the following line in the SLDB window:

      1: [*ABORT] Return to SLIME's top level.

    Then type enter to execute this restart. Alternatively, we can enter a in normal mode to select the abort restart to return to the previous level. At this time, there is no command to return to SLIME's top level.

Trace Function

Trace Function in Slimv

The following steps show how to get started with tracing functions in Slimv:

1. Create a file with Vim, say foo.lisp and enter the following code into it:

   (defun square (x)
     (* x x))
   
   (square (square 2))

2. Enter ,b in normal mode to evaluate the entire buffer.

3. Place the cursor on the function name, i.e. on square and enter ,t in normal mode to toggle tracing for this function. A prompt appears to confirm the function name. Type enter to confirm.

4. While the cursor is on the last expression, enter ,d in normal mode to evaluate the top-level form. The following output appears in the REPL buffer.

   (square (square 2))
     0: (SQUARE 2)
     0: SQUARE returned 4
     0: (SQUARE 4)
     0: SQUARE returned 16
   16

   This output contains information about each call to the traced function, arguments passed to it and the return values.

Trace Function in Vlime

It takes a little more work to start tracing functions in Vlime. The following steps show how to do it:

1. Add the following statement to ~/.vimrc:

   let g:vlime_contribs = ['SWANK-ASDF', 'SWANK-PACKAGE-FU',
                         \ 'SWANK-PRESENTATIONS', 'SWANK-FANCY-INSPECTOR',
                         \ 'SWANK-C-P-C', 'SWANK-ARGLISTS', 'SWANK-REPL',
                         \ 'SWANK-FUZZY', 'SWANK-TRACE-DIALOG']

   The above variable defines the list of Swank contrib modules to load while initialising a Vlime connection. All modules mentioned above except the last one are loaded by default. The SWANK-TRACE-DIALOG module is not loaded by default but this module is necessary for tracing functions, so in order to load it, we define this variable to load this module in addition to all the other modules that are loaded by default.

2. Create a file with Vim, say foo.lisp and enter the following code into it:

   (defun square (x)
     (* x x))
   
   (square (square 2))

3. Save the file, connect to Vlime server and enter \of in normal mode to compile the entire buffer.

4. Enter \TD in normal mode to show the trace dialog in a split window.

5. Enter ctrl + w w in normal mode to go back to the source code window.

6. Place the cursor on the function name, i.e. on square and enter \TT in normal mode to toggle tracing for this function.

7. While the cursor is on the last expression, enter \st in normal mode to evaluate the top-level form.

8. Enter ctrl + w w in normal mode twice to go to the trace window.

9. Under Trace Entries, place the cursor on [refresh] and type enter.

10. Then place the cursor on [fetch next batch] and type enter. Two results should appear for the two square calls that were made due to step 7. The trace information would be folded under each call.

11. Move the cursor to each fold line and enter zo in normal mode to open the fold. After opening both the folds, the following result should be visible:

    0 - COMMON-LISP-USER::SQUARE
        > 2
        < 4
    1 - COMMON-LISP-USER::SQUARE
        > 4
        < 16
        16

    The lines starting with > show the arguments and the ones starting with < show the return values.

Nifty Features

In this section, we will go over some of the nifty features that these plugins offer. Not all features will be covered here. I have chosen only a few features for the discussion here that I felt would be useful to beginners and at the same time also demonstrate the versatility of these plugins.

Evaluate Top-Level Form

In the previous sections, we saw how to evaluate the current expression under the cursor. In this section, we will see how to evaluate the top-level expression around the current cursor position. Let us do a small exercise to see this:

1. Create a file with Vim, say foo.lisp and enter the following code into it:

   (+ 1 (* 2 (/ 6 2)))

2. With Slimv or Vlime connected to Swank, let us do a quick recap of how to evaluate the current expression.

   With Slimv, enter the normal mode command ,e to evaluate the current expression.

   With Vlime, enter the normal mode command \ss to evaluate the current expression.

   The current expression, i.e. (/ 6 2) should get evaluated and the result 3 should appear in the REPL buffer.

3. Let us now see how to evaluate the top-level expression.

   With Slimv, enter the normal mode command ,d to evaluate the top-level expression.

   With Vlime, enter the normal mode command \st to evaluate the top-level expression.

   The top-level expression should get evaluated and the result 7 should appear in the REPL buffer.

Rainbow Parentheses

Rainbow parentheses make it easy to see matching parentheses by colouring different levels of parentheses with different colours. Matching parentheses have the same colour. To enable this feature in Slimv, add this command to ~/.vimrc:

let g:lisp_rainbow=1

This feature is not available in Vlime. But there are several Vim plugins that support rainbow parentheses. Here are the steps to install one such plugin that is quite popular:

git clone https://github.com/junegunn/rainbow_parentheses.vim.git ~/.vim/pack/plugins/start/rainbow_parentheses
echo 'autocmd FileType lisp,scheme,clojure RainbowParentheses' >> ~/.vimrc

In case you ever want to uninstall it, enter these commands:

rm -rf ~/.vim/pack/plugins/start/rainbow_parentheses
sed -i.bkp '/autocmd.*RainbowParentheses/d' ~/.vimrc

Argument List

You must have seen this feature already while trying out the sections earlier. While editing a Lisp source file, after typing a function name, as soon as a space is typed or the enter key is typed, the argument list for the function appears to serve as a reference. In Slimv, the argument list appears in the status line at the bottom. In Vlime, the argument list appears in a split window at the top.

Omni-Completion

Type a function name partially, e.g. form and type tab while still in insert mode. The omni-completion menu should appear with the list of completions if there are multiple choices. Type ctrl+n to select the next choice and ctrl+p to select the previous choice. Selecting a choice also immediately inserts that choice in the buffer. This works in both Slimv and Vlime. In Slimv, we can also type tab to select the next choice.

By default, omni-completion is fuzzy. For example, type wl and type tab and omni-complete should insert write-line automatically as well as show other matching choices.

Describe Symbol

With Slimv, enter the normal mode command ,s to describe the symbol under the cursor. This brings up the documentation of the symbol in the Vim message area. This feature works while editing Common Lisp and Clojure source files but not while editing Scheme source file. This feature is not supported for Scheme at this time. See the Other Lisp Dialects section for details on how to set up Slimv with Clojure and MIT/GNU Scheme.

With Vlime, enter \da in normal mode to describe the symbol under the cursor. This brings up the documentation of the symbol in a split window.

Expand Macro

Here is an excercise that shows how to expand macros interactively while editing a Lisp source file:

1. Create a file with Vim, say foo.lisp and enter the following code into it:

   (defmacro calc (a op b)
     (list op a b))
   
   (defmacro square (x)
     (list 'calc x '* x))
   
   (square 2)

2. With Slimv, enter ,b in normal mode to evaluate the entire buffer.

   With Vlime, save the file, connect to Vlime server and type \of in normal mode to compile the entire buffer.

3. With Slimv, while the cursor is on the last expression, enter ,1 in normal mode to expand the macro form once.

   With Vlime, enter \m1 in normal mode to do the same thing.

   The following expansion should appear as the result:

   (CALC 2 * 2)

   Slimv displays the expansion in the REPL buffer whereas Vlime displays it in a new split window.

4. With Slimv, enter ,m in normal mode to recursively expand the current expression until it is no longer a macro.

   With Vlime, enter \ma in normal mode to do the same thing.

   The following expansion should appear as the result:

   (* 2 2)

Cross Reference

Here is an exercise that shows how to use the cross-reference commands in Slimv and Vlime:

1. Create a file with Vim, say foo.lisp and enter the following code into it:

   (defun square (x)
     (* x x))
   
   (defun square-of-sum (x y)
     (square (+ x y)))
   
   (defun sum-of-squares (x y)
     (+ (square x) (square y)))
   
   (square-of-sum 2 3)
   (sum-of-squares 2 3)

2. With Slimv, enter ,b in normal mode to evaluate the entire buffer.

   With Vlime, save the file, connect to Vlime server and type \of in normal mode to compile the entire buffer.

3. With Slimv, place the cursor on any occurrence of the symbol square and enter ,xl in normal mode. A prompt would appear to confirm the symbol name. Type enter to confirm. The list of all callers should now appear in the REPL buffer.

   With Vlime, place the cursor on any occurrence of the symbol square and enter \xc in normal mode to list all callers of the function. The output appears in a split window containing the cross reference (xref) buffer. Type enter on any item in the xref buffer and Vlime will take you directly to the referenced location.

Other Common Lisp Implementations

The previous sections used SBCL as the implementation of Common Lisp. How well do Slimv and Vlime work with other Common Lisp implementations?

I have found that both plugins are pretty well tested with SBCL. However, they may not be so well tested with other implementations. Due to the lack of sufficient testing with Common Lisp implementations other than SBCL, certain errors may occur while using other implementations. Sometimes it is possible to work around these errors and sometimes it isn't. We will see an example of this in an upcoming section when we try to start Swank server automatically using Vlime and CLISP.

For this section, I choose CLISP and Embeddable Common-Lisp (ECL) as two other implementations of Common Lisp that will be used with Slimv and Vlime. After following the upcoming subsections, you should get the hang of how to make Slimv or Vlime work with other implementations of Common Lisp.

Use Slimv with CLISP

If you have read and tried the steps in the Get Started with Slimv and SBCL section, it is going to be quite easy to use Slimv with CLISP. The steps are similar with a few minor modifications. They are explained below:

1. Uninstall SBCL and install CLISP with these commands:

   sudo apt-get remove sbcl
   sudo apt-get install clisp

2. To start Swank server manually, enter this command:

   clisp ~/.vim/pack/plugins/start/slimv/slime/start-swank.lisp

   Then edit a Lisp source file and enter the normal command ,c to connect to it and bring up the REPL window.

3. To start Swank automatically from Slimv, there is nothing more to be done. Just edit a Lisp source file and enter the normal mode command ,c. While running in GNU Screen, tmux or a desktop environment, Slimv can automatically detect CLISP and start Swank server with it.

In general, to start Swank server manually with another Common Lisp implementation, we need to figure out how to load start-swank.lisp with it.

Use Slimv with ECL

The steps to use Slimv with Embeddable Common-Lisp (ECL) are very similar too. Once again, only if we need to start Swank server manually, we need to figure out the command to do so. Otherwise, there is no other difference. Here are the steps:

1. Ensure that SBCL and CLISP are uninstalled and ECL is installed.

   sudo apt-get remove sbcl clisp
   sudo apt-get install ecl

2. To start Swank server manually, enter this command:

   ecl --load ~/.vim/pack/plugins/start/slimv/slime/start-swank.lisp

   Then edit a Lisp source file and enter the normal command ,c to connect to it and bring up the REPL window.

3. To start Swank automatically from Slimv, there is nothing more to be done. Just edit a Lisp source file and enter the normal mode command ,c. While running in GNU Screen, tmux or a desktop environment, Slimv can automatically detect CLISP and start Swank server with it.

   There is a possible timeout issue to be aware of though. ECL can take a minute or two to compile the code it loads the first time Swank server is started. However, Slimv has a default timeout period of 20 seconds, so Slimv may fail with the following error message:

   SWANK server is not running.  Press ENTER to continue.

   If this happens, just wait for ECL to complete compiling Swank server. Once it starts Swank server, enter the normal mode command ,c again and it should connect immediately.

Use Vlime with CLISP

This subsection assumes that you have already read and tried the Get Started with Vlime and SBCL section, so you are familiar with Vlime basics. Now we will see what more it takes to use Vlime with CLISP in the steps below:

1. Let us assume we want to start afresh with CLISP, i.e. we do not have previous artefacts created by SBCL. To clean up old artefacts, enter these commands:

   rm -rf ~/.sbclrc ~/quicklisp
   sudo apt-get remove sbcl

2. Install CLISP with this command:

   sudo apt-get install clisp

3. Install Quicklisp using CLISP with these commands:

   curl -O https://beta.quicklisp.org/quicklisp.lisp
   clisp -i quicklisp.lisp -x '(quicklisp-quickstart:install)'
   clisp -i ~/quicklisp/setup.lisp -x '(ql:add-to-init-file)'

   Type enter in the end, when prompted, to complete the installation.

4. Add the following code to ~/.vimrc:

   let g:vlime_cl_impl = 'clisp'
   function! VlimeBuildServerCommandFor_clisp(vlime_loader, vlime_eval)
       return ['clisp', '-i', a:vlime_loader,
                      \ '-x', a:vlime_eval,
                      \ '-repl']
   endfunction

   Unlike Slimv, automatic start of Swank server with Common Lisp implementations other than SBCL are not supported out of the box, so the above Vim script tells Vlime how to start Swank server with CLISP. The -repl option is used to work around an issue that is explained in the next point.

5. Vlime is now ready to be used with CLISP. Just edit a Lisp source file and enter the normal mode command \rr to start Swank server and connect to it automatically.

   You may see the following error in the SLIME debugger (sldb) split window:

   SOCKET-STATUS on #1=#<INPUT STRING-INPUT-STREAM> is illegal

   Despite the above error, the following message should appear at the bottom:

   Vlime Connection 1 established.

   If the above message occurs, you can ignore this error, close the debugger window as well as the console output window and continue to use Vlime normally.

   The -repl option used in the previous step ensures that the REPL starts despite this error. Without it, this step would not have succeeded. This is what I meant when I said earlier that we may need to work around certain errors while using these plugins with a Common Lisp implementation other than SBCL.

Use Vlime with ECL

Here are the steps to use Vlime with ECL:

1. Let us assume we want to start afresh with ECL, i.e. we do not have previous artefacts created by SBCL or ECL. To clean up old artefacts, enter these commands:

   rm -rf ~/.sbclrc ~/.clisprc.lisp ~/quicklisp
   sudo apt-get remove sbcl clisp

2. Install ECL with this command:

   sudo apt-get install ecl

3. Install Quicklisp using ECL with these commands:

   curl -O https://beta.quicklisp.org/quicklisp.lisp
   ecl --load quicklisp.lisp --eval '(quicklisp-quickstart:install)' --eval '(quit)'
   ecl --load ~/quicklisp/setup.lisp --eval '(ql:add-to-init-file)' --eval '(quit)'

   Type enter in the end, when prompted, to complete the installation.

4. Add the following code to ~/.vimrc:

   let g:vlime_cl_impl = 'ecl'
   function! VlimeBuildServerCommandFor_ecl(vlime_loader, vlime_eval)
       return ['ecl', '--load', a:vlime_loader,
                    \ '--eval', a:vlime_eval]
   endfunction

5. Edit a Lisp source file and enter the normal mode command \rr to start Swank server and connect to it automatically.

Other Lisp Dialects

So far, we have seen how to use Slimv or Vlime with a Common Lisp implementation. Now let us see how well these plugins work with other Lisp dialects. Vlime does not support other Lisp dialects. It supports Common Lisp only. Slimv supports two other popular dialects of Lisp: Scheme and Clojure. In the next two subsections, we see how

Use Slimv with MIT/GNU Scheme

Slimv is documented to work with MIT/GNU Scheme on Linux only. Enter :help slimv-installation in Vim to read more about it. It says the following under the "Prerequisites" section.

Lisp or Clojure or MIT/GNU Scheme (Linux only) installed.

Further, the Swank loader script for MIT/GNU Scheme named swank-mit-scheme.scm says the following in its source code comments:

You need MIT/GNU Scheme 9.2

At the time of writing this article, I have confirmed that both these requirements indeed need to be met to use Slimv with MIT/GNU Scheme. Here are the steps to use Slimv with MIT/GNU Scheme:

1. Install MIT/GNU Scheme with this command:

   sudo apt-get mit-scheme

   Ensure that vim-nox, tmux and Slimv are installed as explained in the Get Started with Slimv and SBCL subsection earlier.

2. This is an optional step. To start Swank server automatically from Slimv, run Vim in tmux, GNU Screen or a desktop environment. In this article, we use tmux, so start tmux with this command:

   tmux

3. This step is necessary only if you are not using tmux, GNU Screen or a desktop environment. In such a case, enter this command to start Swank server manually:

   scheme --load ~/.vim/pack/plugins/start/slimv/slime/contrib/swank-mit-scheme.scm

4. Create a new Scheme source code file, say, foo.scm with this command:

   vim foo.scm

5. To connect to Swank server, enter the following command in Vim normal mode:

   ,c

6. Type some code into the buffer for the new file. To do so, first type i to enter insert mode and type this code:

   (display "hello, world\n")

   Type esc to return to normal mode. To evaluate, the current expression under the cursor, enter the following command in normal mode:

   ,e

   Both the current expression and its result should appear in the REPL window.

I have confirmed that the steps above work fine with MIT/GNU Scheme 9.1.1 on Debian GNU/Linux 9.11 (stretch). Like I mentioned before, Slimv requires Linux to work with MIT/GNU Scheme. For example, trying to start Swank server with MIT/GNU Scheme 9.2 on macOS High Sierra 10.13.6 fails with this error:

; /usr/local/Cellar/mit-scheme/9.2_2/lib/mit-scheme-c/include/config.h:879:10:
fatal error: 'sys/types.h' file not found

Further, the version of MIT/GNU Scheme really needs to be 9.x. For example, when I try to start Swank with MIT/GNU Scheme 10.1.5 on Debian GNU/Linux 10.1 (buster), the following error occurs:

;The object #[package 12 (user)], passed as an argument to ->environment, is not an environment.

Use Slimv with Clojure

Slimv works fine with Clojure too. However, it may have some trouble locating Clojure on the system if we attempt to start Swank server automatically with Clojure. That is because where and how Clojure is installed varies from operating system to operating system and also depends on the installation procedure chosen to set up Clojure.

On Unix-like systems, Slimv looks for JAR files that match the glob pattern clojure*.jar at paths that match the glob pattern /usr/local/bin/*clojure* and ~/*clojure, in that order. On Windows, it looks for the JAR files at directory paths that match the glob pattern C:\*clojure* and C:\*clojure*\lib. Additionally, Slimv also looks for the JAR files at the paths mentioned in the PATH environment variable. There are a few more strategies too to locate Clojure but we will not get into that here.

In this section, I will show how to build Clojure from source with Maven and install it at ~/clojure/clojure.jar, a path Slimv can easily find, so installing it here would mean that the steps below would work everywhere regardless of the operating system. If you are on Windows, install Clojure at C:\clojure\clojure.jar instead.

Here are the steps to install Clojure at ~/clojure/clojure.jar and use it with Slimv:

1. Choose one of the two sets of commands below to install Maven:

   # On Debian, Ubuntu, etc.
   sudo apt-get install maven
   
   # On macOS
   brew install openjdk maven
   export JAVA_HOME=/usr/local/opt/openjdk
   export PATH="$JAVA_HOME/bin:$PATH"

2. Enter these commands to install Clojure:

   git clone https://github.com/clojure/clojure.git ~/clojure
   git -C ~/clojure checkout clojure-1.10.1
   mvn -f ~/clojure/pom.xml -Plocal -Dmaven.test.skip=true package

3. Ensure that vim-nox, tmux and Slimv are installed as explained in the Get Started with Slimv and SBCL subsection earlier.

4. This is an optional step. To start Swank server automatically from Slimv, run Vim in tmux, GNU Screen or a desktop environment. In this article, we use tmux, so start tmux with this command:

   tmux

5. This step is necessary only if you are not using tmux, GNU Screen or a desktop environment. In such a case, enter these commands to start Swank server manually:

   SWANK_DIR=~/.vim/pack/plugins/start/slimv/swank-clojure
   java -cp "$HOME/clojure/clojure.jar:$SWANK_DIR" clojure.main -i "$SWANK_DIR/swank/swank.clj" -e '(swank.swank/start-repl)' -r

6. Create a new Clojure source code file, say, foo.clj with this command:

   vim foo.clj

7. To connect to Swank server, enter the following command in Vim normal mode:

   ,c

8. Type some code into the buffer for the new file. To do so, first type i to enter insert mode and type this code:

   (println "hello, world")

   Type esc to return to normal mode. To evaluate, the current expression under the cursor, enter the following command in normal mode:

   ,e

   Both the current expression and its result should appear in the REPL window.

Comparison of Slimv and Vlime

Finally, let me provide a comparison of both Slimv and Vlime side by side. This comparison table below is not exhaustive. There are more differences between the tools than what is mentioned below.

Quick Recommendation

If you are looking for a quick recommendation on which plugin to use, I am going to recommend Slimv. It has been around for much longer. It supports a wider variety of Lisp implementations. I find its default key bindings more convenient. A truly interactive REPL buffer is also a bonus. Also, Slimv supports Scheme and Clojure whereas Vlime does not. Having said that, I think it is a good idea to try out both the plugins on your own and then find out which one suits you more.

Disclosure

Four bugs were harmed while writing this article!

While writing this article, I found the following four bugs in Slimv which were then promptly squashed: #87, #88, #89 and #90.

References

• Couple of Emacs hacks
• Vim 8.0 Released
• Slimv Tutorial
• A Tutorial for Vlime

Read on website | #lisp | #programming | #vim | #technology | #how-to

curl -L -w "time_namelookup: %{time_namelookup}
time_connect: %{time_connect}
time_appconnect: %{time_appconnect}
time_pretransfer: %{time_pretransfer}
time_redirect: %{time_redirect}
time_starttransfer: %{time_starttransfer}
time_total: %{time_total}
" https://example.com/

Here is the same command written as a one-liner, so that I can copy it easily from this page with a triple-click whenever I need it in future:

curl -L -w "time_namelookup: %{time_namelookup}\ntime_connect: %{time_connect}\ntime_appconnect: %{time_appconnect}\ntime_pretransfer: %{time_pretransfer}\ntime_redirect: %{time_redirect}\ntime_starttransfer: %{time_starttransfer}\ntime_total: %{time_total}\n" https://example.com/

Here is how the output of the above command typically looks:

$ curl -L -w "namelookup: %{time_namelookup}\nconnect: %{time_connect}\nappconnect: %{time_appconnect}\npretransfer: %{time_pretransfer}\nstarttransfer: %{time_starttransfer}\ntotal: %{time_total}\n" https://example.com/
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
...
</html>
time_namelookup: 0.001403
time_connect: 0.245464
time_appconnect: 0.757656
time_pretransfer: 0.757823
time_redirect: 0.000000
time_starttransfer: 0.982111
time_total: 0.982326

In the output above, I have omitted most of the HTML output and replaced the omitted part with ellipsis for the sake of brevity.

The list below provides a description of each number in the output above. This information is picked straight from the manual page of curl 7.20.0. Here are the details:

• time_namelookup: The time, in seconds, it took from the start until the name resolving was completed.

• time_connect: The time, in seconds, it took from the start until the TCP connect to the remote host (or proxy) was completed.

• time_appconnect: The time, in seconds, it took from the start until the SSL/SSH/etc connect/handshake to the remote host was completed. (Added in 7.19.0)

• time_pretransfer: The time, in seconds, it took from the start until the file transfer was just about to begin. This includes all pre-transfer commands and negotiations that are specific to the particular protocol(s) involved.

• time_redirect: The time, in seconds, it took for all redirection steps include name lookup, connect, pretransfer and transfer before the final transaction was started. time_redirect shows the complete execution time for multiple redirections. (Added in 7.12.3)

• time_starttransfer: The time, in seconds, it took from the start until the first byte was just about to be transferred. This includes time_pretransfer and also the time the server needed to calculate the result.

• time_total: The total time, in seconds, that the full operation lasted. The time will be displayed with millisecond resolution.

An important thing worth noting here is that the difference in the numbers for time_appconnect and time_connect time tells us how much time is spent in SSL/TLS handshake. For a cleartext connection without SSL/TLS, time_appconnect is reported as zero. Here is an example output that demonstrates this:

$ curl -L -w "time_namelookup: %{time_namelookup}\ntime_connect: %{time_connect}\ntime_appconnect: %{time_appconnect}\ntime_pretransfer: %{time_pretransfer}\ntime_redirect: %{time_redirect}\ntime_starttransfer: %{time_starttransfer}\ntime_total: %{time_total}\n" http://example.com/
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
...
</html>
time_namelookup: 0.001507
time_connect: 0.247032
time_appconnect: 0.000000
time_pretransfer: 0.247122
time_redirect: 0.000000
time_starttransfer: 0.512645
time_total: 0.512853

Also note that time_redirect is zero in both outputs above. That is because no redirection occurs while visiting example.com. Here is another example that shows how the output looks when a redirection occurs:

$ curl -L -w "time_namelookup: %{time_namelookup}\ntime_connect: %{time_connect}\ntime_appconnect: %{time_appconnect}\ntime_pretransfer: %{time_pretransfer}\ntime_redirect: %{time_redirect}\ntime_starttransfer: %{time_starttransfer}\ntime_total: %{time_total}\n" https://susam.net/blog
<!DOCTYPE HTML>
<html>
...
</html>
time_namelookup: 0.001886
time_connect: 0.152445
time_appconnect: 0.465326
time_pretransfer: 0.465413
time_redirect: 0.614289
time_starttransfer: 0.763997
time_total: 0.765413

When faced with a potential latency issue in web services, this is often one of the first commands I run several times from multiple clients because the results from this command help to get a quick sense of the layer that might be responsible for the latency issue.

Read on website | #networking | #technology | #how-to

In this article, we discuss how to write our own "hello, world" program into the boot sector. At the time of this writing, most such code examples available on the web are meant for the Netwide Assembler (NASM). Very little material is available that could be tried with the readily available GNU tools like the GNU assembler (as) and the GNU linker (ld). This article is an effort to fill this gap.

Boot Sector

When the computer starts, the processor starts executing instructions at the memory address 0xffff:0x0000 (CS:IP). This is an address in the BIOS ROM. The machine instructions at this address begins the boot sequence. In practice, this memory address contains a JMP instruction to another address, typically 0xf000:0xe05b. This latter address contains the code to perform power-on self test (POST), perform several initialisations, find the boot device, load the code from the boot sector into memory and execute it. From here, the code in the boot sector takes control. In IBM-compatible PCs, the boot sector is the first sector of a data storage device. This is 512 bytes in length. The following table shows what the boot sector contains.

This type of boot sector found in IBM-compatible PCs is also known as master boot record (MBR). The next two sections explain how to write executable code into the boot sector. Two programs are discussed in the these two sections: one that merely prints a character and another that prints a string.

The reader is expected to have a working knowledge of x86 assembly language programming using GNU assembler. The details of assembly language won't be discussed here. Only how to write code for boot sector will be discussed.

The code examples were verified by using the following tools while writing this article:

1. Debian GNU/Linux 4.0 (etch)
2. GNU assembler (GNU Binutils for Debian) 2.17
3. GNU ld (GNU Binutils for Debian) 2.17
4. dd (coreutils) 5.97
5. DOSBox 0.65
6. QEMU 0.8.2

Print Character

The following code prints the character 'A' in yellow on a blue background:

.code16
.section .text
.globl _start
_start:
  mov $0xb800, %ax
  mov %ax, %ds
  mov $0x1e41, %ax
  xor %di, %di
  mov %ax, (%di)
idle:
  hlt
  jmp idle

We save the above code in a file, say a.s, then assemble and link this code with the following commands:

as -o a.o a.s
ld --oformat binary -o a.com a.o

The above commands should generate a 15-byte output file named a.com. The .code16 directive in the source code tells the assembler that this code is meant for 16-bit mode. The _start label is meant to tell the linker that this is the entry point in the program.

The video memory of the VGA is mapped to various segments between 0xa000 and 0xc000 in the main memory. The colour text mode is mapped to the segment 0xb800. The first two instructions copy 0xb800 into the data segment register, so that any data offsets specified is an offset in this segment. Then the ASCII code for the character 'A' (i.e. 0x41 or 65) is copied into the first location in this segment and the attribute (0x1e) of this character to the second location. The higher nibble (0x1) is the attribute for background colour and the lower nibble (0xe) is that of the foreground colour. The highest bit of each nibble is the intensifier bit. Depending on the video mode setup, the highest bit may also represent a blinking character. The other three bits represent red, green and blue. This is represented in a tabular form below.

We can be see from the table that the background colour is dark blue and the foreground colour is bright yellow. We assemble and link the code with the as and ld commands mentioned earlier and generate an executable binary consisting of machine code.

Before writing the executable binary into the boot sector, we might want to verify whether the code works correctly with an emulator. DOSBox is a pretty good emulator for this purpose. It is available as the dosbox package in Debian. Here is one way to run the executable binary file using DOSBox:

dosbox -c cls a.com

The letter A printed in yellow on a blue foreground should appear in the first column of the first row of the screen.

In the ld command earlier to generate the executable binary, we used the extension name com for the binary file to make DOSBox believe that it is a DOS COM file, i.e. merely machine code and data with no headers. In fact, the --oformat binary option in the ld command ensures that the output file contains only machine code. This is why we are able to run the binary with DOSBox for verification. If we do not use DOSBox, any extension name or no extension name for the binary would suffice.

Once we are satisfied with the output of a.com running in DOSBox, we create a boot image file with this command: sector with these commands:

cp a.com a.img
echo 55 aa | xxd -r -p | dd seek=510 bs=1 of=hello.img

This boot image can be tested with DOSBox using the following command:

dosbox -c cls -c 'boot a.img'

Yet another way to test this image would be to make QEMU x86 system emulator boot using this image. Here is the command to do so:

qemu-system-i386 -fda a.img

Finally, if you are feeling brave enough, you could write this image to the boot sector of an actual physical storage device, such as a USB flash drive and then boot your computer with it. To do so, you first need to determine the device file that represents the storage device. There are many ways to do this. A couple of commands that may be helpful to locate the storage device are mount and fdisk -l. Assuming that there is a USB flash drive at /dev/sdx, the boot image can be written to its boot sector using this command:

cp a.img /dev/sdx

CAUTION: You need to be absolutely sure of the device path of the device being written to. The device path /dev/sdx is only an example here. If the boot image is written to the wrong device, access to the data on that would be lost.

Now booting the computer with this device should show display the letter 'A' in yellow on a blue background.

Print String

The following code prints the string "hello, world" in yellow on a blue background:

.code16

.section .text
.globl _start
_start:
  ljmp $0, $start
start:
  mov $0xb800, %ax
  mov %ax, %ds
  xor %di, %di
  mov $message, %si
  mov $0x1e, %ah
print:
  mov %cs:(%si), %al
  mov %ax, (%di)
  inc %si
  inc %di
  inc %di
  cmp $24, %di
  jne print
idle:
  hlt
  jmp idle

.section .data
message:
  .ascii "hello, world"

The BIOS reads the code from the first sector of the boot device into the memory at physical address 0x7c00 and jumps to that address. While most BIOS implementations jump to 0x0000:0x7c00 (CS:IP) to execute the boot sector code loaded at this address, unfortunately there are some BIOS implementations that jump to 0x07c0:0x0000 instead to reach this address. We will soon see that we are going to use offsets relative to the code segment to locate our string and copy it to video memory. While the physical address of the string is always going to be the same regardless of which of the two types of BIOS implementations run our program, the offset of the string is going to differ based on the BIOS implementation. If the register CS is set to 0 and the register IP is set to 0x7c00 when the BIOS jumps to our program, the offset of the string is going to be greater than 0x7c00. But if CS and IP are set to 0x07c0 and 0 respectively, when the BIOS jumps to our program, the offset of the string is going to be much smaller.

We cannot know in advance which type of BIOS implementation is going to load our program into memory, so we need to prepare our program to handle both scenarios: one in which the BIOS executes our program by jumping to 0x0000:0x7c00 as well as the other in which the BIOS jumps to 0x07c0:0x0000 to execute our program. We do this by using a very popular technique of setting the register CS to 0 ourselves by executing a far jump instruction to the code segment 0. The very first instruction in this program that performs ljmp $0, $start accomplishes this.

There are two sections in this code. The text section has the executable instructions. The data section has the string we want to print. The code copies the first byte of the string to the memory location 0xb800:0x0000, its attribute to 0xb800:0x0001, the second byte of the string to 0xb800:0x0002, its attribute to 0xb800:0x0003 and so on until it has advanced to 0xb800:0x0018 after having written 24 bytes for the 12 characters we need to print. The instruction movb %cs:(%si), %al copies one character from the string indexed by the SI register in the code segment into the AL register. We are reading the characters from the code segment because we will place the string in the code segment using the linker commands discussed later.

However, while testing with DOSBox, things are a little different. In DOS, the text section is loaded at an offset 0x0100 in the code segment. This should be specified to the linker while linking so that it can correctly resolve the value of the label named message. Therefore we will assemble and link our program twice: once for testing it with DOSBox and once again for creating the boot image.

To understand the offset at which the data section can be put, it is worth looking at how the binary code looks like with a trial linking with the following commands:

as -o hello.o hello.s
ld --oformat binary -Ttext 0 -Tdata 40 -o hello.com hello.o
objdump -bbinary -mi8086 -D hello.com
xxd -g1 hello.com

The -Ttext 0 option tells the linker to assume that the text section should be loaded at offset 0x0 in the code segment. Similarly, the -Tdata 40 tells the linker to assume that the data section is at offset 0x40.

The objdump command mentioned above disassembles the generated binary file. This shows where the text section and data section are placed.

$ objdump -bbinary -mi8086 -D hello.com

hello.com:     file format binary


Disassembly of section .data:

00000000 <.data>:
   0:   ea 05 00 00 00          ljmp   $0x0,$0x5
   5:   b8 00 b8                mov    $0xb800,%ax
   8:   8e d8                   mov    %ax,%ds
   a:   31 ff                   xor    %di,%di
   c:   be 40 00                mov    $0x40,%si
   f:   b4 1e                   mov    $0x1e,%ah
  11:   2e 8a 04                mov    %cs:(%si),%al
  14:   89 05                   mov    %ax,(%di)
  16:   46                      inc    %si
  17:   47                      inc    %di
  18:   47                      inc    %di
  19:   83 ff 18                cmp    $0x18,%di
  1c:   75 f3                   jne    0x11
  1e:   f4                      hlt
  1f:   eb fd                   jmp    0x1e
        ...
  3d:   00 00                   add    %al,(%bx,%si)
  3f:   00 68 65                add    %ch,0x65(%bx,%si)
  42:   6c                      insb   (%dx),%es:(%di)
  43:   6c                      insb   (%dx),%es:(%di)
  44:   6f                      outsw  %ds:(%si),(%dx)
  45:   2c 20                   sub    $0x20,%al
  47:   77 6f                   ja     0xb8
  49:   72 6c                   jb     0xb7
  4b:   64                      fs

Note that the ... above indicates zero bytes skipped by objdump. The text section is above these zero bytes and the data section is below them. Let us also see the output of the xxd command:

$ xxd -g1 hello.com
00000000: ea 05 00 00 00 b8 00 b8 8e d8 31 ff be 40 00 b4  ..........1..@..
00000010: 1e 2e 8a 04 89 05 46 47 47 83 ff 18 75 f3 f4 eb  ......FGG...u...
00000020: fd 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
00000030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
00000040: 68 65 6c 6c 6f 2c 20 77 6f 72 6c 64              hello, world

Both outputs above show that the text section occupies the first 0x21 bytes (33 bytes). The data section is 0xc bytes (12 bytes) in length. Let us create a binary where the region from offset 0x0 to offset 0x20 contains the text section and the region from offset 0x21 to offset 0x2c contains the data section. The total length of the binary would then be 0x2d bytes (45 bytes). We will create a new binary as per this plan.

However while creating the new binary, we should remember that DOS would load the binary at offset 0x100, so we need to tell the linker to assume 0x100 as the offset of the text section and 0x121 as the offset of the data section, so that it resolves the value of the label named message accordingly. Moreover while testing with DOS, we must remove the far jump instruction at the top of our program because DOS does not load our program at physical address 0x7c00 of the memory. We create a new binary in this manner and test it with DOSBox with these commands:

grep -v ljmp hello.s > dos-hello.s
as -o hello.o dos-hello.s
ld --oformat binary -Ttext 100 -Tdata 121 -o hello.com hello.o

Now we can test this program with DOSBox with the following command:

dosbox -c cls hello.com

If everything looks fine, we assemble and link our program once again for boot sector and create a boot image with these commands:

as -o hello.o hello.s
ld --oformat binary -Ttext 7c00 -Tdata 7c21 -o hello.img hello.o
echo 55 aa | xxd -r -p | dd seek=510 bs=1 of=hello.img

Now we can test this image with DOSBox like this:

dosbox -c cls -c 'boot hello.img'

We can also test the image with QEMU with the following command:

qemu-system-i386 -fda hello.img

Finally, this image can be written to the boot sector as follows:

cp hello.img /dev/sdx

CAUTION: Again, one needs to be very careful with the commands here. The device path /dev/sdx is only an example. This path must be changed to the path of the actual device one wants to write the boot sector binary to.

Once written to the device successfully, the computer may be booted with this device to display the "hello, world" string on the screen.

Read on website | #assembly | #programming | #linux | #technology | #how-to

You open a file, edit it and save it only to get the E45 error message that says:

E45: 'readonly' option is set (add ! to override)

You now realise that only root can edit the file. What do you? Start over? No, instead try this:

:w !sudo tee "%"

I learnt this trick recently from the comment section of Tip #975 on the Vim Tips website.

Explanation

How does the :w !sudo tee "%" trick work? Let us look at the command part-by-part:

• :w !{cmd}

  Execute {cmd} with all lines in buffer as standard input.

• "%"

  The % is replaced with the current filename. The quotes around it keeps the filename as a single argument even if it contains whitespace.

• tee {file}

  The tee command is a Unix command (not a Vim command). It copies standard input to standard output and {file}.

More Information

For more information on this command, enter the following commands in Vim:

:help :w_c
:help current-file
:help :_%

Also, enter the following command in shell:

man tee

I hope this was fun!

Read on website | #vim | #unix | #technology | #how-to

C:\>DEBUG
G =FFFF:0000

In the above example, we start the DOS debugger and then enter the G (go) command to execute the program at FFFF:0000. Just doing this simple operation should reboot the system immediately.

When the computer boots, the x86 microprocessor starts in real mode and executes the instruction at FFFF:0000. This is an address in the BIOS ROM that contains a far jump instruction to go to another address, typically F000:E05B.

C:\>DEBUG
-U FFFF:0000 4
FFFF:0000 EA5BE000F0    JMP     F000:E05B

The address F000:E05B contains the BIOS start-up program which performs a power-on self-test (POST), initialises the peripheral devices, loads the boot sector code and executes it. These operations complete the booting sequence.

The important point worth noting here is that the very first instruction the microprocessor executes after booting is the instruction at FFFF:0000. We can use this fact to create a tiny executable program that can be used to reboot the computer. Of course, we can always perform a soft reboot using the key sequence ctrl+alt+del. However, just for fun, let us create a program to reboot the computer with a JMP FFFF:0000 instruction.

Reboot Program

Here is a complete DEBUG.EXE session that shows how we could write a simple reboot program:

C:\>DEBUG
-A
1165:0100 JMP FFFF:0000
1165:0105
-N REBOOT.COM
-R CX
CX 0000
:5
-W
Writing 00005 bytes
-Q

C:\>

Note that the N (name) command specifies the name of the file where we write the binary machine code to. Also, note that the W (write) command expects the registers BX and CX to contain the number of bytes to be written to the file. When the DOS debugger starts, it already initialises BX to 0 automatically, so we only set the register CX to 5 with the R CX command above.

Now we can execute this 5-byte program like this:

C:>REBOOT

Debugger Scripting

In the previous section, we saw how we can start DEBUG.EXE and type the debugger commands and the assembly language instruction to jump to FFFF:0000. We can also keep these debugger inputs in a separate text file and feed that to the debugger. Here is how the content of such a text file would look:

A
JMP FFFF:0000

N REBOOT.COM
R CX
5
W
Q

If the above input is saved in a file, say, REBOOT.TXT, then we can run the DOS command DEBUG < REBOOT.TXT to assemble the program and create the binary executable file. The following DOS session example shows how this command behaves:

C:\>DEBUG < REBOOT.TXT
-A
1165:0100 JMP FFFF:0000
1165:0105
-N REBOOT.COM
-R CX
CX 0000
:5
-W
Writing 00005 bytes
-Q

C:>

Disassembly

Here is a quick demonstration of how we can disassemble the executable code:

C:\>DEBUG REBOOT.COM
-U 100 104
117C:0100 EA0000FFFF    JMP     FFFF:0000

While we did not really need to disassemble this tiny program, the above example shows how we can use the debugger command U (unassemble) to translate machine code to assembly language mnemonics.

Read on website | #assembly | #programming | #dos | #technology | #how-to

MS-DOS as well as Windows 98 come with a debugger program named DEBUG.EXE that can be used to work with assembly language instructions and machine code. In MS-DOS version 6.22, this program is named DEBUG.EXE and it is typically present at C:\DOS\DEBUG.EXE. On Windows 98, this program is usually present at C:\Windows\Command\Debug.exe. It is a line-oriented debugger that supports various useful features to work with and debug binary executable programs consisting of machine code.

In this post, we see how we can use this debugger program to assemble a few minimal programs that print some characters to standard output. We first create a 7-byte program that prints a single character. Then we create a 23-byte program that prints the "hello, world" string. All the steps provided in this post work well with Windows 98 too.

Contents

• Introduction
• Print Character
• Hello, World
• Debugger Scripting
• Disassembly
• INT 20 vs RET
• Conclusion

Print Character

Let us first see how to create a tiny 7-byte program that prints the character A to standard output. The following DEBUG.EXE session shows how we do it.

C:\>DEBUG
-A
1165:0100 MOV AH, 2
1165:0102 MOV DL, 41
1165:0104 INT 21
1165:0106 RET
1165:0107
-G
A
Program terminated normally
-N A.COM
-R CX
CX 0000
:7
-W
Writing 00007 bytes
-Q

C:\>

Now we can execute this program as follows:

C:\>A
A
C:\>

The debugger command A creates machine executable code from assembly language instructions. The machine code created is written to the main memory at address CS:0100 by default. The first three instructions generate the software interrupt 0x21 (decimal 33) with AH set to 2 and DL set to 0x41 (decimal 65) which happens to be the ASCII code of the character A. Interrupt 0x21 offers a wide variety of DOS services. Setting AH to 2 tells this interrupt to invoke the function that prints a single character to standard output. This function expects DL to be set to the ASCII code of the character we want to print.

The command G executes the program in memory from the current location. The current location is defined by the current value of CS:IP which is CS:0100 by default. We use this command to confirm that the program runs as expected.

Next we prepare to write the machine code to a binary executable file. The command N is used to specify the name of the file. The command W is used to write the machine code to the file. This command expects the registers BX and CX to contain the number of bytes to be written to the file. When the DOS debugger starts, BX is already initialised to 0, so we only set the register CX to 7 with the R CX command. Finally, we use the command Q to quit the debugger and return to MS-DOS.

Hello, World

The following DEBUG.EXE session shows how to create a program that prints a string.

C:\>DEBUG
-A
1165:0100 MOV AH, 9
1165:0102 MOV DX, 108
1165:0105 INT 21
1165:0107 RET
1165:0108 DB 'hello, world', D, A, '$'
1165:0117
-G
hello, world

Program terminated normally
-N HELLO.COM
-R CX
CX 0000
:17
-W
Writing 00017 bytes
-Q

C:\>

Now we can execute this 23-byte program like this:

C:\>HELLO
hello, world

C:\>

In the program above we use the pseudo-instruction DB to define the bytes of the string we want to print. We add the trailing bytes 0xD and 0xA to print the carriage return (CR) and the line feed (LF) characters so that the string is terminated with a newline. Finally, the string is terminated with the byte for dollar sign ('$') because the software interrupt we generate next expects the string to be terminated with this symbol's byte value.

We use the software interrupt 0x21 again. However, this time we set AH to 9 to invoke the function that prints a string. This function expects DS:DX to point to the address of a string terminated with the byte value of '$'. The register DS has the same value as that of CS, so we only set DX to the offset at which the string begins.

Debugger Scripting

We have already seen above how to assemble a "hello, world" program in the previous section. We started the debugger program, typed some commands and typed assembly language instructions to create our program. It is also possible to prepare a separate input file with all the debugger commands and assembly language instructions in it. We then feed this file to the debugger program. This can be useful while writing more complex programs where we cannot afford to lose our assembly language source code if we inadvertently crash the debugger by executing an illegal instruction.

To create a separate input file that can be fed to the debugger, we may use the DOS command EDIT HELLO.TXT to open a new file with MS-DOS Editor, then type in the following debugger commands and then save and exit the editor.

A
MOV AH, 9
MOV DX, 108
INT 21
RET
DB 'hello, world', D, A, '$'

N HELLO.COM
R CX
17
W
Q

This is almost the same as the inputs we typed into the debugger in the previous section. The only difference from the previous section is that we omit the G command here because we don't really need to run the program while assembling it, although we could do so if we really wanted to.

Then we can run the DOS command DEBUG < HELLO.TXT to assemble the program and create the binary executable file. Here is a DOS session example that shows what the output of this command looks like:

C:\>DEBUG < HELLO.TXT
-A
1165:0100 MOV AH, 9
1165:0102 MOV DX, 108
1165:0105 INT 21
1165:0107 RET
1165:0108 DB 'hello, world', D, A, '$'
1165:0117
-N HELLO.COM
-R CX
CX 0000
:17
-W
Writing 00017 bytes
-Q

C:\>

The output is in fact very similar to the debugger session in the previous section.

Disassembly

Now that we have seen how to assemble simple programs into binary executable files using the debugger, we will now briefly see how to disassemble the binary executable files. This could be useful when we want to debug an existing program.

C:\>DEBUG A.COM
-U 100 106
117C:0100 B402          MOV     AH,02
117C:0102 B241          MOV     DL,41
117C:0104 CD21          INT     21
117C:0106 C3            RET

The debugger command U (unassemble) is used to translate the binary machine code to assembly language mnemonics.

C:\>DEBUG HELLO.COM
-U 100 116
117C:0100 B409          MOV     AH,09
117C:0102 BA0801        MOV     DX,0108
117C:0105 CD21          INT     21
117C:0107 C3            RET
117C:0108 68            DB      68
117C:0109 65            DB      65
117C:010A 6C            DB      6C
117C:010B 6C            DB      6C
117C:010C 6F            DB      6F
117C:010D 2C20          SUB     AL,20
117C:010F 776F          JA      0180
117C:0111 726C          JB      017F
117C:0113 64            DB      64
117C:0114 0D0A24        OR      AX,240A
-D 100 116
117C:0100  B4 09 BA 08 01 CD 21 C3-68 65 6C 6C 6F 2C 20 77   ......!.hello, w
117C:0110  6F 72 6C 64 0D 0A 24                              orld..$

INT 20 vs RET

Another way to terminate a .COM program is to simply use the instruction INT 20. This consumes two bytes in the machine code: CD 20. While producing the smallest possible executables was not really the goal of this post, the code examples above indulge in a little bit of size reduction by using the RET instruction to terminate the program. This consumes only one byte: C3. This works because when a .COM file starts, the register SP contains FFFE. The stack memory locations at offset FFFE and FFFF contain 00 and 00 respectively. Further, the memory address offset 0000 contains the instruction INT 20. Here is a demonstration of these facts using the debugger program:

C:\>DEBUG HELLO.COM
-R SP
SP FFFE
:
-D FFFE
117C:FFF0                                            00 00
-U 0 1
117C:0000 CD20          INT     20

As a result, executing the RET instruction pops 0000 off the stack at FFFE and loads it into IP. This results in the instruction INT 20 at offset 0000 getting executed which leads to program termination.

While both INT 20 and RET lead to successful program termination both in DOS as well as while debugging with DEBUG.EXE, there is some difference between them which affects the debugging experience. Terminating the program with INT 20 allows us to run the program repeatedly within the debugger by repeated applications of the G debugger command. But when we terminate the program with RET, we cannot run the program repeatedly in this manner. The program runs and terminates successfully the first time we run it in the debugger but the stack does not get reinitialised with zeros to prepare it for another execution of the program within the debugger. Therefore when we try to run the program the second time using the G command, the program does not terminate successfully. It hangs instead. It is possible to work around this by reinitialising the stack with the debugger command E FFFE 0 0 before running G again.

Conclusion

Although the DOS debugger is very limited in features in comparison with sophisticated assemblers like NASM, MASM, etc., this humble program can perform some of the basic operations involved in working with assembly language and machine code. It can read and write binary executable files, examine memory, execute machine instructions in memory, modify registers, edit binary files, etc. The fact that this debugger program is always available with MS-DOS or Windows 98 system means that these systems are ready for some rudimentary assembly language programming without requiring any additional tools.

Read on website | #assembly | #programming | #dos | #technology | #how-to

Editing Data

Let us first see an example of editing an error message produced by the MODE command. This DOS command is used for displaying and reconfiguring system settings. For example, the following command sets the display to show 40 characters per line:

C:\>MODE 40

The following command reverts the display to show 80 characters per line:

C:\>MODE 80

Here is another example of this command that shows the current settings for serial port COM1:

C:\>MODE COM1

Status for device COM1:
-----------------------
Retry=NONE

C:\>

An invalid parameter leads to an error like this:

C:\>MODE 0

Invalid parameter - 0

C:\>

We will edit this error message to be slightly more helpful. The following debugger session shows how.

C:\>DEBUG C:\DOS\MODE.COM
-S 0 FFFF 'Invalid parameter'
117C:19D1
-D 19D0 19FF
117C:19D0  13 49 6E 76 61 6C 69 64-20 70 61 72 61 6D 65 74   .Invalid paramet
117C:19E0  65 72 0D 0A 20 0D 0A 49-6E 76 61 6C 69 64 20 6E   er.. ..Invalid n
117C:19F0  75 6D 62 65 72 20 6F 66-20 70 61 72 61 6D 65 74   umber of paramet
-E 19D0 12 'No soup for you!' D A
-D 19D0 19FF
117C:19D0  12 4E 6F 20 73 6F 75 70-20 66 6F 72 20 79 6F 75   .No soup for you
117C:19E0  21 0D 0A 0A 20 0D 0A 49-6E 76 61 6C 69 64 20 6E   !... ..Invalid n
117C:19F0  75 6D 62 65 72 20 6F 66-20 70 61 72 61 6D 65 74   umber of paramet
-N SOUP.COM
-W
Writing 05C11 bytes
-Q

C:\>

We first open MODE.COM with the debugger. When we do so, the entire program is loaded into offset 0x100 of the code segment (CS). Then we use the S debugger command to search for the string "Invalid parameter". This prints the offset at which this string occurs in memory.

We use the D command to dump the bytes around that offset. In the first row of the output, the byte value 13 (decimal 19) represents the length of the string that follows it. Indeed there are 19 bytes in the string composed of the text "Invalid parameter" and the following carriage return (CR) and line feed (LF) characters. The CR and LF characters have ASCII codes 0xD (decimal 13) and 0xA (decimal 10). These values can be seen at the third and fourth places of the second row of the output of this command.

Then we use the E command to enter a new string length followed by a new string to replace the existing error message. Note that we enter a string length of 0x12 (decimal 18) which is indeed the length of the string that follows it. After entering the new string, we dump the memory again with D to verify that the new string is now present in memory.

After confirming that the edited string looks good, we use the N command to specify the name of the file we want to write the edited binary to. This command starts writing the bytes from offset 0x100 to the named file. It reads the number of bytes to be written to the file from the BX and CX registers. These registers are already initialised to the length of the file when we load a file in the debugger. Since we have not modified these registers ourselves, we don't need to set them again. In case you do need to set the BX and CX registers in a different situation, the commands to do so are R BX and R CX respectively.

Finally, the W command writes the file and the Q command quits the debugger. Now we can test the new program as follows:

C:\>SOUP 0

No soup for you! - 0

C:\>

Editing Machine Instructions

In this section, we will see how to edit the binary we created in the previous section further to add our own machine instructions to print a welcome message when the program starts. Here is an example debugger session that shows how to do it.

C:\>DEBUG SOUP.COM
-U
117C:0100 E99521        JMP     2298
117C:0103 51            PUSH    CX
117C:0104 8ACA          MOV     CL,DL
117C:0106 D0E1          SHL     CL,1
117C:0108 32ED          XOR     CH,CH
117C:010A 80CD03        OR      CH,03
117C:010D D2E5          SHL     CH,CL
117C:010F 2E            CS:
117C:0110 222E7D01      AND     CH,[017D]
117C:0114 2E            CS:
117C:0115 890E6402      MOV     [0264],CX
117C:0119 59            POP     CX
117C:011A 7505          JNZ     0121
117C:011C EA39E700F0    JMP     F000:E739
-D 300
117C:0300  07 1F C3 18 18 18 18 18-00 00 00 00 00 00 00 00   ................
117C:0310  00 00 FF 00 00 00 00 00-FF 00 00 00 00 00 00 00   ................
117C:0320  00 00 00 00 00 00 00 00-00 00 FF FF 90 00 40 00   ..............@.
117C:0330  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00   ................
117C:0340  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00   ................
117C:0350  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00   ................
117C:0360  00 00 00 FF 00 00 00 00-00 00 00 00 00 00 00 00   ................
117C:0370  02 00 2B C0 8E C0 A0 71-03 A2 BA 07 A2 BC 07 3C   ..+....q.......<
-A
117C:0100 JMP 330
117C:0103
-A 330
117C:0330 MOV AH, 9
117C:0332 MOV DX, 33A
117C:0335 INT 21
117C:0337 JMP 2298
117C:033A DB 'Welcome to Soup Kitchen!', D, A, '$'
117C:0355
-W
Writing 05C11 bytes
-Q

C:\>

At the beginning, we use the debugger command U to unassemble (disassemble) some bytes at the top of the program to see what they look like. We see that the very first instruction is a jump to offset 0x2298. The debugger command D 300 shows that there are contiguous zero bytes around offset 0x330. We replace some of these zero bytes with new machine instructions that print our welcome message. To do this, we first replace the jump instruction at the top with a jump instruction to offset 0x330 where we then place the machine code for our welcome message. This new machine code prints the welcome message and then jumps to offset 0x2298 allowing the remainder of the program to execute as usual.

The debugger command A is used to assemble the machine code for the altered jump instruction at the top. By default it writes the assembled machine code to CS:0100 which is the address at which DOS loads executable programs. Then we use the debugger command A 330 to add new machine code at offset 0x330. We try not to go beyond the region with contiguous zeroes while writing our machine instructions. Fortunately for us, our entire code for the welcome message occupies 37 bytes and and the last byte of our code lands at offset 0x354.

Finally, we write the updated program in memory back to the file named SOUP.COM. Since the debugger was used to load the file named SOUP.COM, we do not need to use the N command to specify the name of the file again. When a file has just been loaded into the debugger, by default the W command writes the program in memory back to the same file that was loaded into the memory.

Now our updated program should behave as shown below:

C:\>SOUP COM1
Welcome to Soup Kitchen!

Status for device COM1:
-----------------------
Retry=NONE

C:\>SOUP 0
Welcome to Soup Kitchen!

No soup for you! - 0

C:\>

That's our modified program that prints a welcome message and our own error message created with the humble DOS debugger.

Read on website | #assembly | #programming | #dos | #technology | #how-to