This blog has moved

The new location is https://blog.pg12.org.

Including source code in papers (and exams)

In march 2015, I wrote about using Pygments when including source code in papers (and on the web). I still use Pygments, but I have never updated the description on how I do it now. In most cases, I use Pygments implicit without interacting with the program myself. I leave it to the LaTeX package minted. You still need to install Pygments, but you do not need to interact with it. The minted package calls Pygments to do the task for you. The only thing you have to do is to ensure that LaTeX or PDFLaTeX (or whatever version of LaTeX you are using) is passed the --shell-escape option. Otherwise, LaTeX is not allowed to do such calls.

I often use minted in combination with the LaTeX package fancyvrb. For example in this excerption from a LaTeX document:

  \usepackage{fancyvrb}
  \usepackage{minted}

  ⁝

  The function \mintinline{python}{dblderivative} should be used
  like this:

    \inputminted[firstline=2]{python}{dblderivative-ex.py}

  The execution of this example will give the following output:

    \VerbatimInput[label=Resultat fra kjøring]{dblderivative-ex.out}

In the preamble I load the two LaTeX packages. The \inputminted command will include the Python code from the file dblderivative.py pretty-printed in the document in the selected style. You can use the command \usemintedstyle to change the style of the code included in the document. To list possible styles run the following command in a shell:

  pygmentize -L styles

The default style is inspired by Emacs. The \VerbatimInput command in the example will include the text file dblderivative-ex.out with a nice layout in the document, including a descriptive text (label). In this example the text file is generated from running the Python program dblderivative-ex.py piping the output from the program to the text file:

  python dblderivative-ex.py > dblderivative-ex.out

This example above can typically end up looking like this in the final PDF-document:

You can also include code directly in LaTeX using the minted environment:

  \begin{minted}{python}
    def dblderivative(f, x, h=1E-5):
        return (f(x + h) - 2*f(x) + f(x - h))/h**2
  \end{minted}

More detailed description on the usage of these packages can be found in the minted documentation and in the fancyvrb documentation.

Qt for Python (and plotting with Matplotlib)

To able to develop GUI-based Python programs using Qt you need to install Qt and PySide2 (Python mapping for Qt 5, see Qt for Python). A good place to start is the Qt for Python getting started document.

 

Install the software

I have Python 3.7 on my Mac (currently, I am not able to get Qt and Python 3.8 to work). I have also installed numpy and matplotlib with pip (since I have many different Python installations on my Mac, including 2.7, 3.7 and 3.8, I will use pip3.7 to be sure the right Python installation is used):

  > pip3.7 install numpy matplotlib 

For my use, the open source license of Qt is OK. I installed Qt 5.12 from source code on my Mac with macOS 10.15 following these steps (descibed in Building Qt 5 from Git):

  > cd path/to/my/git/clone/of/qt
  > git clone git://code.qt.io/qt/qt5.git
  > cd gt5
  > git checkout 5.12
  > perl init-repository
  > cd ..
  > mkdir qt5-build
  > cd qt5-build
  > ../qt5/configure -developer-build -opensource \
      -nomake examples -nomake tests
  > make -j4

In the configure step above you have to accept the license of the software.

It is also possible to download the source code directly, without using git, at http://download.qt.io/official_releases/qt/. In the following example I installed Qt 5.13.2 on my Mac. First, I downloaded the qt-opensource-mac-x64-5.13.2.dmg file from the URL above. Then I used the application qt-opensource-mac-x64-5.13.2.app to install the tools and the source code at my home directory in the folder Qt5.13.2. Finally, I performed the following steps (again, you have to accept to license in the configure step):

  > cd ~/Qt5.13.2/5.13.2/Src/
  > ./configure -prefix $PWD/qtbase -opensource -nomake tests
  > make -j 4

The next step is to install PySide2 using pip:

  > pip3.7 install PySide2

 

A couple Python Qt examples

Once all is installed you can create your own first Qt application:

  import sys
  from PySide2.QtWidgets import QApplication, QLabel

  app = QApplication(sys.argv)
  label = QLabel("Hello World!")
  label.show()
  app.exec_()

Another example with two pop-up windows with buttons:

  import sys
  from PySide2.QtWidgets import QApplication, QPushButton

  def say_hello():
      print("Button clicked, Hello!")

  app = QApplication(sys.argv)

  button1 = QPushButton("Click me")
  button1.clicked.connect(say_hello)
  button1.show()

  button2 = QPushButton("Exit")
  button2.clicked.connect(app.exit)
  button2.show()

  app.exec_()

A larger example with a random multiple language Hello World text:

  import sys
  import random
  from PySide2 import QtCore, QtWidgets

  class MyWidget(QtWidgets.QWidget):

      msgformat = "<font color=red size=40>%s</font>"
      hello = ["Hallo Welt", "你好，世界",
          "Hei maailma", "Hola Mundo", "Привет мир"]

      def __init__(self):
          QtWidgets.QWidget.__init__(self)
          self.button = QtWidgets.QPushButton("Click me!")
          self.text = QtWidgets.QLabel(
              self.msgformat % ("Hello World",))
          self.text.setAlignment(QtCore.Qt.AlignCenter)
          self.layout = QtWidgets.QVBoxLayout()
          self.layout.addWidget(self.text)
          self.layout.addWidget(self.button)
          self.setLayout(self.layout)

          self.button.clicked.connect(self.magic)

      def magic(self):
          self.text.setText(
              self.msgformat % (random.choice(self.hello),))

  app = QtWidgets.QApplication(sys.argv)
  widget = MyWidget()
  widget.show()
  sys.exit(app.exec_())

 

How to use Matplotlib to plot graphs in a Python Qt application

This has been fun, but no plotting of graphs so far. How can we connect Matplotlib with Qt? We start with this code:

  # Qt and matplotlib integration
  import matplotlib
  matplotlib.use("Qt5Agg")
  from matplotlib.backends.backend_qt5agg import FigureCanvas
  from matplotlib.figure import Figure

We use matplotlib.use("Qt5Agg") to select the Qt backend for rendering and GUI integration. FigureCanvas is the area onto which the figure is drawn. Since we are creating a Qt application where we want to plot a graph in the GUI, we have to use the FigurCanvas from Qt. A Matplotlib Figure includes the plot elements. We will in this example add a plot to such a Figure (see add_subplot below).

We will use different Qt widgets in the implementation of this example. QLabel and QLineEdit for text, QPushButton for buttons, and QHBoxLayout and QVBoxLayout to create the layout of the application:

  # Use Qt with the PySide2 mapping (Qt's official Python mapping)
  from PySide2.QtWidgets import QApplication, QWidget
  from PySide2.QtWidgets import QHBoxLayout, QVBoxLayout
  from PySide2.QtWidgets import QPushButton, QLabel, QLineEdit

We will create a widget, MyPlotWidget, for our plotting application. It includes a canvas for the plot:

  class MyPlotWidget(QWidget):

      labeltext = "Write an expression to plot (Python " + \
          "syntax, including numpy mathematical functions "
          "with one variable): "

      def __init__(self, app):
          QWidget.__init__(self)
          self.fig = Figure(
              figsize=(10,8), dpi=100,
              facecolor=(0.8,0.8,0.9), edgecolor=(0,0,0))
          self.ax = self.fig.add_subplot(1, 1, 1)
          self.canvas = FigureCanvas(self.fig)

Then we create all the building blocks for the application, including input text (QLineEdit), buttons (QPushButton) and help text (QLabel):

          # Create the building blocks
          self.setWindowTitle("Plot a graph")
          self.plotlabel = QLabel(self.labeltext)
          self.exprlabel = QLabel("Expression (with variable x):")
          self.expr = QLineEdit("sin(1/x)")
          self.minlabel = QLabel("Min x:")
          self.min = QLineEdit("-pi/4")
          self.min.setFixedWidth(80)
          self.maxlabel = QLabel("Max x:")
          self.max = QLineEdit("pi/4")
          self.max.setFixedWidth(80)
          self.button1 = QPushButton("Plot it")
          self.button2 = QPushButton("Exit")

The overall application layout is specified by at QVBoxLayout, a vertical stack of building blocks (widgets). Inside this box we also have a horizontal line of building blocks created with a QHBoxLayout:

          # Define layout
          self.minmaxline = QHBoxLayout()
          self.layout = QVBoxLayout()
          self.layout.addWidget(self.canvas)
          self.layout.addWidget(self.plotlabel)
          self.minmaxline.addWidget(self.exprlabel)
          self.minmaxline.addWidget(self.expr)
          self.minmaxline.addSpacing(20)
          self.minmaxline.addWidget(self.minlabel)
          self.minmaxline.addWidget(self.min)
          self.minmaxline.addSpacing(10)
          self.minmaxline.addWidget(self.maxlabel)
          self.minmaxline.addWidget(self.max)
          self.minmaxline.addSpacing(30)
          self.minmaxline.addWidget(self.button1)
          self.minmaxline.addWidget(self.button2)
          self.layout.addLayout(self.minmaxline)
          self.setLayout(self.layout)

The final thing we do when a MyPlotWidget is initialised is to connect the GUI-actions to the code (button clicks, and carriage return in the edit fields):

          # Connect button clicks and CR to actions
          self.button1.clicked.connect(self.plotit)
          self.button2.clicked.connect(app.exit)
          self.expr.returnPressed.connect(self.plotit)
          self.min.returnPressed.connect(self.plotit)
          self.max.returnPressed.connect(self.plotit)

The method plotit calculates 100.001 points on the graph for the given x-values and the expression (function). It then clears the plot and updates it with the calculated values. Finally the updated plot is drawn in the canvas:

      def plotit(self):
          x = linspace(
              eval(self.min.text()),
              eval(self.max.text()), 100001)
          y = eval(self.expr.text())
          self.plotlabel.setText(self.label-text)
          self.ax.clear()
          self.ax.plot(x, y)
          self.ax.figure.canvas.draw()
          self.ax.figure.canvas.blit(self.ax.figure.bbox)

That was the complete implementation of the MyPlotWidget class. The last lines of code creates a Qt application with such a visible widget before the execution is handed over to the Qt execution environment:

  # Create Qt app and widget
  app = QApplication(sys.argv)
  widget = MyPlotWidget(app)
  widget.show()
  sys.exit(app.exec_())

The complete code for this Qt and Matplotlib example is available here: qt-matplotlib.py.

A screenshot of the final application.

Emacs on Mac

Until recently, I have been an Aquamacs user on the Mac. However, recently I have been switching back to a more standard Emacs distribution for Mac. After some time of research I ended up with the Emacs Mac Port. This distribution is easily installed and updated through Homebrew, and since I am already using Homebrew for other important software on my Mac (including bsdmake), this was an obvious choice. I use Homebrew to install the Emacs Mac Port from railwaycat with these commands:

  brew tap railwaycat/emacsmacport
  brew install emacs-mac

Later, when I want to update to the latest release, I just do this:

  brew update emacs-mac

Optionally, you can also create a soft-link in the Application directory to be able to access Emacs from there:

  ln -s /usr/local/opt/emacs-mac/Emacs.app /Applications

Tips to run Emacs from command line prompt can be found here.

Since I want to have a more Mac feel to my Emacs on the Mac (used Aquamacs to long, I guess), the final configuration I suggest is to add support for typical Mac key bindings. I use a modified version of this mac-port-keys.el. My remapping of modifiers is changed to the following (use the function modifier key for meta since I need option modifier key for different key modifiers, including typing the Norwegian letters æ, ø and å):

  (when (symbolp 'mac-control-modifier)
  ;; Re-map modifiers
  (setq mac-control-modifier 'control
        ;mac-option-modifier 'meta      ; (AA)
        mac-function-modifier 'meta     ; (AA)
        mac-command-modifier 'super)
  )

On my iMac keyboard, I do not have the function modifier key on a proper location, so I use Karabiner-Elements to map the unused Caps Lock key to the function modifier (the fn-key). I actually do this on all my Macs, since I never use Caps Locks and the meta key is used a lot in Emacs.

And I use Cua-Mode (but I disable the overriding of standard Emacs bindings):

  ;; Cua-mode (AA)
  (cua-mode t)
  (setq cua-enable-cua-keys nil)          ; C-v and M-v to scroll!

Using PlistBuddy when I want to open Emacs from a link in an HTML file on OS X

Previously, I have discussed How to open Emacs from a link in an HTML file on Mac OS X. This post is just to give a complete overview of how this can be done without to many manual tasks. My complete AppleScript program and its Makefile is available for download. It should only be used with your own modification.  It includes the usage of cvs to commit the changes in the bib files to my local cvs-server and the establishment of a VPN connection if I am not on my local university network (since my cvs-server is not available outside the university network). This part should be completely recoded for your usage.

I use BSD make to compile and install the software, and I use PlistBuddy to manipulate the Info.plist file of the AppleScript application. The focus of this post is how to modify the Info.plist file of the AppleScript application using PlistBuddy. A lot of resources for PlistBuddy is available online: A Simple PlistBuddy Tutorial, How To Manage Plist Files With PlistBuddy, and OSX: Dealing with property list files.

In the previous post I explained the changes needed to be done in the Info.plist file of the application. The application is compiled using osacompile:

osacompile -o EmacsURL.app EmacsURL.applescript

The Info.plist is now located in the EmacsURL.app/Contents directory. To add the necessary properties in the Info.plist file we perform the following PlistBuddy commands (in the EmacsURL.app/Contents directory):

PlistBuddy -c 'Add :CFBundleIdentifier string com.apple.AppleScript.EmacsURL’ Info.plist
PlistBuddy -c 'Add :CFBundleURLTypes array’ Info.plist
PlistBuddy -c 'Add :CFBundleURLTypes:0:CFBundleURLName string "Emacs Helper"' Info.plist
PlistBuddy -c 'Add :CFBundleURLTypes:0:CFBundleURLSchemes array' Info.plist
PlistBuddy -c 'Add :CFBundleURLTypes:0:CFBundleURLSchemes:0 string emacs' Info.plist

First we add an identifier for the application (com.apple.AppleScript.EmacsURL). Then we add the URL-handling as an array that contains one dictionary with a name (Emacs Helper) and a scheme. The scheme is an array with a single string element (emacs). The consequence is that this application will handle the emacs URL protocol (handle URLs with the prefix emacs:). To activate this on OS X, the application should be run once after it is installed. For other details regarding compiling, modifying the Info.plist file, and installing the program, see the Makefile file.

Run all Airmail rules

I currently use Airmail as my Mac (and iOS) mail client. With many email accounts I found that Airmail fits my needs OK. And I have created a large number of rules that is automatically performed on new emails in my inbox. However, for different reasons I would sometimes like to perform all my rules manually. As far as I know this is not possible in the Mac version of Airmail. So I had to implement it myself using AppleScript. You can find the program below. A few things you should be aware of: (i) This program do not open the Airmail application (I expect Airmail to be in use on your Mac). (ii) The program needs to be able to control your computer (see System Preferences → Security & Privacy → Privacy → Accessibility). (iii) I have tried to make it robust, but no guarantees. It could mess up.

I suggest to save this code as the application RunAllAirmailRules.app in the Applications folder of your home directory (choose File Format → Application in the save dialog in Script Editor). Enjoy.

-- An Airmail 3 application
tell application "System Events" to tell process "Airmail 3"
  set frontmost to true
  
  -- Activate (and open) "Rules" window of Airmail 3
  set rules to windows where title contains "Rules"
  if rules is {} then
    click menu item "Rules" of menu "Window" of menu bar 1
    delay 1
    set rules to windows where title contains "Rules"
  end if
  if rules is not {} then
    perform action "AXRaise" of item 1 of rules
    
    -- Traverese all rules
    set all_rules to rows of table 1 of scroll area 1 of item 1 of rules
    repeat with rule in all_rules
      
      -- Select rule and go to preview of the rule
      set the value of attribute "AXSelected" of rule to true
      set enabled_rule to the value of checkbox "Enable" of item 1 of rules as boolean
      if enabled_rule then
        click button "Preview" of item 1 of rules
        
        -- Click "Apply to ..." button
        set btn_ok to false
        repeat while not btn_ok
          try
            set btn to button "Apply to previous messages" of sheet 1 of window "Rules"
            if enabled of btn then
              set btn_ok to true
            end if
          on error msg
            delay 0.1
            set btn_ok to false
          end try
        end repeat
        click btn
        
        -- Click "Close" button
        set btn_ok to false
        repeat while not btn_ok
          try
            set btn to button "Close" of sheet 1 of window "Rules"
            if enabled of btn then
              set btn_ok to true
            end if
          on error msg
            delay 0.1
            set btn_ok to false
          end try
        end repeat
        click btn
        
      end if
    end repeat
  end if
end tell

Open Emacs from a link in an HTML file on Mac OS X

Since I do most of my writing in LaTeX, I have all my references (publications I cite when I write papers) in a series of bib-files (for BibTeX). To easier browse and search these references I have created a small Python script that generates a HTML document with all these references. Each reference also includes a link to the bib-file this reference was found in. In the link I include the key of the reference as a fragment identifier. And to specify that I want this link to be open in Emacs, I identify the URL protocol as emacs. The following is an example of such a URL:

emacs:///path/to/file/refs.bib#smith1999a

This link should open Emacs at the start of the reference with the key smith1999a in the file /path/to/file/mybib.bib. The next step is to create an application doing the necessary work and register the URL protocol emacs to activate this application. The easiest approach on a Mac is the write a short AppleScript Application. I have followed the instructions found at these two articles: Launch Scripts from Webpage Links and OS X URL handler to open links to local files. First I created the following AppleScript application:

on open location emacsURL

  -- Find file path and BibTeX key positions in URL
  set p to the offset of "///" in emacsURL
  set i to the offset of "#" in emacsURL
	
  -- File path
  set fn to text (p + 2) thru (i - 1) of emacsURL
	
  -- BibTeX key
  set bk to text (i + 1) thru -1 of emacsURL
	
  -- Line number of BibTeX key, e.g. 
  -- grep -n {smith199a, refs.bib | awk -F:'{ print $1 }'
  set ln to (do shell script "grep -n {" & bk
    & ", " & fn & " | awk -F: '{ print $1 }'")
	
  -- Open file in Emacs on correct line using 
  -- "+line:ln <filename>"
  if application "Emacs" is not running then
    tell application "Emacs" to activate
    delay 2 -- To ensure Emacs is ready
    do shell script "/usr/local/bin/emacsclient -n +"
      & ln & " " & fn
  else
    do shell script "/usr/local/bin/emacsclient -c -n +"
      & ln & " " & fn
  end if
	
end open location

From the Script Editor I saved the above AppelScript code as an Application. Then I had to edit the property list file of the application. In Finder, click on the application with the Control key held down and choose Show Package Contents. Go to the Contents folder and edit the Info.plist file (using Emacs or other suitable programs like Vim, Coda or Xcode). I changed the CFBundleIdentifier and added the following lines to register this application to handle the emacs URLs:

<key>CFBundleURLTypes</key>
<array>
  <dict>
    <key>CFBundleURLName</key>
    <string>Emacs Helper</string>
    <key>CFBundleURLSchemes</key>
    <array>
      <string>emacs</string>
    </array>
  </dict>
</array>

The application has to execute once the register (activate) the URL handling.