Nduli's World - Engineering

Static Website with Pandoc

2021-07-31T00:00:00+03:00

I needed to pull off a quick website that would have various mathematics questions, and thought to try and use pandoc. Here are my learnings:

I first installed pandoc-bin because I didn't to deal with a lot of haskell dependencies.

yay -S pandoc-bin

The first iteration I made generated an index.html file with all the contents and a table of contents. The following bashscript shows this:

mkdir -p output
pandoc --toc -f markdown -t html --mathjax -s -o ./output/index.html *.md
rsync -rav -e "ssh -p PORT" output/ username@domain:/destination_folder

A sample file would look like this:

# Title that will be in table of contents

Content body

The order in the table of contents is on the alphabetical ordering of the files. In my case, the date prefixed all files, so they ended up in the order I wanted them to.

The file however became too large after some time. I didn't want to get a static site manager for this, so I tweaked the bashscript.

I can compile each file to a separate html file with pandoc, but I first need to get a list of all files I want.

find . -maxdepth 1 -type f -name "*.md" -printf "%f\n" | sort | xargs -I % pandoc -f markdown -t html --mathjax --metadata title="questions" -o %.html  %

I need to ensure compile the files to a common directory so that its easy to sync with my server. An index file that links up to all the children files should also be generated dynamically, so that I can easily find the links. I created a custom function gen_html that does this:

gen_html () {
    mkdir -p test_compile
    file_name=$1
    name="${1%.*}"
    echo "[${name}](./${name}.html)" >> index.md
    echo "" >> index.md
    pandoc -f markdown -t html --mathjax --metadata title="questions" -s -o ./test_compile/${name}.html $file_name
}

xargs runs in a different process, so the function gen_html will not be available for that. To fix this, we first export it and then call it with the xargs context. The _ is a place holder for argv[0] and % is the parameter defined by -I. See this link for more information.

export -f gen_html
find . -maxdepth 1 -type f -name "*.md" -printf "%f\n" | sort | xargs -I % bash -c 'gen_html "$@"' _ %

After which I generate the index with:

pandoc -f markdown -t html --mathjax -s -o ./test_compile/index.html index.md

You can find the full code in this gihub gist.

Comment System Using Recursive Queries

2021-06-26T00:00:00+03:00

I was reading SQL antipatterns and got into the chapter about recursive queries and how there are different ways to approach this. I wanted to experiment with this and found that postgresql supports recursive queries. I decided to make a simple commenting database and see how it would work.

Fibonnacci with CTEs

Adopted from https://wiki.postgresql.org/wiki/Fibonacci_Numbers

WITH RECURSIVE fib(a, b) AS (
    -- non recursive part --
    values (0, 1)
    UNION ALL
    -- recursive part --
    select b, a + b from fib
) SELECT a from fib limit 20;

Here's how this would work:

There are three tables, intermediate, working and result table. The non-recursive part is first calculated, setting the working and result table to values (0, 1)
The recursive part of the query select b, a + b from fib is then run on. fib here is the working table, which contains (0, 1). This sets the intermediate table to (1, 1). This gets appended to the result table, and replaces the working table.
In the next run, the working table is (1, 1), resulting in (1, 2) in the intermediate table, which gets appended to the result table, and replaces the working table. This goes on and on.

The recursion should stop when the intermediate table is empty but since fibonacci is infinite this does not happen. Adding a where clause to the recursive part would cause this to occur. For example:

WITH RECURSIVE fib(a, b) AS (
    -- non recursive part --
    values (0, 1)
    UNION ALL
    -- recursive part --
    select b, a + b from fib where b < 100

) SELECT a from fib;

The last select statement: SELECT a from fib limit 20; picks the information from the result table. Limiting this too can cause the infinite recursion to stop.

Detailed explanation of how this works can be found here: CTE Read me

Comment System

Here's the table structure I used:

DROP TABLE IF EXISTS comments;
CREATE TABLE comments (comment_id SERIAL PRIMARY KEY, comment TEXT, date_created DATE, parent_id INT);

INSERT INTO comments (comment_id, comment, date_created, parent_id) VALUES
(1, 'parent 1', '2021-01-01', NULL),
(2, 'parent 2', '2021-01-01', NULL),
(3, 'child of 1', '2021-01-02', 1),
(4, 'another child of 1', '2021-01-03', 1),
(5, 'child of 2', '2021-01-03', 2),
(6, 'child of 3', '2021-01-03', 3),
(7, 'child of 6', '2021-01-04', 6),
(8, 'child of 7', '2021-01-05', 7),
(9, 'child of 1', '2021-01-06', 1),
(10, 'child of 5', '2021-01-05', 5);

A simple recursive query getting all children for comment_id 1 is:

WITH RECURSIVE child_comments AS (
    SELECT * FROM comments WHERE comment_id = 1
    UNION ALL
    SELECT c.* FROM comments c INNER JOIN child_comments cc ON c.parent_id = cc.comment_id
) SELECT * FROM child_comments ;

A good use case is when you have partitioned tables and want to see all descendant of a particular parent.

DROP TABLE IF EXISTS parent;
CREATE TABLE parent (id INT, user_id INT, topic_id INT, comment TEXT) PARTITION BY HASH(user_id);
CREATE TABLE child_1 PARTITION OF parent FOR VALUES WITH (modulus 2, remainder 0) PARTITION BY HASH(topic_id);
CREATE TABLE child_2 PARTITION OF parent FOR VALUES WITH (modulus 2, remainder 1);
CREATE TABLE grand_child_1 PARTITION OF child_1 FOR VALUES WITH (modulus 2, remainder 0);
CREATE TABLE grand_child_2 PARTITION OF child_1 FOR VALUES WITH (modulus 2, remainder 1);

WITH RECURSIVE child_partition AS (
SELECT inhparent, inhrelid FROM pg_catalog.pg_inherits WHERE inhparent = 'parent'::regclass
UNION ALL
SELECT pg_cat.inhparent, pg_cat.inhrelid FROM pg_catalog.pg_inherits pg_cat INNER JOIN child_partition cp ON pg_cat.inhparent = cp.inhrelid
) SELECT inhparent::regclass AS parent, inhrelid::regclass AS child from child_partition;

which results in:

 parent  |     child
---------+---------------
 parent  | child_1
 parent  | child_2
 child_1 | grand_child_1
 child_1 | grand_child_2
(4 rows)

Cycle Prevention

To introduce a cycle in this query we just have to do:

UPDATE comments SET parent_id=8 WHERE comment_id=1;

To prevent this while running our query, we have to keep a state of all the parents we've visited and filter these out in the recursive bit. In this case, we maintain an array of visited parents and ignore all children comments that have that id.

WITH RECURSIVE child_comments AS (
    SELECT *, array[comment_id] as visited_parents FROM comments WHERE comment_id = 1
    UNION ALL
    SELECT c.*, cc.visited_parents || c.comment_id as visited_parents FROM comments c INNER JOIN child_comments cc ON c.parent_id = cc.comment_id WHERE NOT c.comment_id = ANY (cc.visited_parents)
) SELECT * FROM child_comments LIMIT 10;

But how do we prevent cycle creation in the query itself? One method is to have a trigger that gets all parents of a child comment, and doesn't update if the update would cause a cycle.

CREATE OR REPLACE FUNCTION cycle_prevention() RETURNS trigger AS $cycle_prevention$
    DECLARE
        parents_not_allowed int[];
    BEGIN
        -- Check that parent id doesn't cause a cycle
        IF NEW.parent_id IS NOT NULL THEN
            raise notice 'parent id: %, comment_id %', NEW.parent_id, NEW.comment_id;

            WITH RECURSIVE parents AS (
                SELECT parent_id from comments where comment_id = NEW.parent_id
                UNION ALL
                select c.parent_id from comments c INNER JOIN parents p on p.parent_id=c.comment_id
            )
            SELECT ARRAY(SELECT parent_id::int FROM parents LIMIT 10) INTO parents_not_allowed;
            raise notice 'Value: %', parents_not_allowed;

            IF NEW.comment_id = ANY(parents_not_allowed) THEN
                RAISE EXCEPTION 'cycle found in query';
            END IF;
        END IF;
        RETURN NEW;
    END;
$cycle_prevention$ LANGUAGE plpgsql;

CREATE TRIGGER cycle_prevention BEFORE INSERT OR UPDATE ON comments
    FOR EACH ROW EXECUTE PROCEDURE cycle_prevention();

Now the update fails with ERROR: cycle found in query.

Copying Pass Secrets to Another Machine

2021-03-27T00:00:00+03:00

I had a new laptop but couldn't access my previous one, so I had to figure out how to get my pass secrets. I had these problems:

How would I get my old password store?
How would I be able get my old gpg keys into this new laptop?
I had started storing new passwords using a new gpg key, so how would this work the old password store (generated with a different key)?

Getting the old secrets

My home directory was compressed using borg and sent to me using:

borg init /path/to/folder
borg create --progess /path/to/backup::susa /home/username

And on my laptop I ran:

borg mount backup::susa-initial /home/username/borg_mount

This folder is mounted as read only, limiting what I could do in it. Here are some of the errors I got because of this.

gpg --homedir $(pwd) --list-keys
gpg: failed to create temporary file '/home/username/borg_mount/home/username/.password-store/.#lk0x000055b881984170.archlinux.39288': Read-only file system
gpg: keyblock resource '/home/username/borg_mount/home/username/.password-store/pubring.kbx': Read-only file system
gpg: failed to create temporary file '/home/username/borg_mount/home/username/.password-store/.#lk0x000055b881981700.archlinux.39288': Read-only file system
gpg: Fatal: can't create lock for '/home/username/borg_mount/home/username/.password-store/trustdb.gpg'

I copied over the gpg key into a folder outside the mounted borg folder:

cp -r borg_mount/home/username/.gnupg/* gpg_tests/
cd gpg_tests
gpg --homedir $(pwd) --list-keys # this now works

And exported my keys with:

gpg --homedir $(pwd) --output personal_gpg.gpg --armor --export ID_FOR_KEY
gpg --homedir $(pwd) --output personal_sec.gpg --armor --export-secret-keys ID_FOR_KEY

I imported them them with:

gpg --import personal_gpg.gpg
gpg --allow-secret-key-import --import personal_sec.gpg

Working with both password stores

Pass stores the gpg_id in its root folder, so I just copied over the previous pass store into a subdirectory in my new password store.

cp home/username/.password_store ~/.password_store/oldpasswords

Now running pass shows all my passwords, both on from my old laptop and new laptop. It's a little inconvenient because I now have an extra node to go through to access my old passwords but this is an acceptable compromise.

Another unexplored alternative is to convert all the old passwords to use the new gpg key. It should be a simple script that just gets the password from the old gpg key and saves it into the new store.

Setting up Hspec on a Haskell Project

2021-01-23T00:00:00+03:00

I'd started the Kindle-Highlights project without tests. This was a problem because I couldn't add features as fast as I wanted because I'd break something without knowing. I needed to add tests to speed up my development. For haskell, hspec is a tool/library for this. Setting this up on the project was surprisingly challenging, mostly because I didn't properly read the docs.

Some of the things I missed out on or had to do are:

There is auto detection of test files with hspec-discover.
Test files should have the suffix Spec (I'd added them with a Test Suffix) to be auto discovered by hspec-discover.
Each test file has to export a spec of type Spec
I needed to set up some of my modules properly in the cabal file.
I had to move executable to a different directory to prevent warning on missing dependencies, see github stack issue `.
I switched to cabal version 2.2 to use the shared properties field.

Here are the steps I took to having a properly working and tested code base:

Cabal Setup

I modified my cabal config by:

Using cabal version 2.2: this is so that I could use common stanzas preventing duplication of dependencies, see cabal common-stanzas
Adding a library section that exposed the modules that I wanted to test.
Adding a test section that contained hspec options.
Added an executable section and moved the executable code Main.hs to another folder to prevent warning errors on missing dependencies.

cabal-version:       2.2
name:                kindle-highlights
.
.

common shared-properties
  default-language:    Haskell2010
  build-depends:       base >= 4.7 && < 5,
                       parsec,
                       text,
                       containers,
                       time

library
  import:              shared-properties
  exposed-modules:     KindleHighlights,
                       CommandOptions
  hs-source-dirs:      src
  build-depends:       optparse-applicative

executable kindle-highlights
  import:              shared-properties
  hs-source-dirs:      app
  main-is:             Main.hs
  build-depends:       optparse-applicative,
                       kindle-highlights,

test-suite spec
  import:              shared-properties
  type:                exitcode-stdio-1.0
  hs-source-dirs:      test
  main-is:             Spec.hs
  other-modules:       ParserSpec
  build-depends:       kindle-highlights,
                       hspec >= 2.7,
                       hspec-discover >= 2.7
  ghc-options: -Wall
  build-tool-depends: hspec-discover:hspec-discover == 2.*

Key things here are that the test-suite spec points to the 'test' directory, with hspec and hspec-discover added as build-depends.

Writing the Tests

To enable automatic test discovery, the Spec.hs file has:

{-# OPTIONS_GHC -F -pgmF hspec-discover #-}

which means it will automatically pick up tests that I write. The test files has to have the Spec suffix for this to work.

To write a test file, for example, ParserSpec.hs

module ParserSpec (spec) where

spec :: Spec
spec = describe "KindleHighlights" $ do
  it "has a string definition for end of group"
    $          eogString
    `shouldBe` "==========\r\n"

has to export the spec function, which will be called by hspec.

Running the tests with stack test should show:

kindle-highlights> test (suite: spec)

Progress 1/2: kindle-highlights
Parser
  KindleHighlights
    has a string definition for end of group
    highlights
    highlights 2
    groups

Finished in 0.0004 seconds
4 examples, 0 failures

kindle-highlights> Test suite spec passed
Completed 2 action(s).

which shows the tests ran successfully.

Control: Controllability and Observability

2018-02-21T15:00:00+03:00

The state space equations are given by:

\begin{align*} x'(t) = Ax(t) + Bu(t)\\ y(t) = Cx(t) + Du(t) \end{align*}

If we discretize the system, the equation can be written as:

\begin{align*} x(n+1) = A_dx(n) + B_du(n)\\ y(n) = C_dx(n) + D_du(n) \end{align*}

The discretization steps can be found here:discretization

Controllability

A system is said to be controllable if the state can be changed by changing the inputs.

Using the discretized equation, we can come up with the following:

\begin{align*} x(1) &= A_dx(0) + B_du(0)\\ x(2) &= A_dx(1) + B_du(1) = A_d^2 x(0) + A_dB_du(0) + B_du(1)\\ .\\ x(n) &= A_d^n x(0) + A_d^{n-1}B_du(0) + A_d^{n-2}B_du(1) + ... + B_du(n-1) \end{align*}

The last equation can be written as:

\begin{equation*} x(n) - A_d^n x(0)= \begin{bmatrix} B_d & A_dB_d & A_d^2B_d & ... & A_d^{n-2}B_d & A_d^{n-1}B_d \end{bmatrix} * \begin{bmatrix} u(n-1) \\ . \\ . \\ . \\u(2) \\ u(1) \\ u(0) \end{bmatrix} \end{equation*}

Replacing with the controllability matrix with C, we get:

\begin{align*} x(n) - A_d^n x(0) = C * U \\ \therefore\\ U = C^{-1} * (x(n) - A_d^n x(0)) \end{align*}

This means that the equation is solvable when \(C^{-1}\) exists, i.e. its a non singular matrix.

Therefore a system is controllable when:

\begin{equation*} \begin{bmatrix} B_d & A_dB_d & A_d^2B_d & ... & A_d^{n-2}B_d & A_d^{n-1}B_d \end{bmatrix} \end{equation*}

has a determinant.

Observability

Using the discretized equation too:

\begin{align*} y(0) = C_dx(0)\\ y(1) = C_dx(1) = C_dA_dx(0)\\ y(2) = C_dx(2) = C_dA_d^2x(0) \\ y(n) = C_dx(n) = C_dA_d^nx(0) \end{align*}

This can be written as:

\begin{equation*} \begin{bmatrix} y(0)\\y(1)\\y(2)\\.\\y(n) \end{bmatrix} = \begin{bmatrix} C_d \\ C_dA_d \\ C_dA_d^2 \\ . \\C_dA_d^n \end{bmatrix} * x(0) \end{equation*}

The observability matrix is:

\begin{equation*} \begin{bmatrix} C_d \\ C_dA_d \\ C_dA_d^2 \\ . \\C_dA_d^n \end{bmatrix} \end{equation*}

And the systems has a unique solution if the system has a rank n. If the system is a square matrix, the solution is of rank n if the determinant is not 0.

The observability and controllability derivations can be found from here:rutgers

System Clock Prescalers and Their Hazards

2018-02-04T15:00:00+03:00

One means of saving power with the attiny13A is by slowing down the clock frequency of the microcontroller. One option of doing this is by using the system clock prescaler. The attiny13A is shipped with running on a 9.6MHz internal RC Oscillator. It has the CKDIV8 fuse set (which provides a system clock prescaler of 8), thus it runs on a frequency of 1.2MHz.

Supposing we want to slow it further, we can set up the CLKPR (Clock Prescale Register) bits for the same. If we set this to a prescaler of 16 for example, the attiny will run at a frequency of 600KHz. However, there is a special write procedure provided for this to prevent unintentional changes:

Write the Clock Prescaler Change Enable (CLKPCE) bit to one and all other bits in CLKPR to zero.
Within four cycles, write the desired value to CLKPS while writing a zero to CLKPCE.

To follow this, our code will be:

// set CLKPCE to 1 and other bits to zero
CLKPR = 0b10000000;
// set prescaler to 256
CLKPR = 0b00001000;

Furthermore, when compiling we have to use the optimization flags -0s, otherwise the process might take more than four cycles. The microcontroller will be running at a clock speed of 37.5KHz after the operation.

Hazards

If successful, however, programming the microcontroller will be difficult. This is because of syncing issues, whereby the microcontroller is significantly slower than the programmer. To fix this you can set the -B flag using avrdude. However, the usbasp programmer I am using does not have this functionality.

To fix this we have to analyze how the microcontroller sets the new frequency. First, the microcontroller will boot up. At this state CLKPR is still its default value. Then the two instructions setting the prescaler are run. This therefore means that if we can find a way to stop those two instructions from running, we can program the microcontroller with another firmware.

IF we activate the RESET pin, before the clock prescaler is set, the chip can be reprogrammed. Connect Ground signal to the RESET pin and then provide power to the programmer (usbasp). This maintains the microcontroller in Reset mode, thus the instructions changing the prescaler are not carried out. Then immediately flash the attiny. If successful, the error will be gone.

Another way to prevent this is to have a considerable delay before the CLKPCE bits are set. This delay allows for reprogramming if necessary.

Prepare Kicad PCB for Elecrow Production

2017-12-07T08:00:00+03:00

Before designing the PCB, set the following design rules:

Minimum PCB Track : greater than 0.2mm
Minimum Track /Vias space: greater than 0.2mm
Minimum pads space: greater than 0.2mm

The defaults provided by Kicad are OK, so no need to change them unless you have some special requirements.

Once the pcb has been completed, go go:

File>>Plot

Make sure the following options are checked in Layers section:

F.Cu
B.Cu
B.SilkS
F.SilkS
B.Mask
F.Mask
Edge.Cuts

Make sure the Plot format is set to Gerber. It is also advisable to set an OutputDirectory for your gerber files. This prevents mixing up the gerber with normal project files.

Make sure the following are checked in Options:

Plot footprint values
Plot footprint references
Exclude PCB edge layer from other layers

For Gerber options, make sure this is checked:

Use Protel filename extensions

Click on Plot to generate the gerber files.

To generate the drill files, click on "Generate Drill File". I set the Drill Units to Millimeters, Zeros Format to Decimal Format and Dril Map File Format to Gerber. Also Drill Origin was set to Absolute. Then click Drill File.

Kicad outputs files with a different naming convention as that expected by elecrow. So you'll have to rename these files. For examples:

pump-B.Cu.gbl becomes pump.gbl
pump-B.Mask.gbs becomes pump.gbs
pump-B.SilkS.gbo becomes pump.gbo
pump-Edge.Cuts.gm1 becomes pump.gml
pump-F.Cu.gtl becomes pump.gtl
pump-F.Mask.gts becomes pump.gts
pump-F.SilkS.gto becomes pump.gto
pump.drl becomes pump.txt

After renaming the files you can then compress all of them into a zip file and make the order.

Scrolling Text With 5*4 LED Array

2017-10-29T10:20:00+03:00

I usually see lights in Nairobi scrolling text from right to left. It is an interesting thing to see, and if done right is usually beautiful. I wanted to make one, and so this is how I went about it.

Multiplexing LEDs

To light and LED one needs 5V supplied to cathode and GND supplied to anode. This means if we have 2 leds, both receiving 5V and only one receiving GND, only one LED will light. The same occurs when both receive GND and only one receives 5V. This is the basic building block of an LED matrix.

So how does one keep this so. Well, one option is using a microcontroller. The microcontroller's pins can be set to output a HIGH (5V) or a LOW(GND/0V). So having a microcontroller will help us out in choosing which LED to use.

The following is the schematic I then came up with:

To have the scrolling effect, I then wrote code that turns on and off LEDs at a particular pattern. Making this faster will result in less flickering of light. I used an atmega 8 for this and added the respective current limiting resistors. The circuit and code for this can be found here:githublink.

Here is the image of the final fabricated circuit:

You can find the video of the working circuit here. The text being shown is "HAPPY EASTER" in case it is the video is not clear enough.

Disabling Blanking in Raspberry Pi

2017-10-17T15:00:00+03:00

Blanking is whereby a screen just goes blank (i.e. black screen). This usually happens after some time of no usage on the raspberry pi which is about 15 minutes. This can sometimes become frustrating especially if it is a screen set up for monitoring something.

So to disable it temporarily:

xset s off # disables screen saver
xset -dpms #disable power management settings
xset -s noblank

To disable it permanently, open the file /etc/lightdm/lightdm.conf:

sudo vi /etc/lightdm/lightdm.conf

Then add the following line:

xserver-command=X -s 0 -dpms

Setting Up Raspberry Pi

2017-07-16T08:20:00+03:00

The raspberry pi is a mini computer with GPIO pins that can be used to interface with various sensors and actuators. More information of the raspberry pi can be found here

The SD card should be of a size greater than 4GB, especially if you want to install Raspbian Jesse.

Connect the card to your computer and get its id. To do this:

sudo fdisk -l

If you do the above command before inserting the card and after inserting the card, you'll find a new addition which is the card.

Format the card to fat partition. To do this:

sudo umount /dev/sdcardfound
sudo mkfs.fat /dev/sdcardfound

After this the sd card is ready for use.

Download the OS from here:raspbian, and unzip it. I chose Raspbian Jesse with Desktop for my OS.

unzip downloadedraspbian.zip

Then burn the OS into the sd card using dd:

sudo dd if=unzippedraspbian.img of=/dev/sdcardfound bs=4M status= progress

Once this step is completed the OS is ready for use.

With Screen, Keyboard and Mouse

Attach the sd card, screen, keyboard and mouse to the raspberry pi. For power, a supply rated 5V 2.5A is recommended. In my case, I used one rate 5V and 1A and it worked well. When powered, there are two LED lights, a red one and a green one. The red should be on indicating the pi is receiving power. The green light should be blinking. If it is not this indicated either a problem with the sd card or the power supply.

Once the pi is started, wait until the desktop environment is loaded. Then you can then enable vnc and ssh.

To do this, go to the terminal and type:

sudo raspi-config

Go to interface and enable both vnc and ssh.

You can also go to MENU > PREFERENCES > Interfaces. From this you can also enable ssh and vnc.

To set up a virtual keyboard, first connect to a wifi/LAN network and do:

sudo apt update
sudo apt install matchbox-keyboard
sudo reboot

The keyboard should be in accessibility menu. If it isn't go to MENU > Preferences > Main Menu Editor > Accesorites. Disable and enable the Keyboard option.

To ssh into the pi, first get the ip address. To do this, hover on the wifi icon and the ip will be displayed. You can also get the ip address from the terminal by running:

ip address show

From another computer, run:

ssh pi@piipaddress

The password is : raspberry

The pi already comes with a vnc server known as realvnc. So just install the viewer on your laptop and use it. For archlinux:

yaourt -S realvnc-vnc-viewer

Then confirm that vnc is enabled on the pi. To use it:

vncviewer piipaddress

Realvnc can also be setup to run via the cloud. Just visit their website here:realvnc to find out how.