cosarara.me

The blog

Installing OpenBSD on a Scaleway VPS

Yesterday I wanted to set up an OpenBSD vps. And the cheapest vps provider these days seems to be Scaleway, which doesn't allow custom ISO uploads.

So what I did is pick debian, and from grub run with the serial console attached:

set root=(hd0,gpt1)
kopenbsd /bsd.rd
boot

And I got nothing, because this thing is serial not a proper terminal (or so I thought at first. Actually, the problem is the combination of grub+uefi+openbsd ramdisk fails to output anything other than serial):

screenshot of things not working

Then I learned about -h com0:

set root=(hd0,gpt1)
kopenbsd -h com0 /bsd.rd
boot

This works fine in qemu. Output goes to the serial console, installation can start, etc.

On the VPS, the shell never starts: https://p.teknik.io/yEJ0N

I tried different versions of openbsd and found out that versions previous to 5.3 worked. Something changed on 5.3, 5.2 "works" (network card not recognized, gpt/uefi don't work). Of course it doesn't work well enough to do a proper install, but it gets to a shell.

Now I asked myself, is the process hanging here (1), or is the output changing from serial to something that doesn't work through the scaleway tty (2)? And I went to IRC for help. Specifically, freenode #openbsd. And then the couple guys that tried to help me decided it was certainly the second thing. Why? Because I was not using the blessed boot.conf to set tty com0. Because grub was doing it wrong. Because I wasn't using a custom ramdisk with a nice boot.conf.

This was all a big misunderstanding, for various reasons:

  1. I'm getting all the kernel messages but the last in the serial console. The moment the messages stop, the kernel has been chatting serial for a while, and it's just missing one message to send.
  2. Local qemu tests proved that grub with -h com0 could work fine.
  3. I am not using /boot at all, because all it does I'm already doing with grub, successfully.

But still, the people on IRC helped me mess with ramdisks, and messing with ramdisks proved to me that the kernel was really hanging. The first test was modifying /.profile to make it reboot when it got to the shell immediately. This worked correctly on qemu, but did nothing on Scaleway. If it were a problem of display, it would still reboot.

The second test was removing /sbin/init. This makes the kernel panic and reboot... but just after finding the root partition. Ours is hanging just before finding the root, so obviously it didn't reboot with this either.

In short, the situation is that the kernel is hanging somewhere between printing scsibus2 at softraid0: 256 targets and root on rd0a swap on rd0b dump on rd0b The next thing was seeing if the kernel verbose mode would help.

I booted an openbsd vm, ran boot -c (yes, this time from the famous boot> prompt, since it was a full installation) enabled verbose mode and looked at the output. There were no added lines between scsibus... and root on..., so verbosity wasn't going to help.

The only thing left to do then, is build an openbsd kernel (and ramdisk image) with added printfs everywhere, to see where in the code the kernel hangs and finally find the bug. Or is it?

No, first let's compare good and bad boots to see that the devices are different. Then try to boot a local uefi qemu with virtio devices, just as scaleway is doing. See that it fails just the same. Ah, locally reproducing a bug, that sure is nice. Especially so when the vps takes over a minute to reboot every time.

But when booting an actual install66.fs it doesn't fail; it is only when booting from grub /with virtio/ and /uefi/. (Can't boot from grub on uefi without serial, can't see anything)

Some tests:

  • UEFI+GRUB => no image
  • UEFI+GRUB+virtio => no image
  • UEFI+GRUB+com0 => works
  • UEFI+GRUB+com0+virtio => kernel freezes after scsibusN at softraidM: 256 targets

BIOS+GRUB+virtio => works

UEFI+install66.fs+virtio => works

Anyway, there's a bug here somewhere.

How to actually boot the install media

Reboot to the rescue ubuntu thing that boots off the network on the scaleway control panel.

wget http://ftp.hostserver.de/pub/OpenBSD/6.6/amd64/miniroot66.fs
cat miniroot66.fs > /dev/vda
sync
halt

Set to boot again from disk, reboot, attach console. boot> prompt will appear:

set tty com0
boot

I told it to use the whole disk as GPT. The system will warn that "An EFI/GPT disk may not boot. Proceed?". I disregarded the warning and the system booted fine after finishing the installation.

- cosarara

KRunner and D-Bus

Today I pressed my awesome shortcut for layout switching and found that it didn't work. Instead, something called KRunner popped up. BUT WHY?

What followed was an investigation. Typing krunner on google already explained the first thing: KRunner binds to Alt-F2 and Alt-SPACE.

krunner google query

Now why was krunner running?

> ps ax | grep krunner
2343941 ?        Sl     0:00 /usr/bin/krunner
2382396 pts/10   S+     0:00 grep --color=auto krunner

Pretty high PID, it sure didn't start with the system. Who is its parent?

> cat /proc/2343941/status | rg -i ppid
PPid:   1477
> ps ax | grep 1477
1477 ?        Ss     0:00 /usr/lib/systemd/systemd --user

Oh boy, it's our friend systemd --user. Now how did this end up happening?

> systemctl --user status 2343941
● dbus.service - D-Bus User Message Bus
     Loaded: loaded (/usr/lib/systemd/user/dbus.service; static; vendor preset: enabled)
     Active: active (running) since Thu 2020-01-16 17:02:17 CET; 1 weeks 0 days ago
TriggeredBy: ● dbus.socket
       Docs: man:dbus-daemon(1)
   Main PID: 1512 (dbus-daemon)
     CGroup: /user.slice/user-1000.slice/user@1000.service/dbus.service
             ├─   1512 /usr/bin/dbus-daemon --session --address=systemd: --nofork --nop...
             ├─   1578 /usr/lib/ibus/ibus-portal
             ├─   2255 /usr/lib/dconf-service
             ├─2343766 /usr/lib/kactivitymanagerd
             ├─2343773 /usr/bin/kglobalaccel5
             └─2343941 /usr/bin/krunner

D-Bus, that little guy I've never known what exactly it does but is always there. www.freedesktop.org/software/dbus/ tells us:

In addition to interprocess communication, D-Bus helps coordinate process lifecycle; it makes it simple and reliable to code a "single instance" application or daemon, and to launch applications and daemons on demand when their services are needed.

So, someone asked the magic D-Bus to run krunner for them.

> journalctl --user --unit=dbus
[...] Activating service name='org.kde.ActivityManager' requested by ':1.335' \
(uid=1000 pid=2343761 comm="kate -b ")
[...] Successfully activated service 'org.kde.ActivityManager'
[...] Activating service name='org.kde.kglobalaccel' requested by ':1.336' \
(uid=1000 pid=2343766 comm="/usr/lib/kactivitymanagerd ")
[...] Successfully activated service 'org.kde.kglobalaccel

Oh, Kate! I remember running Kate the other day to open some file!

So the process is like that:

  1. Kate starts the KDE Activity Manager
  2. The Activity Manager starts the kglobalaccel thing
  3. The kglobalaccel thing binds Alt-F2 and Alt-SPACE to start krunner.

And basically I hate everything and sometimes the alt+space bind overrides my WM's and sometimes not.

- cosarara

Ludum Dare 40

This last weekend, after partying hard on Friday night, I made a game for the Ludum Dare 40 Compo! (and messed up my sleep schedule even more)

https://ldjam.com/events/ludum-dare/40/to-the-other-side/

screenshot

I used the Godot Engine (v2.1.4, not the v3 beta), the Aseprite pixel art editor (the GPL fork), the sfxr sound effect generator and the MilkyTracker music tracker.

I did everything all the graphics, music, sfx and code myself (except for Godot), and all the project files (with source and all) are public (which reminds me I need to add a proper license).

I exported to Linux, Windows, and the Web. I didn't do macOS because I don't have a mac to test, and macOS users can always use the web version. I didn't do phones because the game only runs at a fixed resolution and doesn't support touchscreen controls. All the download links are in the ldjam post linked above.

My takes on the whole thing would be that:

  • Godot and Aseprite are great
  • I suck at composing music
  • Making a small game in 48h is feasible and fun

The only major problem I had over the whole process was that the web godot export wouldn't play my sound effects. Reading godot's source code I found it was due to the compression godot's importer does by default, which I then disabled, and the bug was fixed.

- cosarara

Disassembling bytecode

Today I rewrote the decompiler in Red Alien. Here is a quick comparison:

From script source:

#dyn 0x800000

#org @start
lock
:label
loadpointer @text
jump :label

#org @text
= \c\h01\h05\v\h01I don't even know what\n
= I'm doing! Ádudududu\p
= \c\h01\03Does this text even make\n
= sense?

Before:

'--------------------
'file name = /home/jaume/RH/ruby.gba
'address = 0x800000
'---

#org 0x800000
lock
loadpointer 0x880001a
jump 0x8800001

#org 0x80001a
= \cÀÈ\vÀI don't even know what\nI'm doing! Ádudududu\p\cÀ03Does this text even ma
= ke\nsense?

#org 0x800001
loadpointer 0x880001a
jump 0x8800001

After:

'--------------------
'file name =  /home/jaume/RH/ruby.gba
'address =  0x800000
'--------------------

#org 0x800000
lock
' joined
#org 0x800001
loadpointer 0x880001a
jump 0x8800001

#org 0x80001a
= \c\h01\h05\v\h01I don't even know what\nI'm doing! Ádudududu\p\c\h01\ha13Does 
= this text even make\nsense?$$

Notice how the code at 0x800001 isn't duplicated any more. Also, the splitting code for strings is now much better, and characters outside the ascii range are detected as control codes depending on the preceding characters (\c, \v). That $$ is the 0xFF string terminator, which I made explicit in August.

...and I spent just as much time getting backtick code blocks with PKS highlighting working on this blogpost than on the work itself.

- cosarara

Blog Technology

The other day, I was asked on IRC what did this website run on.

Let's go from the bottom to the top. First, the hardware:

An Intel Atom (D945GCLF2) with just a power cord and an ethernet cable comming out of it, sitting on my home desk. A single 300GB disk, no raid, no backups, no SAI, the best setup ever. 2GB RAM. Since the fan is noisy, I have it set to never spin below 75°C.

Now, for the software:

  • Arch Linux using the LTS kernel. I update it when I feel like it.
  • Nginx, serving plain text files, and running a reverse proxy for:

Everything is started using systemd service files. The lisp server runs on sbcl, wrapped by rlwrap, inside in a tmux session, and is responsible for http://www.cosarara.me/cl/archive (please forgive me for letting an implementation detail inside a URL).

The cherrypy server runs as is (service Type=simple), and used to serve this blog directly. Right now I use it to preview things before generating the actual static files.

At the time of writing, the script I use to generate the static content looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
#!/usr/bin/env python3
import os
import web
out_dir = "static"

root = web.Root()
blog = root.blog
redalien = root.redalien
tutorial = root.tutorial
pages = [
    ('index.html', root.index()),
    ('blog/index.html', blog.index()),
    ('blog/atom.xml', blog.atom()),
    ('redalien/index.html', redalien.index()),
    ('redalien/manual/index.html', redalien.manual()),
    #('tutorial/index.html', tutorial.index()),
    ('tutorial/fixing/index.html', tutorial.fixing()),
    ('bluespider/index.html', root.bluespider()),
    ('random/index.html', root.random()),
    ('dtops/index.html', root.dtops()),
    ] + [('blog/entry/{}'.format(i), blog.entry(i)) for i in range(1, 10)]
print("rendered")

for path, page in pages:
    print(path)
    fullpath = os.path.join(out_dir, path)
    os.makedirs(os.path.dirname(fullpath), exist_ok=True)
    with open(fullpath, 'w') as f:
    f.write(page)

(gotta automate that range(1, 10))

As for the pages and entries themselves, I write them in either markdown or HTML, and they are rendered automatically inside a mako template. Mako! Mako! Everything runs on mako and these AVALANCHE terrorists won't stop me. All the code is in a nice <200 SLOC web.py file.

I guess I could use something like pelican instead, but this system is already built, it works, and it's flexible.

- cosarara

Today I Solved/Broke ROM-Hacking

GBA Pokémon game ROM-Hacking is messy. It's done using tools like map editors, hex editors or image inserters directly on the ROM file, and the only safety measure are plain file backups.

Normal game or software development, on the other hand, usually keeps all the source data in separate, easily editable files, then provides some means of building the final product automatically.

Wouldn't it be great if ROM Hacking could be done like normal software development? We'd only need some kind of build system, with a linker that could set all the pointers where they belong... Then we'd press a button and all our data would be written on top of a stripped down ROM file, producing the final game.

But we have that! The script compiler does exactly what we need. You give it your code with a bunch of @-prefixed labels, it looks for free space and it links it all together. What we don't have, though, is non-script things in script source form. To start with, we'd need:

  • Map headers and data
  • Graphics
  • Trainer data
  • Pokemon data

Graphics can probably be compressed by grit. I can make Blue Spider output source files. Trainer data, pokémon data, etc. are similar structures - some work, but possible. The only thing left is some way to nicely edit tileset block data, which my map editor doesn't do. Music and ASM modifications are done by the means of binaries already. So, let's give it a go...

I added a script to Blue Spider named dump_pks.py. It takes the map bank list, the list of maps for every bank, and most of the data for every map (including events, but not the tile map itself yet), and outputs it as pks scripts in a directory named map_dump when called as:

$ ./dump_pks.py FR.gba

It also writes a file include_list.pks which has an #include line for every other file, so that running:

$ asc-cli c FR.gba map_dump/include_list.pks

Builds the whole thing (and, if nothing has been changed on the source files, changes nothing on the ROM). But wait! I hear you say, that's full of raw addresses, nobody could work with that! And that's true, which is why dump_pks.py takes a --label option, which makes it spit out nice @labels instead of hex addresses:

$ ./dump_pks.py --label FR.gba

And so, map_dump/3/map_0.pks AKA Pallet Town will look more or less like this:

'map header
#org @map_3_0_map_header
#word @map_3_0_map_data_header 'map_data_ptr
#word @map_3_0_events_header 'event_data_ptr
#word @map_3_0_level_scripts 'level_script_ptr
#word 0x835276c 'connections_ptr
#hword 0x12c 'song_index
#hword 0x4e 'map_ptr_index
#byte 0x58 'label_index
#byte 0x0 'is_a_cave
#byte 0x2 'weather
#byte 0x1 'map_type
#hword 0x601 'null
#byte 0x0 'show_label
#byte 0x0 'battle_type

'map data header
#org @map_3_0_map_data_header
#word 0x18 'w
#word 0x14 'h
#word 0x82dd0f8 'border_ptr
#word 0x82dd100 'tilemap_ptr
#word @map_3_0_t1_header 'global_tileset_ptr
#word @map_3_0_t2_header 'local_tileset_ptr
#byte 0x2 'border_w
#byte 0x2 'border_h

't1 header
#org @map_3_0_t1_header
#byte 0x1 'is_compressed
#byte 0x0 'tileset_type
#hword 0x0 'null
#word 0x8ea1d68 'tileset_image_ptr
#word 0x8ea1b68 'palettes_ptr
#word 0x829f6c8 'block_data_ptr
[rest omitted]

Now, there's a couple things which must be done before this can be compiled. First, a suitable #dyn line must be added at the top of include_list.pks. And second, since Blue Spider doesn't know when to stop finding maps at the end of the last bank, all the bad maps must be removed from the definition of said bank. In the case of Fire Red, that means removing all maps but number 0 at map_dump/bank_42.pks.

And so, this is it for today. There is quite a bit of work ahead, but the future doesn't look bad.

- cosarara

Random Thoughts, part I

Go listen to 'go' by Delilah, remixed by Paralloyd & Too greezey
- cosarara

Fixing broken ROM Hacks the dirty way

People make ROM Hacks using buggy tools, and then one day they find their game freezing on a pokemon evolution, or battle, or whatever. Since the ROM file is just a big mess and they have no idea how to fix it, they either start over or give up. I sometimes fix other people's ROM hacks.

My usual method was going close to the point of failure (freeze, reboot, whatever), making vba-sdl-h start generating a trace, activating that failure and then looking at the 200+ MB trace file using split(1) and less(1) (basic gnu tools).

## Yesterday, this method was proving more ineficcient than usual, so Yesterday, this method proved more ineficcient than usual, so I came up with another one. I created 2 directories, a and b. In a I put a clean Pokemon ruby ROM and the broken hack, named r.gba and d.gba. In b I created the following script:

#!/bin/sh
cp ../a/* .
dd if=r.gba of=d.gba skip=$OFFSET seek=$OFFSET count=$SIZE bs=1 conv=notrunc

What this does is basically to copy SIZE bytes from r.gba to d.gba starting from byte OFFSET, using dd(1). I would run it as (for example):

$ OFFSET=$((0x416000)) SIZE=$((0x1000)) ./do.sh

I had a savestate to run just before the crash, so I just had to change OFFSET and SIZE, run vba-sdl-h, press F1, and see if it worked.

Starting with 0x600000 bytes from the ROM start fixed the bug, as expected, but that would be replacing too many things from the hack with stuff from the original game. So I narrowed it down by halving and tweaking those values until I ended up with a that 0x1000 byte copy shown above - which fixes evolution in the hack and (hopefully) doesn't break anything important in doing so.

The result of all this is here: What would have been, after some more work, Pokemon Dark Blue's beta 2.

- cosarara

VNC over 2 SSH servers

Today I *needed* a graphical connection to my home desktop, from ~100 km away. That means VNC. At home I have 2 computers: a server and said desktop. The server has a port redirected from the router for ssh, and from the server I can access the desktop (ssh as well). Therefore, I need some kind of double tunnel to get from the laptop to the home machine. The final solution looked like this:

home $ x0vncserver -display :0 -passwordfile ~/.vnc/passwd
laptop $ ssh -L 5900:(Laptop local IP):5900 -N (Home IP)
laptop $ vncviewer DotWhenNoCursor=1 :0

All the ssh connections I did here used RSA keys. VNC uses a password, but it's not facing the public internet and the connection is encrypted (it isn't from the home server to the desktop, but I assume there are no rogues inside my house), so it's not actually needed.

Neither of those IP addresses have to be actual addresses here. In my case, the first one was the hostname of my desktop as written in the server's /etc/hosts. The second one was the name of a .ssh/config entry which uses a CNAME domain name, which points to a no-ip domain name (yay!).

That DotWhenNoCursor=1 thing was added because I had no mouse cursor without it (I'm not sure what to blame here). You can put that in ~/.vnc/default.tigervnc too.

Once the connection is open, you can press F8 to open the viewer's menu and use the fullscreen option or close the program, among other options.

- cosarara

Bye bye, database

This blog doesn't use a database any more. Instead, entries are saved in plain text files, and it's all put together using Mako templates.

Now I can have the blog contents under version control, I edit the entries the same way I edit the rest of the website (emacs and ssh), and the code has decreased in size quite a bit. It's also one less service running on the server.

This was my first application using a database, and I learnt a lot, but there is no point in keeping it any more.

- cosarara

Docs

Here it is, lads, a manual for Red Alien. I might or might not write a tutorial as well.

While doing it I realized that writing HTML using emacs with web-mode and evil is not bad at all. Also, I'll see if I can integrate that syntax highlighting I used in the document into the GUI's QScintilla editor.

- cosarara

Beep! Someone said my name

As an IRC client, I run weechat on my server, inside a tmux session, and connect to that using SSH. I had disabled the beep plugin, since I was begining to go crazy every time someone mentioned me. Still, I needed some way to get notified when someone mentioned or /query'ed me. Enter notify-send and twmnd. notify-send is a standard tool to send notifications to the notification server in a linux desktop. twmnd is a nice notification server that blends well with i3 and is not annoying. The first obvious step is to start running twmnd, then.

Now, how do you tell weechat to send your desktop a notify-send over ssh every time someone says your name? You write a plugin. Here's mine: pastebin. Actually, as you'll see, I didn't write all of that. I took a script named lnotify.py from the the weechat website, and modified it to my needs. If you want to use it, you'll have to replace "altair" on line 104 with whatever hostname or IP address your desktop has.

- cosarara

Upset

I'm so upset. jk. https://github.com/cosarara97/upset

I think the original UPS patcher had a command line interface already... Anyway, this was both easy and fun to do.

- cosarara

Here we go again

The server stopped working, like a week or two ago. The hard drives were too hot and died. Some info was lost, like everything on the database, but I saved the website code (yay!).
- cosarara