jehanne/doc/hacking/overview.md

title
Hacking Jehanne

Jehanne is a work in progress that still needs a lot of effort to become useful.

Here you find a quick tour of the project organization.

Further help can be obtained from the mailing list: you are welcome to challenge my assumptions.

Coding Styles

Jehanne is a small operating system, but it's composed of a kernel, a few libraries and a minimal set of commands coded in C.

Most of libraries and commands come either from Bell Labs' Plan 9 or from 9front, and thus they loosely follow the coding conventions described in the style(6) manual page there. It's wise to stick to that convention for such code, in the hope to share improvements between the projects in a friendly manner.

Other libraries and commands may come from the Unix ecosystem: we stick with the existing conventions there too.

Here I describe the arbitrary conventions that are followed for new original C components and for the kernel. This unfortunately means that, depending on the age of a kernel file, you will find either the loosely followed Pike' style or my ugly rules and conventions.

Fortunately, if you contribute a new interesting program, you are free to choose your favourite coding style, declare it clearly in a README file and stick with it. Note however that this won't apply to libraries.

Use good sense

Le bon sens est la chose du monde la mieux partagée; car chacun pense en être si bien pourvu, que ceux même qui sont les plus difficiles à contenter en toute autre chose n’ont point coutume d’en désirer plus qu’ils en ont.

-- René Descartes, Discours de la méthode

In Jehanne good sense is both strictly enforced and loosely defined.
This way nobody will try to sell you books, lectures or TED conferences about it.

These are my rules of thumb:

Keep it simple

Never add features just because you can. Remove redundant features. Decouple unrelated features. Use obvious names for files and folders.

Encapsulate

Use properly scoped functions to access structures' members.

Do not abstract

Replace abstractions used less than 3 times. Remove unused code.

Aesthetics

I do not care too much about aesthetics, but readability matters.
Unfortunately, just like any other programmer, what I find readable largely depends on the code that I had to debug in the past.

The conventions I try to honor are:

1. Tabs are 8 characters.

2. Lines should be no longer than readable. You can use macros to improve readability.

3. Format blocks like these:

       if(x == nil)
       	do_something();
   
       if(x == y){
       	...
       } else {
       	...
       }
   
       switch(v){
       case AnOption:
       	...
       	break;
       case AnotherOption:
       	...
       	break;
       default:
       	...
       	break;
       }
   

4. Format functions like this:

       /* will wlock/wunlock pool_lock */
       static void
       freelist_add(ImagePointer ptr, ElfImage *img)
       {
       	...
       }
   

5. Use one space around = + - < > * / % | & ^ <= >= == != ? :, but no space between unary operators (& * + - ~ ! sizeof typeof alignof __attribute__ defined ++ --) and their operand, and obviously no space around the . and -> structure member operators

6. Use short names in local variables and module functions when the meaning is obvious in the context using them (tmp, i, j).

7. Use descriptive names for globally visible functions and variables (eg proc_segment_detach). In Jehanne's kernel a few frequently used global variables are allowed to violate this rule: up (current user process), m (current processor) and sys.

8. Use typedefs for struct and enums (CamelCase) but not for pointers.

9. Functions should be short, do one thing, hold few local variables and goto a centralized cleanup section on error. Keep in mind errors when designing the return values of your functions. Use Plan9's error() machinery only in functions directly called by other modules (like Dev methods and exported ones), not just to easily unroll the stack.

Hacking Tools

To fasten development, a set of simple tools have been created. As tools to a greater goal, they are disposable and can only evolve as required by Jehanne's development.

Development Environment

Currently, Jehanne is coded and cross compiled from Linux (I work on a stable Debian GNU/Linux).

To build it you need bash, git, golang, build-essential flex, bison and qemu-system.

Inside the root of the repository you can enter the development environment with ./hacking/devshell.sh that will start a new Bash:

• your $PS1 will be prepended with "JehanneDEV "
• the environment variable $JEHANNE will hold the path of the root of the repository
• the environment variable $ARCH will be "amd64" (aka x86_64, the only supported architecture so far)
• $PATH will include $JEHANNE/hacking/bin and $JEHANNE/hacking/cross/toolchain/bin
• the environment variable $TOOLPREFIX will contain the prefix of the cross compiling toolchain

devshell.sh also gives you an hook to customize your development environment without touching the repository: if the $JEHANNE_DEVELOPER_DIR (default: ~/.jehanne/) exists and contains a script named devshell.sh, such script will be sourced. For example my own devshell.sh starts a couple of terminals.

To build the cross compiler you "only" need to run (cd $JEHANNE/hacking/cross/; ./init.sh; git clean -xdf src/). It will automatically download and compile Binutils and GCC (and their obsolete build dependencies). The process requires around 30 minutes (depending on your hardware) and 4 GB of free disk space, but fortunately it's seldom required during development.

All other development tools can be built with the command ./hacking/buildtools.sh. You can also invoke it with --no-drawterm or --no-tools: since building drawterm is slower than all the other tools together, I usually build it alone.

The build system

Jehanne's build system is an evolution of the Harvey's original one based on Go and Json. It violates the principle of least surprise, so that I was originally pretty skeptic about it, but it turns out that Aki was right: a general purpose language provide both power and painless evolution to a build system.

Thus, to build Jehanne you use the build commands. Its source code is at ./hacking/src/jehanne/cmd/ and its documentation can be obtained with build --help.

It consumes small JSON files (usually named build.json) describing the build process. Some example to get started are the commands one and the libc one.

When I need to rebuild the entire system (for example after a change to libc) I simply run cd $JEHANNE; git clean -xdf . && build.

However you can always build just a specific component (or component set), for example build sys/src/cmd/rc/ or build sys/src/cmds.json Cmds or even cd sys/src/cmd/hmi/console/ && build screenconsole.json && cd -.

Note that the Jehanne's build system does not track or enforce dependencies or file changes: it always run the entire build described in the provided JSON and nothing more. Thus it's simple, fast enough and fully predictable.

Qemu (and friends)

Jehanne has been tested on Qemu, Bochs, VMVare and Hyper-V, but the day to day testing is done with Qemu.

To run the system in Qemu you can run:

./hacking/runOver9P.sh

that connects a 9P2000 server running on the linux host to mount $JEHANNE as the root file system

./hacking/runDisk.sh [path/to/disk/image]

that uses the disk image provided (or $DISK) to as the root file system

./hacking/QA.sh

used by runqemu to start the workhorse or to execute the QA checks (it should not be executed directly).

These scripts react to a few environment variables:

$KERNEL

kernel to load (default: jehanne.32bit)

$KERNDIR

directory containing $KERNEL (default: $JEHANNE/arch/$ARCH/kern/)

$KAPPEND

additional parameters for the kernel

$NCPU

number of simmetric processors to use

Qemu will multiplex the terminal I/O between Jehanne's serial console and Qemu monitor. To switch between the two use Ctrl-a x. To stop Qemu use Ctrl-a c.

To create or update a bootable usb stick (or a disk image to be used with Bochs or Qemu) you can use:

./hacking/disk-create.sh

creates a raw disk image at $DISK (default ./hacking/sample-disk.img). It uses syslinux, its bios files (looked up at $SYSLINUXBIOS) and fdisk, but it can be run as a user without sudo. The image will contains two separate partitions, one for syslinux, the kernel and the initial ram disk (the dos partition) and one for the rest of the system (the plan9 partition) in a hjfs file system.

./hacking/disk-boot-update.sh

updates syslinux, the kernel and the initial ram disk in the appropriate partition of $DISK

./hacking/disk-update.sh file1 file2 ...

copy the files provided as arguments to the hjfs partition of $DISK.

./hacking/disk-get.sh file1 files2 ...

copy the files provided as arguments from the hjfs partition of $DISK to $JEHANNE/usr/glenda/tmp.

Note that the whole process does NOT require root privileges: you don't need to trust Jehanne's developers but you have to dd the usb stick yourself.

Debugging

Once you get used to the codebase, debugging Jehanne is pretty simple.

First start the system in Qemu with either ./hacking/runOver9P.sh or ./hacking/runDisk.sh. If $KAPPEND contains the string "waitgdb", Jehanne will stop at an early stage after the boot and will wait for a gdb connection.

To start such connection you can use the script ./hacking/gdb.sh that will provide you a small but useful set of functions to ease your session:

jhn-connect [host:port]

will connect to host:port (default localhost:1234); it's better than a simple target remote :1234 because it integrates well with the waitgdb kernel argument and it is faster to type (jh TAB c TAB)

jhn-log-syscalls

will log all syscalls.

jhn-log-errors

will log errors

jhn-break-cmd arch/amd64/path/to/cmd "cmd" [address]

will set a breakpoint at the provided address in the user space program named "cmd" (default address: 0x4000c0, aka _main)

jhn-break-pid arch/amd64/path/to/cmd pid [address]

will set a breakpoint at the provided address in the user space program running at pid (default address: 0x4000c0)

Note how in Jehanne you can debug any program or library running in user space with few simple gdb functions.

If $JEHANNE_DEVELOPER_DIR/gdbinit exists it is sourced, providing another hook to ease your debug as you like.

If $JEHANNE_GDB_LOGS is defined the whole session will be logged there, prepended with the current commit hash and a brief summary of the repository status.

The workhorse

A simplified kernel is built before the others: the workhorse.

It's used during the compilation, whenever we need to run a Jehanne's program that is impractical to port to Linux. For example it's run in a qemu instance to create the initial ram disk for the other kernels.

Custom Go tools

Here is a brief summary of the other custom tools in ./hacking/src/jehanne/cmd/:

runqemu

runs Jehanne in a qemu instance and send commands to it. It is used both during compilation (to create the initial ram disk, for example) and to run quality checks.

ksyscalls and usyscalls

produce the boring code for system calls, in kernel and in libc respectively. It reads sysconf.json.

mksys

produces several headers reading from the sysconf.json too

data2c and elf2c

embeed in programs in kernels (mainly in the workhorse).

fetch

downloads external resources listed in a fetch.json file

telnet

can connect a running instance of Jehanne. It was used before drawterm was available.

preen

pretty print the JSON files used by build

Inside the development shell, these tools are available in $PATH.

Miscellaneous utilities

Among devtools you can also find several shell scripts and files that are designed to be used only from the repository root.

In the default configuration (see cfg/startup), Jehanne is started as a cpu server owned by glenda. You can connect it with the ./hacking/drawterm.sh script. The password is "demodemo".

Finally ./hacking/continuous-build.sh and ./hacking/coverity-scan.sh are used during continuous build.

Third parties

In the hacking/third_party directory you can file the Go 9P2000 server used during development and drawterm, both downloaded as git submodules and compiled by ./hacking/buildtools.sh.