Linux Programming and Cloud Computing Study Notes

Linux Evolution, GNU, GPL and System Architecture

What this topic covers and why it matters

Almost every big-data platform you will meet in this course — Hadoop clusters, Spark workers, Kafka brokers, cloud virtual machines — runs on Linux. Before typing commands it helps to know where Linux came from, why it is free to use and modify, why it dominates servers, and how its parts (hardware, kernel, system calls, libraries, shell and applications) fit together. This topic builds that mental model; later topics use it constantly.

Prerequisite: what an operating system does

A computer performs four basic functions: data storage (saving data on storage devices), data processing (computing and transforming data), data movement (transferring data between components) and control (coordinating all of these). A user cannot drive the hardware directly — nobody wants to send electrical signals to a disk controller to save a file. There must be an interface between the user and the hardware. That interface is the operating system (OS).

An operating system is system software that:

• manages hardware and software resources;
• provides a common interface between users (and their programs) and the hardware;
• controls and coordinates the use of hardware among many programs and many users;
• acts as a resource manager, allocating CPU time, memory and I/O devices to running programs.

Applications (user programs)
           ↕
    OPERATING SYSTEM
           ↕
Hardware (CPU, memory, I/O devices)

The core program of the OS, which stays in main memory the whole time the machine runs, is called the kernel. Programs that come with the OS (file managers, compilers, command utilities) are system programs; programs installed later (browsers, office suites, IDEs) are application programs; and programs you write yourself are user programs. Most users see the OS through system programs rather than through the kernel itself.

From UNIX to Linux: the evolution

UNIX (1969 onwards)

UNIX was created at AT&T Bell Labs in 1969 by Ken Thompson, Dennis Ritchie and colleagues. It first ran on a DEC PDP-7 minicomputer. In 1973 it was largely rewritten in the C language, which Ritchie had developed; this made UNIX portable to other hardware, which was unusual at the time. UNIX was multi-user and multitasking, was designed "by developers for developers", and introduced ideas still used today: a hierarchical filesystem, small tools combined with pipes, and "everything is a file". Its design strongly influenced later Unix-like systems such as BSD, Solaris, macOS and Linux.

UNIX source code was licensed commercially, so by the 1980s it was expensive and restricted. Universities produced their own variants (notably BSD from Berkeley), and in 1987 Professor Andrew Tanenbaum released MINIX, a small Unix-like teaching OS.

Linux (1991 onwards)

In 1991 Linus Torvalds, a student at the University of Helsinki, wanted a free Unix-like kernel for his Intel 386 PC. On 25 August 1991 he announced his "just a hobby" project on the comp.os.minix newsgroup, and version 0.01 followed in September 1991. Strictly speaking, Linux is the kernel — the core that manages processes, memory, devices and filesystems. Version 1.0 was released in March 1994.

A kernel alone is not a usable system: you also need a shell, a compiler, a C library and hundreds of utilities. These already existed thanks to the GNU project, described next. Combining the Linux kernel with GNU tools and libraries produced a complete, free, Unix-like operating system — which is why the Free Software Foundation prefers the name GNU/Linux.

A community-driven operating system

Linux grew through open collaboration. Anyone can read the source, propose a change (a "patch") on public mailing lists such as the Linux Kernel Mailing List, and have it reviewed by subsystem maintainers; Linus Torvalds still merges the final mainline releases, which appear roughly every two to three months. Today thousands of individual developers and hundreds of companies (hardware vendors, cloud providers, distribution makers) contribute. This model gives Linux fast bug fixing, broad hardware support and no single vendor lock-in — major reasons it now dominates servers and supercomputers (every system on the TOP500 supercomputer list has run Linux since November 2017).

The GNU movement and the GPL

The GNU project

In 1983 Richard Stallman announced the GNU project ("GNU's Not Unix"), whose goal was a complete, free, Unix-compatible software environment. In 1985 he founded the Free Software Foundation (FSF) to support it. GNU produced many of the tools you will use daily: the bash shell, the GCC compiler, the GNU C library (glibc), core utilities such as ls, cp and mv (coreutils), GNU tar, GNU make and the Emacs editor. By 1991 GNU had almost everything except a finished kernel (its own kernel, GNU Hurd, was not ready) — exactly the gap Linux filled.

"Free" here means freedom, not price ("free as in free speech, not free beer"). The FSF defines four freedoms:

The GNU General Public License (GPL)

The GPL is the licence that protects these freedoms. It permits anyone to use, study, modify and redistribute the software. Its key rule is copyleft: if you distribute the program or a derivative of it, you must distribute it under the same GPL terms and make the corresponding source code available. You may not add restrictions that take the freedoms away from the next person.

Important consequences that students often get wrong:

• Selling GPL software is allowed; you must still provide the source to the people you distribute it to.
• Modifying GPL software for private internal use does not oblige you to publish anything — the obligations are triggered by distribution.
• The Linux kernel is licensed under GPL version 2 only. Many GNU programs use GPL version 3, and libraries such as glibc use the LGPL (Lesser GPL), which lets non-GPL programs link to them.

The phrase open source (popularised from 1998) describes the same kind of software from a practical, development-model angle; "free software" emphasises the ethical freedoms. Both describe Linux.

Linux distributions

A Linux distribution ("distro") packages the Linux kernel together with GNU and other user-space tools, system libraries, an init system (usually systemd), a package manager, an installer and default configuration, and tests them to work together. You install a distribution, not "the kernel".

Distributions differ in release model (fixed LTS releases versus rolling updates), default software and support period, but they share the same kernel and the same core commands, so skills transfer between them.

Linux operating as a server

A Linux server is a Linux system configured to provide network or local services — web pages, databases, file sharing, remote login — under controlled user identities, resource limits and operational policies. Servers usually run headless (no graphical desktop) and are administered remotely over SSH.

A process is a running instance of a program. A service is a managed background capability, usually implemented by a long-running process called a daemon (names often end in d, such as sshd, httpd, crond). On modern distributions services are started, stopped and supervised by the service manager systemd (Topic 2).

Why Linux dominates server environments:

• Cost and licensing — no per-server licence fees; scales to thousands of cluster nodes cheaply.
• Stability — servers run for months between reboots; services can be restarted individually.
• Security model — separate users, file permissions, privilege separation, quick community patches.
• Remote, scriptable administration — everything can be done from a shell over SSH and automated with scripts.
• Small footprint — a minimal install without a GUI leaves more RAM and CPU for workloads and reduces attack surface.
• Openness — source can be audited and tuned; runs on everything from embedded boards to cloud VMs.

Read-only commands you can try on any Linux server (output varies by system):

uname -r                                   # kernel release, e.g. 6.8.0-45-generic
cat /etc/os-release                        # distribution name and version
uptime                                     # how long the system has been running, load averages
systemctl list-units --type=service --state=running   # running services

Architecture and structure of Linux

Linux is organised in layers. Each layer uses only the services of the layer below it.

+------------------------------------------------------------+
|  USER SPACE                                                |
|   Applications: browsers, databases, Hadoop, your scripts  |
|   Shell (bash) and utilities (ls, cp, grep ...)            |
|   C library (glibc): printf(), fopen(), malloc() ...       |
+---------------- System call interface ---------------------+
|  KERNEL SPACE                                              |
|   Process management and scheduler | Memory management     |
|   Virtual filesystem (VFS) and filesystems | Networking    |
|   Inter-process communication (pipes, signals, sockets)    |
|   Device drivers                                           |
+------------------------------------------------------------+
|  HARDWARE: CPU | memory | disks | network cards | terminal |
+------------------------------------------------------------+

Classic UNIX descriptions divide the kernel into two major subsystems: the file subsystem (files, directories, inodes, buffer cache) and the process control subsystem (scheduling, memory management, inter-process communication). Linux keeps the same ideas with more components.

Kernel

The Linux kernel is monolithic: the scheduler, memory manager, filesystems, network stack and drivers all run together in kernel space for speed. It is also modular: drivers and features can be loaded and unloaded at run time as loadable kernel modules (lsmod lists them, modprobe loads one). This contrasts with a microkernel design (MINIX, GNU Hurd), where most services run as separate user-space processes — a design difference Tanenbaum and Torvalds famously debated in 1992.

Shell

The shell is a command interpreter: it reads what you type, finds the program, starts it and shows the result. It is an ordinary user-space program, not part of the kernel. bash (Bourne Again SHell, from GNU) is the default on most distributions; others include sh, zsh and ksh. A prompt ending in $ usually means an ordinary user; # means the superuser root. When you type ls, the shell creates a child process (fork), loads the ls program into it (exec), and ls asks the kernel — through system calls — to read the directory and write the listing to your terminal.

Key UNIX/Linux concepts

Kernel space, user space and dual-mode operation

Kernel space and user space

Because the kernel runs continuously in main memory, memory is divided logically into kernel space (used only by the kernel) and user space (used by applications). Kernel space executes privileged code that manages memory, devices, processes and filesystems. A user-space program cannot read or write kernel memory or touch hardware directly; if it could, one buggy program could crash the whole machine or read another user's data.

When a program runs it becomes a process. The kernel keeps a control record for it (the process control block, PCB) in kernel space, while the program's own image lives in user space:

+------------------+  high address
|      Stack       |  function call frames, local variables (grows down)
|        ↓         |
|        ↑         |
|       Heap       |  dynamically allocated memory (malloc, new)
+------------------+
|       BSS        |  uninitialised global/static variables
+------------------+
|       Data       |  initialised global/static variables (plus read-only data)
+------------------+
|   Text (code)    |  program instructions
+------------------+  low address

A useful phrase: a file has space, a process has life — a program is a passive file on disk; a process is that program in execution, with CPU time, memory and open files.

Dual-mode operation

To enforce this separation the CPU itself runs in two modes:

In the textbook model a hardware mode bit records the current mode (0 = kernel mode, 1 = user mode). On x86 processors the same idea is implemented as privilege rings: the kernel runs in ring 0 and applications in ring 3. Throughout a program's execution the CPU keeps switching between the two modes, which is why this is called dual-mode operation.

User program runs ............ user mode
     | system call, interrupt or exception (trap)
     v
Kernel handles the request ... kernel mode
     | return from system call
     v
User program resumes ......... user mode

Dual mode is one of several protection mechanisms. I/O protection makes all I/O instructions privileged, so I/O must go through system calls. Memory protection prevents one program from corrupting another program's memory or the kernel's interrupt tables. CPU protection uses a timer interrupt so no program can keep the CPU forever. When hardware detects an error (such as an illegal memory access) it raises an interrupt and the OS takes action, for example terminating the faulty program.

System calls

What a system call is

A system call is the protected entry point through which a user-space process asks the kernel to do privileged work: open a file, read data, create a process, allocate memory, send network data. System calls are defined in C (with small assembly stubs) and are the only legitimate way into kernel services.

Programs can make system calls directly (through thin wrapper functions such as write()) or indirectly through higher-level library functions. For example, printf() is a C library function: it formats text in a user-space buffer and then calls the write() system call. The wrapper executes a special trap instruction (syscall on 64-bit x86; int 0x80 on older 32-bit x86) that switches the CPU to kernel mode and jumps to the kernel's system call handler.

User program:     printf("hello")
                       |
C library (glibc): formats text, calls write() wrapper
                       |  trap instruction (syscall / int 0x80) → kernel mode
Kernel:           system call handler → sys_write() → device driver → I/O
                       |
Back in user mode: printf returns to the program

Manual section 2 documents system calls and section 3 documents library functions, so man 2 write and man 3 printf show the two layers.

Categories of system calls

System calls differ between operating systems (Windows has its own API), which is one reason programs must be recompiled or ported between them. A general OS design principle also applies here: mechanisms decide how something is done (the system call that switches processes), while policies decide what is done (which process runs next).

Worked example: system calls from C

This program makes a direct write() call, then uses fork(), execlp() (one of the exec family) and waitpid():

#include <stdio.h>
#include <string.h>
#include <sys/wait.h>
#include <unistd.h>

int main(void) {
    const char *msg = "hello from write()\n";
    write(1, msg, strlen(msg));          /* file descriptor 1 = standard output */

    pid_t pid = fork();                  /* create a child process */
    if (pid == 0) {
        execlp("echo", "echo", "child replaced by echo", (char *)NULL);
        perror("execlp");                /* reached only if exec fails */
        return 1;
    }
    int status;
    waitpid(pid, &status, 0);            /* parent waits for the child */
    printf("parent: child exited with %d\n", WEXITSTATUS(status));
    return 0;
}

Compile and run:

gcc syscalls.c -o syscalls
./syscalls

Expected output:

hello from write()
child replaced by echo
parent: child exited with 0

fork() returns 0 in the child and the child's PID in the parent. In the child, execlp replaces the program image with echo, so the line after it runs only if exec fails. This fork-then-exec pattern is exactly how the shell runs every command you type. On Linux, strace ./syscalls would list each system call as it happens.

Common misconceptions

Mini-lab: explore your Linux system (read-only)

These commands only display information and are safe on any Linux machine:

uname -a              # kernel name, release and hardware architecture
cat /etc/os-release   # which distribution and version
cat /proc/version     # kernel version and the compiler that built it
echo $SHELL           # your login shell, e.g. /bin/bash
lsmod | head -5       # first few loaded kernel modules
man 2 write           # manual page of the write system call (press q to quit)
man 3 printf          # manual page of the printf library function

Note which distribution family you are on, the kernel release, and whether your prompt ends in $ or #.

Key points

• UNIX (Bell Labs, 1969; rewritten in C in 1973) inspired Linux; Linus Torvalds released the Linux kernel in 1991.
• The GNU project (Stallman, 1983) supplied the shell, compiler, C library and utilities; GNU + Linux kernel = a complete free OS.
• The GPL grants the four freedoms and uses copyleft: distributed derivatives must stay under the GPL with source available. The Linux kernel is GPLv2.
• A distribution bundles kernel, tools, libraries, init system, package manager and installer (Ubuntu/Debian use .deb/apt; RHEL family uses .rpm/dnf).
• Linux dominates servers through cost, stability, security, remote scriptable administration and community development.
• Architecture: hardware → kernel (monolithic, modular) → system call interface → C library → shell and utilities → applications.
• Kernel space vs user space and kernel mode vs user mode protect the system; system calls are the controlled gateway between them.