Assignment 3 (30 marks)
Due: March 22, 2010 (11:59pm)

The purpose of this assignment is to learn about memory management in the Linux operating system. In particular, you will study the physical memory layout, the virtual memory system, shared memory segments, and the structure of process address spaces. You will do so via a combination of systems-level programming in the OS kernel, as well as some application-level programming using some new kernel system calls. The intention is for you to do your development and testing within User Mode Linux (UML), but if you have root (superuser) access to your own dedicated Linux system, you are welcome to try your luck there!

Memory Scanner

Calgary International Airport recently added a full-body scanner to enhance the security check on flights destined to the United States. This scanner allows airport security staff to conduct a detailed head-to-toe scan on selected passengers, and reveal any naughty bits that they might be carrying on their person.

In this assignment, the goal is to develop an analogous "scanner" that can scan through the memory used by active Linux processes, and report any unusual phenomena or naughty bits. Rather than borrow the full-body scanner from the Calgary Airport, we will instead use kernel system calls to extract the information that we need.

Simple Scan (10 marks)

Write a new kernel system call scanner that reports the current usage of the memory system in UML. The information returned by this system call should indicate the total number of processes that are resident in memory, their memory resource usage (e.g., size), and possibly other information of your own choosing. The intent of this system call is to provide you an overview of the current level of activity on the system as a whole.

Write a corresponding user-level program that allows the root superuser to invoke the memory scanning system call at any time to provide a view of memory system usage. Demonstrate your memory scanner on a lightly loaded Linux system with several user processes running. Simple text-based ASCII formatting in columns is fine.

User Scan (10 marks)

Extend your system call from above to support targeted scanning of processes owned by a particular user. Your system call will take one parameter, specifying the user id number (e.g., 501) of the user of interest.

Modify your user-level program so that the root superuser can specify which user should be scanned. If a specific user id is specified, then only processes owned by that user should be scanned. If no user id number is specified, then all user and system processes should be scanned. Demonstrate your memory scanning utility on a Linux system with processes from at least 2 different users running.

Shared Memory Scan (10 marks)

Extend your system call from above to report user processes that are using shared memory segments (e.g., to communicate with each other). These processes are to be considered suspicious for the purposes of this assignment, and should be flagged as such in the output. Compile and run one or more programs that use shared memory segments. Show sample output for a specific user, process, or set of processes when shared memory segments are being used.

Bonus (up to 5 marks)

Extend your system call and user-level application program to allow the root superuser to specify and scan for any suspicious text (e.g., "botnet", "spyware", "Windows") in the text segment or data segment of an active process. These processes are to be considered suspicious for the purposes of this assignment, and should be flagged as such in the output. Show sample output for a specific user, process, or set of processes when such suspicious text strings are observed. (Note: I don't know how to do this yet myself!)

Comments, Tips, and Hints

This assignment is more difficult than Assignment 2, but still very doable based on your growing knowledge of UML. Be sure to attend the tutorial on Linux memory system organization, and start working on the assignment as early as you can!

The incremental development steps indicated above are meant to be helpful. Get the basics of the system call working first, and then worry about extending it with parameters and added features.

Please take care when traversing kernel data structures containing processes, threads, scheduling entities, and memory allocation information. Be conscious of the amount of time you spend in the kernel recording information, and think carefully about how you transfer the information out. Please use locking where appropriate, without unduly impeding the operation of the kernel itself.

As with Assignment 2, kernel development work is best done outside the UML environment, while kernel testing has to be done inside the UML environment.

You will definitely end up crashing your kernel multiple times. Be prepared for this possibility, and remember how to get back to a proper working state when needed.

Submitting Your Assignment

When you are finished, send your solutions directly to your assigned TA via email, using a single email attachment (e.g., gzipped tar file, including a README file, relevant source code, and sample output). Multiple repeated submissions from the same student are frowned upon, as are multiple email attachments. Please put an appropriate subject line on your email. The email subject line should be in the following format: {Tutorial section}_{Assignment number}_{First name}_{Last name} (e.g., T01_A3_Ali_Abedi) and the name of the attached file should be: {First name}_{Last name} (e.g., Ali_Abedi.tar). Submissions must be received on or before the stated submission deadline, otherwise a late penalty of 10% (3 marks) per day will apply.