Tech Kaizen: KERNEL DEVELOPMENT

Showing posts with label KERNEL DEVELOPMENT. Show all posts

Windows User-Mode Driver Framework (UMDF) ..

Windows Driver Frameworks (WDF) is a set of libraries that you can use to develop device drivers that are interoperable with Windows. WDF is comprised of (KMDF) and (UMDF). UMDF is a framework for the creation of user-mode drivers. Like Kernel-Mode Driver Framework (KMDF), UMDF provides an abstraction layer from , handling much of the Plug and Play (PnP) and functionality, and allowing the driver to opt in for specific functionality and event handling.

Writing a driver using UMDF version 1.x requires using the model to write C++ code. While UMDF version 1 is based on the same conceptual driver programming model as KMDF, UMDF 1 implements the model with different components, device driver interfaces (DDIs), and data structures. In contrast, starting in UMDF version 2, you can write a UMDF driver in the C programming language that calls many of the methods that are available to KMDF drivers. All of the interfaces that are shared between UMDF version 2 and KMDF have the same names, parameters, and structure definitions. If your driver uses only shared functionality, or uses conditional macros around calls that are only supported in one framework, you can write a single driver that you can compile with either UMDF or KMDF.

UMDF drivers abstract hardware functionality, run in the user-mode environment, and can access various services. UMDF drivers operate as part of a stack of drivers that manage a device. File system drivers, display drivers, and print drivers cannot be UMDF drivers.

A UMDF driver interacts with the following system-supplied components:

Driver host process:
The driver host process loads vendor-supplied UMDF drivers and framework DLLs, provides an execution environment for user-mode drivers, and routes messages between drivers in a user-mode stack. The driver host process (Wudfhost.exe) is a child process of the service. Wudfhost.exe usually runs in the , which has minimum privileges on the local computer. An instance of Wudfhost.exe loads one or more UMDF driver DLLs, in addition to the framework DLLs. The driver host process provides a runtime environment that handles interprocess communication (IPC) between the driver manager and the , as well as I/O dispatching, driver loading, driver layering, and thread pool management.

Driver manager:
The driver manager is a Windows service that manages all instances of the Wudfhost driver host process. The driver manager launches and tracks information about each driver host process. Each host is a child process of the driver manager. Only one driver manager exists per system. The driver manager starts during installation of the first UMDF device and runs on the system thereafter.

Reflector:
The reflector is a kernel-mode driver that permits an application and a driver host process (and user-mode device stacks) to communicate. The reflector creates a separate device object for each device instance and handles Plug and Play (PnP) and power I/O requests associated with each device instance. All communication between the application and the driver host process happens through the reflector.

If your driver requires one of these features, you must write a KMDF driver:
While UMDF version 2.0 offers a significant subset of functionality that was previously available only to KMDF drivers, the following features are available only to KMDF drivers.

Direct memory access (DMA)
Bus enumeration
Functional power states (limited support is available in UMDF)
Access to WDM objects
Neither Buffered Nor Direct I/O
Internal device control requests (IOCTLs)
Remove lock opt-in for I/O requests

Advantages of Writing UMDF Drivers:

UMDF drivers contribute to greater operating system stability because they have access only to the address space of the process in which they run.
Because UMDF drivers run under the LocalService account, they have limited access to a user's data or to system files.
User-mode drivers operate in a much simpler environment than kernel-mode drivers. For example, kernel-mode drivers must take into account IRQL, page faults, and thread context. In user mode, however, these issues do not exist. User-mode drivers always run in a different thread from the requesting process and can always take page faults.
UMDF version 2 offers feature parity with KMDF in most areas. For a full comparison, see Comparing UMDF 2.0 Functionality to KMDF.
UMDF version 2 facilitates converting between KMDF and UMDF. See How to convert a KMDF driver to a UMDF 2.0 driver (and vice-versa).
You can debug UMDF drivers by using either a user-mode debugger or, starting with UMDF version 2, a kernel-mode debugger.
You can use the Wdfkd.dll debugger extension commands with KMDF and starting with UMDF version 2. For more info, see Debugger Extensions.

A fundamental goal of the overall WDF model is to provide intelligent defaults, so that you can focus on your device hardware and avoid writing code to perform tasks that are common to most drivers. To achieve this goal, the framework is designed to work with drivers on an "opt-in" basis. When you write a UMDF driver, you provide callback routines for only the events that affect your device. For example, some devices require intervention immediately after they are turned on and just before they are turned off. The driver for such a device can implement callback functions that the framework calls at those times.

The driver includes code to handle only those events for which its device requires device-specific support. All other events can be handled by framework defaults.
In addition, a driver can configure its I/O request queues so that the framework stops dispatching requests while the device is in a low-power state and resumes dispatching after the device has returned to the operational state. Similarly, if an I/O request arrives while the device is in a low-power state, the framework can automatically turn on the device.

Misc Info:

The Windows 10 WDF source code is now available as open source on GitHub. This means that you can debug your driver using WDF source code. Download it from http://github.com/Microsoft/Windows-Driver-Frameworks.
The Windows Driver Kit (WDK) 10 samples are also now published to GitHub. Download them from http://github.com/Microsoft/Windows-Driver-Samples.
Want to benefit from the universal capabilities of UMDF 2? To learn how to port your old UMDF 1 driver, see Porting a Driver from UMDF 1 to UMDF 2.

ref:

Getting started with Windows drivers -

Getting started with UMDF - https://msdn.microsoft.com/en-us/library/windows/hardware/dn384105(v=vs.85).aspx

UMDF overview - https://msdn.microsoft.com/en-us/library/windows/hardware/ff560442%28v=vs.85%29.aspx

Choosing driver model - https://msdn.microsoft.com/en-us/library/windows/hardware/ff554652%28v=vs.85%29.aspx

Comparing UMDF 2.0 Functionality to KMDF - https://msdn.microsoft.com/en-us/library/windows/hardware/dn376882(v=vs.85).aspx

How to convert a KMDF driver to a UMDF 2.0 driver (and vice-versa) - https://msdn.microsoft.com/en-us/library/windows/hardware/dn265585(v=vs.85).aspx

Windows Driver Framework(WDF) development guide - https://msdn.microsoft.com/en-us/library/windows/hardware/ff557565(v=vs.85).aspx

What's New for WDF Drivers in Windows 10 Insider Preview - https://msdn.microsoft.com/en-us/library/windows/hardware/ff544296(v=vs.85).aspx

Linux Kernel Development(miscellaneous)

A kernel module(.ko) is a bit of compiled code that can be inserted into the kernel at run-time, such as with insmod or modprobe. A driver is a bit of code that runs in the kernel to talk to some hardware device. It "drives" the hardware. Most every bit of hardware in your computer has an associated driver. A large part of a running kernel is driver code; the rest of the code provides generic services like , , , etc.

A driver may be built statically into the kernel file on disk. (The one in /boot, loaded into RAM at boot time by the early in the boot process.) A driver may also be built as a kernel module so that it can be dynamically loaded later. (And then maybe unloaded.)

Standard practice is to build drivers as kernel modules where possible, rather than link them statically to the kernel, since that gives more flexibility. There are good reasons not to, however:

Sometimes a given driver is absolutely necessary to help the system boot up. That doesn't happen as often as you might imagine, due to the initrd feature.
Statically built drivers may be exactly what you want in a system that is statically scoped, such as an embedded system. That is to say, if you know in advance exactly which drivers will always be needed and that this will never change, you have a good reason not to bother with dynamic kernel modules.

Not all kernel modules are drivers. For example, a relatively recent feature in the is that you can load a different . One exception to this broad statement is the , which has no "driver" per se. Your computer may also contain hardware for which you have no driver.

Linux Header packages only contain the header part of the above (and not all of that - only the "exported" headers), and some of the build infrastructure. So what you are seeing is expected. Header packages do not contain C source code (except for some stubs and build infrastructure code). The whole point of having this type of package is to save space (and bandwidth) - the whole Linux kernel source tree is rather large, and completely unnecessary if you don't intend to compile the kernel yourself. The header packages are built and shipped by distributions to provide just the right things necessary to build modules, but no more. (They certainly do not contain the compiled kernel.)

Installed linux kernel binaries are usually installed in the /boot directory, along with bootloader binaries and configuration files. (This is sometimes an independent filesystem, not mounted by default.) The exact name of the files depends on the kernel and distribution. (So does the bootloader.)

Installed linux kernel modules reside in sub-directories /lib/modules/`uname -r`/

Full kernel source code: /usr/src/linux is a traditional place to put kernel sources, but nothing prevents you from putting kernel sources elsewhere. This path is also often just a symbolic link to a directory. The symlink is there to simplify building applications that depend on the kernel source. KConfig files are a description of the kernel configuration options (and their dependencies) that are available for a given directory/module. Apart from that, it's all (mostly) C source code, header files and Makefiles. There are a few helper scripts here and there, and assembly source too.

Linux device drivers commands:

Know the linux kernel version from kernel source code =>

make kernelversion
Check the top-level Makefile contents

Miscellaneous commands

Linux command syntax	Linux command description
ls -R /lib/modules/$(uname -r)	Command to list all modules available for a given linux system
modinfo /path/to/module.ko	Display module information
insmod kernel-module-name	Install a module to a running kernel. NOTE: this command does not resolve module dependencies
modprobe kernel-module-name	Install a module to a running kernel inlcuding dependencies
depmod -a	Rebuild module dependancy database using /lib/modules/$(uname -r)/modules.dep
insmod --force kernel-module-name	Force insmod to load module even if its build for a defferent module version
modprobe -n -v kernel-module-name	Display insmod commands to load module and its dependencies. Useful when modprobe gives up due to dependency problem
lsmod	Display all modules currently loaded into a kernel
rmmod kernel-module-name	Command to remove a module from a running kernel

General linux commands:

list all the dependent libraries of a binary => ldd $$(NAME_OF_BINARY)

list all API exposed by shared library =>

nm –D –defined-only name_of_binary (Exported sumbols are indicated by a T. Required symbols that must be loaded from other shared objects have a U)
objdump –T $(NAME_OF_BINARY)

find the bitness of a file =>

readelf –h $(NAME_OF_BINARY)
objdump –a $(NAME_OF_BINARY)

Print CRC checksum and byte counts of each file =>

Print MD5 hash sum of a file =>

Print SHA1 hash sum of a file =>

Estimate file space usage => du –h $(NAME_OF_DIRECTORY)

Display amount of free and used memory in the system(in mega bytes) =>

Find the process ID of a running program =>

Outputs file status =>

Print the of printable characters in files => strings

Locate the binary, source, and manual page files for a command =>

Display a tree of processes =>

ref:

Kernel newbies - http://kernelnewbies.org/

Kernel coverage at LWN.net - http://lwn.net/Kernel/

Linux kernel documentation(all in one) - https://code.google.com/p/kernel-all-in-one/source/browse/trunk/Docs/?r=70

Unreliable Guide To Hacking The Linux Kernel - http://kernelbook.sourceforge.net/kernel-hacking.pdf

Linux kernel development 3rd edition by Robert Love - https://archive.org/details/pdfy-PjVB7QjMXCW8xzZj, http://reiber.org/nxt/pub/Linux/LinuxKernelDevelopment/Linux.Kernel.Development.3rd.Edition.pdf

Linux Device drivers 3rd edition - http://lwn.net/Kernel/LDD3/

Understanding the Linux Kernel, 3rd Edition By Daniel P. Bovet, Marco Cesati - http://gauss.ececs.uc.edu/Courses/c4029/code/memory/understanding.pdf

The Linux Kernel Module Programming Guide - http://www.tldp.org/LDP/lkmpg/2.6/lkmpg.pdf

Linux kernel in a Nutshell - https://aligunduz.org/random/LinuxKernelInANutshell.pdf, http://www.kroah.com/lkn/

Linux Device Driver Dos and Don'ts - http://kernel-janitor.sourceforge.net/kernel-janitor/docs/driver-howto.html

Kernel APIs, Part 1: Invoking user-space applications from the kernel - http://www.ibm.com/developerworks/library/l-user-space-apps/

Linux Device Drivers: Kernel to User Space communication

Inside Kernel space code:

To convert data from User space to Kernel space =>

To send data from Kernel space to User space =>

To read/write data to IO device below Device Driver(Firmware) =>

The file_operations structure is defined in linux/fs.h, and holds pointers to functions defined by the driver that perform various operations on the device. Each field of the structure corresponds to the address of some function defined by the driver to handle a requested operation.

C99 way of assigning to elements of a structure. You should use this syntax in case someone wants to port your driver. It will help with compatibility:

struct file_operations fops = {

.read = device_read,

.write = device_write,

.open = device_open,

.release = device_release

};

debugfs is a special filesystem (technically referred as a kernel space-user-space interface) available in Linux kernel since version 2.6.10-rc3. It is a simple to use RAM-based file system specially designed for debugging purposes. debugfs exists as a simple way for kernel developers to make information available to user space.

Inside User space code:

Open device directly from User space and read/write.

example: open(“/dev/device_name”, O_RDWR);

Shared Memory:

Shared memory in user space => mmap(), munmap()

Shared memory in kernel space => remap_pfn_range()

PCI Device Drivers:

- register a new pci driver
- unregister a new pci driver

- network device drivers
- char device drivers
- block device drivers

- driver initialization entry point
- driver exit entry point

Miscellaneous commands:

=> lists all pci device drivers

=> lists all Loadable kernel modules(LKM)

=> add/remove a kernel module

ref:

User space memory access from Kernel => http://www.ibm.com/developerworks/library/l-kernel-memory-access/

Linux Kernel to User space communication(copy_from-user, copy_to_user) - http://www.makelinux.net/ldd3/chp-3-sect-7

Google Research(papers ...) - http://research.google.com/

Kernel to User space communication - http://people.ee.ethz.ch/~arkeller/linux/multi/kernel_user_space_howto-2.html

Use mmap() shared memory instead of copy_to_user => http://www.linuxforums.org/forum/kernel/158548-avoid-memory-copying-between-user-space-kernel-space.html

Debugfs -

Debugfs Sample Code -

Debugging Kernel with debgufs -

Miscellaneous Character Drivers -

Linux Device Driver Overview - http://inbasudhakar.blogspot.com/2011/08/pci-driver-flow.html

Practical and effective sandboxing for non-root users - http://people.csail.mit.edu/nickolai/papers/kim-mbox.pdf

packetdrill: Scriptable Network Stack Testing, from Sockets to Packets - http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en/us/pubs/archive/41316.pdf

http://linuxforthenew.blogspot.com/2013/01/why-doshould-we-use-copyfromuser-or.html

File System Filter Drivers vs Device Drivers

The following subsections describe some of the differences between file system filter drivers and device drivers.

No Power Management

Because file system filter drivers are not device drivers and thus do not control hardware devices directly, they do not receive IRP_MJ_POWER requests. Instead, power IRPs are sent directly to the storage device stack. In rare circumstances, however, file system filter drivers might interfere with power management. For this reason, file system filter drivers should not register dispatch routines for IRP_MJ_POWER in theDriverEntry routine, and they should not call PoXxx routines.

No WDM

File system filter drivers cannot be Windows Driver Model (WDM) drivers. The Microsoft Windows Driver Model is only for device drivers. For more information about file system driver development in Windows Me, Windows 98, and Windows 95, see the Windows Me Driver Development Kit (DDK).

No AddDevice or StartIo

Because file system filter drivers are not device drivers and thus do not control hardware devices directly, they should not have AddDevice or StartIo routines.

Different Device Objects Created

Although file system filter drivers and device drivers both create device objects, they differ in the number and kinds of device objects that they create.

Device drivers create physical and functional device objects to represent devices. The Plug and Play (PnP) Manager builds and maintains a global device tree that contains all device objects that are created by device drivers. The device objects that file system filter drivers create are not contained in this device tree.

File system filter drivers do not create physical or functional device objects. Instead, they create control device objects and filter device objects. The control device object represents the filter driver to the system and to user-mode applications. The filter device object performs the actual work of filtering a specific file system or volume. A file system filter driver normally creates one control device object and one or more filter device objects.

Other Differences

Because file system filter drivers are not device drivers, they do not perform direct memory access (DMA).

Unlike device filter drivers, which can attach above or below a target device's function driver, file system filter drivers can attach only above a target file system driver. Thus, in device-driver terms, a file system filter driver can be only an upper filter, never a lower filter.

ref:

http://msdn.microsoft.com/en-us/library/ff548075(v=vs.85).aspx

Sample Filter Driver code -

http://read.pudn.com/downloads2/sourcecode/windows/system/4397/WinntInternalFS/filesys/src/sfsdinit.c__.htm

http://read.pudn.com/downloads27/sourcecode/windows/system/86694/CallMsgRing0/CallMsgRing0.c__.htm

http://www.google.com/url?sa=t&source=web&cd=17&ved=0CDYQFjAGOAo&url=http%3A%2F%2Fwww.mglife.ru%2Fpub%2Finfo%2F%25D0%25A3%25D1%2587%25D0%25B5%25D0%25B1%25D0%25BD%25D0%25B8%25D0%25BA%25D0%25B8%2F3%2F6%2FTemp%2FTrueCrypt%2520v4.2a%2520source%2FTrueCrypt%2FDriver%2FNtdriver.c&rct=j&q=IoRegisterShutdownNotification%20C%20code&ei=iz8BTpCZHIKutwfi5LyDDg&usg=AFQjCNGp87K_UESdkXWKxQ-NPRZTe5CL8Q&sig2=O9yhjBMU4klYADle9A7mKQ

Windows Minifilter Driver - User mode to Kernel mode Communication

The supports through "communication ports". The controls security on the port by specifying a to be applied to the . Communication through a communication port is not buffered, so it is faster and more efficient. A user-mode application or service can reply to messages from a minifilter driver for .

ref:

User mode to Kernel mode communication (Communication Ports) - http://msdn.microsoft.com/en-us/library/ff539277(v=vs.85).aspx

User-Mode Interactions: Guidelines for Kernel-Mode Drivers(KM-UMGuide.doc) -

http://msdn.microsoft.com/en-us/windows/hardware/gg487414.aspx

- http://www.osronline.com/showThread.cfm?link=90062

- http://sww-it.ru/wp-content/uploads/2011/University%20of%20Tokyo%20Windows%20Internals%20Lectures/10-LPC/LPC.pdf

Kernel Mode to User Mode Communication WFP Driver - http://www.osronline.com/showthread.cfm?link=205452

User to Kernel Communication Model - http://muglin.ru/messagesupport.ppt

- http://muglin.ru/UserModeLib.ppt

Windows FileSystem Mini Filter Driver Development

A File system filter driver intercepts requests targeted at a file system or another file system filter driver. By intercepting the request before it reaches its intended target, the filter driver can extend or replace functionality provided by the original target of the request. Examples of file system filter drivers include anti-virus filters, backup agents, and encryption products. To develop file systems and file system filter drivers, use the IFS (Installable File System) Kit, which is provided with the Windows Driver Kit (WDK).

Filter Manager and Minifilters Basics:

The Filter Manager is a file system filter driver provided by Microsoft that simplifies the development of third-party filter drivers and solves many of the problems with the existing legacy filter driver model, such as the ability to control load order through an assigned altitude. A filter driver developed to the Filter Manager model is called a minifilter. Every minifilter driver has an assigned altitude, which is a unique identifier that determines where the minifilter is loaded.

A minifilter driver can be loaded at any time while the system is running. If a minifilter driver's specifies a driver start type of SERVICE_BOOT_START, SERVICE_SYSTEM_START, or SERVICE_AUTO_START, the minifilter driver is loaded according to existing load order group definitions for file system filter drivers, to support interoperability with legacy filter drivers. While the system is running, a minifilter driver can be loaded through a service start request (sc start, net start, or the service APIs), or through an explicit load request (fltmc load, FltLoadFilter, orFilterLoad).

A minifilter driver's DriverEntry routine is called when the minifilter driver is loaded, so the minifilter driver can perform initialization that will apply to all instances of the minifilter driver. Within its DriverEntry routine, the minifilter driver calls FltRegisterFilter to register callback routines with the filter manager and FltStartFiltering to notify the filter manager that the minifilter driver is ready to start attaching to volumes and filtering I/O requests.

Minifilter driver instances are defined in the INF file used to install the minifilter driver. A minifilter driver's INF file must define a , and it can define additional instances. These definitions apply across all volumes. Each instance definition includes the instance name, its altitude, and flags that indicate whether the instance can be attached automatically, manually, or both. The default instance is used to order minifilter drivers so that the filter manager calls the minifilter driver's mount and instance setup callback routines in the correct order. The default instance is also used with explicit attachment requests when the caller doesn't specify an instance name.

Excerpts:

1. loads FilterDriver …. It’s equivalent of () API
2. calls DriverEntry() API
3. StopService() API calls the DriverUnloadCallback() registered with FltRegisterFilter() API … This is not a real DriverUnload ; it’s kind of stopping the driver to work !
4. DeleteService() API calls the () API & which really unloads the driver.

i.e

FilterLoad() => CreateService() + StartService()

FilterUnload() => DeleteService()

StartService() => DriverEntry()

ref:

An Introduction To Writing TDI Filter Drivers - http://www.iseclab.org/papers/Writing_TDI_Drivers.pdf

Sample TDI Driver Firewall Opensource code - http://sourceforge.net/projects/tdifw/

File System Filter Driver - http://msdn.microsoft.com/en-us/windows/hardware/gg462968

FileSystem Mini Filter Drivers - http://msdn.microsoft.com/en-us/library/ff540402(v=vs.85).aspx

FileSystem Mini Filter Driver development Guide - http://download.microsoft.com/download/e/b/a/eba1050f-a31d-436b-9281-92cdfeae4b45/filterdriverdeveloperguide.doc

Developing FileSystem Mini filter drivers - http://www.osr.com/filters.pdf

Filter Driver Development Kit - http://www.osr.com/fddk.html