Big Data

Big data is an all-encompassing term for any collection of data sets so large and complex that it becomes difficult to process using on-hand data management tools or traditional data processing applications. The trend to larger data sets is due to the additional information derivable from analysis of a single large set of related data, as compared to separate smaller sets with the same total amount of data.

MapReduce is a programming model and an associated implementation for processing and generating large data sets with a parallel, distributed algorithm on a cluster. A MapReduce program is composed of a Map() procedure that performs filtering and sorting (such as sorting students by first name into queues, one queue for each name) and a Reduce() procedure that performs a summary operation (such as counting the number of students in each queue, yielding name frequencies). The "MapReduce System" (also called "infrastructure" or "framework") orchestrates the processing by marshalling the distributed servers, running the various tasks in parallel, managing all communications and data transfers between the various parts of the system, and providing for redundancy and fault tolerance.

MapReduce actually refers to two separate and distinct tasks that Hadoop programs perform. The first is the map job, which takes a set of data and converts it into another set of data, where individual elements are broken down into tuples (key/value pairs). The reduce job takes the output from a map as input and combines those data tuples into a smaller set of tuples. As the sequence of the name MapReduce implies, the reduce job is always performed after the map job.

ref:

Big Data - http://en.wikipedia.org/wiki/Big_data, http://en.wikipedia.org/wiki/MapReduce

Hadoop tutorial - http://www.coreservlets.com/hadoop-tutorial/

What is Hadoop - http://www-01.ibm.com/software/data/infosphere/hadoop/

MapReduce: Simplified Data Processing on Large Clusters - http://static.googleusercontent.com/media/research.google.com/en/us/archive/mapreduce-osdi04.pdf

Google’s MapReduce Programming Model(Revisited) - http://userpages.uni-koblenz.de/~laemmel/MapReduce/paper.pdf

MapReduce: Simplified Data Processing on Large Clusters - http://www.cs.utexas.edu/~pingali/CS395T/2012sp/lectures/MR-nikhil-panpalia.pdf

Hadoop/MapReduce - http://www.cs.colorado.edu/~kena/classes/5448/s11/presentations/hadoop.pdf

Apache's implementation of Google's MapReduce framework - https://www.defcon.org/images/defcon-17/dc-17-presentations/defcon-17-calca-anguiano-hadoop.pdf

Intel big data - http://www.intel.com/bigdata

Apache Hadoop Framework Spotlights - http://www.intel.com/content/www/us/en/big-data/big-data-apache-hadoop-framework-spotlights-landing.html

Linux Kernel Development(miscellaneous)

A kernel module(.ko) is a bit of compiled code that can be inserted into the kernel at run-time, such as with insmod or modprobe. A driver is a bit of code that runs in the kernel to talk to some hardware device. It "drives" the hardware. Most every bit of hardware in your computer has an associated driver. A large part of a running kernel is driver code; the rest of the code provides generic services like memory management, IPC, scheduling, etc.

A driver may be built statically into the kernel file on disk. (The one in /boot, loaded into RAM at boot time by the boot loader early in the boot process.) A driver may also be built as a kernel module so that it can be dynamically loaded later. (And then maybe unloaded.)

Standard practice is to build drivers as kernel modules where possible, rather than link them statically to the kernel, since that gives more flexibility. There are good reasons not to, however:

• Sometimes a given driver is absolutely necessary to help the system boot up. That doesn't happen as often as you might imagine, due to the initrd feature.
• Statically built drivers may be exactly what you want in a system that is statically scoped, such as an embedded system. That is to say, if you know in advance exactly which drivers will always be needed and that this will never change, you have a good reason not to bother with dynamic kernel modules.

Not all kernel modules are drivers. For example, a relatively recent feature in the Linux kernel is that you can load a different process scheduler. One exception to this broad statement is the CPU chip, which has no "driver" per se. Your computer may also contain hardware for which you have no driver.

Linux Header packages only contain the header part of the above (and not all of that - only the "exported" headers), and some of the build infrastructure. So what you are seeing is expected. Header packages do not contain C source code (except for some stubs and build infrastructure code). The whole point of having this type of package is to save space (and bandwidth) - the whole Linux kernel source tree is rather large, and completely unnecessary if you don't intend to compile the kernel yourself. The header packages are built and shipped by distributions to provide just the right things necessary to build modules, but no more. (They certainly do not contain the compiled kernel.)

Installed linux kernel binaries are usually installed in the /boot directory, along with bootloader binaries and configuration files. (This is sometimes an independent filesystem, not mounted by default.) The exact name of the files depends on the kernel and distribution. (So does the bootloader.)

Installed linux kernel modules reside in sub-directories /lib/modules/`uname -r`/

Full kernel source code: /usr/src/linux is a traditional place to put kernel sources, but nothing prevents you from putting kernel sources elsewhere. This path is also often just a symbolic link to a directory. The symlink is there to simplify building applications that depend on the kernel source. KConfig files are a description of the kernel configuration options (and their dependencies) that are available for a given directory/module. Apart from that, it's all (mostly) C source code, header files and Makefiles. There are a few helper scripts here and there, and assembly source too.

Linux device drivers commands:

1. Know the linux kernel version from kernel source code =>
1. make kernelversion
2. Check the top-level Makefile contents
2. Miscellaneous commands

Linux command syntax	Linux command description
ls -R /lib/modules/$(uname -r)	Command to list all modules available for a given linux system
modinfo /path/to/module.ko	Display module information
insmod kernel-module-name	Install a module to a running kernel. NOTE: this command does not resolve module dependencies
modprobe kernel-module-name	Install a module to a running kernel inlcuding dependencies
depmod -a	Rebuild module dependancy database using /lib/modules/$(uname -r)/modules.dep
insmod --force kernel-module-name	Force insmod to load module even if its build for a defferent module version
modprobe -n -v kernel-module-name	Display insmod commands to load module and its dependencies. Useful when modprobe gives up due to dependency problem
lsmod	Display all modules currently loaded into a kernel
rmmod kernel-module-name	Command to remove a module from a running kernel

General linux commands:

1. list all the dependent libraries of a binary => ldd $$(NAME_OF_BINARY)
2. list all API exposed by shared library =>
1. nm –D –defined-only name_of_binary (Exported sumbols are indicated by a T. Required symbols that must be loaded from other shared objects have a U)
2. objdump –T $(NAME_OF_BINARY)
3. find the bitness of a file =>
1. readelf –h $(NAME_OF_BINARY)
2. objdump –a $(NAME_OF_BINARY)
4. Print CRC checksum and byte counts of each file => cksum
5. Print MD5 hash sum of a file => md5sum
6. Print SHA1 hash sum of a file => sha1sum
7. Estimate file space usage => du –h $(NAME_OF_DIRECTORY)
8. Display amount of free and used memory in the system(in mega bytes) => free -m
9. Find the process ID of a running program => pidof
10. Outputs file status => stat
11. Print the strings of printable characters in files => strings
12. Locate the binary, source, and manual page files for a command => whereis
13. Display a tree of processes => pstree

ref:

Kernel newbies - http://kernelnewbies.org/

Kernel coverage at LWN.net - http://lwn.net/Kernel/

Linux kernel documentation(all in one) - https://code.google.com/p/kernel-all-in-one/source/browse/trunk/Docs/?r=70

Unreliable Guide To Hacking The Linux Kernel - http://kernelbook.sourceforge.net/kernel-hacking.pdf

Linux kernel development 3rd edition by Robert Love - https://archive.org/details/pdfy-PjVB7QjMXCW8xzZj,  http://reiber.org/nxt/pub/Linux/LinuxKernelDevelopment/Linux.Kernel.Development.3rd.Edition.pdf

Linux Device drivers 3rd edition - http://lwn.net/Kernel/LDD3/

Understanding the Linux Kernel, 3rd Edition By Daniel P. Bovet, Marco Cesati - http://gauss.ececs.uc.edu/Courses/c4029/code/memory/understanding.pdf

The Linux Kernel Module Programming Guide - http://www.tldp.org/LDP/lkmpg/2.6/lkmpg.pdf

Linux kernel in a Nutshell - https://aligunduz.org/random/LinuxKernelInANutshell.pdf, http://www.kroah.com/lkn/

Linux Device Driver Dos and Don'ts - http://kernel-janitor.sourceforge.net/kernel-janitor/docs/driver-howto.html

Kernel APIs, Part 1: Invoking user-space applications from the kernel - http://www.ibm.com/developerworks/library/l-user-space-apps/