Popular Open Source Tools: git, gerrit, jenkins, jira

git - source code repository tool
gerrit - code review tool
jenkins - build automation tool
jira - bug tracking tool


Git is an open-source version-control system designed to handle very large projects that are distributed over multiple repositories.

Gerrit is a web-based code review system for projects that use git. Gerrit encourages more centralized use of Git by allowing all authorized users to submit changes, which are automatically merged if they pass code review. In addition, Gerrit makes reviewing easier by displaying changes side by side in-browser and enabling inline comments.

Jenkins is an open source continuous integration tool written in Java. The project was forked from Hudson after a dispute with Oracle. Jenkins provides continuous integration services for software development. It is a server-based system running in a servlet container such as Apache Tomcat. 

JIRA is a proprietary issue tracking product. It provides bug tracking, issue tracking, and project management functions. The Apache Software Foundation uses JIRA and Bugzilla. JIRA includes tools allowing migration from competitor Bugzilla.

ref:

Test Management tools - http://en.wikipedia.org/wiki/Test_management_tools

Android development(git, gerrit) - https://source.android.com/source/developing.html

Gerrit code review - https://review.openstack.org/Documentation/intro-quick.html

Jenkins - http://jenkins-ci.org/

Jenkins overview - http://www.vogella.com/tutorials/Jenkins/article.html

Security Test Tools - http://www.opensourcetesting.org/security.php

Image Processing Libraries: OpenCV, OpenMAX

OpenCV (Open Source Computer Vision) is a library of programming functions mainly aimed at real-time computer vision. The library is cross-platform. It focuses mainly on real-time image processing. If the library finds Intel's Integrated Performance Primitives on the system, it will use these proprietary optimized routines to accelerate itself.

OpenVX (Open Vision Acceleration) is an open, royalty-free standard for cross platform acceleration of computer vision applications. It is designed by the Khronos Group to facilitate portable, optimized and power-efficient processing of methods for vision algorithms. This is aimed for embedded and real-time programs within computer vision and related scenarios. It uses a connected graph representation of operations.

OpenMAX Integration Layer (IL) is a standard API to access Multimedia Components on mobile platforms. It focuses mainly on audio, video, and still images processing. It has been defined by the Khronos group. By means of the OpenMAX IL API, multimedia frameworks can access hardware accelerators on platforms that provide it. Bellagio is an opensource implementation of the OpenMAX IL API that runs on Linux.It is intended to show the usage of the IL API and to allow people to start developing components.This package provides the OpenMAX IL core shared library with a "reference" component.

OpenMAX DL is a OpenMAX DL (Development Layer) APIs contains a comprehensive set of audio, video and imaging functions that can be implemented and optimized on new CPUs, hardware engines, and DSPs and then used for a wide range of accelerated codec functionality such as MPEG-4, H.264, MP3, AAC and JPEG.

Eigen is a C++ template library for linear algebra: matrices, vectors, numerical solvers, and related algorithms. They currently cover the two following feature sets, and more will come in the future: Dense matrix and array manipulations, Sparse linear algebra. Eigenfaces is a popular Facial Recognition library.

Festival library(&Edinburgh Speech Tools library) is a C++ Text to Voice generation library.

Digital Audio Programming

Digital Audio is the most commonly used method to represent sound inside a computer. In this method sound is stored as a sequence of samples taken from the audio signal using constant time intervals. The quality of digital audio signal depends on the time (recording rate) and voltage resolution (usually in an linear integer representation with basic unit one bit). A sample represents volume of the signal at the moment when it was measured. In uncompressed digital audio each sample require one or more bytes of storage. Number of bytes required depends on number of channels (mono, stereo) and sample format (8 or 16 bits, mu-Law, etc.). The length of this interval determines the sampling rate. Normally used sampling rates are between 8 kHz (telephone quality) and 48 kHz (DAT tapes).

Digital audio can be stored in a wide range of formats. Generally speaking, audio comes in two flavors: compressed and uncompressed. Compressed audio can further be subdivided into different kinds of compression: lossless, which preserves the original content exactly, and lossy which achieves more compression at the expense of degrading the audio. Uncompressed PCM audio, on the other hand, is defined by two parameters: the sample rate and the bit-depth. Loosely speaking, the sample rate limits the maximum frequency that can be represented by the format, and the bit-depth determines the maximum dynamic range that can be represented by the format. You can think of bit-depth as determining how much noise there is compared to signal.

In the Linux kernel, there have historically been two uniform sound APIs used. One is OSS(Open Sound System); the other is ALSA (Advanced Linux Sound Architecture). ALSA is available for Linux only, and as there is only one implementation of the ALSA interface, ALSA refers equally to that implementation and to the interface itself.

Android Native code makes uses of OpenSL ES (Open Sound Library for Embedded Systems) for handling Audio processing.

ref:

The ABCs of PCM (Uncompressed) digital audio - http://blog.bjornroche.com/2013/05/the-abcs-of-pcm-uncompressed-digital.html

WAVE PCM soundfile format - https://ccrma.stanford.edu/courses/422/projects/WaveFormat/

An introduction to Linux sound systems and APIs - http://archive09.linux.com/articles/113775

A Tutorial on Using the ALSA Audio API - http://equalarea.com/paul/alsa-audio.html

How to write ALSA driver - http://c-qs.blogspot.com/2014/05/writing-of-alsa-driver.html

alsa vs tinyalsa - http://blog.csdn.net/myzhzygh/article/details/8468210

Open Sound System(OSS) - 

1. http://en.wikipedia.org/wiki/Open_Sound_System
2. https://wiki.archlinux.org/index.php/Open_Sound_System
3. http://www.4front-tech.com/pguide/audio.html

Advanced Linux Sound Architecture(ALSA) -

1. ALSA pcm -  http://www.alsa-project.org/alsa-doc/alsa-lib/pcm.html
2. ALSA -  alsa-lib, libsound2.so - http://www.alsa-project.org/alsa-doc/alsa-lib/pcm.html
3. ALSA Pulse Audio - http://freedesktop.org/software/pulseaudio/doxygen/simple.html
4. Pulse Audio(libpulse.so)  - http://freedesktop.org/software/pulseaudio/doxygen/simple.html
5. Linux sound drivers -  http://www.tldp.org/HOWTO/Sound-HOWTO/

Introduction to Audio programming -

1. How Audio Data is Represented - http://blogs.msdn.com/b/dawate/archive/2009/06/22/intro-to-audio-programming-part-1-how-audio-data-is-represented.aspx
2. Demystifying the WAV Format - http://blogs.msdn.com/b/dawate/archive/2009/06/23/intro-to-audio-programming-part-2-demystifying-the-wav-format.aspx
3. An Introduction to Audio Processing Objects - http://msdn.microsoft.com/en-us/magazine/dn201755.aspx
4. Synthesizing Simple Wave Audio using C# - http://blogs.msdn.com/b/dawate/archive/2009/06/24/intro-to-audio-programming-part-3-synthesizing-simple-wave-audio-using-c.aspx
5. Algorithms for Different Sound Waves in C# - http://blogs.msdn.com/b/dawate/archive/2009/06/25/intro-to-audio-programming-part-4-algorithms-for-different-sound-waves-in-c.aspx

C/C++ wav processing sample code - https://android.googlesource.com/platform/frameworks/av/+/jb-dev/cmds/stagefright/

Linux pcm audio sample code - http://www.alsa-project.org/alsa-doc/alsa-lib/_2test_2pcm_8c-example.html

Android audio native programming using OpenSL ES - http://audioprograming.wordpress.com/2012/03/03/android-audio-streaming-with-opensl-es-and-the-ndk/

Android API for record/play audio(example: pcm): AudioRecord, AudioTrack - 

1. http://www.edumobile.org/android/android-development/audio-recording-in-wav-format-in-android-programming/
2. http://stackoverflow.com/questions/5245497/how-to-record-wav-format-file-in-android
3. http://stackoverflow.com/questions/4871149/how-to-record-voice-in-wav-formt-using-android

Internet of Things(IoT)

The Internet of Things (IoT) is the interconnection of uniquely identifiable embedded computing devices within the existing Internet infrastructure. Typically, IoT is expected to offer advanced connectivity of devices, systems, and services that goes beyond machine-to-machine communications(M2M) and covers a variety of protocols, domains, and applications. The interconnection of these embedded devices is expected to usher in automation in nearly all fields.

The Internet of Things (IoT) is the network of physical objects accessed through the Internet as defined by technology analysts and visionaries. These objects contain embedded technology to interact with internal states or the external environment. In other words, when objects can sense and communicate, it changes how and where decisions are made, and who makes them. A common first step toward the IoT is converting networks on proprietary protocols to IP-based networks.

The IoT needs standard protocols. Popular Open source IoT protocols are CoAP, MQTT, ETSI SmartM2M or LwM2M. Two of the most promising for small devices are -

• CoAP(Constrained Application Protocol)
• MQTT(Message Queuing Telemetry Transport)

CoAP is designed for interoperability with the web. MQTT gives flexibility in communication patterns and acts purely as a pipe for binary data.

CoAP

CoAP is the Constrained Application Protocol from the CoRE (Constrained Resource Environments) IETF group. Like HTTP, CoAP is a document transfer protocol. Unlike HTTP, CoAP is designed for the needs of constrained devices. CoAP packets are much smaller than HTTP TCP flows. Bitfields and mappings from strings to integers are used extensively to save space. Packets are simple to generate and can be parsed in place without consuming extra RAM in constrained devices.

CoAP runs over UDP not TCP. Clients and servers communicate through connectionless datagrams. Retries and reordering are implemented in the application stack. Removing the need for TCP may allow full IP networking in small microcontrollers. CoAP allows UDP broadcast and multicast to be used for addressing. CoAP follows a client/server model. Clients make requests to servers, servers send back responses. Clients may GET, PUT, POST and DELETE resources. CoAP is designed to interoperate with HTTP and the RESTful web at large through simple proxies. As CoAP is datagram based, it may be used on top of SMS and other packet based communications protocols. tinydtls is a library for Datagram Transport Layer Security (DTLS) covering both the client and the server state machine. It is implemented in C and provides support for the mandatory cipher suites specified in CoAP.

MQTT

MQTT is a publish/subscribe messaging protocol designed for lightweight M2M communications. It was originally developed by IBM and is now an open standard. MQTT has a client/server model, where every sensor is a client and connects to a server, known as a broker over TCP. MQTT is message oriented. Every message is a discrete chunk of data, opaque to the broker. Every message is published to an address, known as a topic. Clients may subscribe to multiple topics. Every client subscribed to a topic receives every message published to the topic.

Popular Open Source IoT Operating Systems -

• Contiki
• TinyOS
• RIOT
• mbedOS
• FreeRTOS
• Google Brillo

ref:

Wiki - http://en.wikipedia.org/wiki/Internet_of_Things

Open Interconnect Consortium(OIC) - http://openinterconnect.org/

AllSeen Alliance - https://allseenalliance.org/

Qualcomm Allseen Alliance source code download - https://allseenalliance.org/developers/download

Intel iotivity source code download - https://www.iotivity.org/downloads

IoT standards - iot.eclipse.org— Standards

Google Thread IoT - http://www.threadgroup.org/

Industrial Internet Consortium - http://www.iiconsortium.org/

Google Nest and others group - http://www.threadgroup.org/

Apple HomeKit - https://developer.apple.com/homekit/

IoT overview -

• http://www.cisco.com/web/solutions/trends/iot/overview.html
• http://www.microsoft.com/windowsembedded/en-us/internet-of-things.aspx
• http://www-01.ibm.com/software/info/internet-of-things/
• http://www.businessinsider.com/enterprise-growth-in-the-internet-of-things-market-2014-8
• http://www.mckinsey.com/insights/high_tech_telecoms_internet/the_internet_of_things

IoT protocols(MQTT, CoAP) - http://www.eclipse.org/community/eclipse_newsletter/2014/february/article2.php

libcoap: CoAP implementation in C - http://sourceforge.net/projects/libcoap/

tinyDtls - https://projects.eclipse.org/projects/iot.tinydtls

micro-ecc - https://github.com/kmackay/micro-ecc

jCoAP: CoAP implementation in Java - https://code.google.com/p/jcoap/

CoAP.net: CoAP implementation in C# - http://sourceforge.net/projects/coapnet/

CoRE (Constrained Resource Environments) - https://datatracker.ietf.org/doc/charter-ietf-core/

Open IoT stack for Java - http://iot.eclipse.org/java/download/

Intel IoT developer kit - https://software.intel.com/en-us/blogs/2014/09/08/announcing-intel-iot-developer-kit

Eclipse C/C++ IDE for Intel Galileo/Edison - https://software.intel.com/en-us/getting-started-for-c-c-plus-plus-eclipse-galileo-and-edison

Arduino Development

Arduino is an open-source electronics prototyping platform based on flexible, easy-to-use hardware and software. It's intended for artists, designers, hobbyists, and anyone interested in creating interactive objects or environments.

Arduino is a single-board microcontroller, intended to make building interactive objects or environments more accessible. The hardware consists of an open-source hardware board designed around an 8-bit Atmel AVR microcontroller, or a 32-bit Atmel ARM. Current models feature a USB interface, 6 analog input pins, as well as 14 digital I/O pins that accommodate various extension boards.

Arduino can sense the environment by receiving input from a variety of sensors and can affect its surroundings by controlling lights, motors, and other actuators. The microcontroller on the board is programmed using the Arduino programming language and the Arduino development environment. Arduino projects can be stand-alone or they can communicate with software on running on a computer (e.g. Flash, Processing, MaxMSP. The boards can be built by hand or purchased preassembled; the software can be downloaded for free. The hardware reference designs (CAD files) are available under an open-source license, you are free to adapt them to your needs.

avr-gcc, avr-g++, avr-objcopy, avrdude are the tools used in compiling 'C' code on arduino.

ref:

wiki - http://en.wikipedia.org/wiki/Arduino

Arduino website - http://www.arduino.cc/

Opensource Hardware Group(All about Arduino) - http://opensourcehardwaregroup.com/

Arduino IDE - http://arduino.cc/en/main/software

Arduino development environment - http://arduino.cc/en/guide/Environment

Arduino Development tools - http://playground.arduino.cc/Main/DevelopmentTools

Arduino library - https://github.com/arduino/Arduino/tree/master/hardware/arduino/cores/arduino

Arduino Makefile - https://github.com/sudar/Arduino-Makefile

Programming Arduino Uno in pure C/C++ - 

• https://code.google.com/p/arduino/wiki/BuildProcess
• https://balau82.wordpress.com/2011/03/29/programming-arduino-uno-in-pure-c/
• https://www.ashleymills.com/node/327
• http://www.wikihow.com/Write-Arduino-Software-in-C

Arduino IDE for Microsoft Visual Studio and Atmel Studio - http://www.visualmicro.com/

Introduction to Arduino - http://www.instructables.com/id/Intro-to-Arduino/

Intel Arduino compatible Galileo board - http://www.intel.com/content/www/us/en/do-it-yourself/galileo-maker-quark-board.html

Arduiono IDE and sketch development overview - http://opensourcehardwaregroup.com/tutorial-3-arduino-ide-and-sketch-overview/

Arduino boards summary - http://www.hobbytronics.co.uk/arduino-summary

Advanced Arduino Hacking - https://pragprog.com/magazines/2011-04/advanced-arduino-hacking

Arduino Books(https://drive.google.com/folderview?id=0B5BO97vHUkN2VzEzTXFtRS1tSXc&usp=sharing_eid) -

• Beginning Arduino, 2nd Edition: https://drive.google.com/file/d/0B_hC-3L4eq17T0tsMmxwSk1fUVk/view?usp=sharing_eid
• Getting Started with Arduino v2: https://drive.google.com/file/d/0B_hC-3L4eq17aTdtWUxsRm85Wm8/view?usp=sharing_eid
• Exploring Arduino: Tools and Techniques for Engineering Wizardry: https://drive.google.com/file/d/0B_hC-3L4eq17NzdxSmZTbVk1dE0/view?usp=sharing_eid
• Arduino - A Quick Start Guide Quick Start Guide: https://drive.google.com/file/d/0B_hC-3L4eq17NjR4U1NaWktqMEk/view?usp=sharing_eid

Arduino Cookbooks -

Arduino Cookbook 2nd edition - 

• https://drive.google.com/file/d/0B_hC-3L4eq17cVdwTTlDMUFCM1k/view?usp=sharing_eid
• http://www.mogi.bme.hu/letoltes/MECHATRONIKAI%20&%20IR%C3%81NY%C3%8DT%C3%81STECHNIKAI%20T%C3%81RGYAK/MECHATRONIKAI%20SZIMULACIOK/Ardunio/Arduino_CookBook_2nd_edition.pdf

Arduino Cookbook 1st edition - 

• https://drive.google.com/file/d/0B_hC-3L4eq17djIyeGttX1p1bXc/view?usp=sharing_eid
• http://juniorfall.files.wordpress.com/2011/11/arduino-cookbook.pdf

Functional Programming Overview

Functional programming is a style of programming which models computations as the evaluation of expressions. Functional Programming is when functions, not objects or procedures, are used as the fundamental building blocks of a program. Functions in this sense, not to be confused with 'C' language functions which are just procedures, are analogous to mathematical equations: they declare a relationship between two or more entities. Functional Programming, however, is not about mathematics but about abstraction and reducing complexity: as such, it provides a powerful paradigm in which to tackle complex, real-world programming tasks.

From pragmatic perspective, a beginner needs to understand only two concepts(Immutability and statelessness) in order to use it for everyday Functional Programming applications - 

First, data in functional programs should be immutable, which sounds serious but just means that it should never change. At first, this might seem odd (after all, who needs a program that never changes anything?), but in practice, you would simply create new data structures instead of modifying ones that already exist. For example, if you need to manipulate some data in an array, then you’d make a new array with the updated values, rather than revise the original array. Easy!

Secondly, functional programs should be stateless, which basically means they should perform every task as if for the first time, with no knowledge of what may or may not have happened earlier in the program’s execution. Combined with immutability, this helps us think of each function as if it were operating in a vacuum, blissfully ignorant of anything else in the application besides other functions. In more concrete terms, this means that your functions will operate only on data passed in as arguments and will never rely on outside values to perform their calculations.

The special characteristics and advantages of functional programming are often summed up more or less as follows. Functional programs contain no assignment statements, so variables, once given a value, never change. More generally, functional programs contain no side-effects at all. A function call can have no effect other than to compute its result. This eliminates a major source of bugs, and also makes the order of execution irrelevant — since no side effect can change an expression’s value, it can be evaluated at any time. This relieves the programmer of the burden of prescribing the flow of control. Since expressions can be evaluated at any time, one can freely replace variables by their values and vice versa - that is, programs are “referentially transparent”.

Prominent functional programming languages are Scheme, Haskell, OCaml, Erlang, Clojure, Racket, F#, Common Lisp, Scala ..

ref:

Functional Programming - http://en.wikipedia.org/wiki/Functional_programming

Introduction to Functional programming - http://www.smashingmagazine.com/2014/07/02/dont-be-scared-of-functional-programming/

Why Functional Programming matters - http://www.cse.chalmers.se/~rjmh/Papers/whyfp.pdf, http://worrydream.com/refs/Hughes-WhyFunctionalProgrammingMatters.pdf

Conception, Evolution, and Application of Functional Programming  Languages - http://www.utdallas.edu/~kXh060100/cs6301fa14/p359-hudak.pdf

Functional programming: A step backward - http://www.javaworld.com/article/2078610/java-concurrency/functional-programming--a-step-backward.html

erlang documentation - http://www.erlang.org/doc/pdf/otp-system-documentation.pdf

Haskell - http://www.haskell.org/haskellwiki/Haskell

Misc - 

• http://www.haskell.org/haskellwiki/Functional_programming
• http://c2.com/cgi/wiki?FunctionalProgramming

Bluetooth Technology Overview

Bluetooth is a wireless technology for creating personal networks operating in the 2.4 GHz unlicensed band, with a range of 10 meters. Networks are usually formed ad-hoc from portable devices such as cellular phones, handhelds, and laptops. Unlike Wi-Fi wireless technology, Bluetooth offers higher level service profiles, such as FTP-like file servers, file pushing, voice transport, serial line emulation, and more. RFCOMM(Radio Frequency Communication) protocol provides emulation of serial ports over the L2CAP protocol. RFCOMM is a simple transport protocol, with additional provisions for emulating the 9 circuits of RS-232 (EIATIA-232-E) serial ports. It supports up to 60 simultaneous connections (RFCOMM channels) between two Bluetooth devices. RFCOMM is intended to cover applications that make use of the serial ports of the devices in which they reside. The communication segment is a direct connect Bluetooth link from one device to another.

Bluetooth Low Energy(BLE) or Bluetooth LE, marketed as Bluetooth Smart, is a wireless personal area network technology designed and marketed by the Bluetooth Special Interest Group. Compared to Classic Bluetooth, Bluetooth Smart is intended to provide considerably reduced power consumption and cost while maintaining a similar communication range. Bluetooth Smart uses the same 2.4 GHz radio frequencies as Classic Bluetooth, which allows dual-mode devices to share a single radio antenna. LE does, however, use a simpler modulation system. Bluetooth v4.0 standard is the creation of dual-mode capabilities that allow for classic Bluetooth operation as well as Bluetooth low energy support – the best of both worlds.  The latest Bluetooth specification uses a service-based architecture based on the attribute protocol (ATT). All communication in low energy takes place over the Generic Attribute Profile (GATT). An application or another profile uses the GATT profile so a client and server can interact in a structured way.The server contains a number of attributes, and the GATT Profile defines how to use the Attribute Protocol to discover, read, write and obtain indications. These features support a service-based architecture. The services are used as defined in the profile specifications. GATT enables you to expose service and characteristics defined in the profile specification.The GATT profile is also part of the core and defined in the core specification.

BlueZ: BlueZ is the Bluetooth stack for Linux kernel-based family of operating systems.

BlueZ BLE "C" API:

hci_le_set_scan_enable

hci_le_set_scan_parameters

hci_open_dev

BlueZ Bluetooth "C" API:

hci_inquiry

hci_open_dev

BLE "C" Programming Overview:

Call hci_le_set_scan_parameters to set options, setsockopt(SOL_HCI, HCI_FILTER, filter) to just get LE scan events and then callhci_le_set_scan_enable to turn on scanning(there's a function called hci_get_route to get an available device number). Each device connection was made with a socket(AF_BLUETOOTH, SOCK_SEQPACKET, BTPROTO_L2CAP) which you then tell to connect to a particular device by calling connect on the socket with a struct sockaddr_l2 that has the particular device address in it.

ref:

Bluetooth specifications - 

• https://www.bluetooth.org/en-us/specification/adopted-specifications
• http://chapters.comsoc.org/vancouver/BTLER3.pdf

Bluetooth SIG(Special Interest Group) - https://www.bluetooth.org/en-us

Bluetooth developer portal - https://developer.bluetooth.org/TechnologyOverview/Pages/RFCOMM.aspx

BLE(Bluetooth Low Energy) - http://en.wikipedia.org/wiki/Bluetooth_low_energy

Bluetooth libraries - http://www.jenkinssoftware.com/raknet/manual/bluetooth.html

Linux Bluetooth library , BlueZ - http://www.bluez.org/

Introduction to Bluetooth programming - http://people.csail.mit.edu/albert/bluez-intro/

Best Practices for Bluetooth Development - http://commondatastorage.googleapis.com/io-2013/presentations/129%20-%20Bluetooth%20Best%20Practices.pdf

Bluetooth Specification RFCOMM with Application - http://hccc.ee.ccu.edu.tw/courses/bt/vg/rfcomm.pdf

Linux Bluetooth code - 

• http://lxr.free-electrons.com/source/net/bluetooth/
• http://lxr.free-electrons.com/source/include/net/bluetooth/rfcomm.h

Android Bluetooth overview - http://developer.android.com/guide/topics/connectivity/bluetooth.html

Android audio native programming using OpenSL ES - http://audioprograming.wordpress.com/2012/03/03/android-audio-streaming-with-opensl-es-and-the-ndk/

BlueNMEA is an Android application which sends location data over Bluetooth (RFCOMM) or TCP in the NMEA format - http://max.kellermann.name/projects/blue-nmea/

Misc - http://www.bluetooth.com/Pages/Bluetooth-Home.aspx

BlueZ Ble Server(Advertiser) - https://gist.github.com/AlexanderLea/8e4a7ff46dcfa42ce565

BlueZ Ble Client - https://github.com/carsonmcdonald/bluez-experiments/blob/master/experiments/scantest.c

BlueZ - 

1. BlueZ - http://www.bluez.org/, http://sourceforge.net/projects/bluez/
2. How to setup Bluetooth - http://www.thinkwiki.org/wiki/How_to_setup_Bluetooth
3. BlueZ - https://wiki.archlinux.org/index.php/bluetooth
4. Linux BlueZ programming introduction - http://people.csail.mit.edu/albert/bluez-intro/
5. Linux BlueZ Howto - http://www.tik.ee.ethz.ch/~beutel/pub/bluezhowto.pdf
6. Linux VHCI(Virtual Host Controller Interface) driver - https://sourceforge.net/projects/usb-vhci/

Big Data

Big data is an all-encompassing term for any collection of data sets so large and complex that it becomes difficult to process using on-hand data management tools or traditional data processing applications. The trend to larger data sets is due to the additional information derivable from analysis of a single large set of related data, as compared to separate smaller sets with the same total amount of data.

MapReduce is a programming model and an associated implementation for processing and generating large data sets with a parallel, distributed algorithm on a cluster. A MapReduce program is composed of a Map() procedure that performs filtering and sorting (such as sorting students by first name into queues, one queue for each name) and a Reduce() procedure that performs a summary operation (such as counting the number of students in each queue, yielding name frequencies). The "MapReduce System" (also called "infrastructure" or "framework") orchestrates the processing by marshalling the distributed servers, running the various tasks in parallel, managing all communications and data transfers between the various parts of the system, and providing for redundancy and fault tolerance.

MapReduce actually refers to two separate and distinct tasks that Hadoop programs perform. The first is the map job, which takes a set of data and converts it into another set of data, where individual elements are broken down into tuples (key/value pairs). The reduce job takes the output from a map as input and combines those data tuples into a smaller set of tuples. As the sequence of the name MapReduce implies, the reduce job is always performed after the map job.

ref:

Big Data - http://en.wikipedia.org/wiki/Big_data, http://en.wikipedia.org/wiki/MapReduce

Hadoop tutorial - http://www.coreservlets.com/hadoop-tutorial/

What is Hadoop - http://www-01.ibm.com/software/data/infosphere/hadoop/

MapReduce: Simplified Data Processing on Large Clusters - http://static.googleusercontent.com/media/research.google.com/en/us/archive/mapreduce-osdi04.pdf

Google’s MapReduce Programming Model(Revisited) - http://userpages.uni-koblenz.de/~laemmel/MapReduce/paper.pdf

MapReduce: Simplified Data Processing on Large Clusters - http://www.cs.utexas.edu/~pingali/CS395T/2012sp/lectures/MR-nikhil-panpalia.pdf

Hadoop/MapReduce - http://www.cs.colorado.edu/~kena/classes/5448/s11/presentations/hadoop.pdf

Apache's implementation of Google's MapReduce framework - https://www.defcon.org/images/defcon-17/dc-17-presentations/defcon-17-calca-anguiano-hadoop.pdf

Intel big data - http://www.intel.com/bigdata

Apache Hadoop Framework Spotlights - http://www.intel.com/content/www/us/en/big-data/big-data-apache-hadoop-framework-spotlights-landing.html

Linux Kernel Development(miscellaneous)

A kernel module(.ko) is a bit of compiled code that can be inserted into the kernel at run-time, such as with insmod or modprobe. A driver is a bit of code that runs in the kernel to talk to some hardware device. It "drives" the hardware. Most every bit of hardware in your computer has an associated driver. A large part of a running kernel is driver code; the rest of the code provides generic services like memory management, IPC, scheduling, etc.

A driver may be built statically into the kernel file on disk. (The one in /boot, loaded into RAM at boot time by the boot loader early in the boot process.) A driver may also be built as a kernel module so that it can be dynamically loaded later. (And then maybe unloaded.)

Standard practice is to build drivers as kernel modules where possible, rather than link them statically to the kernel, since that gives more flexibility. There are good reasons not to, however:

• Sometimes a given driver is absolutely necessary to help the system boot up. That doesn't happen as often as you might imagine, due to the initrd feature.
• Statically built drivers may be exactly what you want in a system that is statically scoped, such as an embedded system. That is to say, if you know in advance exactly which drivers will always be needed and that this will never change, you have a good reason not to bother with dynamic kernel modules.

Not all kernel modules are drivers. For example, a relatively recent feature in the Linux kernel is that you can load a different process scheduler. One exception to this broad statement is the CPU chip, which has no "driver" per se. Your computer may also contain hardware for which you have no driver.

Linux Header packages only contain the header part of the above (and not all of that - only the "exported" headers), and some of the build infrastructure. So what you are seeing is expected. Header packages do not contain C source code (except for some stubs and build infrastructure code). The whole point of having this type of package is to save space (and bandwidth) - the whole Linux kernel source tree is rather large, and completely unnecessary if you don't intend to compile the kernel yourself. The header packages are built and shipped by distributions to provide just the right things necessary to build modules, but no more. (They certainly do not contain the compiled kernel.)

Installed linux kernel binaries are usually installed in the /boot directory, along with bootloader binaries and configuration files. (This is sometimes an independent filesystem, not mounted by default.) The exact name of the files depends on the kernel and distribution. (So does the bootloader.)

Installed linux kernel modules reside in sub-directories /lib/modules/`uname -r`/

Full kernel source code: /usr/src/linux is a traditional place to put kernel sources, but nothing prevents you from putting kernel sources elsewhere. This path is also often just a symbolic link to a directory. The symlink is there to simplify building applications that depend on the kernel source. KConfig files are a description of the kernel configuration options (and their dependencies) that are available for a given directory/module. Apart from that, it's all (mostly) C source code, header files and Makefiles. There are a few helper scripts here and there, and assembly source too.

Linux device drivers commands:

1. Know the linux kernel version from kernel source code =>
1. make kernelversion
2. Check the top-level Makefile contents
2. Miscellaneous commands

Linux command syntax	Linux command description
ls -R /lib/modules/$(uname -r)	Command to list all modules available for a given linux system
modinfo /path/to/module.ko	Display module information
insmod kernel-module-name	Install a module to a running kernel. NOTE: this command does not resolve module dependencies
modprobe kernel-module-name	Install a module to a running kernel inlcuding dependencies
depmod -a	Rebuild module dependancy database using /lib/modules/$(uname -r)/modules.dep
insmod --force kernel-module-name	Force insmod to load module even if its build for a defferent module version
modprobe -n -v kernel-module-name	Display insmod commands to load module and its dependencies. Useful when modprobe gives up due to dependency problem
lsmod	Display all modules currently loaded into a kernel
rmmod kernel-module-name	Command to remove a module from a running kernel

General linux commands:

1. list all the dependent libraries of a binary => ldd $$(NAME_OF_BINARY)
2. list all API exposed by shared library =>
1. nm –D –defined-only name_of_binary (Exported sumbols are indicated by a T. Required symbols that must be loaded from other shared objects have a U)
2. objdump –T $(NAME_OF_BINARY)
3. find the bitness of a file =>
1. readelf –h $(NAME_OF_BINARY)
2. objdump –a $(NAME_OF_BINARY)
4. Print CRC checksum and byte counts of each file => cksum
5. Print MD5 hash sum of a file => md5sum
6. Print SHA1 hash sum of a file => sha1sum
7. Estimate file space usage => du –h $(NAME_OF_DIRECTORY)
8. Display amount of free and used memory in the system(in mega bytes) => free -m
9. Find the process ID of a running program => pidof
10. Outputs file status => stat
11. Print the strings of printable characters in files => strings
12. Locate the binary, source, and manual page files for a command => whereis
13. Display a tree of processes => pstree

ref:

Kernel newbies - http://kernelnewbies.org/

Kernel coverage at LWN.net - http://lwn.net/Kernel/

Linux kernel documentation(all in one) - https://code.google.com/p/kernel-all-in-one/source/browse/trunk/Docs/?r=70

Unreliable Guide To Hacking The Linux Kernel - http://kernelbook.sourceforge.net/kernel-hacking.pdf

Linux kernel development 3rd edition by Robert Love - https://archive.org/details/pdfy-PjVB7QjMXCW8xzZj,  http://reiber.org/nxt/pub/Linux/LinuxKernelDevelopment/Linux.Kernel.Development.3rd.Edition.pdf

Linux Device drivers 3rd edition - http://lwn.net/Kernel/LDD3/

Understanding the Linux Kernel, 3rd Edition By Daniel P. Bovet, Marco Cesati - http://gauss.ececs.uc.edu/Courses/c4029/code/memory/understanding.pdf

The Linux Kernel Module Programming Guide - http://www.tldp.org/LDP/lkmpg/2.6/lkmpg.pdf

Linux kernel in a Nutshell - https://aligunduz.org/random/LinuxKernelInANutshell.pdf, http://www.kroah.com/lkn/

Linux Device Driver Dos and Don'ts - http://kernel-janitor.sourceforge.net/kernel-janitor/docs/driver-howto.html

Kernel APIs, Part 1: Invoking user-space applications from the kernel - http://www.ibm.com/developerworks/library/l-user-space-apps/