Linux Device Drivers: Kernel to User Space communication

Inside Kernel space code:

To convert data from User space to Kernel space => copy_from_user

To send data from Kernel space to User space => copy_to_user

To read/write data to IO device below Device Driver(Firmware) => ioread32/iowrite32

The file_operations structure is defined in linux/fs.h, and holds pointers to functions defined by the driver that perform various operations on the device. Each field of the structure corresponds to the address of some function defined by the driver to handle a requested operation.

C99 way of assigning to elements of a structure.  You should use this syntax in case someone wants to port your driver. It will help with compatibility:

    struct file_operations fops = {

       .read = device_read,

       .write = device_write,

       .open = device_open,

       .release = device_release

    };

debugfs is a special filesystem (technically referred as a kernel space-user-space interface) available in Linux kernel since version 2.6.10-rc3. It is a simple to use RAM-based file system specially designed for debugging purposes. debugfs exists as a simple way for kernel developers to make information available to user space.

Inside User space code:

Open device directly from User space and read/write. 

example: open(“/dev/device_name”, O_RDWR);

Shared Memory:

Shared memory in user space => mmap(), munmap()

Shared memory in kernel space => remap_pfn_range()

PCI Device Drivers:

pci_register_driver - register a new pci driver
pci_unregister_driver - unregister a new pci driver

register_netdev - network device drivers
misc_register - char device drivers
register_blkdev - block device drivers 

module_init - driver initialization entry point
module_exit - driver exit entry point

Miscellaneous commands:

lspci –k => lists all pci device drivers

lsmod => lists all Loadable kernel modules(LKM)

modprobe => add/remove a kernel module

ref:

User space memory access from Kernel => http://www.ibm.com/developerworks/library/l-kernel-memory-access/

Linux Kernel to User space communication(copy_from-user, copy_to_user) -  http://www.makelinux.net/ldd3/chp-3-sect-7

Google Research(papers ...) - http://research.google.com/

Kernel to User space communication - http://people.ee.ethz.ch/~arkeller/linux/multi/kernel_user_space_howto-2.html

Use mmap() shared memory instead of copy_to_user => http://www.linuxforums.org/forum/kernel/158548-avoid-memory-copying-between-user-space-kernel-space.html

Debugfs - 

• http://en.wikipedia.org/wiki/Debugfs, 
• http://www.linuxforu.com/2010/10/debugging-linux-kernel-with-debugfs/

Debugfs Sample Code -  

• http://people.ee.ethz.ch/~arkeller/linux/code/mmap_simple_kernel.c
• http://people.ee.ethz.ch/~arkeller/linux/code/mmap_user.c

Debugging Kernel with debgufs - 

• http://www.linuxforu.com/2010/10/debugging-linux-kernel-with-debugfs/
• http://www.tldp.org/LDP/lkmpg/2.4/html/c577.htm 

Miscellaneous Character Drivers - 

• http://www.linuxjournal.com/article/2920
• http://syprog.blogspot.com/2011/10/hijack-linux-system-calls-part-ii.html
• http://www.embeddedlinux.org.cn/EssentialLinuxDeviceDrivers/final/ch05lev1sec7.html

Linux Device Driver Overview - http://inbasudhakar.blogspot.com/2011/08/pci-driver-flow.html

Practical and effective sandboxing for non-root users -  http://people.csail.mit.edu/nickolai/papers/kim-mbox.pdf

packetdrill: Scriptable Network Stack Testing, from Sockets to Packets - http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en/us/pubs/archive/41316.pdf

http://linuxforthenew.blogspot.com/2013/01/why-doshould-we-use-copyfromuser-or.html

Android Development Overview

Android is a Linux-based operating system designed primarily for touchscreen mobile devices such as smartphones and tablet computers. Initially developed by Android, Inc., which Google backed financially and later bought in 2005, Android was unveiled in 2007 along with the founding of the Open Handset Alliance: a consortium of hardware, software, and telecommunication companies devoted to advancing open standards for mobile devices. 

Android software development is the process by which new applications are created for the Android operating system. Applications are usually developed in the Java programming language using the Android Software Development Kit, but other development tools are available. 

ref:

Android Wiki - 

• http://en.wikipedia.org/wiki/Android_(operating_system)
• http://en.wikipedia.org/wiki/Android_software_development
• http://wiki.processing.org/w/Android

Android Developer - http://developer.android.com/index.html

Android Source - http://source.android.com/

Android Platform Developer's guide - http://www.kandroid.org/online-pdk/guide/index.html

Intel Android resources - http://software.intel.com/en-us/android

Marakana(NewCircle) resources - https://thenewcircle.com/s/tags/android

Browse Android source code online -

• http://grepcode.com/project/repository.grepcode.com/java/ext/com.google.android/android
• https://code.google.com/p/android-source-browsing/source/browse?repo=platform--bionic
• https://gitorious.org/android-enablement/system-core/source/e2bf2ea4d2846031edfc52b942ad53e5467243f6:
• https://android.googlesource.com/
• http://androidxref.com/
• http://androidxref.com/source/
• http://developer.android.com/reference/packages.html
• https://github.com/android
• http://omapzoom.org/

Android Books - https://drive.google.com/folderview?id=0B5BO97vHUkN2ZmV0cE5wMXFabFU&usp=sharing_eid

Android Tools - http://developer.android.com/tools/help/logcat.html

Android Compatibility - https://source.android.com/compatibility/index.html

Android Compatibility Test Suit (CTS) - https://www.google.com/search?q=3.3.4+Pass+Compatibility+Test+Suite+(CTS)&oq=3.3.4+Pass+Compatibility+Test+Suite+(CTS)&aqs=chrome..69i57.528j0&sourceid=chrome&ie=UTF-8

Android Compatibility Definition Documentation (CDD) v4.2 - http://static.googleusercontent.com/external_content/untrusted_dlcp/source.android.com/en/us/compatibility/android-4.2-cdd.pdf

Developing Code for Android - http://www.cprogramming.com/android/android_getting_started.html

Android SDK - http://developer.android.com/sdk/index.html

Downloading and building Android - http://source.android.com/source/building.html

Android SDK ADT bundle - http://developer.android.com/sdk/index.html#download

Android Development Intelpedia Wiki - https://intelpedia.intel.com/SFF_Android_Development

Mobile Application enablement at Intel - https://employeecontent.intel.com/contentdelivery/getcontent.aspx?webpath=cn_Active/BizSites/MobileApp/MobileApp_LandingPage.htm

Intel Application Mobilization Framework - https://employeecontent.intel.com/contentdelivery/getcontent.aspx?webpath=cn_Active/BizSites/MobileApp/MobileApp_FrameWork.htm

Intel for Android Developers Learning Series Landing Page - http://software.intel.com/en-us/blogs/2012/11/28/intel-for-android-developers-learning-series

Development and Optimization for NDK-based Android Game Application on Platforms based on Intel® Architecture - http://software.intel.com/en-us/articles/development-and-optimization-for-ndk-based-android-game-application-on-platforms-based-on

Building Android NDK applications with Intel® IPP - http://software.intel.com/en-us/articles/building-android-ndk-applications-with-intel-ipp

Creating and Porting NDK based Android Apps for IA - http://software.intel.com/en-us/articles/creating-and-porting-ndk-based-android-apps-for-ia/

Android NDK for Intel chipset - http://software.intel.com/en-us/articles/android-ndk-for-intel-architecture

Android System-Level Java/C++ Code - http://software.intel.com/en-us/articles/android-system-level-javac-code-debugging

Installing the Android SDK for Intel® Architecture - http://software.intel.com/en-us/articles/installing-the-android-sdk-for-ia/

How to debug an App for Android x86 and the tools to use - http://software.intel.com/en-us/articles/how-to-debug-an-app-for-android-x86-and-the-tools-to-use/

Remote Application Debug on Android OS - http://software.intel.com/en-us/articles/application-debug-android/

Android 4.3 (Jelly Bean) x86 Emulator System Image - http://software.intel.com/en-us/articles/android-43-jelly-bean-x86-emulator-system-image

Intel ships high-powered C++ compiler for native Android apps - http://www.theregister.co.uk/2013/08/28/intel_cpp_compiler_for_android/

Intel compilers for Windows and Linux are available for internal use here - softwareproducts.intel.com/ILC

How to add a native lib to apk - http://stackoverflow.com/questions/4365227/add-a-native-lib-to-an-apk, http://stackoverflow.com/questions/8295728/how-to-add-a-shared-library-to-my-apk

Where can I browse Android source code on-line - http://stackoverflow.com/questions/449763/where-can-i-browse-android-source-code-on-line

Linux Device Drivers - ttp://lwn.net/Kernel/LDD3/

Porting Unix to Windows - http://msdn.microsoft.com/en-us/library/y23kc048.aspx

Windows to Unix Porting - 

• http://www.ibm.com/developerworks/aix/library/au-porting/
• http://www.ibm.com/developerworks/aix/library/au-porting2/

Porting Windows to Unix - http://www.woodtennis.com/win2unix.html

Porting Windows to Unix - http://www.woodtennis.com/win2unix.html

Unix man pages - http://www.unix.com/man-page/All/3/libm/

Popular Java/JavaScript frameworks/tools

JAX-RS Implementations: 

• Jersey ( JAX-RS Reference Implementation from Sun )
• RESTEasy ( JBoss Application Server's JAX-RS project )
• Apache CXF
• RESTlet

JAX-WS Implementations: 

• Apache CXF
• Apache Axis2
• Metro ( from Sun )
• Spring-WS

ORM Frameworks:

• Hibernate ( from JBoss )
• TopLink ( from Oracle )
• MyBatis ( google code project )
• Cayenne ( from Apache )

Java Messaging Service(JMS) Implementations:

• Apache ActiveMQ
• IBM MQSeries ( Websphere MQ )
• Oracle OpenMQ ( Open Message Queue )
• JBoss HornetQ

Opensource Java Enterprise Service Bus(ESB):

• Mule
• Apache ServiceMix
• Open ESB

Java to XML Bindings:

• JAXB
• Castor
• XMLBeans

Java Search Engines:

• Lucene
• Solr ( uses Lucense internally )

Unit Testing Frameworks:

• TestNG
• JUnit

Javascript Frameworks:

• AngularJS
• Dojo
• JQuery

JavaScript(WebUI) Unit Testing Frameworks:

• Jasmine
• QUnit
• JSTestDriver
• JsUnit

Opensource Web Application UI Testing(Automating Web Browsers) Frameworks:

• WebDriver ( Selenium 2.0 )
• Selenium
• Watij

Analyse Security Vulnerabilities in Code:

• HP Fortify - Identify, fix and protect from security vulnerabilities in software applications
• IBM Rational AppScan - Intended to test Web applications for security vulnerabilities during the development process
• Coverity - Static code analysis tools for C/C++/Java/C# used to find defects and security vulnerabilities in source code

Technology Update

1. AngularJS website - http://angularjs.org/

2. AngularJS Batarang – http://blog.angularjs.org/2012/07/introducing-angularjs-batarang.html

Chrome extension to debug AngularJS application. Extends the Developer Tools, adding tools for debugging and profiling AngularJS applications.

3. Bootstrap .css file => http://twitter.github.com/bootstrap/

Twitter Bootstrap is a free collection of tools for creating websites and web applications. It contains HTML and CSS-based design templates for typography, forms, buttons, charts, navigation and other interface components, as well as optional JavaScript extensions

4. AMD (requirejs) - http://requirejs.org/docs/whyamd.html

The Asynchronous Module Definition (**AMD**) API specifies a mechanism for defining modules such that the module and its dependencies can be asynchronously loaded. This is particularly well suited for the browser environment where synchronous loading of modules incurs performance, usability, debugging, and cross-domain access problems.

5. Playground for WebDevelopers – http://jsfiddle.net/

jsFiddle is a playground for web developers, a tool which may be used in many ways. You can use it as an online editor for snippets build from HTML, CSS and JavaScript.

7. WebStorm Application can be used to perform ‘Remote Debugging’ web applications.

8. Chef is a configuration management tool written in Ruby and Erlang. It uses a pure-Ruby, domain-specific language (DSL) for writing system configuration "recipes" or "cookbooks". Chef was written by Opscode and is released as open sourceunder the Apache License 2.0. Chef is a DevOps tool used for configuring cloud services or to streamline the task of configuring a company's internal servers. Chef automatically sets up and tweaks the operating systems and programs that run in massive data centers.

9. Erlang is a general-purpose concurrent, garbage-collected programming language and runtime system. The sequential subset of Erlang is a functional language, with strict evaluation, single assignment, and dynamic typing. It was designed by Ericsson to support distributed, fault-tolerant, soft-real-time, non-stop applications. It supports hot swapping, so that code can be changed without stopping a system. Facebook uses Erlang to power the backend of its chat service, handling more than 100 million active users.It can be observed in some of its HTTP response headers.

10. Splunk - Splunk captures, indexes and correlates real-time data in a searchable repository from which it can generate graphs, reports, alerts, dashboards and visualizations. It aims tomake machine data accessible across an organization and identifies data patterns,provides metrics, diagnoses problems and provides intelligence for business operation.

11. Nagios - Nagios is an open source computer system monitor, network monitoring and infrastructure monitoring software application. Nagios offers monitoring and alerting for servers, switches, applications, and services. It alerts users when things go wrong and alerts them again when the problem has been resolved.

12. Apache POI - Pure Java libraries for reading and writing files in Microsoft Office formats, such as Word, PowerPoint and Excel.

The Apache POI project(http://en.wikipedia.org/wiki/Apache_POI) contains sub components like - 

• HSSF (Horrible SpreadSheet Format) – reads and writes Microsoft Excel (XLS) format files. It can read files written by Excel 97 onwards; this file format is known as the BIFF 8 format. As the Excel file format is complex and contains a number of tricky characteristics, some of the more advanced features cannot be read.
• XSSF (XML SpreadSheet Format) – reads and writes Office Open XML (XLSX) format files. Similar feature set to HSSF, but for Office Open XML files.

Apache Cassandra: An open source distributed database management system

Apache Cassandra is an open source distributed database management system. It is an Apache Software Foundation top-level project designed to handle very large amounts of data spread out across many commodity servers while providing a highly available service with no single point of failure. It is a NoSQL solution that was initially developed by Facebook and powered their Inbox Search feature until late 2010. Jeff Hammerbacher, who led the Facebook Data team at the time, has described Cassandra as a BigTable data model running on an Amazon Dynamo-like infrastructure.

Apache Cassandra is a distributed storage system for managing structured/unstructured data while providing reliability at a massive scale. Cassandra database is the right choice when you need scalability and high availability without compromising performance. Linear scalability and proven fault-tolerance on commodity hardware or cloud infrastructure make it the perfect platform for mission-critical data. Cassandra's support for replicating across multiple datacenters is best-in-class, providing lower latency for your users and the peace of mind of knowing that you can survive regional outages.

Cassandra's Column Family data model offers the convenience of column indexes with the performance of log-structured updates, strong support for materialized views, and powerful built-in caching.

Cassandra is designed to scale to a very large size across many commodity servers, with no single point of failure.  The philosophy behind the design of the storage portion of Cassandra is that it be able to satisfy the requirements of applications that demand storage of large amounts of structured data. Cassandra aims to run on top of an infrastructure of hundreds of nodes (possibly spread across different datacenters). At this scale, small and large components fail continuously; the way Cassandra manages the persistent state in the context of these failures enables the reliability and scalability of the software systems relying on this service. 

HBase vs Cassandra:

• HBase is based on BigTable (Google)
• Cassandra is based on DynamoDB (Amazon).   Initially developed at Facebook by former Amazon engineers.  This is one reason why Cassandra supports multi data center.  Rackspace is a big contributor to Cassandra due to multi data center support.

Prominent users:

• Cisco's WebEx uses Cassandra to store user feed and activity in near real time.
• Facebook used Cassandra to power Inbox Search, with over 200 nodes deployed. This was abandoned in late 2010 when they built Facebook Messaging platform on HBase.
• IBM has done research in building a scalable email system based on Cassandra
• Netflix uses Cassandra as their back-end database for their streaming services
• Formspring uses Cassandra to count responses, as well as store Social Graph data 
• Twitter announced it is planning to use Cassandra because it can be run on large server clusters and is capable of taking in very large amounts of data at a time.Twitter continues to use it but not for Tweets themselves.
• WalmartLabs (previously Kosmix) uses Cassandra with SSD

ref:

Apache Cassandra - http://cassandra.apache.org/, http://en.wikipedia.org/wiki/Apache_Cassandra

Cassandra - https://wiki.intuit.com/display/ARCH/Cassandra

Cassandra NoSQL Database: Getting Started - http://msdn.microsoft.com/en-us/magazine/jj553519.aspx

HBase vs Cassandra - http://bigdatanoob.blogspot.com/2012/11/hbase-vs-cassandra.html

Use Cassandra to Run Hadoop MapReduce - http://architects.dzone.com/articles/use-cassandra-run-hadoop

Cassandra vs HBase - http://ria101.wordpress.com/2010/02/24/hbase-vs-cassandra-why-we-moved/

Running Hadoop MapReduce With Cassandra NoSQL - http://allthingshadoop.com/2010/04/24/running-hadoop-mapreduce-with-cassandra-nosql/