Tech Kaizen: VIRTUALIZATION

Showing posts with label VIRTUALIZATION. Show all posts

Inter Virtual Machine Communication

Virtual Machine(VM) to Virtual Machine(VM) Communication:

A new Linux socket family called VM sockets (AF_VSOCK address family) enables bi-directional, many-to-one, communication between a hypervisor and its virtual machines, using the classic BSD sockets API. Although VM sockets were originally introduced by VMware, they can be used with QEMU+KVM virtual machines as well.

https://github.com/torvalds/linux/blob/master/include/uapi/linux/vm_sockets.h

https://lwn.net/Articles/536167/

https://medium.com/@mdlayher/linux-vm-sockets-in-go-ea11768e9e67

libvirt is Hypervisor Management API spec/standard that all hypervisors implement.

https://www.ibm.com/developerworks/library/l-libvirt/index.html

https://developer.ibm.com/articles/os-python-kvm-scripting1/

https://superuser.com/questions/1121796/prevent-direct-guest-to-guest-communication-on-a-virtual-network

VirtioVsock is a Host/Guest communication API. VirtioVsock internally makes use of VM sockets. To take advantage of VM sockets (using virtio-vsock), the Linux kernel (on both the hypervisor and guest) and QEMU must be fairly up-to-date. Kernel 4.8+ is required on both machines, and QEMU 2.8+ is required to execute the VM.

https://wiki.qemu.org/Features/VirtioVsock

https://vmsplice.net/~stefan/stefanha-kvm-forum-2015.pdf

https://medium.com/@mdlayher/linux-vm-sockets-in-go-ea11768e9e67

All VMs(Guests) on the same machine(host) can use Shared Memory as a VM is basically a process running on the host. All the Virtual Machines(VMs) can be assigned static IPs.

Miscellaneous:

Virtualization - https://www.cse.unsw.edu.au/~cs9242/08/lectures/11-virt.pdf

A high performance inter-domain communication approach for virtual machines - https://ac.els-cdn.com/S0164121212002476/1-s2.0-S0164121212002476-main.pdf?_tid=55c90ff8-b3df-440b-8ac9-f4cb9bb25681&acdnat=1548280536_1cd3c00cba3abaa654c9cb40de656dde

Efficient Shared Memory Message Passing for Inter-VM Communications @ https://www.researchgate.net/publication/29609820_Efficient_Shared_Memory_Message_Passing_for_Inter-VM_Communications

Shared-Memory Optimizations for Inter Virtual Machine Communication -
https://www.cc.gatech.edu/~lingliu/papers/2016/ACM-ComputingSurveys-YiRen.pdf

https://www.semanticscholar.org/paper/Shared-Memory-Optimizations-for-Communication-Ren-Liu/03ed30028164bd7b5215da3fb431f4402071a49f

Efficient shared memory message passing for inter-VM communications - https://hal.archives-ouvertes.fr/hal-00368622/document

Challenges for Inter Virtual Machine Communication - https://pdfs.semanticscholar.org/4295/889cc3e8b3f46fa7eb8e34a793433cdd0929.pdf

A Survey on the Security of Virtual Machines - https://www.cse.wustl.edu/~jain/cse571-09/ftp/vmsec/

Fido: Fast Inter-Virtual-Machine Communication for Enterprise Appliances - https://www.usenix.org/legacy/event/usenix09/tech/full_papers/burtsev/burtsev_html/index.html

nc-vsock - a netcat-like utility for AF_VSOCK - https://github.com/stefanha/nc-vsock

Virtualization Blog - http://blog.vmsplice.net/

Stefan Hajnoczi, Virtualization expert details:

https://vmsplice.net/~stefan/stefanha-kvm-forum-2015.pdf

Blog: http://blog.vmsplice.net/
Specification: http://goo.gl/mi6LCR
Code:
● https://github.com/stefanha/linux => vsock
● https://github.com/stefanha/qemu => vsock
Slides available on my website: http://vmsplice.net/

Hypervisor: KVM, XEN

KVM(Kernel-based Virtual Machine) & XEN are bare metal (type-1) hypervisors. Any Hypervisor(whether type-1 or type-2) needs a OS.

Installation of KVM hypervisor => Install minimal Ubuntu (or) CentOS and install KVM module(.ko).

KVM converts Linux into a Type-1 hypervisor. There is only one kernel that is used (and that is the Linux kernel, which has KVM included). On the flip side, I can make an argument that Xen is not a Type-1 hypervisor, because the CPU and memory is scheduled by the hypervisor, but IO is scheduled by Dom0, which is a guest (so it's not bare metal). In the KVM architecture, the CPU, memory, and IO are scheduled by the Linux kernel with KVM.

references:

KVM -

https://en.wikipedia.org/wiki/Kernel-based_Virtual_Machine

https://www.linux-kvm.org/page/Main_Page

Xen -

https://en.wikipedia.org/wiki/Xen

https://www.xenproject.org/developers/teams/hypervisor.html

https://xen-orchestra.com/#!/xo-home

KVM: Bare-Metal Hypervisor - https://virtualizationreview.com/Blogs/Mental-Ward/2009/02/KVM-BareMetal-Hypervisor.aspx?m=1

KVM: Bare metal virtualization on Ubuntu with KVM - https://fabianlee.org/2018/08/27/kvm-bare-metal-virtualization-on-ubuntu-with-kvm/

Hyper-visors overview - https://www.sciencedirect.com/topics/computer-science/hypervisors

Linux Containers - https://linuxcontainers.org/

Miscellaneous links:

Virtualization - https://www.cse.unsw.edu.au/~cs9242/08/lectures/11-virt.pdf

https://wiki.ubuntu.com/KvmWithBridge

https://jaredkipe.com/blog/programming/general/ubuntu-kvm-1-bridge-network/

https://blog.ubuntu.com/2015/11/05/canonical-launches-worlds-fastest-hypervisor-lxd

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/virtualization_deployment_and_administration_guide/sect-starting_suspending_resuming_saving_and_restoring_a_guest_virtual_machine-starting_a_defined_domain

https://searchservervirtualization.techtarget.com/definition/hosted-hypervisor-Type-2-hypervisor

https://searchservervirtualization.techtarget.com/tip/Open-source-Xen-hypervisor-in-the-Linux-kernel-Does-it-matter

Vagrant, Chef ... Virtual environments Management

Vagrant is an open-source (MIT) tool for building and managing virtualized development environments. It can be seen as a wrapper around virtualization software such as VirtualBox, KVM, VMware and around configuration management software such as Ansible, Chef, Salt or Puppet.

Chef is a configuration management tool written in Ruby and Erlang. Chef is a powerful automation platform that transforms complex infrastructure into code, bringing your servers and services to life. Whether you’re operating in the cloud, on-premises, or a hybrid, Chef automates how applications are configured, deployed, and managed across your network, no matter its size. Chef is used to streamline the task of configuring and maintaining a company's servers, and can integrate with cloud-based platforms such as Rackspace, Amazon EC2, Google Cloud Platform, OpenStack, SoftLayer and Microsoft Azure to automatically provision and configure new machines.

ref:

Vagrant wiki - http://en.wikipedia.org/wiki/Vagrant_%28software%29

Vagrant - https://www.vagrantup.com/

Vagrant documentation - http://docs.vagrantup.com/v2/

Vagrant 'Getting started' - http://docs-v1.vagrantup.com/v1/docs/getting-started/

Chef tool - https://www.chef.io/chef/

Chef 'Getting started' - https://learn.chef.io/

Vagrant & Chef provisioning - https://docs.vagrantup.com/v2/provisioning/chef_client.html

Building and managing Hadoop with Chef - https://www.chef.io/blog/chefconf-talks/building-and-managing-hadoop-with-chef-john-martin/

Introduction to Vagrant -

VagrantBox - http://www.vagrantbox.es/

Application Virtualization & Application Streaming

Application virtualization is an umbrella term that describes software technologies that improve portability, manageability and compatibility of applications by encapsulating them from the underlying operating system on which they are executed. A fully virtualized application is not installed in the traditional sense, although it is still executed as if it were. The application is fooled at runtime into believing that it is directly interfacing with the original operating system and all the resources managed by it, when in reality it is not. In this context, the term "virtualization" refers to the artifact being encapsulated (application), which is quite different to its meaning in hardware virtualization, where it refers to the artifact being abstracted (physical hardware)

The basic concept of Application Streaming has its foundation in the way modern computer programming languages and operating systems produce and run application code. Only specific parts of a computer program need to be available at any instance for the end user to perform a particular function. This means that a program need not be fully installed on a client computer, but parts of it can be delivered over the network as and when they are required.

ref:

Application Virtualization - http://en.wikipedia.org/wiki/Application_virtualization

Application Streaming - http://en.wikipedia.org/wiki/Application_streaming

MSDN Webcast: Application Virtualization And Software Streaming Within Systems Management Server (Level 200) - http://www.techrepublic.com/whitepapers/msdn-webcast-application-virtualization-and-software-streaming-within-systems-management-server-level-200/355156

Application Virtualization for Developers - http://msdnrss.thecoderblogs.com/2011/04/application-virtualization-for-developers/

Microsoft Application Virtualization (App-V) - http://www.microsoft.com/en-us/windows/enterprise/products-and-technologies/virtualization/app-v.aspx

Application Virtualization Smackdown - http://www.brianmadden.com/blogs/rubenspruijt/archive/2010/09/22/application-virtualization-smackdown-head-to-head-analysis-of-endeavors-citrix-installfree-microsoft-spoon-symantec-and-vmware.aspx

App streaming vs installation - http://www.theregister.co.uk/2011/04/12/app_streaming/

Application Virtualization with Microsoft Dynamics NAV 2009 R2 - http://msdn.microsoft.com/en-us/library/gg670186.aspx

VMware vShield EndPoint Security: Agentless VM Security Solution

Strengthen security for virtual machines and their hosts while improving performance by orders of magnitude for endpoint protection, with VMware vShield Endpoint, part of the VMware vShield family. Offload antivirus and anti-malware processing to dedicated security-hardened virtual machines delivered by VMware partners. Leverage existing investments and manage antivirus and anti-malware policies for virtualized environments with the same management interfaces as physical environments.

VMware vShield Endpoint comprises a hardened, security virtual machine that hosts the third-party anti-virus software. Each virtual machine requiring AV protection only requires a small-footprint, driver software, deployed as a loadable kernel module per vSphere host.

How Does VMware vShield Endpoint Work ?

vShield Endpoint protects virtual machines and their hosts against viruses, malware and other threats. vShield Endpoint plugs directly into vSphere and consists of three components:

1. Hardened security virtual machine (delivered by VMware partners)

2. Driver for virtual machines to offload file events

3. VMware Endpoint Security (EPSEC) loadable kernel module (LKM) to link the first two components at the hypervisor layer

Partner Integrations :

Integration of VMware vShield Endpoint with security virtual machine solutions from VMware partners is facilitated through VMware EPSEC, which provides a library and API for introspection into file activity at the hypervisor layer. vShield Endpoint monitors virtual machine file events and notifies the antivirus engine, via VMware EPSEC, which scans and returns a disposition. It also supports scheduled full and partial file scans initiated by the antivirus engine in the security virtual machine.

ref

vShield Endpoint Security - http://www.vmware.com/products/vshield-endpoint/features.html

VMware vShield Endpoint - http://www.vmware.com/files/pdf/vmware-vshield-endpoint-ds-en.pdf & http://www.vmware.com/files/pdf/vmworld/security-media-backgrounder-en.pdf

Understanding VMware vShield Endpoint And Agentless Malware Protection -

http://www.cloudave.com/4583/vmware-vshield-endpoint-api-and-agentless-malware-protection/?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+CloudAve+%28CloudAve%29

vShield Programming API - http://www.vmware.com/pdf/vshield_41_api.pdf

Trend Micro integration with VMware endpoint security - http://www.vmwareforum2011.com/library/documents/finland_1100_trendmicro_session.pdf

Full Virtualization vs Para Virtualization

There are several ways to implement virtualization. Two leading approaches are full virtualization and para-virtualization.

Full virtualization is designed to provide total abstraction of the underlying physical system and creates a complete virtual system in which the guest operating systems can execute. No modification is required in the guest OS or application; the guest OS or application is not aware of the virtualized environment so they have the capability to execute on the VM just as they would on a physical system. This approach can be advantageous because it enables complete decoupling of the software from the hardware. As a result, full virtualization can streamline the migration of applications and workloads between different physical systems. Full virtualization also helps provide complete isolation of different applications, which helps make this approach highly secure. Microsoft® Virtual Server and VMware® ESX Server™ software are examples of full virtualization.

Para-virtualization presents each VM with an abstraction of the hardware that is similar but not identical to the underlying physical hardware. Para-virtualization techniques require modifications to the guest operating systems that are running on the VMs. As a result, the guest operating systems are aware that they are executing on a VM—allowing for near-native performance. Para-virtualization methods are still being developed and thus have limitations, including several insecurities such as the guest OS cache data, unauthenticated connections, and so forth. Xen is an open source virtualization software based on paravirtualization technology.

Xen 3.0 Aerchitecture: The Intel x86 architecture provides four levels of privilege modes. These modes, or rings, are numbered 0 to 3, with 0 being the most privileged. In a non-virtualized system, the OS executes at ring 0 and the applications at ring 3. Rings 1 and 2 are typically not used. In Xen para-virtualization, the VMM executes at ring 0, the guest OS at ring 1, and the applications at ring 3. This approach helps to ensure that the VMM possesses the highest privilege, while the guest OS executes in a higher privileged mode than the applications and is isolated from the applications. Privileged instructions issued by the guest OS are verified and executed by the VMM.

ref:

vmware paravirtualization - http://www.vmware.com/files/pdf/VMware_paravirtualization.pdf

Xen Para virtualization - http://www.dell.com/downloads/global/power/ps3q05-20050191-Abels.pdf

Para virtualization with ESX - http://www.virtuallifestyle.nl/2008/10/paravirtualization-with-esx/

Compare Vmware Esx, Microsoft Hyper-V & Citrix Xen - http://www.vmware.com/technical-resources/advantages/robust-foundation.html

Misc -

http://virtualization.info/en/news/2008/02/long-vmware-road-to-cpu-para.html

http://www.vmware.com/pdf/VMware_VMI_performance.pdf

http://blogs.vmware.com/performance/2008/02/vmi-performance.html

http://www.electricrelaxation.com/2007/05/27/building-a-vmi-enabled-kernel-for-debian-lenny-testing

VMware software products - ESXi, vCenter, vMotion

ESX Server

VMware’s ESX server is at the highest end of features and price of all the VMware server applications. The ESX actually loads right on to “bare-metal” servers. Thus, there is no need to first load an underlying operating system prior to loading VMware ESX. What is unique about ESX is that it comes with its own modified Linux Kernel called VMKernel (based on Red Hat Enterprise Linux). ESX is now sold in a suite of products called VMware Infrastructure.

VMWare Server

VMware’s Server is a FREE VMware virtualization product built for use in production servers. Unlike ESX, VMware Server still uses the underlying host operating system. With VMware Server, you loose the some of the functionality and performance of the ESX server but don’t have as great of price tag (its free!)

Workstation

VMware’s Workstation is for use on a client workstation. For example, say that I want to run both Windows 2003 server and Linux Fedora Core 5 on my desktop workstation, which is running Windows XP. VMware Workstation would be the program I would use to do this. This would allow me the flexibility to run these guest operating systems to test various applications and features. I could also create snapshots of them to capture their configuration at a certain point in time and easily duplicate them to create other virtual machines (such as moving them to a VMware Server).

VirtualCenter

Virtual Center provides a centralized management console for all VMware servers. If you plan to grow your farm of virtualized servers into the ten’s and hundreds, over time, you should have Virtual Center in your plans.

Vmotion

Like magic, VMotion can move a running virtual server to another physical server, without interrupting that server’s requests. This can be done for maintenance of hardware or to better balance workload. VMotion requires a SAN be used for these virtual machines. Both Virtual Center and VMotion are part of the VMWare vManage offering.

VMware Converter

The VMware Converter is used to move physical machine operating systems to virtual machines. This tool automates the migration to virtual machines. Besides migrating, the Converter can be used to create images of physical systems for disaster recovery purposes. Currently in Beta, the VMware Converter will replace the P2V Assistant.

ref:

VMware software products - http://en.wikipedia.org/wiki/List_of_VMware_software

VMware product's comparision - http://www.petri.co.il/virtual_vmware_versions_compared.htm

VMware APIs and SDK - http://www.vmware.com/support/pubs/sdk_pubs.html

vShield Programming API - http://www.vmware.com/pdf/vshield_41_api.pdf

vShield Endpoint Security - http://www.vmware.com/products/vshield-endpoint/features.html

VMware VI (vSphere) Java API - http://sourceforge.net/projects/vijava/ & http://vijava.sourceforge.net/

Java sample code using VI Java API - http://humairahmed.com/blog/?p=1026

Automate VMware tasks using C# with VIX API - http://www.codeproject.com/KB/library/VMWareTasks.aspx

Introduction to Virtualization - http://www.kernelthread.com/publications/virtualization/

Beyond VMware - http://www.petri.co.il/virtual_virtualization_options_compared.htm

Virtual Desktop Infrastructure (VDI)

Desktop virtualization is the act of decoupling the different computing layers and storing some or all of them in a data center. Virtual desktop infrastructure (VDI) is the practice of hosting a desktop operating system within a virtual machine (VM) running on a centralized server. VDI is a variation on the client/server computing model, sometimes referred to as server-based computing (SBC).

Let’s look at VDI from a high level and compare it to more traditional server-based computing models.

The image on the right shows a traditional SBC model. The users all have access to a desktop GUI via an individual session on the terminal server. This server has a single OS installed; an instance of Terminal Services to provide the sessions and session management, and a set of applications that can be used by all the users on the server.

In the VDI model on the left, a single server is used again, but a hardware virtualization layer is added to this server in place of a more traditional OS like Windows Server. The Virtualization layer provides numerous Virtual Machines that are each supplied with an operating system, applications, and a unique GUI / desktop environment for each user.

As you can tell by the image, a VDI solution provides the same basic functionality of a traditional SBC solution. This functionality is (primarily) to provide a centralized desktop via a protocol like RDP or ICA. Besides the Virtualization layer and numerous OSes, the two solutions look almost identical. (The key word is ’almost’.)

With VDI you gain a couple of benefits that you cannot achieve in a SBC environment, including things like:

The ability to provide a unique environment for each and every user.
Each of these environments can be completely customized with different apps and settings without impacting other users.
Users can be granted more control of their own “virtual” desktop to allow them to install and modify applications if needed.
Applications that were not multi-user friendly (i.e. “we can’t get this to run on Citrix”) can be run in this environment since each instance is just like installing the app on a new desktop.

ref:

Virtualization Wiki - http://en.wikipedia.org/wiki/Virtualization

Introduction to Virtualization - http://www.kernelthread.com/publications/virtualization/

Virtualization Best Practises - http://support.ca.com/phpdocs/0/common/impcd/r11/virtualization/doc/virtualization_best_practices.pdf , http://support.ca.com/phpdocs/0/common/impcd/r11/virtualization/virt_main.htm

Virtualization: Fueling Green Data Centers - http://msdn.microsoft.com/en-us/library/dd347164.aspx

Microsoft Virtualization Portal - http://www.microsoft.com/virtualization/en/us/default.aspx

Virtualization White Papers - http://www.microsoft.com/virtualization/en/us/resources.aspx

Virtualization information on Microsoft Technet - http://technet.microsoft.com/en-us/virtualization/default.aspx

Virtualization Case Studies - http://www.microsoft.com/virtualization/en/us/case-studies-featured.aspx

Microsoft Virtualization learning portal - http://www.microsoft.com/learning/en/us/training/virtualization.aspx

Virtualization implementations and optimization guide - http://searchservervirtualization.techtarget.com/tutorial/Virtualization-implementations-and-optimization-guide

VMWorld 2010 Presentations - http://www.vmworld.com/community/conferences/2010/knowledge-experts