Uber Jaeger Distributed Tracing Platform

Distributed Tracing is a way to see and understand the whole chain of events in a complex interaction between microservices. It’s often run as part of a service mesh, which is a way to manage and observe micro-services. Distributed Tracing also called distributed request tracing, is a method used to profile and monitor applications, especially those built using a microservices architecture. Distributed tracing helps pinpoint where failures occur and what causes poor performance.

OpenTracing is a vendor-agnostic API to help developers easily instrument tracing into their code base. It’s open because no one company owns it. In fact, many tracing tooling companies are getting behind OpenTracing as a standardized way to instrument distributed tracing.

Why Jaeger ?
    As on-the-ground microservice practitioners are quickly realizing, the majority of operational problems that arise when moving to a distributed architecture are ultimately grounded in two areas: networking and observability. It is simply an orders of magnitude larger problem to network and debug a set of intertwined distributed services versus a single monolithic application.

The Jaeger tracing system is an open-source tracing system for microservices, and it supports the OpenTracing standard. Jaeger uses distributed tracing to follow the path of a request through different microservices. Rather than guessing, we can see a visual representation of the call flows. Jaeger was initially published as open source by Uber Technologies and has evolved since then. The system gives you distributing tracing, root cause analysis, service dependency analysis, and more. Jaeger has tooling for Go, Java, JavaScript (Node.js,) Python, and C++.

ref:

Distributed Tracing - https://opentracing.io/docs/overview/what-is-tracing/

OpenTracing specification -
    1. https://opentracing.io/
    2. https://github.com/opentracing/specification

Jaeger Distributed Tracing Platform =>

    1. Tracing - https://www.jaegertracing.io/

    2. github source code - https://github.com/jaegertracing

    3. Client libraries Documentation - https://www.jaegertracing.io/docs/client-libraries/

            golang client github code - https://github.com/jaegertracing/jaeger-client-go

    4. Architecture - https://www.jaegertracing.io/docs/architecture/

    5. Documentation - https://www.jaegertracing.io/docs

    6. Tutorial - https://www.scalyr.com/blog/jaeger-tracing-tutorial/

    7. Overview - https://www.redhat.com/en/topics/microservices/what-is-jaeger

Swagger OpenAPI Specification - Describing RESTful APIs(using JSON)

OpenAPI Specification(formerly Swagger Specification) defines a standard, language-agnostic interface to RESTful APIs which allows both humans and computers to discover and understand the capabilities of the service without access to source code, documentation, or through network traffic inspection. An OpenAPI definition can then be used by documentation generation tools to display the API, code generation tools to generate servers and clients in various programming languages, testing tools, and many other use cases.

API specifications can be written in YAML or JSON. The format is easy to learn and readable to both humans and machines.

OpenAPI Generator allows generation of API client libraries (SDK generation), server stubs, documentation and configuration automatically given an OpenAPI Spec (both 2.0 and 3.0 are supported).

The core of OpenAPI Generator is based on Mustache Templates. Every line of code in your generated SDK is created by reading a template file in and substituting values into it based on your OpenAPI descriptions.

ref:

OpenAPI specification(swagger) - https://swagger.io/specification/

github OpenAPI Specification - https://github.com/OAI/OpenAPI-Specification/blob/master/versions/3.0.2.md

Generate Swgager API HTML online from YAML - https://editor.swagger.io/

Generate Swgager API JSON online from YAML(from File menu) - https://editor.swagger.io/

 

OpenAPI V3 code generation -  

    1.  https://mux.com/blog/an-adventure-in-openapi-v3-api-code-generation/

    2. https://swagger.io/tools/swagger-codegen

Usage of openapi-generator-cli - https://github.com/OpenAPITools/openapi-generator

Use mustache templates with Openapi generator code - https://openapi-generator.tech/docs/templating/ 

Openapi generator source code - https://github.com/OpenAPITools/openapi-generator/tree/master/modules/openapi-generator/src/main

swagger openapi generator mustache resource templates - https://github.com/swagger-api/swagger-codegen/tree/master/modules/swagger-codegen/src/main/resources

swagger openapi generator mustache templates for golang client -
https://github.com/swagger-api/swagger-codegen/tree/master/modules/swagger-codegen/src/main/resources/go

swagger openapi generator mustache templates golang server - https://github.com/swagger-api/swagger-codegen/tree/master/modules/swagger-codegen/src/main/resources/go-server
 

openapi Custom code generator & template - https://openapi-generator.tech/docs/customization/  

Misc =>

     1. https://www.openapis.org/

     2. https://swagger.io/docs/specification/about/ 

Istio Service Mesh

A Service Mesh is the network of micro-services that make up applications and the interactions between them. As a service mesh grows in size and complexity, it can become harder to understand and manage. Its requirements can include discovery, load balancing, failure recovery, metrics, and monitoring.

Developers must use micro-services to architect for portability, meanwhile operators are managing extremely large hybrid and multi-cloud deployments. Istio’s diverse feature set lets you successfully, and efficiently, run a distributed micro-service architecture, and provides a uniform way to secure, connect, and monitor micro-services. At a high level, Istio helps reduce the complexity of these deployments, and eases the strain on your development teams. It is a completely open source service mesh that layers transparently onto existing distributed applications. It is also a platform, including APIs that let it integrate into any logging platform, or telemetry or policy system.

You add Istio support to services by deploying a special sidecar proxy(envoy) throughout your environment that intercepts all network communication between micro-services, then configure and manage Istio using its control plane functionality. Istio’s security capabilities free developers to focus on security at the application level. Istio provides the underlying secure communication channel, and manages authentication, authorization, and encryption of service communication at scale. With Istio, service communications are secured by default, letting you enforce policies consistently across diverse protocols and runtimes - all with little or no application changes.

Istio’s robust tracing, monitoring, and logging features give you deep insights into your service mesh deployment. Gain a real understanding of how service performance impacts things upstream and downstream with Istio’s monitoring features, while its custom dashboards provide visibility into the performance of all your services and let you see how that performance is affecting your other processes.

While Istio is platform independent, using it with Kubernetes (or infrastructure) network policies, the benefits are even greater, including the ability to secure pod-to-pod or service-to-service communication at the network and application layers.

Command above downloads the latest release (numerically) of Istio => curl -sL https://istio.io/downloadIstioctl | sh -

To download a specific version(say  version 1.4.3) => curl -L https://istio.io/downloadIstio | ISTIO_VERSION=1.4.3 sh -

Command to enable istio sidecar injection=> kubectl label namespace default istio-injection=enabled

After successful integration of istio in Kubernetes(K8s) environment, a kubernetes namspace called "istio-system" will be created.
Command to check all pods, svcs, deployments created under istio-system namespace => kubectl -n istio-system get all

Istio commandline => istioctl

istio visualization tools => kiali, Netflix’s Vizceral(https://github.com/nmnellis/vistio)
 
Tool troubleshooting and Monitoring Docker & Kubernetes => Weave Scope

TimeSeries Database to store Metrics => Prometheus

Metrics Visualization Tool => Grafana

Open source tracing tool, to visualize the latency of requests within your mesh => Zipkin

Vizceral is an open source project released by Netflix to monitor network traffic between applications and clusters in near real time. Vistio is an adaptation of Vizceral for Istio and mesh monitoring. It utilizes metrics generated by Istio Mixer which are then fed into Prometheus. Vistio queries Prometheus and stores that data locally to allow for the replaying of traffic. 


ref:

Istio documentation - https://istio.io/docs/

Istio Blog - https://istio.io/blog/

Istio Architecture - https://istio.io/docs/ops/deployment/architecture/

Demystifying Istio sidecar model - https://istio.io/blog/2019/data-plane-setup/

Istio routing Basics - https://www.tigera.io/blog/istio-routing-basics/ 

Kubernetes Istio overview - https://itnext.io/kubernetes-istio-simply-visually-explained-58a7d158b83f 

Kubernetes Documentation - https://kubernetes.io/docs 

Kubernetes Blog - https://kubernetes.io/blog/ 

What is Istio - https://istio.io/docs/concepts/what-is-istio/

Istio Service Mesh source code - https://github.com/istio/istio

Envoy Proxy source code - https://github.com/envoyproxy/envoy

Envoy Proxy Blog - https://blog.envoyproxy.io/

Vistio Tutorial - https://itnext.io/vistio-visualize-your-istio-mesh-using-netflixs-vizceral-b075c402e18e

Istio Commandline tool(istioctl) - https://istio.io/docs/reference/commands/istioctl/

Istio installation - https://istio.io/docs/setup/install/multicluster/gateways/ 

How To Install and Use Istio With Kubernetes - https://www.digitalocean.com/community/tutorials/how-to-install-and-use-istio-with-kubernetes 

Running Istio on Kubernetes(k8s):

    1. https://www.tigera.io/blog/running-istio-on-kubernetes-in-production-part-i/ 

    2. https://www.tigera.io/blog/istio-and-kubernetes-in-production-part-2-tracing/ 

Istio installation with shared control plane - https://istio.io/docs/setup/install/multicluster/shared/  

Customizable Install with Istioctl - https://istio.io/docs/setup/install/istioctl/ 

Istio Traffic Management issues - https://istio.io/docs/ops/common-problems/network-issues/ 

Istio diagnostic tools - https://istio.io/docs/ops/diagnostic-tools/ 

Istio debugging Envoy - https://istio.io/docs/ops/diagnostic-tools/proxy-cmd/

IBM tutorial on Istio Multi Cluster support:

1. https://www.ibm.com/cloud/blog/istio-multicluster-support
2. https://www.infoq.com/articles/kubernetes-multicluster-comms/ 

setup multi-cluster istio dns setup - https://preliminary.istio.io/docs/setup/install/multicluster/gateways/#setup-dns

istio coredns issues -

1. https://github.com/coredns/coredns/issues/3412 
2. https://github.com/istio/istio.io/pull/5303

Istio Service Discovery(or Traffic routing) - https://istio.io/docs/concepts/traffic-management/

Istio Service Discovery(or Traffic routing) Architecture - https://istio.io/docs/ops/deployment/architecture/ 

Demystifying Istio's Sidecar Injection Model - https://istio.io/blog/2019/data-plane-setup/

Kubernetes Istio - https://platform9.com/blog/kubernetes-service-mesh-a-comparison-of-istio-linkerd-and-consul/

Istio and CoreDns - https://istio.io/docs/setup/install/multicluster/gateways/ 

Kubernetes coredns istio grpc plugin - https://github.com/istio-ecosystem/istio-coredns-plugin 

Istio diagnostic tools - https://istio.io/docs/ops/diagnostic-tools/ 

Istio observability - https://istio.io/docs/tasks/observability/

Istio Metrics - https://istio.io/docs/tasks/observability/metrics/

Istio + Kiali - https://medium.com/kialiproject/intro-to-service-mesh-and-istio-part-i-a27984726fdb

Istio Monitoring tools - https://www.datadoghq.com/blog/istio-monitoring-tools/

Istio WebAssesmbly(wasm) - https://istio.io/blog/2020/wasm-announce/ 

Envoy Proxy - https://github.com/envoyproxy

Envoy Proxy Envoy - https://github.com/envoyproxy/envoy

Envoy Get Started - https://www.envoyproxy.io/docs/envoy/latest/start/start

Envoy with Ambassador - https://www.envoyproxy.io/docs/envoy/latest/start/distro/ambassador  

Proxy Wasm - https://github.com/proxy-wasm/spec

Envoy Wasm - https://github.com/envoyproxy/envoy-wasm

Kubernetes and IPTables

route is a command that displays, adds and deletes entries from the kernel's TCP/IP routing table (aka "Forwarding Information Base"). iptables is a command that displays, adds, and deletes entries from Netfilter, the Linux kernel's packet filtering and manipulating subsystem. It handles NAT.

 

A firewall is a type of network security tool that controls the inbound and outbound network traffic according to its predefined rule set. We can use a firewall along with other safety measures to protect our servers from hackers' pries and attacks.

iptables is a user-space utility program that allows a system administrator to configure the IP packet filter rules of the Linux kernel firewall, implemented as different Netfilter modules. The filters are organized in different tables, which contain chains of rules for how to treat network traffic packets. iptables allows configuring the tables provided by the Linux kernel firewall, as well as the chains and rules it stores. The kernel module currently used for iptables only applies to IPv4 traffic, to configure firewall rules for IPv6 connections instead use ip6tables, which respond to the same command structures as iptables.

iptables is made up of some basic structures, as seen below:

    1. TABLES
    2. CHAINS
    3. TARGETS
 
TABLES:
    TABLES are the major pieces of the packet processing system, and they consist of FILTER, NAT, and MANGLE. FILTER is used for the standard processing of packets, and it’s the default table if none other is specified. NAT is used to rewrite the source and/or destination of packets and/or track connections. MANGLE is used to otherwise modify packets, i.e. modifying various portions of a TCP header, etc.

CHAINS:
    CHAINS are then associated with each table. Chains are lists of rules within a table, and they are associated with “hook points” on the system, i.e. places where you can intercept traffic and take action. Here are the default table/chain combinations:

FILTER: Input, Output, Forward
NAT: Prerouting, Postrouting, Output
MANGLE: Prerouting, Postrouting, Input, Output, Forward
And here’s when the different chains do their thing:

PREROUTING: Immediately after being received by an interface.
POSTROUTING: Right before leaving an interface.
INPUT: Right before being handed to a local process.
OUTPUT: Right after being created by a local process.
FORWARD: For any packets coming in one interface and leaving out another.
In other words, if you want to process packets as they leave your system, but without doing any NAT or MANGLE(ing), you’ll look to the OUTPUT chain within the FILTER table. If you want to process packets coming from the outside destined for your local machine, you’ll want to use the same FILTER table, but the INPUT chain. See the image below for a visual representation of this.

TARGETS:
    TARGETS determine what will happen to a packet within a chain if a match is found with one of its rules. A two most common ones are DROP and ACCEPT. So if you want to drop a packet on the floor, you write a rule that matches the particular traffic and then jump to the DROP target. Conversely, if you want to allow something, you jump to the ACCEPT target — simple enough.

How Packets Move:
    Packets move through netfilter by traversing chains. Each non-empty chain has a list of rules in it, which packets are checked against one after another. If a match is found, the packet is processed. If no match is found, the default action is taken. The default action for a chain is also called its policy. By default, chain policies are to jump to the ACCEPT target, but this can be set to DROP if so desired (I suggest it).

Quality Of Service(QoS) packet marking can be performed by adding rules to the PREROUTING chain in the mangle table.

    iptables -t mangle -A PREROUTING -p icmp -j MARK --set-mark 0x1
    iptables -t mangle -A PREROUTING -p icmp -j RETURN

iptables rules are in memory only and won’t survive reboot. To preserve the firewall rules there is a tool to save/restore(iptables-save/iptables-restore) the rules to a file and a service(iptables-persistent on Debian/Ubuntu) to automatically save/restore the rules.

Kubernetes networking uses iptables to control the network connections between pods (and between nodes), handling many of the networking and port forwarding rules. Also, port mapping is greatly simplified (and mostly eliminated) since each pod has its own IP address and its container can listen on its native port. Be careful of using services that may create conflicting iptables rules. You can check the rules by running iptables-save, which dumps the rule set to STDOUT.


If you intend to expose application services externally, by either using the NodePort or LoadBalancing service types, traffic forwarding must be enabled in your iptables rule set. If you find that you are unable to access a service from outside of the network used by the pod where your application is running, check that your iptables rule set does not contain a rule similar to the following:

:FORWARD DROP [0:0]

In the case of Istio Service Mesh, istio-init This init container is used to setup the iptables rules so that inbound/outbound traffic will go through the sidecar proxy.

ref:

iptables -

    1.https://www.karlrupp.net/en/computer/nat_tutorial

    2. https://wiki.archlinux.org/index.php/Iptables

    3. https://medium.com/swlh/manage-iptables-firewall-for-docker-kubernetes-daa5870aca4d  

 
   4. http://www.adminsehow.com/2011/09/iptables-packet-traverse-map/
 

iptables firewall - http://linux-training.be/networking/ch14.html

 

Redhat Security guide - https://web.mit.edu/rhel-doc/4/RH-DOCS/rhel-sg-en-4/s1-firewall-ipt-fwd.html

 

Beginner's guide to iptables - https://www.howtogeek.com/177621/the-beginners-guide-to-iptables-the-linux-firewall/

iptables primer - https://danielmiessler.com/study/iptables/ 

How to add iptables on CentOS -

    1. https://wiki.centos.org/HowTos/Network/IPTables

    2. https://upcloud.com/community/tutorials/configure-iptables-centos/

iptables cheat sheet -

    1. https://www.andreafortuna.org/2019/05/08/iptables-a-simple-cheatsheet/
    
    2. https://gist.github.com/davydany/0ad377f6de3c70056d2bd0f1549e1017

    3. https://medium.com/faun/cheatsheet-iptables-c32d203772b7

 
kube-iptables-tailer is a service that gives you better visibility on networking issues in your Kubernetes cluster by detecting the traffic denied by iptables and surfacing corresponding information to the affected Pods via Kubernetes events -  https://github.com/box/kube-iptables-tailer

Kubernetes iptables rules -
https://camo.githubusercontent.com/4194f82b206a9b40bf50a18760eb84b9d338b521/68747470733a2f2f63646e2e7261776769742e636f6d2f63696c69756d2f6b38732d69707461626c65732d6469616772616d2f6d61737465722f6b756265726e657465735f69707461626c65732e737667

How Kubernetes Networking works -

    1.  https://neuvector.com/network-security/kubernetes-networking/#:~:text=Kubernetes%20networking%20uses%20iptables%20to,networking%20and%20port%20forwarding%20rules.&text=Also%2C%20port%20mapping%20is%20greatly,listen%20on%20its%20native%20port.

     2. https://docs.oracle.com/en/operating-systems/oracle-linux/kubernetes/kube_admin_config_iptables.html

     3. https://www.stackrox.com/post/2020/01/kubernetes-networking-demystified/

     4. https://www.digitalocean.com/community/tutorials/how-to-inspect-kubernetes-networking

Istio iptables - https://github.com/istio/cni/blob/master/tools/packaging/common/istio-iptables.sh

Istio sidecar injection model using iptables - https://istio.io/blog/2019/data-plane-setup/