Docker research
Docker research dimensions :
- Organize and manage cluster
- OpenShift
- Symphony Swarm
- Azure Swarm
- Docker Datacenter Swarm
- Build application image
- Jenkins CI
- TFS CI
- OpenShift CI
- Deploy application bundle
- Java UI and – Java Service
- Node Service – Kafka Producer
- .NET Core – SQL on Linux
- Build DevOps imsage (CI)
- Linux , JDK 1.8 , Node, npm, grunt, JUnit
- Linux, NET Core, Node, npm , NUnit
- Deploy DevOps bundle (CI)
- Jenkins CI – Linux DevOps image – java application
- Jenkins CI – Linux DevOps image – node application
- Jenkins CI – Linux DevOps image – .NET Core aplication
- TFS CI – Linux DevOps image – java application
- TFS CI – Linux DevOps image – node application
- TFS CI – Linux DevOps image – .NET Core aplication
- OpenShift CI – Linux DevOps image – .NET Core aplication
- Develop Docker DevOps pipeline
- Jenkins
- TFS
- Evaluate Docker development environment
- Windows 10 – Nano docker container
- Windows 10 – Linux docker container
- Docker cluster monitoring
- OpenShift
- UCP
- Azure MSOMS
Docker use cases
|
User Story
|
Helpful Links
|
Base images
|
Comments
|
|---|---|---|---|
| As an ASG hosting engineer I would like to automate Swarm cluster creation so I would focus on the application deliverable | Using Docker Engine 1.12 | ||
| As an ASG Technical lead I would like to dockernize Java microservice implementation and publish it to the Artifactory docker repository | |||
| As an ASG Technical lead I would like to dockernize .NET Core microservice implementation and publish it to the Artifactory docker repository | |||
| As an ASG Technical lead I would like to dockernize Node microservice implementation and publish it to the Artifactory docker repository | |||
| As an ASG DevOps Infrastructure engineer I would like to dynamically invoke docker container specified in the docker-compose.ci.build.yml using Jenkins workflow so that I could resolve all dependencies required by the application build process and use Docker cluster elasticity in order to make my DevOps practices more agile and resilient | Focus on designing and implementing Jenkins Docker workflow | ||
| As anASG DevOps Infrastructure engineer I would like to dynamically invoke docker container specified in the docker-compose.ci.build.yml using TFS workflow so that I could resolve all dependencies required by the application build process and use Docker cluster elasticity in order to make my DevOps practices more agile and resilient | Focus on designing and implementing TFS Docker workflow |
Images information
|
Base Image
|
OS
|
Dockerfile Location
|
command
|
Comments
|
|---|---|---|---|---|
| Microsoft official images | https://hub.docker.com/u/microsoft/ | Library of Microsoft images | ||
| microsoft/aspnetcore-build | Linux | https://hub.docker.com/r/microsoft/aspnetcore-build/ | docker pull microsoft/aspnetcore-build | Official images for building ASP.NET Core applications. |
| microsoft/azure-cli | Linux | https://hub.docker.com/r/microsoft/azure-cli/ |
|
Docker image for Microsoft Azure Command Line Interface |
| microsoft/vsts-agent | Linux | https://hub.docker.com/r/microsoft/vsts-agent/ | docker pull microsoft/vsts-agent | Official images for the Visual Studio Team Services (VSTS) agent. |
| microsoft/mssql-server-linux | Linux | https://hub.docker.com/r/microsoft/mssql-server-linux/ | docker pull microsoft/mssql-server-linux | Official images for Microsoft SQL Server on Linux for Docker Engine. |
| CloudBees Official images | https://hub.docker.com/u/cloudbees/ | Library of CloudBees Images | ||
| IBM images | https://hub.docker.com/u/ibmcom/ | Library of IBM Images |
Kubernetes terminology and concepts
Kubernetes aims to decouple applications from machines by leveraging the foundations of distributed computing and application containers. At a high level Kubernetes sits on top of a cluster of machines and provides an abstraction of a single machine.
CLUSTERS
Clusters are the set of compute, storage, and network resources where pods are deployed, managed, and scaled. Clusters are made of nodes connected via a “flat” network, in which each node and pod can communicate with each other. A typical Kubernetes cluster size ranges from 1 – 200 nodes, and it’s common to have more than one Kubernetes cluster in a given data center based on node count and service SLAs.
PODS
Pods are a colocated group of application containers that share volumes and a networking stack. Pods are the smallest units that can be deployed within a Kubernetes cluster. They are used for run once jobs, can be deployed individually, but long running applications, such as web services, should be deployed and managed by a replication controller.
REPLICATION CONTROLLERS
Replication Controllers ensure a specific number of pods, based on a template, are running at any given time. Replication Controllers manage pods based on labels and status updates.
SERVICES
Services deliver cluster wide service discovery and basic load balancing by providing a persistent name, address, or port for pods with a common set of labels.
LABELS
Labels are used to organize and select groups of objects, such as pods, based on key/value pairs.
The Kubernetes Control Plane
The control plane is made up of a collection of components that work together to provide a unified view of the cluster.
ETCD
etcd is a distributed, consistent key-value store for shared configuration and service discovery, with a focus on being: simple, secure, fast, and reliable. etcd uses the Raft consensus algorithm to achieve fault-tolerance and high-availability. etcd provides the ability to “watch” for changes, which allows for fast coordination between Kubernetes components. All persistent cluster state is stored in etcd.
KUBERNETES API SERVER
The apiserver is responsible for serving the Kubernetes API and proxying cluster components such as the Kubernetes web UI. The apiserver exposes a REST interface that processes operations such as creating pods and services, and updating the corresponding objects in etcd. The apiserver is the only Kubernetes component that talks directly to etcd.
SCHEDULER
The scheduler watches the apiserver for unscheduled pods and schedules them onto healthy nodes based on resource requirements.
CONTROLLER MANAGER
There are other cluster-level functions such as managing service end-points, which is handled by the endpoints controller, and node lifecycle management which is handled by the node controller. When it comes to pods, replication controllers provide the ability to scale pods across a fleet of machines, and ensure the desired number of pods are always running.
Each of these controllers currently live in a single process called the Controller Manager.
The Kubernetes Node
The Kubernetes node runs all the components necessary for running application containers and load balancing service end-points. Nodes are also responsible for reporting resource utilization and status information to the API server.
DOCKER
Docker, the container runtime engine, runs on every node and handles downloading and running containers. Docker is controlled locally via its API by the Kubelet.
KUBELET
Each node runs the Kubelet, which is responsible for node registration, and management of pods. The Kubelet watches the Kubernetes API server for pods to create as scheduled by the Scheduler, and pods to delete based on cluster events. The Kubelet also handles reporting resource utilization, and health status information for a specific node and the pods it’s running.
PROXY
Each node also runs a simple network proxy with support for TCP and UDP stream forwarding across a set of pods as defined in the Kubernetes API.
File based CI trigger
GitLab has introduced file based ci trigger. The ci is using yaml notation in order to describe build pipeline and system recognized .gitlab-ci.yml file. I think this is very power concept for following reasons : It bring text-based dsl ( yaml in this case) and the file nature of the build definition allows to version and branch it along with code. Immediate benefit is that by the branching code base , the branch comes with the cloned build definition from the parent . Microsoft could improve on by adding more notation other than yaml , adding run-time incline information like $agent and $Build and also bring interactive intelligence using collaboration (bot) platform.
Announcing IdentityServer for ASP.NET 5 and .NET Core — leastprivilege.com
Over the last couple of years, we’ve been working with the ASP.NET team on the authentication and authorization story for Web API, Katana and ASP.NET 5. This included the design around claims-based identity, authorization and token-based authentication. In the Katana timeframe we also reviewed the OAuth 2.0 authorization server middleware (and the templates around it) […]
via Announcing IdentityServer for ASP.NET 5 and .NET Core — leastprivilege.com
Why Git?
Technical answer
Very often the answer to this question can be provided by highlighting multiple technical advantages of the distributed version control system, such as local repositories, differnt topologies,out of the box DR,efficient branching and merging and other.
Sceptic
An experienced developer could argue that modern centralized version control systems offer backup capability, personal branches in a form of shelves, change set management, labeling and other advanced features. Benefits of the DVCS might not overcome the simplicity and practicality of the centralized version control system. The transition to Git could result in the an increased process complexity and a temporary loss of productivity.
Social aspect of the Git
Developers, engaged into verity of the Open Source projects, have quickly realized that Git is not just a convenient geo accessible repository – it is also a powerful collaboration environment. De facto, Git has become very important social media channel and an idea exchange host for developers. Git mirrors the human thought process branching and provides mechanisms for individual or collaborative memories. “Why Git ?” – DevOps is a collaborative culture and Git fits it perfectly.
Git and Corporation
The transition to Git is very similar to a transition from COTS to DevOps – instead of multiple isolated teams working within closed boundaries, the focus is on the end product and the collaborative contribution to a platform evolution. The goal of adopting the Git culture is to bring Open Source style of development into the corporate environment.
Latest trends in the application test practices
Paradigm shift
New approach in the application testing is innovative and would require some time to digest. The biggest paradigm shift is to stop chasing bugs and focus on the identifying and changing processes and practices yielding high frequencies of production problems. According to Microsoft, it is possible to achieve practically acceptable threshold of bugs in production only by tuning up SDLC pipeline. The previous statement requires a clarification:
- Limited number of bugs are allowed in the production
- The major source of bugs is not implementation but processes and practices
- If amount of bugs does not exceed the threshold there is no need for the QA organization. The rational is that Developers writing automated test, feedback from Insiders/Preview/Production users , green-blue deployment and the implementation of the feature toggle practices is enough framework to deliver quality at speed.
Functional testing
The message for functional testing is very clear:
- Test has to be automated
- Test has to be written and not recorded
- Test has to be written only by a developer who introduces a change and is fully responsible for the application working in production
- Test has to be written before or right after a change.
Non-Functional tests
The value of the non-functional tests has to be re-visited through a prism of new emerging concepts such as Continuous insight, elastic scaling and fabric deployment. Continuous Insight has three parts: Availability check, Telemetry and Usage. Availability check performs intelligent application pings and is a substitute for any type of connectivity tests. The application telemetry and usage in the new cloud-born architecture are connected to the stream analytics and integrated with elastic services in order to notify self-balanced , self-healing , resilient fabric cluster with provisioning or de-provisioning events. It seems to me that classic performance testing goals – identifying application breaking or throttling points in pre-production environment and resource planning are becoming obsolete. The recommendation is to identify resources consumption anomalies within telemetry stream that might be due to poor application design or implementation and convert them into technical debt backlog items.
“What types of functional tests are necessary?” is not a correct question. The only type of automated test to start with is an isolated requisite based test . All other tests should be a result of an application evolution – found bugs, common inconsistencies, consumer’s feedback, etc..
In conclusion, I would suggest the elimination of the multiple obsolete test practices; along with documentation waste and expensive tooling makes application lifecycle management much chipper , much simple , much cleaner and much faster.
Microsoft cross-platform offerings
Run-time Cross-platform
The natural focus of a .NET developer is typically narrowed to a runtime built on the top of the CLR. However, there are different programmable runtimes supported by Windows outside of the .NET platform. Among all of the choices of platforms – JavaScript is most recognized, supported and adopted by developers. Below is an ECMA world diagram for the ECMAScript
JavaScript is supported by every browser, OS or mobile device, creating a perfect candidate for Cross platform development. Client side JavaScript libraries, such as AngularJS, React.js and Express.js, have earned recognition within IT Industry, and should be a part of the mix for any .NET Web Application development. MEAN (MongoDB, Express, Angular, Node) is an end-to-end JavaScript based call stack and is one of the most popular cross-platform development choices.
Hosting Cross-platform
Microsoft and Java are have been pursuing interoperable hosting platform for more than a decade. Java community offers multiple JVM languages such as Java, Scala, Clojure and JVM oriented application servers – Jboss, WebLogic, WebSphere. Microsoft hosting platform is built on top of CLR and provides COM+, IIS, NT Services and self hosting. It also supports any .NET language such as C#, VB.NET, IronPython, ClojureCLR and others.
Lately there are a couple of other developments that require community attention – Microsoft Docker support and HttpPlatformHandler for IIS hosting.
HttpHendlers and HttpModules are classic reliable IIS hosting extensions mechanism. HttpPlatformHandler is a module that allows to redirect HTTP traffic to any HTTP listener such as Tomcat, Ruby or Kestrel without losing IIS comprehensive process run-time and security management. It has been used by Microsoft Azure cloud services and now is offered as a separate install for IIS 8.
Docker is an emerging technology that has a potential to impact all aspects of SDLC. Docker architecture allows to package business logic, runtime and configuration aspects and to ship it as a single image to the docker registry for future consumption by any runtime. Microsoft offering in this space would be Windows Container Server, Nano Server (No UI scriptable Windows Server with CLI, REST, WinRT, PowerShell DSC access). .NET community would definitely need to catch up with understanding of the Docker infrastructure, PowerShell DSC, SSH, YAML manifest format and docker related Windows Server architecture.
Windows Cross-Platform
Microsoft OS system is going through a dramatic change in order to remove legacy code, legacy architecture and over 30 years of dependencies. During those years Microsoft OS was supporting multiple kernels such as Windows, Windows Server, Xbox and others which have proved difficult to manage and secure.
With the Windows 10 and Windows Server 2016 Microsoft is in the process of completing a convergence to a much cleaner model which in a lot of sense looks like a Linux Distributions. The Core OS will be hosting a kernel and will be a part of any Windows Distribution – Client, Server, Mobile, Tablet and Xbox.
As a result, applications written with the scope of CoreOS will be able to run on any Windows device without changes. Such applications are branded as a Universal. Universal Application has to be considered as the platform of choice for any smart/rich/thick desktop application as well as tablet or mobile client.
.NET Cross-Platform
.NET platform is an abstraction layer on a top of the Windows, goes through the significant re-write. While .NET 4.6 is the new fully featured Window .NET implementation of the CLR, Microsoft has also put full commercial backing of the Mono-Linux CLR implementation. It has also introduced CoreCLR which is a subset of the CLR capable to run cross-platform (Windows, Linux, Mac OS). Both Mono and CoreCLR are open sourced. However Mono focuses explicitly on the maturity of the Linux implementation, while CoreCLR on the portability.
.NET community should react to the current Microsoft and Industry trends and start to extend Windows oriented skills set to the unfamiliar territories such as Linux and dnx/dnvm. Selecting CoreCLR libraries and tools will make the applications portable across multiple platforms and devices, creating a significant return on investment. ASP.NET 5, MVC 6, EF 7 and Kestrel web server are a complete re-write of corresponding technologies using only CoreCLR subset of the .NET 4.6.
Cloud Cross-Platform
A Cloud Cross-Platform concept is known as PaaS – Platform as a Service. The idea behind PaaS is the abstraction from the technical aspects of the hardware and OS and focuses on the service offering only. Below is a diagram showing PaaS scope in comparison:
A good example which highlights PaaS mindset is an implementation of asynchronous message interchange scenario between B2B or EAI applications. This enterprise pattern usually requires a reliable queue with duplicate message detection and poising messages routing. Azure’s “Service Bus” queue and AWS’s “SQS” provide such services as well as REST or AMQP channels to access them. Artifacts such as hardware, OS, runtime are irrelevant. Moreover, protocol abstruction like REST over HTTP or AMQP allows to switch from one cloud to another without the lost of the functionality
