DevOps

Introduction

1. Devops allows us to create a best mix of practices from Development and operations into a unified discipline.
2. DevOps is short for Development and Operations. It concentrates on the collaboration between developers and other parties involved in building, deploying, operating and maintaining software systems.
3. More Collaboration between the Development and Operations team to automate the entire code issuance and Infrastructure Provisioning Process through Larger Pipeline.
4. This Pipeline contains various upstream and downstream jobs.
5. These jobs contain a sequence of tasks which were performed manually before.
6. History of DevOps
1. Patric Debios, a Belgian consultant, Project Manager, Agile practitioner is one of the initiators of DevOps.
2. A presentation on "10+ Deploys Per Day: Dev and Ops Cooperation" at Flicker helped with idea of DevOps.
3. DevOps resolved the conflict of "It's not my code it's your machines".
4. DevOps Blends Lean thinking with agile Philosphy. 
7. DevOps is required to handle changes in Agile.
8. DevOps acts as a catalyst to the Agile by helping in automating the process of handing issues.
1. Issues could be related to code issues
2. Issues could be related to infrastructure provisioning.
9. Developer Side
1. Challanges
1. Waiting time for code deployment.
2.  Pressure of work on old pending and new code.
2. Solution
1. Continuous integration ensures that there is quick deployment of code,faster testing and speedy feedback mechanism.
2. Thus there is no waiting time to deploy the code .Hence developer focuses on building the current code.
3. Tools that help in continuous integration
1. Bamboo
2. Jenkins
10. Operations Side
1. Challanges
1. Difficult to maintain uptime of the production environment.
2. Tools to automate infrastructure management are not effective.
3. No servers to be monitored increases.
4. Difficult to diagnose and provide feedback on the product.
2. Solution
1. Containerization/Virtualization ensures that there is a simulated environment created to run the software as containers offer great reliability for service.
1. Using Docker we can maintain uptime of a System using techniques like Server less Technology and Auto scaling.
2. Configuration Management helps you to organize and execute configuration plans,consistently provision the system, and proactively manage their infrastructure.
1. Various tools used are
1. Ansible
2. Chef
3. Puppet
4. Helps us to plan and execute infrastructure provisioning process.
3. Continuous Monitoring: Effective monitoring and feedbacks system is established through Nagios. This effective administration is assured.
1.  Tools used for continuous Monitoring are
1. Nagios
2. Splunk
2. We cannot monitor infrastructure 24X7 and set alerts write custom events on which an automatic ticket is generated.
11. DevOps Life Cycle
1. Continuous Development
1. Planning,Creating,Coding and Building
2. Tools
1. Planning and Coding :Git,SVN,Jira
2. Building : Ant,Maven,Gradle
2. Continuous Testing
1. Verify the things we create
2. Developed software is continuously tested for Bugs
3. Tools:Selenium,Junit,TestNG
3. Continuous Integration 
1. New functionality is integrated with the existing code
2. The changed code. should also ensure that there are no errors in the run-time environment.
3. Allowing us to test the changes and check how it reacts with other changes.
4. Tools:Jenkins,Hudson,Bamboo
4. Package Up developed artifacts or features.
5. Prepare a release.
1. Operations work starts off from here.
6. Configure the environment for acceptable release.
7. Continuous Deployment
1. Code is deployed in Production Environment.
2. Correctly deployed on all servers. 
3. Tools:Puppet,Ansible,Chef,Saltstack,Docker,Vagrant
8. Continuous Monitoring
1. Monitor what  goes on in environment after release happens 
2. Monitor the application and the servers closely to check the health of the system proactively.
3. Improve productivity and increase the reliability of the systems,reducing IT support costs.
4. Major Issues found could be reported to the development team.
5. Tools: Splunk,ELK Stack, Nagios, NewRelic.
12. Devops Activities
1. Devops and DevSec ops methodologies and key concepts
2. Different types of version control systems, continuous integration tools,continuous monitoring tools, and cloud models.
1. LIke SVN,Git,Jenkins,ibana,Nagios,AWS,Kubernetes,Team City
3. Configure private instances of Jenkins, Team City tools and Jenkins with Java, Git, and Maven.
4. Setting up a TDD framework with Junit 5 and a behaviour driven development framework with Cucumber.
5. Create Docker container, a Docker registry, and work with Docker compose, Docker Hub and Docker Networking.
6. Describe the importance of cloud and cloud provider in Devops. Take Devops approach when working with AWS,GCP or any such cloud platform and deploy a private Kubernetes cluster.
7. Setup a complete private infrastructure using version control and CICD tools.
13. DevOps is an agile relation between Development and ITOps
14. DevOps includes Plan,Create,Verify and Package of Artifacts from development side and Operations side include Release,Configure and Moniter.
15. Components of a Devops
1. Version Control Systems
2. CI,CD and Build tools
3. Software and Automation testing frameworks
4. Configuration Management tools.
5. Containerization with Docker.
6. Continuous Monitoring tools.
7. Cloud services.
16. Agile and Devops
1. Agile starts with a product backlog
1. Product Backlog is all the things that business wants to get from Software Development teams.
2. Then comes Sprint Planning where we decide what we a re going to work on during a period of Software Development.
1. Leaderships from Development,Operations,QA,Business,Product will brainstorm regarding features which have more priority than others.
2. The features are selected from Backlog and Sprint Backlog.
3. All the features decided to be worked on are a part of that Sprint for that period.
4. Agile Practitioners time-box a sprint which may last from 1 week to 4 weeks.
5. Sprint is a time-box for certain set of features to be completed.
3. A Sprint is considered as passed when all the features from the backlog introduced in Sprint have been created shipped and deployed.
4. A failing Sprint is when not all features can be deployed.
1. Such a Sprint is analysed by Devops team of cause and concerns so that it can be improved in future.Such an analysis is called an retrospective analysis.
2. Retrospective is done to improve Work Practices so future Sprints pass.
1. This may be done by reducing numbers of features in a Sprint
2. Introducing automation techniques.
5. Brainstorming between teams is important so that everyone is in synchronized with work done and status of Sprint.
1. Communication is one of the key features of Devops.
2. Communication is more important than specification in Devops.
6. Sprint review is done for the passing Sprint at the end of cycle if the Sprint fails Sprint Retrospective is done to improve things for future Sprints.
17. Overview of Devops Tools
1. Code
1. Git,Jira,Teamcity,Confluence
2. Confluence is a common knowledge repository of all the ideas we are working with and all the concepts & technologies we are working on.It's a wiki like product
3. We need to look for Event Driven paradigms we can use for communication between web services and external services
1. http://seimplant.blogspot.com/p/event-driven-api-paradigms.html
2. Source code repository
1. Manage tasks features status
2. Project/Sprint Management Tools
3. Build
1. Java - Maven,Gradle
2. Node- NPM(Node Package Manager)
4. Test
1. Junit(to execute unit tests and integration tests)
2. Selenium(for end testing)
3. Appium(for mobile testing)
4. Mocha
5. Jmeter(Performance Testing)
5. Release
1. Jenkins
1. Has plugin feature.
2. Code Ship
6. Deployment
1. Deployment is about packaging
2. Docker
1. Dockers creates containers and containers are isolated deployment units which are easy to manage.
3. DC/OS
4. AWS
1. This is deployment environment typically for doing serverless computing under AWS lambda
5. Google Cloud
1. Google functions(serverless deployment)
6. Azure 
1. Azure functions (serverless deployment env)
7. Operations
1. Need to create computing environments to deploy them into
1. We use orchestration tool for this such as
1. chef
1. pull orchestrator
2. Ansible
1. push orchestrator
3. Kubernetes
1. Container orchestration
8. Monitering
1. Things can go wrong and performance can degrade because of unexpected circumstances thus code needs to be monitored
1. Data Log
1. Monitoring Logging tool
2.  Splunk is an aggregation tool
3. Nagios
18. Categories of DevOps Tools
1. Tools to Communicate within teams and with other Teams
1. Issue and Task Management Tools
2. UML diagram tools
3. Wiki tools for knowledge management of company
4. Agile Sprint Management tools
1. Manage/Track hours of features being worked upon of Product
2. Blockers which stops work.
5. Shared Calender's
6. Shared Documenting tools such as Word Processors and Spreadsheets.
2. Tools required during the code development stage
   
1. Editors
1. Vim
2. IDE-Visual Studio
3. Eclipse
4. Webstorm
5. Pycharm
6. Visual Code(Cross Platform Editor)
7. Intellij
2. Compilers and Interpreters
3. Integrated Development Environments
4. Unit Testing and Mocking Frameworks
1. Mocking is a way of emulating system behavior without having to have a whole system preset.
2. Unit testing ensures that what the code developers are making actually meets the features part of requirement.
5. User Interface, Integration and Database Mapping(ORM) frameworks
1. ORM takes a database schema and turns it into an object
6. REST API design and validation tools
7. Code quality Tools
8. Code Review Tools
1.  Github
3. Build Tools
1. Jenkins
2. Bundler
3. Apache Ant
4. Ms Build
5. Meta builders such as Grunt
6. Direct Builders like Gradle
1. Databases and General persistence tools
1. Both on cloud and private environments.
1. Private Environments like Sql Server,Mysql,Oracle,Postgres
2. Cloud Based Environments like DynmoDb, Microsoft Database
2. Artifact Creation Tools
1. Artifacts can be anything from a single binary used as a dependency by other applications to a complete executable
3. Source Control Systems
1. How do I get the source code used to build the artifact
4. Continuous Integration Tools
4. Tools to Checkout Code
1. Git
1. Linus Torvald built this.
2. Github
3. Getlab
4. BitBucket
5. Subversion
5. Documentation Gathering Tools
1. Wiki
2. Swagger Specification
1. Describes an API
1. Tool happens to be self documenting, using a swagger yml file that confirms to open API specification we can feed that yml file or json file into a documentation interpreter and it automatically creates the documentation for api on the fly.
6. Tools required for integration tests and user acceptance tests
1. Unit Testing
1. Mocha for nodejs
2. mstest for .net
3. junit or testing for Java
2. Integration Testing
1. Selanium
2. SauceLabs
3. Cucumber
4. Testng-Java framework
5. MS Test
6. Test rails
7. Mocha - nodejs
8. Django Test-Python
9. Appium XCode for Apple Mobile Developers
3. Mocking Frameworks
1. easy mock or js mock
2. scripting tools
3. test harness tools
4. user interface playback tools
1. Selanium
4. Code quality Analysis(Automated testing tools)
1. Fortify
2. Coverity
3. Sonar
4. jslint
5. ymllint
7. Packaging and Archiving
1. Some Binary's need to be stored in artifact repository.
2. Nexus is an artifactory.
3. When a code goes past integration testing and is put back to binary format that binary format will be stored in a binary repository or artifact repository.
1. Binary repository may cause security concerns and thus may be use amicably.
8. Configure Management and Deployment tools
1. Help in getting code in environment and configure environment
2. These are done with help of an orchestration manager such as 
1. chef
1. Works on pull paradigm every VM created will have chef and puppet agent put directly on machine and that machine is smart enough to configure the environment with stuff like oracle database,Rabbit MQ Server. 
2. Applications are pulled directly from the repository into the computing environments.
2. cf engine
3. puppet labs
1. Also work on pull paradigm.
4. RunDeck
5. Ansible 
1. Has the ability to push required components onto the computing environments, works on push paradigm.
2. Making code, code artifacts and binaries as one part of the equation and other part in preparing virtual machines in which the code needs to run.
9. Deployment Envirnoments
1. Deployment tools work in coordination with provisioning tools or configuration and management tools
2. Some cloud environments are 
1. Rackspace
2. Vmware
3. Openstack
4. Amazon Web Services
5. Google Cloud Services
6. Microsoft Azure
3. The deployment environment can be an internal,private cloud in which company owns all the hardware and networking capabilities.
4. The provisioning software ensures that artifacts are deployed accordingly as these environments are known before hand only.
10. Continuous Integration Servers
1. Build servers perform movement of  code from branch to branch,branch to server, one environment to another environment.
2. Also called as continuous delivery environments
1. Bamboo
2. Travis
3. Jenkins 
19. Activities in DevOps process
1. Monitoring Performance
2. Releasing into Production
3. Building Applications
4. Code development and Unit Testing
5. Configuration  Management
6. Integration and Performance Testing
7. Packing the Application
20. Standard DevOps Deployment Process
1. Checkout code and build
1. Once the developer has worked on a feature it needs to be integrated to latest code.
2. In SVN this is done via checking out latest code resolving conflicts if any and again build the code.
3. In Git this is done by merging the branch into development branch resolving conflicts if any and again building the code.
4. All the binaries and dependencies are put together and built are in the scope of the feature being implemented this is called as an integrated build.
2. Unit Testing
1. Small pieces of code is unit tested for checking basic functionalities.
3. Quality Control
1. Code is moved from the development branch of the source code repository to the quality control branch
2. The quality control team checks the code on this environment/branch.
4. Deploy to Test Environment
1. Code is tested in the test environment to which code is deployed since this environment has specialized testing tools which development environment does not has.
2. Test Environment is close emulation of Production Environment.
5. Fetch the latest build from the quality control branch.
1. This code is then merged with the code and integration testing is performed.
2. All the artifacts and components of given application are put together and comprehensive testing is performed.It includes following components.
1. Frontend
2. Middleware
3. Backend
4. Database
5. Test Messaging
3. We use testing frameworks to perform these testing.The purpose of these testing framework is it allows execution of testing to happen easily.
4. Humans should perform testing such as high volume performance testing and high volume load testing.
5. For Integration testing we should use testing frameworks.
6. Packaging and Archiving
1. Source Code is built into a deployment unit.
1. Java has java files
2. C# has dll files
3. C++ has executable
4. Python, Javascript, node, group of related java classes has archives.
2. The release of this package is then fetched and build is made ready.
7. Deploy to pre-production environment
1. Preproduction is like real world environment where exact set of instance as in production are setup.
2. The configuration type such as services with common data or less sensitive data are same as in production.
1. Example A QA environment may not have all the countries as in database but preproduction has all.
3. Networking infrastructure matches production environment.
8. Acceptance testing or UAT is preformed on this environment.
1. This is done either by customer or representative of customer who has real interest and can guarantee this is what they have been looking for.
2. Customer can be an Internal customer or external customer.
9. Next it is deployed to production environment
1.  Part of the production release process is to ensure that code rollback can happen
21. Devop's ToDo's
1. Deploying an Angular Application with Docker
2. Dockerizing a Jenkins Pipeline
3. Managing Docker container by using the DRY method.
4. Deploying Applications such as Wordpress in Kubernetes via Jenkins.