System metrics with Docker, Telegraf, Influxdb and Grafana

We’ll use Docker, for the quick deployment our monitoring system, also it’ll gives us a freedom to use any software with dependency free and keep our system clean after. Topical, You can look at Docker from this side too, as a cross-platform package system.

Also, we’ll use part of Tick stack, namely an Influxdb base to store our metrics and Telegraf, like an agent on remote system, for nice and pretty graphs we’ll take Grafana.

Docker in Action

With Docker you can easily deploy a web application along with it’s dependencies, environment variables, and configuration settings - everything you need to recreate your environment quickly and efficiently.

We’ll start by creating a Docker container for running a Python Flask application. From there, we’ll look at a nice development workflow to manage the local development of an app as well as continuous integration and delivery, step by step …

Offline Python Deployments With Docker

In cases when a production server does not have access to the Internet or to the internal network, you will need to bundle up the Python dependencies (as wheel files) and interpreter along with the source code.

Objectives

By the end of this post, you will be able to…

1. Describe the difference between a Python wheel and egg
2. Explain why you may want to build Python wheel files within a Docker container
3. Spin up a custom environment for building Python wheels using Docker
4. Bundle and deploy a Python project to an environment without access to the Internet
5. Explain how this deployment setup can be considered immutable

What is Continuous Delivery?

Continuous Delivery is the ability to get changes of all types—including new features, configuration changes, bug fixes and experiments—into production, or into the hands of users, safely and quickly in a sustainable way.

Our goal is to make deployments—whether of a large-scale distributed system, a complex production environment, an embedded system, or an app—predictable, routine affairs that can be performed on demand.

We achieve all this by ensuring our code is always in a deployable state, even in the face of teams of thousands of developers making changes on a daily basis. We thus completely eliminate the integration, testing and hardening phases that traditionally followed “dev complete”, as well as code freezes.

Continuous Integration With Python: An Introduction

What Is Continuous Integration?

Continuous integration (CI) is the practice of frequently building and testing each change done to your code automatically and as early as possible. Prolific developer and author Martin Fowler defines CI as follows:

“Continuous Integration is a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily - leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible.” (Source)

Profiling Python in Production - How We Reduced CPU Usage by 80% through Python Profiling with Nylas

We recently reduced CPU usage across our fleet by 80%. One key technique that made this possible was a lightweight profiling strategy that we could run in production. This post is about the ways we approached instrumentation, the tradeoffs involved, and some tools you can use to optimize your own apps (including code!).

Background

Nylas is a developer platform that provides APIs to integrate with email, contacts, and calendar. At the core of this is a technology we call the Sync Engine. It’s a large Python application (~30k LOC) which handles syncing via IMAP, SMTP, ActiveSync, and other protocols. The code is open source on GitHub with a solid community.

Comandos frecuentes de Docker ;)

Los contenedores son para Maquinas Virtuales como los threads son para los procesos. O puedes pensar en ellos como chroots en esteroides.

Lifecycle

• docker create crea un contenedor pero no lo comienza.
• docker rename permite renombrar al contenedor.
• docker run crea y comienza un contenedor en una operación.
• docker rm borra un contenedor.
• docker update actualiza los recursos limitados de un contenedor.

Lifecycle of docker container

Image Source and Credits: http://docker-saigon.github.io/post/Docker-Internals/

3 warning flags of DevOps metrics

Metrics. Measurements. Data. Monitoring. Alerting. These are all big topics for DevOps and for cloud-native infrastructure and application development more broadly. In fact, acm Queue, a magazine published by the Association of Computing Machinery, recently devoted an entire issue to the topic.

I've argued before that we conflate a lot of things under the "metrics" term, from key performance indicators to critical failure alerts to data that may be vaguely useful someday for something or other. But that's a topic for another day. What I want to discuss here is how metrics affect behavior.

In 2008, Daniel Ariely published Predictably Irrational, one of a number of books written around that time that introduced behavioral psychology and behavioral economics to the general public. One memorable quote from that book is the following: "Human beings adjust behavior based on the metrics they're held against. Anything you measure will impel a person to optimize his score on that metric. What you measure is what you'll get. Period."

This shouldn't be surprising. It's a finding that's been repeatedly confirmed by research. It should also be familiar to just about anyone with business experience. It's certainly not news to anyone in sales management, for example. Base sales reps' (or their managers'!) bonuses solely on revenue, and they'll discount whatever it takes to maximize revenue even if it puts margin in the toilet. Conversely, want the sales force to push a new product line—which will probably take extra effort—but skip the spiffs? Probably not happening.

Ex. What can we develop by using Python for DevOps?

Problem: Monitor web server logs and send out an email (or notification) when 500’s or 400’s appear more than 10 times in last 10 minutes.

Solution: Write a python script to parse the web server logs and keep a count of status codes. Set a cronjob or run this script as a daemon.

---

Problem: MySQL server is throwing too many connections.

Solution: Write a python script that connects to MySQL and parses `show processlist` to find the number of connections per user per host. Send an email with this information to team. This can again be a cronjob on the MySQL server itself or any server which can connect to MySQL.

---

Python in DevOps

Nowadays everyone is looking at DevOps and Python. So if you want to know why the python is important in DevOps this article may help you.

In DevOps culture, we are required to do a lot of automation using tools and scripts. Python will provide huge open libraries and modules to help in that automation.

If we want to write a script to automate a task Python is a better option in terms of platform independence, easy to write and integrations with all tools. In Python there are so many modules available as open-source which supports several tools.

Agile Vs. DevOps

Stakeholders and communication chain in a typical IT process.

Agile addresses gaps in Customer and Developer communications

¿CÓMO SE RELACIONAN LAS METODOLOGÍAS AGILE CON DEVOPS?

Primero empecemos definiendo qué es Ágile y qué es Devops.

¿Qué es y en qué consiste la metodología Agile?

Es un método con una base más filosófica que metodológica porque no es un sistema que dicte a priori cómo hacer las cosas, sino que es un conjunto de principios y valores a tener en cuenta para evitar los problemas más comunes a la hora de desarrollar un software.

Agile vs DevOps. ¿Cuáles son las diferencias?

Tanto Agile como DevOps son metodologías de desarrollo de software, es decir, constituyen pasos, pautas, mecanismos, procesos, plantillas preestablecidas de cómo se debe desarrollar un proceso de desarrollo de software, así como los profesionales que intervienen en cada una de las etapas.

Aunque suelen tener muchos aspectos en común, lo cierto es que también presentan diferencias y en este artículo te ayudaremos a descubrirlas. ¡Acompáñanos! 

Metodología Ágil

Se trata de una metodología de desarrollo que ha ganado mucho espacio en los últimos años, desplazando a las tradicionales. Esta metodología tiene como eje central el desarrollo iterativo con enfoque incremental y evolutivo.

Esto quiere decir que el desarrollo del producto final es dividido o fragmentado en tareas más pequeñas. Estas tareas se desarrollan y prueban hasta que no presenten errores. De este modo, al finalizarlas, se integran para realizar las pruebas finales. A la hora de implementar esta metodología existen diferentes marcos de trabajo que realizan modificaciones y la ajustan a su entorno y necesidades. 

Definition of DevOps

DevOps is a term for a group of concepts that, while not all new, have catalyzed into a movement and are rapidly spreading throughout the technical community.  Like any new and popular term, people may have confused and sometimes contradictory impressions of what it is.  Here’s my take on how DevOps can be usefully defined; I propose this definition as a standard framework to more clearly discuss the various areas DevOps covers. Like “Quality” or “Agile,” DevOps is a large enough concept that it requires some nuance to fully understand.

DevOps is a new term emerging from the collision of two major related trends. The first was also called “agile infrastructure” or “agile operations”; it sprang from applying Agile and Lean approaches to operations work.  The second is a much expanded understanding of the value of collaboration between development and operations staff throughout all stages of the development lifecycle when creating and operating a service, and how important operations has become in our increasingly service-oriented world (cf. Operations: The New Secret Sauce).