The Humble Programmer - by Edsger W. Dijkstra

As a result of a long sequence of coincidences I entered the programming profession officially on the first spring morning of 1952 and as far as I have been able to trace, I was the first Dutchman to do so in my country. In retrospect the most amazing thing was the slowness with which, at least in my part of the world, the programming profession emerged, a slowness which is now hard to believe. But I am grateful for two vivid recollections from that period that establish that slowness beyond any doubt.

Has DevOps morphed into ITSM?

The more that DevOps matures, the more it looks like IT service management (ITSM).

What? For some time, DevOps has been challenging the old ways of doing ITSM. DevOps has been shifting the traditional ITSM mindset from one of "control" to one of "enablement."

But the disciplines are increasingly merging. Here's why and how.

El TPS (Toyota Production System)

Este post, decidi hacerlo en castellano. Xq se preguntaran? Xq seguramente pierda mucho del vocabulario y la escencia que quiero impactar en el mismo, con la visita que he tenido la oportunidad de realizar.
Me ha gustado mucho pasear por la fabica y conversar con las personas que son parte de Toyota y que hacen que mi auto particular sea parte de ellos también en este momento ;)
Simplemente, pienso y creo, que mucho de lo que van a leer, es el ideal que muchas empresas buscan.
Gracias por haberme invitado a aprender y verlos en accion :D

DevOps Metrics and Key Performance Indicators

When tracking key DevOps metrics, focus less on the perceived success or failure according to any one indicator, but rather, on the story these metrics tell when examined together. A result that seems problematic on its own could look completely different when analyzed alongside additional data.

Careful tracking of the KPIs highlighted bellow can ensure not only greater efficiency in development and production, but more importantly, the best possible end-user experience. Embrace DevOps metrics, and you could see vast improvements in application deployment and feedback.

Using DevOps Metrics Effectively

If you’re a DevOps leader, you have a very tough job. You have to radically transform processes and behaviors that have been ingrained in IT for decades. You have to adopt new technologies quickly while still delivering code quickly and cost-efficiently. And you have to do all this better than your peers at other companies. Otherwise, you’ll simply slow your company’s progress in the new digital marketplace while your better-performing competitors gain advantage.

DevOps & KPIs is making an impact?

DevOps encourages incremental changes and faster releases while also improving quality and satisfaction. But how do we know if DevOps is making an impact? How do we decide what needs to change? We need to measure and this is where DevOps metrics come in.

DevOps Metrics and KPIs

DevOps may seem like the new kid on the block with all the buzz surrounding it, but it’s actually been around for a while and has proven itself a valuable method for improving software enterprises’ ability to deliver better products in less time. It achieves this lofty goal by leveraging the power of cross-discipline teams and empowering transparent communication throughout the organizations. A culture of communication goes a long way in improving morale as well as efficiency.

Gitflow & Gitflow Workflow

GitFlow is one of the popular branching strategies if you using Git version control. Most of the teams I have worked with generally are very satisfied using it. With fixed branch names, pull requests for code reviews, teams find that it fits very nicely with their day to day workflow.

Gitflow Workflow is a Git workflow design that was first published and made popular by Vincent Driessen at nvie. The Gitflow Workflow defines a strict branching model designed around the project release. This provides a robust framework for managing larger projects.

How BDD and Continuous Delivery Help Developers Maintain Flow

Programming is cognitive work, and programmers perform their best work under intense concentration. While there are external factors that can affect this, such as having a quiet office, controllable ways of communication, etc., there are also some internal factors that need to be taken into account. In fact, the way we work can influence the quality of the outcome the most.

Testing Python Applications with Pytest

Testing applications has become a standard skill set required for any competent developer today. The Python community embraces testing, and even the Python standard library has good inbuilt tools to support testing. In the larger Python ecosystem, there are a lot of testing tools. Pytest stands out among them due to its ease of use and its ability to handle increasingly complex testing needs.

Docker Image Size – Does It Matter?

Does Docker image size matter? The answer I usually hear is “yes”.
The logical question that follows is “why?”.

I’ve heard the following two answers too often for my liking:

The Cracking Monolith: The Forces That Call for Microservices

The microservice architecture has recently been gaining traction, with many companies sharing their positive experiences with applying it. The early adopters have been tech behemoths such as Amazon and Netflix, or companies with huge user bases like SoundCloud. Based on the profiles of these companies and the assumption that there’s more complexity to running and deploying many things than to deploying a single application, many people understand microservices as an interesting idea that does not apply to them. It’s something that mere mortals could qualify for in the far distant future, if ever.

Continuous Delivery Patterns with Serverless Applications

Serverless is a new paradigm in which we need to neither administer nor scale the servers that run our code. The promise is that much less operations work is needed to deliver value to customers.

The catch is that the unit of code deployment is now a function, not an application of arbitrary complexity. Deployment also includes configuring managed services and infrastructure to which functions connect. This means that new tools and practices are needed to achieve optimal developer productivity, which, as always, revolves around being able to move fast. Here’s how this is achieved:

Two Key Docker Benefits and How to Attain Them

In 2013, Solomon Hykes showed a demo of the first version of Docker during the PyCon conference in Santa Clara. Since then, the benefits of Docker containers have spread to seemingly every corner of the software industry. While Docker (the project and the company) made containers so popular, they were not the first project to leverage containers out there; and they are definitely not the last either.

7 Ways Continuous Delivery Helps Build a Culture of Learning

At the heart of continuous delivery is a fast feedback loop that immediately shows developers the effects of their work. Mistakes are found and fixed quickly, while beneficial changes can be released and deployed to customers without having to wait for a distant future release date. This rapid feedback helps build an organizational culture of learning and responsibility.

Elements of a Continuous Deployment Workflow

Continuous delivery is a broad set of software development practices designed to ensure that every change is rapidly deployable to production, while the system performs its business function uninterrupted and as intended.

CI/CD principles

CI/CD enables the best tech companies to improve their products many times per day. Here’s what you need to know to do the same.

Why We Need Continuous Integration

The Problem
First, we will take a look at the source of the problem, which lies in the software development cycle. Next, we will cover some of the change conflicts that can take place during that process, and finally we will explore the main factors that can make these problems escalate, followed by an explanation of how continuous integration solves these issues.

What’s the Difference Between Continuous Integration, Continuous Deployment and Continuous Delivery?

Continuous integration, continuous deployment, and continuous delivery are like vectors that have the same direction, but different magnitude. Their goal is the same: make our software development and release process faster and more robust.

Continuous Integration (CI) Explained

Continuous integration enables iterative software development, reduces risks from defects and makes developers highly productive.

What is Continuous Integration?
Continuous integration (CI) is a software development practice in which developers merge their changes to the main branch many times per day. Each merge triggers an automated code build and test sequence, which ideally runs in less than 10 minutes. A successful CI build may lead to further stages of continuous delivery.

CI/CD Pipeline: A Gentle Introduction

Do you want your engineering team to deliver bug-free code at high velocity? A fast and reliable CI/CD pipeline is crucial for doing that sustainably over time.

What is a CI/CD pipeline?
A CI/CD pipeline helps you automate steps in your software delivery process, such as initiating code builds, running automated tests, and deploying to a staging or production environment. Automated pipelines remove manual errors, provide standardized development feedback loops and enable fast product iterations.

What is Proper Continuous Integration?

Continuous integration (CI) is confusing. As with all ideas, everybody does their own version of it in practice.

CI is a solution to the problems we face while writing, testing and delivering software to end users. Its core promise is reliability.

A prerequisite for continuous integration is having an automated test suite, and a clean/ordered repo too ;). This is not a light requirement. Learning to write automated tests and mastering test-driven development takes years of practice. And yet, in a growing app, the tests we’ve developed can become an impediment to our productivity.

6 Steps To Survive A DevOps Transformation

From measurable goals to targeting quick wins and sharing lessons learned, here are the steps that can lead to DevOps success.
DevOps is everywhere, and if you're not already in the middle of a DevOps adoption program, there's probably one coming your way. Here's how to survive, in six only moderately painful steps.

SSH Examples, Tips & Tunnels

Practical SSH examples to take your remote system admin game to the next level. Commands and tips to not only use SSH but master ways to move around the network.

Knowing a few ssh tricks will benefit any system administrator, network engineer or security professional.

Strategies for Organizing Code

This article outlines four different strategies for organizing code: by component, by toolbox, by layer, and by kind. I think these four form a kind of hierarchy with regards to which kind of cohesion they favor and in my experience they cover most of the real-world code I’ve worked with.

Dinner with friends: How tech startups test, organize, and review their code

Building software is hard. In a casual dinner with ten developers (friends) from leading startup software teams, shared a behind-the-scenes look at their processes (and frustrations): testing, deployment, and code reviews.

The Problem with Shared Code

The Problem

To be clear, sharing code is not hard. There are many different ways to share code in general and most of them are relatively easy to implement. The hard part is doing it efficiently at scale which means:

Multiple code modules (that share some code)
Multiple team members
High rate of change
Little to no loss of individual productivity

What’s a monorepo? - Why Google Stores Billions of Lines of Code in a Single Repository

What’s a monorepo? Imagine if every time you needed to make a new source control repository, you just made a new top-level folder in your current repository instead. You’re doing monorepo! All the software written by everyone in your company stored together, versioned together.

Why every development team needs continuous delivery....

Continuous delivery helps you move faster and retain talent. Yet only 28% of dev teams do it! Find out why continuous delivery could be your competitive advantage.
28%. That is the number of teams currently practicing continuous delivery (CD) across all projects, company wide. That means a whopping 72% of software teams are likely to feel dread instead of relief when they hear the term “release”

Monitoring Linux Logs with Kibana and Rsyslog

Monitoring Linux logs is crucial and every DevOps engineer should know how to do it. Here’s why :

You have real-time visual feedback about your logs : probably one of the key aspects of log monitoring, you can build meaningful visualizations (such as datatables, pies, graphs or aggregated bar charts) to give some meaning to your logs.
You are able to aggregate information to build advanced and more complex dashboards : sometimes raw information is not enough, you may want to join it with other logs or to compare it with other logs to identify a trend. A visualization platform with expression handling lets you perform that.
You can quickly filter for a certain term, or given a certain time period : if you are only interested in SSH logs, you can build a targeted dashboard for it.
Logs are navigable in a quick and elegant way : I know the pain of tailing and greping your logs files endlessly. I’d rather have a platform for it.

Python Modules and Packages

There are actually three different ways to define a module in Python:

A module can be written in Python itself.
A module can be written in C and loaded dynamically at run-time, like the re (regular expression) module.
A built-in module is intrinsically contained in the interpreter.

A module’s contents are accessed the same way in all three cases: with the importstatement.

Absolute vs Relative Imports in Python

A Python module is a file that has a .py extension, and a Python package is any folder that has modules inside it (or, in Python 2, a folder that contains an __init__.pyfile).

What happens when you have code in one module that needs to access code in another module or package? You import it!

Memory Management in Python

Memory Is an Empty Book

You can begin by thinking of a computer’s memory as an empty book intended for short stories. There’s nothing written on the pages yet. Eventually, different authors will come along. Each author wants some space to write their story in.

Since they aren’t allowed to write over each other, they must be careful about which pages they write in. Before they begin writing, they consult the manager of the book. The manager then decides where in the book they’re allowed to write.

Since this book is around for a long time, many of the stories in it are no longer relevant. When no one reads or references the stories, they are removed to make room for new stories.

Notificaciones de Grafana en Slack

Como ya tengo algo saturada la bandeja de correo, acabé decidiéndome por enviar las notificaciones de Grafana a Slack :D.

En primer lugar tendremos que crearnos una cuenta en Slack si no la tenemos todavía. Crearemos un espacio de trabajo (workspace) y un canal al que en mi caso he llamado #grafana. Todas las notificaciones de mi servidor Grafana irán a parar a ese canal.

System metrics with Docker, Telegraf, Influxdb and Grafana

We’ll use Docker, for the quick deployment our monitoring system, also it’ll gives us a freedom to use any software with dependency free and keep our system clean after. Topical, You can look at Docker from this side too, as a cross-platform package system.

Also, we’ll use part of Tick stack, namely an Influxdb base to store our metrics and Telegraf, like an agent on remote system, for nice and pretty graphs we’ll take Grafana.

Docker in Action

With Docker you can easily deploy a web application along with it’s dependencies, environment variables, and configuration settings - everything you need to recreate your environment quickly and efficiently.

We’ll start by creating a Docker container for running a Python Flask application. From there, we’ll look at a nice development workflow to manage the local development of an app as well as continuous integration and delivery, step by step …

Offline Python Deployments With Docker

In cases when a production server does not have access to the Internet or to the internal network, you will need to bundle up the Python dependencies (as wheel files) and interpreter along with the source code.

Objectives

By the end of this post, you will be able to…

Describe the difference between a Python wheel and egg
Explain why you may want to build Python wheel files within a Docker container
Spin up a custom environment for building Python wheels using Docker
Bundle and deploy a Python project to an environment without access to the Internet
Explain how this deployment setup can be considered immutable

What is Continuous Delivery?

Continuous Delivery is the ability to get changes of all types—including new features, configuration changes, bug fixes and experiments—into production, or into the hands of users, safely and quickly in a sustainable way.

Our goal is to make deployments—whether of a large-scale distributed system, a complex production environment, an embedded system, or an app—predictable, routine affairs that can be performed on demand.

We achieve all this by ensuring our code is always in a deployable state, even in the face of teams of thousands of developers making changes on a daily basis. We thus completely eliminate the integration, testing and hardening phases that traditionally followed “dev complete”, as well as code freezes.

Continuous Integration With Python: An Introduction

What Is Continuous Integration?

Continuous integration (CI) is the practice of frequently building and testing each change done to your code automatically and as early as possible. Prolific developer and author Martin Fowler defines CI as follows:

“Continuous Integration is a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily - leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible.” (Source)

Profiling Python in Production - How We Reduced CPU Usage by 80% through Python Profiling with Nylas

We recently reduced CPU usage across our fleet by 80%. One key technique that made this possible was a lightweight profiling strategy that we could run in production. This post is about the ways we approached instrumentation, the tradeoffs involved, and some tools you can use to optimize your own apps (including code!).

Background

Nylas is a developer platform that provides APIs to integrate with email, contacts, and calendar. At the core of this is a technology we call the Sync Engine. It’s a large Python application (~30k LOC) which handles syncing via IMAP, SMTP, ActiveSync, and other protocols. The code is open source on GitHub with a solid community.

Comandos frecuentes de Docker ;)

Los contenedores son para Maquinas Virtuales como los threads son para los procesos. O puedes pensar en ellos como chroots en esteroides.

Lifecycle

docker create crea un contenedor pero no lo comienza.
docker rename permite renombrar al contenedor.
docker run crea y comienza un contenedor en una operación.
docker rm borra un contenedor.
docker update actualiza los recursos limitados de un contenedor.

Lifecycle of docker container

Image Source and Credits: http://docker-saigon.github.io/post/Docker-Internals/

3 warning flags of DevOps metrics

Metrics. Measurements. Data. Monitoring. Alerting. These are all big topics for DevOps and for cloud-native infrastructure and application development more broadly. In fact, acm Queue, a magazine published by the Association of Computing Machinery, recently devoted an entire issue to the topic.

I've argued before that we conflate a lot of things under the "metrics" term, from key performance indicators to critical failure alerts to data that may be vaguely useful someday for something or other. But that's a topic for another day. What I want to discuss here is how metrics affect behavior.

In 2008, Daniel Ariely published Predictably Irrational, one of a number of books written around that time that introduced behavioral psychology and behavioral economics to the general public. One memorable quote from that book is the following: "Human beings adjust behavior based on the metrics they're held against. Anything you measure will impel a person to optimize his score on that metric. What you measure is what you'll get. Period."

This shouldn't be surprising. It's a finding that's been repeatedly confirmed by research. It should also be familiar to just about anyone with business experience. It's certainly not news to anyone in sales management, for example. Base sales reps' (or their managers'!) bonuses solely on revenue, and they'll discount whatever it takes to maximize revenue even if it puts margin in the toilet. Conversely, want the sales force to push a new product line—which will probably take extra effort—but skip the spiffs? Probably not happening.

Ex. What can we develop by using Python for DevOps?

Problem: Monitor web server logs and send out an email (or notification) when 500’s or 400’s appear more than 10 times in last 10 minutes.

Solution: Write a python script to parse the web server logs and keep a count of status codes. Set a cronjob or run this script as a daemon.

Problem: MySQL server is throwing too many connections.

Solution: Write a python script that connects to MySQL and parses `show processlist` to find the number of connections per user per host. Send an email with this information to team. This can again be a cronjob on the MySQL server itself or any server which can connect to MySQL.

Python in DevOps

Nowadays everyone is looking at DevOps and Python. So if you want to know why the python is important in DevOps this article may help you.

In DevOps culture, we are required to do a lot of automation using tools and scripts. Python will provide huge open libraries and modules to help in that automation.

If we want to write a script to automate a task Python is a better option in terms of platform independence, easy to write and integrations with all tools. In Python there are so many modules available as open-source which supports several tools.

Agile Vs. DevOps

Stakeholders and communication chain in a typical IT process.

Agile addresses gaps in Customer and Developer communications

¿CÓMO SE RELACIONAN LAS METODOLOGÍAS AGILE CON DEVOPS?

Primero empecemos definiendo qué es Ágile y qué es Devops.

¿Qué es y en qué consiste la metodología Agile?

Es un método con una base más filosófica que metodológica porque no es un sistema que dicte a priori cómo hacer las cosas, sino que es un conjunto de principios y valores a tener en cuenta para evitar los problemas más comunes a la hora de desarrollar un software.

Agile vs DevOps. ¿Cuáles son las diferencias?

Tanto Agile como DevOps son metodologías de desarrollo de software, es decir, constituyen pasos, pautas, mecanismos, procesos, plantillas preestablecidas de cómo se debe desarrollar un proceso de desarrollo de software, así como los profesionales que intervienen en cada una de las etapas.

Aunque suelen tener muchos aspectos en común, lo cierto es que también presentan diferencias y en este artículo te ayudaremos a descubrirlas. ¡Acompáñanos!

Metodología Ágil

Se trata de una metodología de desarrollo que ha ganado mucho espacio en los últimos años, desplazando a las tradicionales. Esta metodología tiene como eje central el desarrollo iterativo con enfoque incremental y evolutivo.

Esto quiere decir que el desarrollo del producto final es dividido o fragmentado en tareas más pequeñas. Estas tareas se desarrollan y prueban hasta que no presenten errores. De este modo, al finalizarlas, se integran para realizar las pruebas finales. A la hora de implementar esta metodología existen diferentes marcos de trabajo que realizan modificaciones y la ajustan a su entorno y necesidades.

Definition of DevOps

DevOps is a term for a group of concepts that, while not all new, have catalyzed into a movement and are rapidly spreading throughout the technical community. Like any new and popular term, people may have confused and sometimes contradictory impressions of what it is. Here’s my take on how DevOps can be usefully defined; I propose this definition as a standard framework to more clearly discuss the various areas DevOps covers. Like “Quality” or “Agile,” DevOps is a large enough concept that it requires some nuance to fully understand.

DevOps is a new term emerging from the collision of two major related trends. The first was also called “agile infrastructure” or “agile operations”; it sprang from applying Agile and Lean approaches to operations work. The second is a much expanded understanding of the value of collaboration between development and operations staff throughout all stages of the development lifecycle when creating and operating a service, and how important operations has become in our increasingly service-oriented world (cf. Operations: The New Secret Sauce).

Suscribirse a: Entradas (Atom)