Introduction

I am a DevOps Evangelist and became team coach about a year ago. My mission is to motivate and guide the team to a higher set of skills, both on technical as personal level. The team consist of technical and business consultants with focus on ECM product implementations (OpenTexts Captiva, Documentum, InfoArchive and BOX).

We were delivering projects on time and within budget, but moral was not on the expected level. Primary reasons were:

• Many projects running at the same time (little focus)
• Lead time for projects due to waiting times between hand-overs
• Stressful deployment periods as well as integration periods.
• An awaiting attitude from the team, relying too much on management

It became clear pretty fast that our current project methodology could use a refresh, it was time to change.

In the months before I started the team lead role, I was involved with integrating Continuous delivery pipelines (see other older posts) for a large international bank. With all best practices learned so far, I was called back to the house to help refresh our working method, culture and motivation.

What is DevOps

“DevOps is the practice of operations and development engineers participating together in the entire service lifecycle, from design through the development process to production support. “  

Generally, people hear DevOps and automatically think about, continuous delivery, agile and infrastructure as code. But it is much more than that. The books “The DevOps Handbook” and “The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win” complete the definition of DevOps with the ‘Three ways’:

1. The First Way emphasizes the performance of the entire system
2. The Second Way is about creating the right to left feedback loops

3. The Third Way is about creating a culture that fosters experimentation and continuous learning

4. Benefits and advantages

   DevOps is all about improving the flow from development to production and back. Improving, deployment speed, quality, better and faster feedback internally as externally from the customer. 

Key benefits we experienced are:

• Faster resolution of problems thanks to better feedback loops and automation
• Happier, more productive teams because of focus, commitment and getting people involved in the complete development lifecycle.
• Automation instead of cookbooks helped in getting a steadier output to our customers.
• The cross-functional teams improve communication, knowledge sharing, learning, innovation and experimentation

  1. The plan forward implementation

     Luckily, I did not embark on this quest alone. Together with two other team coaches and a manager willing to change, we set out to introduce DevOps and change the culture and way of working in our company. But always keeping our human capital in mind. People are the most important part of our company.

• First Steps: Definition of Done and Guilds
• The Agile Way of Working
• Automating Everything s
• Adjusting the Training plan and knowledge sharing
• Deadlines, Focus and delivery

First Steps: Definition of Done and Guilds

In the first town hall meeting, we set the following goals or targets:

• Automate everything: We focus on creating repeatable processes for building code and deploying to production.
• Controlled and Optimized: The team keep everything in version control (GIT, SVN,…) and optimizes the use of Team environments (JIRA, SLACK, O365,…).
• Responsibility: Everyone becomes responsible for delivery. There will be no more pointing fingers.
• Improve: Practicing new techniques and continuous improvement becomes our second nature.

With those goals in mind, we started in forming a definition of done and introduced guilds with the full support of the team.

Definition of Done

With ‘everyone is responsible’ in mind, the definition of done was defined, communicated and put in place.

DEVELOPER

Coding Standards are followed

Everything is checked in SVN or GIT either in our own or at the customer Site

Write at least 1 automated Test (Unit or functional)

BUSINESS SIDE

Create release Notes (Integration and service instructions)

Acceptance criteria are checked on the acceptance environment

EVERYBODY

User Story and Issue Tracking is Up-to-date either in our own JIRA or at the customer Site

With a clear definition of done expectations were transparent and uniform. All stakeholders (business and technical) knew what to expect or what needed to be delivered. We started repeating this message on every town hall meeting. Making sure everybody got the message.

Implementing Guilds

On top of defining a definition of done, we introduced the Guild approach as well. Based on the Spotify model, we wanted to use this initiative to get people together across teams and projects, working on similar issues or questions, …

Guilds provide a horizontal communication layer across our Product Engineering teams. Our engineers, testers, and other staff use them to set their own missions, to establish technical roadmaps, to take on joint tasks for their grassroots initiatives, and to promote education through experiential learning. This collective action benefits their members, their craft, and our organization.

Guilds is all about knowledge sharing and giving the opportunity to experiment. Like the Definition of done, we use our monthly town hall meetings as moment to share our vision and strategy. Repeating the mission trying to engage them to work together and learn from each other.

Agile or Scrum

Together with DevOps comes Agile. Teams and activities merging into 1, following a waterfall method does not fit this picture. Being a certified Product Owner and Scrum master, I’ve been using the SCRUM approach for several years now. It is also the most widely spread and well know Agile method. So this was the most obvious choice when we had to choose an agile methodology .

The first thing that we noticed was that everyone was stating: “yes we work agile/scrum”…. “Not!”

There was still a formal project manager and architect. This is probably the most common mistake. There were still formal hand-overs between stages in the development lifecycle. Lack of team commitment in delivering a potentially releasable product at the end of an iteration. Potentially releasable according to the definition of done.

So it was back to basics. We use the scrum rules from www.scrum.org

You can find the guide here: https://www.scrum.org/resources/scrum-guide

But this is SCRUM in a nutshell.

• Roles: Product owner, Scrum master, scrum team member
• Events: Sprint, Planning meeting, Review (demo), Retrospective, Daily scrum
• Artifacts: Backlog, Sprint backlog, increment (product)

We took the following actions:

• Get rid of titles other than team member, Product owner(PO) and scrum master(SM) so no more project managers, architects, business analysts or developer.
  Assign a PO and SM. Make sure that this person understands what it means to act as a servant leader. Those two must focus on the product and process. They lead the team in delivering value. Important is to emphasize the leader part in this. I haven’t seen a good autonomous team in my whole career, without a formal leader. I challenge everybody to show me one!

• The hardest part were the events. Why do we need a retro, review and planning meeting with the whole team.? We started with sprints of 2 weeks and quickly changed to 1-week sprints. I must admit that senior management had some reservations with this because of time consuming. But when product results were shown at the end of each sprint, the overall team moral and sense of accomplishment was growing.

• AMPLEXOR uses Jira for issue tracking. A great tool for any type of project and collaboration. Artifact or backlog management was in place. This needed only optimization. The PO’s had to become more aware of the crucial role they are playing.

Luckily we have people with great sense of responsibility. The first transition went smooth. All they really need were a good, simple set of rules. It was the complexity of processes and roles that was killing productivity and creativity.

The next step was updating our training and knowledge sharing plan to make them more efficient. A lot of team members have been working as lone rangers for quite some time. The needed to learn to work as a team again.

1. Adjusting the Training plan and knowledge sharing

   We did a survey amongst the team and almost everybody said that they needed more guidance and training if we wanted to succeed. Everybody had heard about DevOps, but nobody had heard about the Three Ways.

Most of the team are tech guys not worrying about the total picture and working method.

After the intro we organized a session based on “The DevOps Handbook”, to describe and explain the three ways . Most impressing fact I remember is that they were all really surprised. Code is not everything ….…. Value is. any value. Providing feedback and communication leads to improvements which can be supported and accomplished by experimenting and training.

The team had to know that they must organize themselves and are able to experiment and investigate cool stuff. This as long as they are delivering value and can visualize this value to the business.

In order to enable appropriate training, I introduced non-mandatory Scrum 101 sessions for interested colleagues. This is a 30-minute time boxed forum where everybody who participates commits to getting at least one official scrum.org certificate. During those sessions we discuss 1 specific topic of the scum guide and monitor the progress of the members.

To broaden and strengthen the knowledge transfer even more and on top of our monthly team gathering, another new concept was created, called Open Podium. In those weekly sessions we stimulate a “show and tell” within a 30-minute time box for cool, experimental stuff the team is doing. People or teams are working on stuff in their own projects which gives them the opportunity to show off to the rest of the company. Initially it had a slow start and not every session is organized or full of information, but with people putting triggers at their colleagues this is improving.

Now we are doing at least three per month. Topics vary from showing a piece of code, introducing a new framework to explaining how documentation is done on a specific external project.

Guilds was the first initiative we took to engage people in knowledge sharing.

Looking back, we see that commitment is there, but we are facing more practical difficulties. The guilds are no longer active. The number of people at customer sites increased thanks to new business. This makes scheduling meetings hard, and let’s face it, the customer commitment comes first. We are adjusting our knowledge sharing ways continuously. The Definition of Done is getting more and more integrated, developers and business consultants are more aware of the quality that they need to deliver and can expect from others.

Automating Everything

Another part of the new strategy was to automate everything, maybe the most ambitious part of the journey. AMPLEXOR is a consulting company implementing proprietary products but also common of the shelve applications and solutions.

It has always been that Operations(OPS) manages infrastructure and development gives them products to release on the servers. Having cookbooks for al kind of infrastructure implementations, one-day OPS had a massive and not manageable backlog. So, it’s obvious that developers were asked to assist. Two days later I found the developers frustrated as hell!

So, reaching back to the principles of DevOps an important question pops up: “why not treat infra as code or automate the work you are doing?” Both in Linux as Windows, there are many ways on approaching this: Chef, Puppet, Nolio, Jenkins, Bash, Powershell, Team Foundation Server(TFS),…

I already wrote articles on automation and continuous delivery. https://www.linkedin.com/pulse/continuous-delivery-emc-captiva-document-capture-dennis-van-aelst

http://blog.amplexor.com/enterprisecontent/en/how-a-stronger-technology-focus-contributes-to-faster-delivery-of-business-value

https://community.emc.com/people/dennisvanaelst/blog

1. Infra structure as code and Jenkins

   Our infra mainly consist of about 100 Virtual machines. For build and deploy automation we are using Jenkins and automated testing via Selenium . One of our products MyInsight is a reporting and dash-boarding plugin for Documentum (http://www.amplexor.com/content/myinsight/en.html ). As every product, Documentum has many versions and releases which must be tested on compatibility.

Budget is always an issue. So, in order to optimize the deploy pipelines for infra and software, the team started to automate almost all possible infra and installation scenarios with PowerShell or bash implementations. Software specific stuff is done with custom java components.

There is still a long way to go. But changing the mindset from following a guide without thinking, to automating the guide in the same process, is actually starting to sink in.

1. Deadlines, Focus and delivery

   As I mentioned earlier, we had a lot items on our backlog. We had so many projects, that none of them were ever finished and new tasks kept popping up. Getting things done, showing results and then doing it again. That is a backbone of SCRUM. It was clear, we needed to fix our development lifecycle.

2. Fixing the Development cycle

   Using the SCRUM we identified the roles in the team but now we had to go to the main stakeholders: management and sales. They also needed to change the way they requested new features, setting deadlines and stop interfering with the developers work anymore. We instructed them that the Product Owner (PO) is the point of contact and not the lead developer. If they wanted to get something done, talk to the PO.

The team wasn’t focused enough and were working on so many things at the same time. To improve this we started working with stakeholder deadlines. The PO must be able to plan or prioritize the work on the backlog. Sometimes we had to play it hard due to lack of flexibility. No deadline. Nothing gets done. Stakeholders on every branch and level in any organization need to see and feel the responsibility they have towards a project. An organization must be open enough that open, honest and transparent communication can thrive between people in a non-blame culture.

As a result of this:

• The number of projects reduced or were put on hold because there was no clear deadline or even sometimes budget or scope.
• Existing backlogs of ongoing projects were cleaned up. A lot of people were removing a bunch of initially asked features.

Not only “The business” or stakeholders had to learn to commit to the project. The team needed to learn to control their activities and deliveries. Agile dictates self-organization. This was exactly what the team missed.

1. Decisiveness and commitment

   Thanks to the renewed PO role, the team got to choose the deliverables. At least, we can state that this was confronting for several of our team members. In the beginning nothing was delivered in time. Definition of Done: What is that? The Sprints were too full, User Stories were not ready but still they were added in the sprint. After some time, we were finalizing stuff and delivering on time, with attention on our human capital. The most important tools for this to change are

• Sprint Review Nothing worse than to tell in front of a group you have nothing to show. The first few times the team will think “O, well” but I guarantee that the third time will be different. The power of presenting is not a skill that everyone has developed, but then again, do it as a team.
• Sprint Retrospective Even for a team that has been working for a long time this is the most uncomfortable event. The silences, people staring at each other, Fascinating. Everybody trying to be politically correct. Something I hate the most. Brutal honesty is sometimes needed, dare to tell your colleague he did a lousy job. Ventilate your anger, but also praise and celebrate at the local bar with beer and snacks.

  1. Let’s wrap up

     It is safe to say that starting a DevOps movement is not easy. But it is thanks to an awesome team that we are making good progress in making all of our professional lives easier and more fun. Together we are delivering cool products, business value and increasing our customer satisfaction

As team leads must keep a strong focus on our human capital and DevOps. We have to guide people into reaching their full potential through learning, experimenting, trying to innovate. Only then we can create an open and safe work environment with a non-blame culture.

1. Credit

   Special thanks to Tim Vernaillen, Marco van Schaijk and Stella Swinkels for contributing.

2. Resources

   • https://theagileadmin.com/what-is-devops
   • https://itrevolution.com/the-three-ways-principles-underpinning-devops
   • https://newrelic.com/devops/benefits-of-devops
   • http://code.hootsuite.com/guilds/
   • https://www.amazon.com/Phoenix-Project-DevOps-Helping-Business/dp/0988262509
   • https://www.amazon.com/DevOps-Handbook-World-Class-Reliability-Organizations/dp/1942788002

|www.AMPLEXOR.com| - 27/10/2017|8 / 8| | :-: | :-: | -: |