Somehow in my previous blog entry Docker: production usefulness. I gave a negative impression (I don’t know why, it isn’t as if I were using NOs in red, bold and supersized for answering main questions). But Docker it’s really a disruptive technology which works, so let’s use it for a near future platform change, the migration of our app servers to Red Hat Jboss 7 (present wildfly) from a classic infrastructure to a cloud/dockerized one.
The goal is an image that can be instanced as domain controller (dc) or node, each execution adding its service/container to the JBoss domain seamlessly. This domain and their services must meet production quality.
Existing Docker Images problematic.
The first problem comes from its origin: “jboss/wildfly” depends on “jboss/base-jdk:7” which depends on “jboss/base:latest” , which depends on “fedora:20“. Fedora isn’t a production software (at least not here in Spain), those images are tainted, we must run our services in supported systems. For our research we’d use Centos so we could change it for RHEL later.
In any case, this means we had to recreate a new Dockerfile without sharing images with the official repo but importing the inner workings of all those dockerfiles. There’s another option, fork all those repos changing only their FROM, in this way we have the advantage of pulling and merging updates. For simplicity and not creating/uploading anything to github I chose the former.
FROM centos:centos7 # ENV http_proxy <PROXY_HOST>:<PROXY_PORT> # ENV https_proxy <PROXY_HOST>:<PROXY_PORT> # imported from jboss/base # Execute system update RUN yum -y update # Install packages necessary to run EAP (from jboss/base) RUN yum -y install xmlstarlet saxon augeas unzip tar bzip2 xz # Imported from jboss/jdk7 # Install necessary packages (from JDK7 Base) RUN yum -y install java-1.7.0-openjdk-devel RUN yum clean all # Imported from jboss/base # Create a user and group used to launch processes # The user ID 1000 is the default for the first "regular" user on Fedora/RHEL, # so there is a high chance that this ID will be equal to the current user # making it easier to use volumes (no permission issues) RUN groupadd -r jboss -g 1000 && useradd -u 1000 -r -g jboss -m -d /opt/jboss -s /sbin/nologin -c "JBoss user" jboss # Set the working directory to jboss' user home directory WORKDIR /opt/jboss # Specify the user which should be used to execute all commands below USER jboss # Imported from jboss/jdk7 # Set the JAVA_HOME variable to make it clear where Java is located ENV JAVA_HOME /usr/lib/jvm/java # Imported from jboss/wildfly # Set the WILDFLY_VERSION env variable ENV WILDFLY_VERSION 8.1.0.Final # Add the WildFly distribution to /opt, and make wildfly the owner of the extracted tar content # Make sure the distribution is available from a well-known place RUN cd $HOME; /curl http://download.jboss.org/wildfly/$WILDFLY_VERSION/wildfly-$WILDFLY_VERSION.tar.gz / | tar zx && mv $HOME/wildfly-$WILDFLY_VERSION $HOME/wildfly # Set the JBOSS_HOME env variable ENV JBOSS_HOME /opt/jboss/wildfly # Expose the ports we're interested in EXPOSE 8080 9990 # Set the default command to run on boot # This will boot WildFly in the standalone mode and bind to all interface CMD ["/opt/jboss/wildfly/bin/standalone.sh", "-b", "0.0.0.0"]
This dockerfile will build us an image similar to the official one but using centos instead of fedora .
$ sudo docker build --tag ackward/wildfly . $ sudo docker images REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE ackward/wildfly latest 1c4572ac308d 4 days ago 680.8 MB <none> <none> 102ff2048023 4 days ago 467.7 MB centos centos7 ae0c2d0bdc10 2 weeks ago 224 MB
Executing them would be so easy as (following the documentation):
$ sudo docker run -it ackward/wildfly /opt/jboss/wildfly/bin/domain.sh -b 0.0.0.0 -bmanagement 0.0.0.0
or for a standalone instance:
$ sudo docker run -it ackward/wildfly
They execute and works, kinda, because it isn’t configured but that’s expected. So we continue to the next step, configure the JBoss Domain Service.
Configuring JBoss Domain Service.
For the neophytes, JBoss App Server has two operation modes, standalone where each server manages itself and domain mode where there is a domain controller managing several nodes similar to how WebSphere Deployment Manager or Weblogic are implemented.
The first thing to consider is how are we going to manage the configuration files, how are going to be accessed, tracked or versioned, how logs could be accessed, how the apps could be deployed, etc.
It’s important because the docker paradigm tries to achieve independence of the data, each exec of a image is fresh and containers should be the most flexible to create and destroy on the fly (tying them with data isn’t the best way to achieve it).
The solution is already implemented in our environments (prod included), when I designed the migration (or runaway) from WebSphere to JBoss I separated the configuration files from the application files (for accountability and because all the configuration files are textfiles, can be gitted and had them in a remote/central repo; an upgrade doesn’t touch the config files, can be merged and diffed between them). This solution works because we can pass the config dir (jboss.<>.base.dir) as an external volume, all the changes would be persistent, we have straight access to the log files or the deployment directory, can be versioned with git and synced with a master/central repo and creating new nodes is dead easy (let’s see it).
Instead of modifying our wildfly dockerfile we extend it, in this way we customize an image while maintaining backward compatibility.
FROM ackward/wildfly RUN mkdir -p /opt/jboss/LOCAL_JB/node VOLUME ["/opt/jboss/LOCAL_JB/node"] EXPOSE 8080 9990 9999 CMD ["/opt/jboss/wildfly/bin/domain.sh", "-b", "0.0.0.0", "-bmanagement", "0.0.0.0", "-Djboss.domain.base.dir=/opt/jboss/LOCAL_JB/node"]
To build it:
$ sudo docker build --tag ackward/dc-wildfly .$ sudo docker imagesREPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE ackward/dc-wildfly latest 1a9de7c0734f 3 days ago 680.8 MB ackward/wilfly latest 1c4572ac308d 5 days ago 680.8 MB <none> <none> 102ff2048023 5 days ago 467.7 MB centos centos7 ae0c2d0bdc10 2 weeks ago 224 MB $ sudo touch: cannot touch '/opt/jboss/LOCAL_JB/node/tt': Permission denied $ sudo chcon -Rt svirt_sandbox_file_t /opt/jboss/
There is some issues with SELinux, we need to change a context in the host dir we’re exposing to the containers :
$ ls -l /opt/jboss/ total 0 drwxr-xr-x. 7 jboss jboss 71 nov 21 08:17 dc1 drwxr-xr-x. 7 jboss jboss 71 nov 24 11:06 node-tplt drwxr-xr-x. 7 jboss jboss 71 nov 21 08:40 node1 drwxr-xr-x. 7 jboss jboss 71 nov 21 11:51 node2 $ ls -l /opt/jboss/dc1/ total 4 drwxr-xr-x. 4 jboss jboss 4096 nov 21 10:59 configuration drwxr-xr-x. 3 jboss jboss 20 nov 21 08:17 data drwxr-xr-x. 2 jboss jboss 61 nov 21 09:23 log drwxr-xr-x. 4 jboss jboss 40 nov 21 08:17 servers drwxr-xr-x. 3 jboss jboss 17 nov 21 08:17 tmp $ ls -l /opt/jboss/node1 total 4 drwxr-xr-x. 3 jboss jboss 4096 nov 21 11:16 configuration drwxr-xr-x. 3 jboss jboss 20 nov 21 08:22 data drwxr-xr-x. 2 jboss jboss 61 nov 21 10:50 log drwxr-xr-x. 4 jboss jboss 40 nov 21 11:12 servers drwxr-xr-x. 3 jboss jboss 17 nov 21 08:22 tmp
$ sudo docker run -d -P -p 9990:9990 -v /opt/jboss/dc1:/opt/jboss/LOCAL_JB/node -h dc1 --name=wf-dc1 ackward/dc-wildfly
If we want to create a new node we’ve an empty (no servers, no groups, just the slave service account needed to join the domain) template in “node-tplt” and a cp :
$ sudo cp /opt/jboss/node-tplt /opt/jboss/node2 $ sudo chown -R jboss:jboss /opt/jboss/node2 $ sudo docker run -P -d --link=wf-dc1:ldc1 -v /opt/jboss/node3:/opt/jboss/LOCAL_JB/node --name=node2 / -h node2 ackward/dc-wildfly /opt/jboss/wildfly/bin/domain.sh -b 0.0.0.0 -Djboss.domain.master.address=ldc1 / -Djboss.domain.base.dir=/opt/jboss/LOCAL_JB/node $ sudo docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 665e986ad585 ackward/dc-wildfly:latest "/opt/jboss/wildfly/ 2 days ago Up 2 days 0.0.0.0:49187->8080/tcp, 0.0.0.0:49188->9990/tcp, 0.0.0.0:49189->9999/tcp node2 1229904dd9d2 ackward/dc-wildfly:latest "/opt/jboss/wildfly/ 2 days ago Up 2 days 0.0.0.0:49184->8080/tcp, 0.0.0.0:49185->9990/tcp, 0.0.0.0:49186->9999/tcp node1 d0cdba99520b ackward/dc-wildfly:latest "/opt/jboss/wildfly/ 3 days ago Up 3 days 0.0.0.0:9990->9990/tcp, 0.0.0.0:49170->9999/tcp, 0.0.0.0:49171->8080/tcp node1/ldc1,node2/ldc1,wf-dc1
Where are we?
A long road yet, the foundations look solid and this post is long enough than most people will just TL;DR.
- We have a way to create nodes, accessing/modifying their config and their logs easily.
- The images and containers would be easily upgraded or customized.
The images and containers have zero configuration. (GREAT!)
- We can use git repos for accountability of our configuration files and creating new nodes.
- With a private docker hub we could share the images on all the docker nodes.
- A way to execute containers in remote servers using Ansible, we could create a role with some docker tasks.
A simple Ansible tasks for executing a container from an image locally:
- hosts: localhost tasks: - name: run wildfly dc docker: image=ackward/wildfly name=wf-dc1 command="/opt/jboss/wildfly/bin/domain.sh -b 0.0.0.0 -bmanagement 0.0.0.0" / hostname=dc1 publish_all_ports=yes volumes=/opt/jboss/dc1:/opt/jboss/LOCAL_JB/node ports=9990
- An infrastructure for provisioning docker hosts (physical/virtual nodes) or at least a fully Ansible configuration for managing the underlying servers and their services.
- Including dependencies and orchestration of those micro-services.
- A map of external dependencies and how they must be managed in a cloud paradigm (with floating services).
- External resources like accessing a mainframe, SAS, fw rules, access control permissions, etc.
- All the deployment (continuous delivery) platform. How are we going to connect, build and deploy apps in this dockerized Jboss Domain.
- All the end2end experience!!!:
- How the services are accessed. And this one is big, now they are exposing a dynamic tunneled port there is only one known port exposed 9990 (the dc admin console) but they are public services with known urls and ports.
- How developers interact with these environments, how they publish, debug, etc… the applications that run on them.
- How the platform is monitored (nodes, services, apps,…)
- Contingency and HA plans.
- A business case! What would happen in 1, 3, 5 years, associated costs, etc. Do we really need to implement it? are there any advantage, benefits of doing so rather than otherwise?.