Derya (Dorian) Sezen

by

Surprisingly my 4-year-old blog post is still on Google #1 Page in organic search, doesn’t it deserve an update?  Just updated with minor changes:

• Windows container support
• Forrester report update

Here is the latest version:

https://www.kloia.com/blog/openshift-vs-rancher

by

I have published an article regarding our Dojo approach at kloia: Click here!

by

Recently i had to dig into a JVM thead wait/blocking/locking troubleshooting and i decided to write down my notes just for future reference for myself and for you!:)

Pre requirements to be known:

1. JVM Memory Space: This is divided to 3 following parts:

1.a Native Heap/Code Cache:  

– Byte Code for JIT

– Native code which are already converted from Byte Code

– Mmap(Memory Map) files

1.b PermGen/Metaspace: Beginning from Java8, this area has no upper limit in order to avoid OutOfMemory errors. Operating System will use SWAP area(Virtual Memory) if the usafe goes beyond Physical Memory.

1.b.i. Size: This is defined my XX parameter 

1.b.ii. Heap Structure: The following are stored under Metaspace:

• Class Definitions Metadata: Class Name, Object Arrays, Internal Object used by JVM, optimization information
• Static Member Variables (Variable itself / Object Reference) 

 1.c Heap: All runtime created stateless&stateful objects, app data, caches are stored in Heap.

1.c.i. Heap Size: Heap size is defined by two parameters:

• Xms(Initial Memory Allocation) 
• Xmx(Maximum Memory Allocation) 

1.c.ii. Heap Structure: Heap is divided into following spaces:

• Young: Divided into 2 parts:
  • Eden: First space used when the object is created.
  • Survival: The objects which are survived from GC(Garbage Collection) are stored on so called S0/S1
• Tenured/OldGen: Objects reacted to max tenured threshold are moved to this space

Here is the model as a schema:

Ref. schema

2. Footprint Requirement: This can be calculated considering the following formula:

• Number of ear, jar, war files that the single JVM process will handle
• Number of Java classes to be loaded during runtime
• Data cache(file, DB …) footprint
• Number of threads that are allowed to be created

Usually heap size 3-4 GB. is a starting point….

3. GC(Garbage Collection): Minimizing the GC frequency is a key factor for performance. Concurrent users and requests generate JVM GC HeartBeat. The frequency should be monitored. 

4. Young vs Tenured: The typical ratio is 1/3, for example if you have 4 GB. Heap, 1GB for Young, 3GB for Tenured is expected. But this is totally depent to your business rules and traffic patterns. 

5. Thread Dump Analyzer: There are various tools that you can use, the ones that i prefer are:

• https://spotify.github.io/threaddump-analyzer/
• http://fastthread.io/

Use XX:+HeapDumpOnOutOfMemoryError in order to create the Heap Dump in case of OutofMemory.

Here are the most popular focus areas that should be analyzed:

• Memory Leak
• Too many open files
• CPU usage
• Deadlocks
• ClassNotFoundException
• NoClassDefFoundError

As a last experience, there is an intersting jar which makes fullGC every 300 sec.:

https://github.com/jelastic-jps/payara/tree/master/payara-micro-cluster 

That can be considered for non performance required applications…. 

by

Routing Mesh is not magic, it just uses ha-proxy inside to route the traffic to the related container.

What is the use-case of Routing Mesh? 

You have plenty of services/apps working on HTTP/HTTPS and you do not want to deal with ports. Routing Mesh simply, manages the "Virtual Hosting" 

1. In case you have Docker Datacenter UCP: 

Step1: Enable Routing Mesh

UCP –> Admin Settings –> Routing Mesh –> Enable HTTP Routing Mesh –> Update (I used port 8090 just for test, normally it should be 80)

Step2: Create a service

UCP –> Resources –> Services –> Create a Service 

Service Name: meshtest

Image Name: nginx:latest

Next –> Rosources –> Networks

Choose the network "ucp-hrm"

Next –> Environment

Create a Service Label "com.docker.ucp.mesh.http.80" with label "external_route=http://meshtest,internal_port=80"

2. In case you just have Docker 1.12+:

Add the following to you docker-compose YAML under the necessary service:

Save Settings and it works!

by

During the installation of Docker Engine and UCP(Universal Control Plane), there is a risk that the networks it chooses by default like 172.17.0.0/16, 172.18.0.0/16, 172.19.0.0/16, 172.20.0.0/16 may overlap with your existing LANs in your organization. Although there are some related posts:

https://forums.docker.com/t/dtr-network-overlaps-corporate-lan/22193/2

https://blog.docker.com/2016/03/docker-networking-design-philosophy/

https://github.com/docker/docker.github.io/blob/master/engine/userguide/networking/default_network/build-bridges.md

IMHO they do not provide a practical solution for that particular case…

Beside, there are the following open issues under github:

https://github.com/docker/docker/issues/21776

https://github.com/docker/docker/pull/29376

Here is a workaround we have founded and applied and working:

• Before the installtion of Docker Engine/UCP, create a virtual interface or extend the netmask of the current interface covering all corporate networks:

ifconfig eth0:0 <yourinternalcorporateIPAddress> netmask <netmask> up

• After you finalize the Docker Engine, swarm or UCP, you will notice that it uses A Class(10.x.x.x/x) or C Class(192.168.x.x/x) rather than 172.x.x.x !!!! Docker installation is smart that it jumps to other network classes.
• Revert back your network interface you the initial state ad that's all!! 

by

I had many unused images appeared because of my fault:

Deryas-MacBook-Pro:etugra dsezen$ docker images

REPOSITORY                  TAG                 IMAGE ID            CREATED             SIZE

etugra                      latest              0e0485e9abbf        5 minutes ago       704.1 MB

<none>                      <none>              8a2ffaa937f3        8 minutes ago       704.1 MB

<none>                      <none>              d13dd077234c        9 minutes ago       704.1 MB

<none>                      <none>              017cb8b846b7        17 minutes ago      704.1 MB

<none>                      <none>              f671aaefbeb8        18 minutes ago      704.1 MB

<none>                      <none>              280ea3ac4ae5        44 minutes ago      704.1 MB

<none>                      <none>              97dcf929d414        45 minutes ago      704.1 MB

<none>                      <none>              019bcd4d779e        53 minutes ago      704.1 MB

<none>                      <none>              053a969a5a22        55 minutes ago      704.1 MB

<none>                      <none>              94be3db991db        58 minutes ago      704.1 MB

<none>                      <none>              cd284b979277        59 minutes ago      704.1 MB

<none>                      <none>              56181d2f6edf        59 minutes ago      704.1 MB

<none>                      <none>              a55ec92c3137        About an hour ago   704.1 MB

<none>                      <none>              7e9afad720aa        About an hour ago   704.1 MB

<none>                      <none>              907bbba168bf        About an hour ago   704.1 MB

<none>                      <none>              f42d87e744f6        About an hour ago   704.1 MB

<none>                      <none>              804aa0feedf2        About an hour ago   704.1 MB

<none>                      <none>              1036b6db0f20        About an hour ago   704.1 MB

<none>                      <none>              4868a31676cb        About an hour ago   704.1 MB

<none>                      <none>              7ade30482774        About an hour ago   704.1 MB

<none>                      <none>              353b07a482be        About an hour ago   704.1 MB

<none>                      <none>              d827f90112d1        About an hour ago   704.1 MB

<none>                      <none>              efb1e94f2c4f        About an hour ago   704.1 MB

<none>                      <none>              9e1210fd7d8f        2 hours ago         684.3 MB

<none>                      <none>              63eaa30ce706        2 hours ago         684.3 MB

<none>                      <none>              b89d5acf03e5        2 hours ago         684.3 MB

<none>                      <none>              c9905abf50b8        2 hours ago         709.6 MB

<none>                      <none>              af6393bc4b96        2 hours ago         696.2 MB

<none>                      <none>              4bb0534b3bd0        21 hours ago        675.1 MB

python                      3-onbuild           61bd1f884054        11 days ago         684.3 MB

Seems docker rmi does not accept standard input, so here is the way i found so that you can overcome this:

docker images|grep none|awk {'print $3'} > /tmp/dockertmp

for i in `cat /tmp/dockertmp`; do docker rmi $i; done