Increasing container startup timeout of testcontainers Mysql container

Recently I moved my Docker from SSD drive to HDD drive as per my post https://blog.bigdatawithjasvant.com/2023/01/installing-docker-in-secondary-drive-on.html

After doing this change I faced a problem that my code build was failing because docker containers launched by testcontainers were slow to come up and testcontainers was only waiting for 120 seconds for it to come up, after that it was killing the container and starting an new one. The new container was also taking more than 120 second so it was also killed and after 3 attempts the build failed. I got the following error in my logs:

[ERROR] Could not start container
java.lang.IllegalStateException: Container is started, but cannot be accessed by (JDBC URL: jdbc:mysql://localhost:51209/test), please check container logs
    at org.testcontainers.containers.JdbcDatabaseContainer.waitUntilContainerStarted (JdbcDatabaseContainer.java:165)
    at org.testcontainers.containers.GenericContainer.tryStart (GenericContainer.java:466)
    at org.testcontainers.containers.GenericContainer.lambda$doStart$0 (GenericContainer.java:329)
    at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess (Unreliables.java:81)
    at org.testcontainers.containers.GenericContainer.doStart (GenericContainer.java:327)
    at org.testcontainers.containers.GenericContainer.start (GenericContainer.java:315)
    at org.testcontainers.jdbc.ContainerDatabaseDriver.connect (ContainerDatabaseDriver.java:118)
    at org.jooq.codegen.GenerationTool.run0 (GenerationTool.java:362)
    at org.jooq.codegen.GenerationTool.run (GenerationTool.java:236)
    at org.jooq.codegen.GenerationTool.generate (GenerationTool.java:231)
    at org.jooq.codegen.maven.Plugin.execute (Plugin.java:207)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:301)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:211)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:165)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:157)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:121)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:127)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:294)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:960)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:293)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:196)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:566)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
[ERROR] Log output from the failed container:

I was looking for a option to increase this timeout but did not find any provision for doing that in the source code of testcontainers. So I had to take a fork from https://github.com/testcontainers/testcontainers-java and change the code to add new provision to override the timeout. The enhanced code is available at following location:

https://github.com/jasvant6thstreet/testcontainers-java-jasvant

You can take a clone of above repository and build it using following command:

gradle build -x test

This command will compile the code on your local machine without running the testcases. It takes some time to build so be patient and wait for it to complete. Once build is complete you can push it to your local maven repository using following command:

gradle publishToMavenLocal

The above command will push enhanced testcontainers to your maven repository and after that you can use it in your maven projects. The version number is 1.17.6-j1 for this fix.

For setting new timeout following new properties are added:

tc.jdbc.startuptimeout=120
tc.jdbc.connectiontimeout=120

You can set it up at three places as per documentation provide in page

https://www.testcontainers.org/features/configuration/

I used the first option to set it via environment variables. I set up the following environment variables in the shall where I was running my build:

TESTCONTAINERS_TC_JDBC_CONNECTIONTIMEOUT=600
TESTCONTAINERS_TC_JDBC_STARTUPTIMEOUT=600

Using the above environment variables testcontainers Mysql connector was waiting for 600 seconds and the docker container was able to come up successfully in this time and build was successfully completed.

I have submitted a pull request to testcontainers for incorporating this change.

Installing Docker in secondary drive on Windows

 Recently I faced a problem that my C drive on widows was almost full. This was due to the fact that docker was consuming 57 GB of space. I had 900 GB in my secondary drive free and my C drive was just 165 GB in total. But the docker installer for Windows does not provide any option to install it on any alternate drive setting "data-root" in configuration file did not help but it made docker desktop hung and I had to manually remove that entry to make docker desktop come up again successfully.

But there is solution to this problem. On windows Docker uses WSL module for running its virtual machine and run a linux distribution inside that. This virtual machine's virtual hard disk is stored in "C:\Users\<Username>\AppData\Local\Docker\wsl\data" folder. There are two linux distributions which runs to support Docker on windows.

PS C:\Windows\system32> wsl -l
Windows Subsystem for Linux Distributions:
docker-desktop (Default)
docker-desktop-data
PS C:\Windows\system32>

docker-desktop is the default and takes less space. docker-desktop-data is the one which takes a lot of space and grows with time. We need to move docker-desktop-data to secondary drive which is D: for my case. We need to stop docker-desktop before moving docker-desktop-data. We don't need to stop docker-desktop-data. We need to run the following commands on priviledged  power shell of window for which you need to run it with "Run as administrator". These call hangs if docker desktop is running. It is difficult to make sure that docker is not running so I recommend to run the following commands just after restarting your computer. 

wsl --shutdown

wsl --export docker-desktop-data docker-desktop-data.tar

wsl --unregister docker-desktop-data

mkdir D:\docker-desktop-data

wsl --import docker-desktop-data D:\docker-desktop-data  .\docker-desktop-data.tar --version 2

After running the above command you will notice that a VHD file is created under "D:\docker-desktop-data" folder. This is the virtual hard disk where Docker stores all the data. Now you can start the docker and start using it with no data loss.

After doing that I faced one problem that docker was running slow because I moved it from SSD to HDD. In normal case this problem is bearable but in my case the build started failing because I was using Mysql in testcontainers and it gives 120 seconds for a container to start and if it does not start in 120 sec it kills it and start a new container. The second container also get killed after 120 seconds and after 3 retries it fails the build. 

There was no provision in testcontainers for increasing the timeout so I had to take a fork of testcontainers and add a provision for doing that. After enhancing testcontainers and increasing timeout to 600 sec my build started working. I described that in my post  

https://blog.bigdatawithjasvant.com/2023/01/increasing-container-startup-timeout-of.html

Maximum sum such that no two elements are adjacent

Recently I came accross an interesting computer science problem.  The problem goes like this:

You are given an array of integers e.g. { 5, 5,  10, 100, 10, 5} you need to find maximum sum which can be achieved using non adjecent elements. In this example maximum sum is 110 which is acheived by including elements { 5, 100, 5 }. We will start with finding maximum sum and later extend it to find elements contributing to maximum sum.

The idea is to calculate maximum sum which can be achived using elements till i and store it in dp[i] . Now for next element i+1 we can include it or exclude it. If we include arr[i+1] then we need to exclude arr[i] so max sum including arr[i+1] can be dp[i-1]+arr[i+1] and maximum sum which can be achived excluding arr[i+1] will be dp[i] so dp[i+1]= max(dp[i-1]+arr[i+1], dp[i]) .  The code for this solution is given below.

public class MaxNonAdjSum1 {
	
    int FindMaxSum(int arr[], int n)
    {
    	int[] dp = new int[n];
    	
    	if(arr[0]>0) {
    		dp[0] = arr[0];
    	} else {
    		dp[0]= 0;
    	}
 
        for (int i = 1; i < n; i++)
        {
        	if((i>=2?dp[i-2]:0)+arr[i]>dp[i-1]) {
        		dp[i]= (i>=2?dp[i-2]:0)+arr[i];
        	} else {
        		dp[i] = dp[i-1];
        	}
        }
        return dp[n-1];
    }
 
    // Driver program to test above functions
    public static void main(String[] args)
    {
    	MaxNonAdjSum1 sum = new MaxNonAdjSum1();
        int arr[] = new int[]{5, 5, 10, 100, 10, 5};
        System.out.println(sum.FindMaxSum(arr, arr.length));
    }

}

In the above code we are using dp[i], dp[i-1] and dp[i-2] only inside the loop. We can replace them with three variables and we can avoid allocating array dp which will reduce space complexity from O(N) to O(1).

But for now let us concentrate on getting elements who are contributing to maximum sum. The idea it to keep a boolean array to keep track if it is included in max sum. If we include arr[i] then arr[i-1] will have to be excluded. So when we set incFlag[i]=true then we set incFlag[i-1]=false. The code for this solution is provided below.

import java.util.ArrayList;

public class MaxNonAdjSum {
	
    ArrayList<Integer> FindMaxSum(int arr[], int n)
    {
    	boolean[] incFlag = new boolean[n];
    	int[] dp = new int[n];
    	
    	if(arr[0]>0) {
    		dp[0] = arr[0];
    		incFlag[0] = true;
    	} else {
    		dp[0]= 0;
    		incFlag[0] = false;
    	}
 
        for (int i = 1; i < n; i++)
        {
        	
        	if((i>=2?dp[i-2]:0)+arr[i]>dp[i-1]) {
        		incFlag[i-1] = false;
        		dp[i]= (i>=2?dp[i-2]:0)+arr[i];
        		incFlag[i] = true;
        	} else {
        		dp[i] = dp[i-1];
        	}
        }
 
        ArrayList<Integer> result = new ArrayList<>();
        for(int i=0; i<n; i++) {
        	if(incFlag[i]) {
        		result.add(arr[i]);
        	}
        }
        return result;
    }
 
    // Driver program to test above functions
    public static void main(String[] args)
    {
    	MaxNonAdjSum sum = new MaxNonAdjSum();
        int arr[] = new int[]{5, 5, 10, 100, 10, 5};
        System.out.println(sum.FindMaxSum(arr, arr.length));
    }

}

One optimization which is still left is to replace dp[i-2], dp[i-1] and dp[i] with three variables and shift there values in every iteration. This will reduce space complexity from O(N) to O(1).

Httpd mod_proxy throwing 503 error

Recently I was debugging an issue of Apache server throwing 503 when we are trying to connect to a nodeJS server running on port 5000. We had proxy setting correctly added into out httpd.conf

<IfModule mod_proxy.c>
    ProxyRequests Off
    SSLProxyEngine On

    <Proxy *>
            Order deny,allow
            Allow from 127.0.0.1
    </Proxy>

    <Location /nodeserver>
        ProxyPass http://localhost:5000
        ProxyPassReverse http://localhost:5000
    </Location>

</IfModule>

The server was throwing 503 error and we were not able to figure out the reason for it because node server was up and was responding to all requests at port 5000. But when we were trying to access it via Apache server it was throwing 503 error. 

After a lot of debugging we found that it was selinux which was preventing Apache server from connecting to node server at port 5000. Although Apache server was able to connect tomcat server running at port 8009. 

The reason of this selective behavior was that selinux defines some ports as http ports and httpd server is allowed to connect to those port. You can get a list of these ports by running following command.

$semanage port -l |grep http_port_t
http_port_t                    tcp      80, 81, 443, 488, 8008, 8009, 8443, 9000

After changing port from 5000 to 9000 for our node server and changing mod_proxy setting in httpd.conf our application started working.

 

Tomcat 9 creates different sessions for different directories

Recently we were testing one application with tomcat 9.0.43 where one application path was starting with sso/ and one was starting with normal/. We were redirecting to normal/ path from sso/ path. We found that the values we set in HttpSession were missing in normal/ path. 

After a lot of debugging we found out that Tomcat 9 was creating two sessions for same client one for sso/ path and one for normal/ path. That is why values set in HttpSession were missing. I wrote following sample jsp file to demonstrate the behavior.

<%@ page language="java" contentType="text/html; charset=UTF-8"
    pageEncoding="UTF-8"%>
<!DOCTYPE html>
<html>
<body>
<%
	String sessionStr = request.getSession().toString();
%>
Session:<%=sessionStr%>

</body>
</html>

I placed the file at two locations. One under sso/index.jsp and one under normal/index.html and hit both of them from same browser and output was following.

SSO:    Session:org.apache.catalina.session.StandardSessionFacade@2ca7ac8e
Normal: Session:org.apache.catalina.session.StandardSessionFacade@4b992f7e

This problem looks like to be fixed in tomcat 9.0.54