Fixing JBOSS-LOCAL-USER: javax.security.sasl.SaslException: ELY05128: Failed to read challenge file

 Recently I was trying to access EJBs hosted in wildfly 23 from a remote machine. Earlier I tested the client running on same machine and it was working fine. But when I put the client on a remote machine it started failing with a strange FileNotFoundError.

Caused by: javax.security.sasl.SaslException: Authentication failed: all available authentication mechanisms failed:
   JBOSS-LOCAL-USER: javax.security.sasl.SaslException: ELY05128: Failed to read challenge file [Caused by java.io.FileNotFoundException: /xxx/wildfly/standalone/tmp/auth/local3418030740192890591.challenge (No such file or directory)]
        at org.jboss.remoting3.remote.ClientConnectionOpenListener.allMechanismsFailed(ClientConnectionOpenListener.java:109) ~[jboss-client.jar:20.0.1.Final]
        at org.jboss.remoting3.remote.ClientConnectionOpenListener$Capabilities.handleEvent(ClientConnectionOpenListener.java:445) ~[jboss-client.jar:20.0.1.Final]
        at org.jboss.remoting3.remote.ClientConnectionOpenListener$Capabilities.handleEvent(ClientConnectionOpenListener.java:244) ~[jboss-client.jar:20.0.1.Final]
        at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92) ~[jboss-client.jar:20.0.1.Final]
        at org.xnio.conduits.ReadReadyHandler$ChannelListenerHandler.readReady(ReadReadyHandler.java:66) ~[jboss-client.jar:20.0.1.Final]
        at org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:89) ~[jboss-client.jar:20.0.1.Final]
        at org.xnio.nio.WorkerThread.run(WorkerThread.java:591) ~[jboss-client.jar:20.0.1.Final]
        at ...asynchronous invocation...(Unknown Source) ~[?:?]
        at org.jboss.remoting3.EndpointImpl.connect(EndpointImpl.java:599) ~[jboss-client.jar:20.0.1.Final]
        at org.jboss.remoting3.EndpointImpl.connect(EndpointImpl.java:565) ~[jboss-client.jar:20.0.1.Final]
        at org.jboss.remoting3.ConnectionInfo$None.getConnection(ConnectionInfo.java:82) ~[jboss-client.jar:20.0.1.Final]
        at org.jboss.remoting3.ConnectionInfo.getConnection(ConnectionInfo.java:55) ~[jboss-client.jar:20.0.1.Final]
        at org.jboss.remoting3.EndpointImpl.doGetConnection(EndpointImpl.java:499) ~[jboss-client.jar:20.0.1.Final]
        at org.jboss.remoting3.EndpointImpl.getConnectedIdentity(EndpointImpl.java:445) ~[jboss-client.jar:20.0.1.Final]
        at org.jboss.remoting3.UncloseableEndpoint.getConnectedIdentity(UncloseableEndpoint.java:52) ~[jboss-client.jar:20.0.1.Final]
        at org.wildfly.naming.client.remote.RemoteNamingProvider.getFuturePeerIdentityPrivileged(RemoteNamingProvider.java:151) ~[jboss-client.jar:20.0.1.Final]
        at org.wildfly.naming.client.remote.RemoteNamingProvider.lambda$getFuturePeerIdentity$0(RemoteNamingProvider.java:138) ~[jboss-client.jar:20.0.1.Final]
        at org.wildfly.naming.client.remote.RemoteNamingProvider$$Lambda$80/601221733.run(Unknown Source) ~[?:?]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_31]
        at org.wildfly.naming.client.remote.RemoteNamingProvider.getFuturePeerIdentity(RemoteNamingProvider.java:138) ~[jboss-client.jar:20.0.1.Final]
        at org.wildfly.naming.client.remote.RemoteNamingProvider.getPeerIdentity(RemoteNamingProvider.java:126) ~[jboss-client.jar:20.0.1.Final]
        at org.wildfly.naming.client.remote.RemoteNamingProvider.getPeerIdentityForNaming(RemoteNamingProvider.java:106) ~[jboss-client.jar:20.0.1.Final]
        ... 90 more
        Suppressed: javax.security.sasl.SaslException: ELY05128: Failed to read challenge file
                at org.wildfly.security.sasl.localuser.LocalUserClient.evaluateMessage(LocalUserClient.java:108) ~[jboss-client.jar:20.0.1.Final]
                at org.wildfly.security.sasl.util.AbstractSaslParticipant.evaluateMessage(AbstractSaslParticipant.java:219) ~[jboss-client.jar:20.0.1.Final]
                at org.wildfly.security.sasl.util.AbstractSaslClient.evaluateChallenge(AbstractSaslClient.java:98) ~[jboss-client.jar:20.0.1.Final]
                at org.wildfly.security.sasl.util.AbstractDelegatingSaslClient.evaluateChallenge(AbstractDelegatingSaslClient.java:54) ~[jboss-client.jar:20.0.1.Final]
                at org.wildfly.security.sasl.util.PrivilegedSaslClient.lambda$evaluateChallenge$0(PrivilegedSaslClient.java:55) ~[jboss-client.jar:20.0.1.Final]
                at org.wildfly.security.sasl.util.PrivilegedSaslClient$$Lambda$128/635454149.run(Unknown Source) ~[?:?]
                at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_31]
                at org.wildfly.security.sasl.util.PrivilegedSaslClient.evaluateChallenge(PrivilegedSaslClient.java:55) ~[jboss-client.jar:20.0.1.Final]
                at org.jboss.remoting3.remote.ClientConnectionOpenListener$Authentication.lambda$handleEvent$0(ClientConnectionOpenListener.java:649) ~[jboss-client.jar:20.0.1.Final]
                at org.jboss.remoting3.remote.ClientConnectionOpenListener$Authentication$$Lambda$129/1032360688.run(Unknown Source) ~[?:?]
                at org.jboss.remoting3.EndpointImpl$TrackingExecutor.lambda$execute$0(EndpointImpl.java:991) ~[jboss-client.jar:20.0.1.Final]
                at org.jboss.remoting3.EndpointImpl$TrackingExecutor$$Lambda$127/1508635946.run(Unknown Source) ~[?:?]
                at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) ~[jboss-client.jar:20.0.1.Final]
                at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982) ~[jboss-client.jar:20.0.1.Final]
                at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486) ~[jboss-client.jar:20.0.1.Final]
                at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377) ~[jboss-client.jar:20.0.1.Final]
                at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_31]

Initially the problem looked like to be a bug in LocalUserClient class. But later I found that it is a functionality. For authenticating local users using sasl server creates a challenge file on server and send path of the file to client and client is supposed to return content of that file. If client is also running on same machine then it is able to read content of the file and return and authentication passes. If you are doing it from a remote machine it fails.

So how to authenticate from a remote client?

You need comment out local authentication line from your standalone-full.xml file to force it use remote authentication mechanism.

            <security-realm name="ApplicationRealm">
                <server-identities>
                    <ssl>
                        <keystore path="application.keystore" relative-to="jboss.server.config.dir" keystore-password="xxx" alias="xxx" key-password="xxx" generate-self-signed-certificate-host="localhost"/>
                    </ssl>
                </server-identities>
                <authentication>
                   <!-- <local default-user="$local" allowed-users="*" skip-group-loading="true"/>-->
                    <properties path="application-users.properties" relative-to="jboss.server.config.dir"/>
                </authentication>
                <authorization>
                    <properties path="application-roles.properties" relative-to="jboss.server.config.dir"/>
                </authorization>
            </security-realm>

After that you need to use add-user.sh command to add a user to wildfly. Now create wildfly-config.xml file with your user's credential on client machine.

<configuration>
    <authentication-client xmlns="urn:elytron:1.0">
        <authentication-rules>
            <rule use-configuration="default"/>
        </authentication-rules>
        <authentication-configurations>
            <configuration name="default">
                <sasl-mechanism-selector selector="#ALL"/>
                <set-user-name name="user"/>
                <credentials>
                    <clear-password password="password"/>
                </credentials>
            </configuration>
        </authentication-configurations>
    </authentication-client>
</configuration>

Now you can pass this credentials file to your EJB client using  parameter:

-Dwildfly.config.url=<your dir>/wildfly-config.xml

Now your client should start working without any authentication error. 

Please comment if you need any more information on this.

java.lang.ClassNotFoundException: Could not load requested class : org.postgresql.Driver in Wildfly

Recently I was trying migrate our application from wildfly 10 to wildfly 20. We faced the following error:

Unable to create requested service [org.hibernate.engine.jdbc.env.spi.JdbcEnvironment]
        at org.jboss.as.jpa.service.PersistenceUnitServiceImpl$1$1.run(PersistenceUnitServiceImpl.java:198)
        at org.jboss.as.jpa.service.PersistenceUnitServiceImpl$1$1.run(PersistenceUnitServiceImpl.java:128)
        at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:658)
        at org.jboss.as.jpa.service.PersistenceUnitServiceImpl$1.run(PersistenceUnitServiceImpl.java:212)
        at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
        at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
        at java.lang.Thread.run(Thread.java:748)
        at org.jboss.threads.JBossThread.run(JBossThread.java:485)
Caused by: org.hibernate.service.spi.ServiceException: Unable to create requested service [org.hibernate.engine.jdbc.env.spi.JdbcEnvironment]
        at org.hibernate.service.internal.AbstractServiceRegistryImpl.createService(AbstractServiceRegistryImpl.java:275)
        at org.hibernate.service.internal.AbstractServiceRegistryImpl.initializeService(AbstractServiceRegistryImpl.java:237)
        at org.hibernate.service.internal.AbstractServiceRegistryImpl.getService(AbstractServiceRegistryImpl.java:214)
        at org.hibernate.id.factory.internal.DefaultIdentifierGeneratorFactory.injectServices(DefaultIdentifierGeneratorFactory.java:152)
        at org.hibernate.service.internal.AbstractServiceRegistryImpl.injectDependencies(AbstractServiceRegistryImpl.java:286)
        at org.hibernate.service.internal.AbstractServiceRegistryImpl.initializeService(AbstractServiceRegistryImpl.java:243)
        at org.hibernate.service.internal.AbstractServiceRegistryImpl.getService(AbstractServiceRegistryImpl.java:214)
        at org.hibernate.boot.internal.InFlightMetadataCollectorImpl.<init>(InFlightMetadataCollectorImpl.java:179)
        at org.hibernate.boot.model.process.spi.MetadataBuildingProcess.complete(MetadataBuildingProcess.java:119)
        at org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl.metadata(EntityManagerFactoryBuilderImpl.java:1215)
        at org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl.build(EntityManagerFactoryBuilderImpl.java:1246)
        at org.jboss.as.jpa.hibernate5.TwoPhaseBootstrapImpl.build(TwoPhaseBootstrapImpl.java:44)
        at org.jboss.as.jpa.service.PersistenceUnitServiceImpl$1$1.run(PersistenceUnitServiceImpl.java:170)
        ... 9 more
Caused by: org.hibernate.boot.registry.classloading.spi.ClassLoadingException: HHH010003: JDBC Driver class not found: org.postgresql.Driver
        at org.hibernate.c3p0.internal.C3P0ConnectionProvider.configure(C3P0ConnectionProvider.java:130)
        at org.hibernate.boot.registry.internal.StandardServiceRegistryImpl.configureService(StandardServiceRegistryImpl.java:100)
        at org.hibernate.service.internal.AbstractServiceRegistryImpl.initializeService(AbstractServiceRegistryImpl.java:246)
        at org.hibernate.service.internal.AbstractServiceRegistryImpl.getService(AbstractServiceRegistryImpl.java:214)
        at org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator.buildJdbcConnectionAccess(JdbcEnvironmentInitiator.java:145)
        at org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator.initiateService(JdbcEnvironmentInitiator.java:66)
        at org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator.initiateService(JdbcEnvironmentInitiator.java:35)
        at org.hibernate.boot.registry.internal.StandardServiceRegistryImpl.initiateService(StandardServiceRegistryImpl.java:94)
        at org.hibernate.service.internal.AbstractServiceRegistryImpl.createService(AbstractServiceRegistryImpl.java:263)
        ... 21 more
Caused by: org.hibernate.boot.registry.classloading.spi.ClassLoadingException: Unable to load class [org.postgresql.Driver]
        at org.hibernate.boot.registry.classloading.internal.ClassLoaderServiceImpl.classForName(ClassLoaderServiceImpl.java:134)
        at org.hibernate.c3p0.internal.C3P0ConnectionProvider.configure(C3P0ConnectionProvider.java:127)
        ... 29 more
Caused by: java.lang.ClassNotFoundException: Could not load requested class : org.postgresql.Driver
        at org.hibernate.boot.registry.classloading.internal.AggregatedClassLoader.findClass(AggregatedClassLoader.java:210)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at org.hibernate.boot.registry.classloading.internal.ClassLoaderServiceImpl.classForName(ClassLoaderServiceImpl.java:131)
        ... 30 more

Here I am going to provide solution to the problem. For loading postgresql classes you need to create a module for postgres and reference that module in your application jar.

Steps for creating modue is:

1. Copy postgres jar to /home/wildfly/modules/system/layers/base/org/postgres/main/postgresql-42.2.2.jar
2. Create /home/wildfly/modules/system/layers/base/org/postgres/main/module.xml file with following content.

<module xmlns="urn:jboss:module:1.7" name="org.postgres">
        <resources>
                <resource-root path="postgresql-42.2.2.jar" />
        </resources>
        <dependencies>
                <module name="javax.api"/>
                <module name="javax.transaction.api"/>
        </dependencies>
</module>

Now you need to refer the module from your application JAR. For doing that you need to create jboss-deployment-structure.xml file with following content:

<?xml version="1.0" encoding="UTF-8"?>
  <jboss-deployment-structure>
      <ear-subdeployments-isolated>false</ear-subdeployments-isolated>
          <deployment>
                <resources>
                        <resource-root path="myapplication.jar" />
                </resources>
                <dependencies>
                    <module name="org.hibernate" export="true"/>
                    <module name="javax.api" />
                    <module name="org.postgres" export="true"/>
                    <module name="org.infinispan" />
                    <module name="org.apache.commons.lang" />
                </dependencies>
                <exclusions>
                        <module name="org.jboss.ejb-client"/>
                </exclusions>

          </deployment>
  </jboss-deployment-structure>

Please note that export="true" is required, without this class will not be loaded.

Fixing EJB error in Wildfly 20

 Recently I was working on porting my company application to wildfly. We were stuck at an EJB exception when we tried to access any EJB method. We were accessing EJB from Wildfly application itself.

2020-08-29 13:26:20,833 [ServerService Thread Pool -- 102] ERROR {DataUpdaterImpl.java:1903} [] - EJBCLIENT000079: Unable to discover destination for request for EJB StatelessEJBLocator for "A/B/TestEJB", view is interface com.A.B.Test, affinity is None
javax.ejb.NoSuchEJBException: EJBCLIENT000079: Unable to discover destination for request for EJB StatelessEJBLocator for "ctools/ctservices/UniqueIdGeneratorEJB", view is interface com.tk20.ejb.api.util.UniqueIdGenerator, affinity is None
        at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:622) ~[jboss-ejb-client-4.0.33.Final.jar!/:4.0.33.Final]
        at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:553) ~[jboss-ejb-client-4.0.33.Final.jar!/:4.0.33.Final]
        at org.jboss.ejb.protocol.remote.RemotingEJBClientInterceptor.handleInvocationResult(RemotingEJBClientInterceptor.java:57) ~[jboss-ejb-client-4.0.33.Final.jar!/:4.0.33.Final]
        at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:624) ~[jboss-ejb-client-4.0.33.Final.jar!/:4.0.33.Final]
        at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:553) ~[jboss-ejb-client-4.0.33.Final.jar!/:4.0.33.Final]
        at org.jboss.ejb.client.TransactionPostDiscoveryInterceptor.handleInvocationResult(TransactionPostDiscoveryInterceptor.java:148) ~[jboss-ejb-client-4.0.33.Final.jar!/:4.0.33.Final]
        at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:624) ~[jboss-ejb-client-4.0.33.Final.jar!/:4.0.33.Final]
        at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:553) ~[jboss-ejb-client-4.0.33.Final.jar!/:4.0.33.Final]
        at org.jboss.ejb.client.DiscoveryEJBClientInterceptor.handleInvocationResult(DiscoveryEJBClientInterceptor.java:137) ~[jboss-ejb-client-4.0.33.Final.jar!/:4.0.33.Final]
        at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:624) ~[jboss-ejb-client-4.0.33.Final.jar!/:4.0.33.Final]
        at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:553) ~[jboss-ejb-client-4.0.33.Final.jar!/:4.0.33.Final]
        at org.jboss.ejb.client.NamingEJBClientInterceptor.handleInvocationResult(NamingEJBClientInterceptor.java:87) ~[jboss-ejb-client-4.0.33.Final.jar!/:4.0.33.Final]
        at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:624) ~[jboss-ejb-client-4.0.33.Final.jar!/:4.0.33.Final]
        at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:553) ~[jboss-ejb-client-4.0.33.Final.jar!/:4.0.33.Final]
        at org.jboss.ejb.client.TransactionInterceptor.handleInvocationResult(TransactionInterceptor.java:212) ~[jboss-ejb-client-4.0.33.Final.jar!/:4.0.33.Final]
        at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:624) ~[jboss-ejb-client-4.0.33.Final.jar!/:4.0.33.Final]
        at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:553) ~[jboss-ejb-client-4.0.33.Final.jar!/:4.0.33.Final]
        at org.jboss.ejb.client.EJBClientInvocationContext.awaitResponse(EJBClientInvocationContext.java:995) ~[jboss-ejb-client-4.0.33.Final.jar!/:4.0.33.Final]
        at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:191) ~[jboss-ejb-client-4.0.33.Final.jar!/:4.0.33.Final]
        at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:125) ~[jboss-ejb-client-4.0.33.Final.jar!/:4.0.33.Final]
        at com.sun.proxy.$Proxy124.getUniqueIdString(Unknown Source) ~[?:?]

This error took a lot of time to fix. There was no information available on internet. The problem is the way JNDI lookup is done. We were looking up "ejb:A/B/TestEJB" in JNDI. The lookup was successful but method invocation failed. The problem was fixed after changing the JNDI lookup to "java:global/A/B/Test".

The remote lookup for "java:global/A/B/Test" in JNDI failed. So for remote invocation the JNDI lookup have to be done without any qualifier that is "/A/B/Test". After doing this remote EJB calls also started working fine.

The local lookup and remote lookup need to be done differently.

Please refer to following post for accessing EJBs from remote machine.

https://blog.bigdatawithjasvant.com/2021/06/fixing-jboss-local-user.html