Ceph RGW Object Storage couldn't be added (Invalid credentials or URL)

##### ISSUE TYPE

 * Bug Report
 * Improvement Request
 * Enhancement Request
 * Feature Idea
 * Documentation Report
 * Other

##### COMPONENT NAME

~~~
Infrastructure > Object Store
~~~

##### CLOUDSTACK VERSION


~~~
4.20.0.0
~~~

##### CONFIGURATION



##### OS / ENVIRONMENT


Ceph version: 19.2.0 squid (stable) 

##### SUMMARY



##### STEPS TO REPRODUCE




1. Create a Object Gateway user on Ceph Dashboard. The user is added under a Tenant.
2. Try to add object store using the Ceph provider.

![image](https://github.com/user-attachments/assets/bc8630d6-e5b7-4ba5-9a4c-11029fa6b2ca)




##### EXPECTED RESULTS


~~~
Being able to add a Ceph object store to my infrastructure.
~~~

##### ACTUAL RESULTS



~~~
Failed with error:

Request failed. (530)
Failed to add object store: Error while initializing Ceph RGW Object Store. Invalid credentials or URL
~~~

##### ERROR LOG

```
2024-12-05 05:54:03,154 DEBUG [c.c.a.ApiServlet] (qtp2038105753-397:[ctx-d6792894]) (logid:de1e0941) ===START===  10.0.113.1 -- GET  name=Ceph+Test+1&provider=Ceph&url=https:%2F%2F<URL_HERE>%2F&command=addObjectStoragePool&response=json&sessionkey=_IgdzVMijOi6635kzbViqZtkR2A
2024-12-05 05:54:03,154 DEBUG [c.c.a.ApiServlet] (qtp2038105753-397:[ctx-d6792894]) (logid:de1e0941) Two factor authentication is already verified for the user 2, so skipping
2024-12-05 05:54:03,161 DEBUG [c.c.a.ApiServer] (qtp2038105753-397:[ctx-d6792894, ctx-4ac36717]) (logid:de1e0941) CIDRs from which account 'Account [{"accountName":"admin","id":2,"uuid":"c2a81095-5469-11ef-8cfa-005056a0391c"}]' is allowed to perform API calls: 0.0.0.0/0,::/0
2024-12-05 05:54:03,163 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp2038105753-397:[ctx-d6792894, ctx-4ac36717]) (logid:de1e0941) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
2024-12-05 05:54:03,163 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp2038105753-397:[ctx-d6792894, ctx-4ac36717]) (logid:de1e0941) API rate limiting is disabled. We will not use ApiRateLimitService.
2024-12-05 05:54:03,176 INFO  [o.a.c.s.d.l.CephObjectStoreLifeCycleImpl] (qtp2038105753-397:[ctx-d6792894, ctx-4ac36717]) (logid:de1e0941) Attempting to connect to Ceph RGW at https://<URL_HERE>/ with access key <ACCESS_KEY_HERE>
2024-12-05 05:54:03,199 DEBUG [o.a.c.s.d.l.CephObjectStoreLifeCycleImpl] (qtp2038105753-397:[ctx-d6792894, ctx-4ac36717]) (logid:de1e0941) Error while initializing Ceph RGW Object Store: Cannot invoke "java.util.List.stream()" because the return value of "org.twonote.rgwadmin4j.impl.RgwAdminImpl.listBucketInfo()" is null
2024-12-05 05:54:03,199 DEBUG [c.c.s.StorageManagerImpl] (qtp2038105753-397:[ctx-d6792894, ctx-4ac36717]) (logid:de1e0941) Failed to add object store: Error while initializing Ceph RGW Object Store. Invalid credentials or URL java.lang.RuntimeException: Error while initializing Ceph RGW Object Store. Invalid credentials or URL
        at org.apache.cloudstack.storage.datastore.lifecycle.CephObjectStoreLifeCycleImpl.initialize(CephObjectStoreLifeCycleImpl.java:88)
        at com.cloud.storage.StorageManagerImpl.discoverObjectStore(StorageManagerImpl.java:4140)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:105)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at jdk.proxy3/jdk.proxy3.$Proxy130.discoverObjectStore(Unknown Source)
        at org.apache.cloudstack.api.command.admin.storage.AddObjectStoragePoolCmd.execute(AddObjectStoragePoolCmd.java:115)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:173)
        at com.cloud.api.ApiServer.queueCommand(ApiServer.java:831)
        at com.cloud.api.ApiServer.handleRequest(ApiServer.java:652)
        at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:354)
        at com.cloud.api.ApiServlet$1.run(ApiServlet.java:157)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:154)
        at com.cloud.api.ApiServlet.doGet(ApiServlet.java:108)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:645)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
        at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.Server.handle(Server.java:516)
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
        at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:555)
        at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:410)
        at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:164)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
        at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
        at java.base/java.lang.Thread.run(Thread.java:840)

2024-12-05 05:54:03,200 WARN  [o.a.c.m.w.WebhookServiceImpl] (qtp2038105753-397:[ctx-d6792894, ctx-4ac36717]) (logid:de1e0941) Skipping delivering event [ID: null, description: {"event":"OBJECT.STORE.CREATE","status":"Completed"}] to any webhook as account ID is missing
2024-12-05 05:54:03,202 WARN  [o.a.c.f.e.EventDistributorImpl] (qtp2038105753-397:[ctx-d6792894, ctx-4ac36717]) (logid:de1e0941) Failed to publish event [category: ActionEvent, type: OBJECT.STORE.CREATE] on bus webhookEventBus
2024-12-05 05:54:03,213 ERROR [o.a.c.a.c.a.s.AddObjectStoragePoolCmd] (qtp2038105753-397:[ctx-d6792894, ctx-4ac36717]) (logid:de1e0941) Exception: com.cloud.utils.exception.CloudRuntimeException: Failed to add object store: Error while initializing Ceph RGW Object Store. Invalid credentials or URL
        at com.cloud.storage.StorageManagerImpl.discoverObjectStore(StorageManagerImpl.java:4145)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:105)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at jdk.proxy3/jdk.proxy3.$Proxy130.discoverObjectStore(Unknown Source)
        at org.apache.cloudstack.api.command.admin.storage.AddObjectStoragePoolCmd.execute(AddObjectStoragePoolCmd.java:115)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:173)
        at com.cloud.api.ApiServer.queueCommand(ApiServer.java:831)
        at com.cloud.api.ApiServer.handleRequest(ApiServer.java:652)
        at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:354)
        at com.cloud.api.ApiServlet$1.run(ApiServlet.java:157)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:154)
        at com.cloud.api.ApiServlet.doGet(ApiServlet.java:108)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:645)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
        at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.Server.handle(Server.java:516)
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
        at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:555)
        at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:410)
        at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:164)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
        at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
        at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.lang.RuntimeException: Error while initializing Ceph RGW Object Store. Invalid credentials or URL
        at org.apache.cloudstack.storage.datastore.lifecycle.CephObjectStoreLifeCycleImpl.initialize(CephObjectStoreLifeCycleImpl.java:88)
        at com.cloud.storage.StorageManagerImpl.discoverObjectStore(StorageManagerImpl.java:4140)
        ... 67 more

2024-12-05 05:54:03,213 INFO  [c.c.a.ApiServer] (qtp2038105753-397:[ctx-d6792894, ctx-4ac36717]) (logid:de1e0941) Failed to add object store: Error while initializing Ceph RGW Object Store. Invalid credentials or URL
2024-12-05 05:54:03,214 INFO  [c.c.a.ApiServlet] (qtp2038105753-397:[ctx-d6792894, ctx-4ac36717]) (logid:de1e0941) (userId=2 accountId=2 sessionId=node0a1m8rqhb8vwl17sdi3x5937fp3) 10.0.113.1 -- GET name=Ceph+Test+1&provider=Ceph&url=https:%2F%2F<URL_HERE>%2F&command=addObjectStoragePool&response=json&sessionkey=_IgdzVMijOi6635kzbViqZtkR2A 530 Failed to add object store: Error while initializing Ceph RGW Object Store. Invalid credentials or URL
2024-12-05 05:54:03,214 DEBUG [c.c.a.ApiServlet] (qtp2038105753-397:[ctx-d6792894, ctx-4ac36717]) (logid:de1e0941) ===END===  10.0.113.1 -- GET  name=Ceph+Test+1&provider=Ceph&url=https:%2F%2F<URL_HERE>%2F&command=addObjectStoragePool&response=json&sessionkey=_IgdzVMijOi6635kzbViqZtkR2A
```

It might be something related to `org.twonote.rgwadmin4j.impl.RgwAdminImpl.listBucketInfo()`. Note that I'm inserting the base URL of my Ceph RGW endpoint. The endpoint should support both path and virtual host based URL (tested with WinSCP). No bucket created at all for the user.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ceph RGW Object Storage couldn't be added (Invalid credentials or URL) #10044

ISSUE TYPE

COMPONENT NAME

CLOUDSTACK VERSION

CONFIGURATION

OS / ENVIRONMENT

SUMMARY

STEPS TO REPRODUCE

EXPECTED RESULTS

ACTUAL RESULTS

ERROR LOG

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Ceph RGW Object Storage couldn't be added (Invalid credentials or URL) #10044

Description

ISSUE TYPE

COMPONENT NAME

CLOUDSTACK VERSION

CONFIGURATION

OS / ENVIRONMENT

SUMMARY

STEPS TO REPRODUCE

EXPECTED RESULTS

ACTUAL RESULTS

ERROR LOG

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions