Internal server error., back-off: 4.34s

I’m running a newly created local production deployment and want to upload some databases using alephclient to test the system When I run alephclient write-entities --infile entities.ftm.json --foreign-id opensanctions-peps, I get the following error: WARNING:alephclient.util:Error: Internal server error., back-off: 4.34s

I also get an error when I try to upload documents through the UI.

I don’t know if it matters, but I’m seeing this error in Docker as well:

Are there any error logs in the API service?

It seems like alephclient is not able to access the API.

I’m seeing the following in the api container:

2024-12-20 17:10:22 {“logger”: “aleph.views.base_api”, “timestamp”: “2024-12-20 22:10:22.001255”, “exception”: “Traceback (most recent call last):\n File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 2190, in wsgi_app\n response = self.full_dispatch_request()\n File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1486, in full_dispatch_request\n rv = self.handle_user_exception(e)\n File "/usr/local/lib/python3.8/dist-packages/flask_cors/extension.py", line 176, in wrapped_function\n return cors_after_request(app.make_response(f(*args, **kwargs)))\n File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1484, in full_dispatch_request\n rv = self.dispatch_request()\n File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1469, in dispatch_request\n return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)\n File "/aleph/aleph/views/collections_api.py", line 310, in bulk\n queue_task(collection, SETTINGS.STAGE_INDEX, job_id=job_id, entity_ids=entity_ids)\n File "/aleph/aleph/queues.py", line 26, in queue_task\n taskqueue.queue_task(\n File "/usr/local/lib/python3.8/dist-packages/servicelayer/taskqueue.py", line 823, in queue_task\n rmq_channel.basic_publish(\n File "/usr/local/lib/python3.8/dist-packages/werkzeug/local.py", line 318, in get\n obj = instance._get_current_object()\n File "/usr/local/lib/python3.8/dist-packages/werkzeug/local.py", line 526, in _get_current_object\n return get_name(local())\n File "/usr/local/lib/python3.8/dist-packages/servicelayer/taskqueue.py", line 721, in get_rabbitmq_channel\n connection = pika.BlockingConnection(\n File "/usr/local/lib/python3.8/dist-packages/pika/adapters/blocking_connection.py", line 360, in init\n self._impl = self._create_connection(parameters, _impl_class)\n File "/usr/local/lib/python3.8/dist-packages/pika/adapters/blocking_connection.py", line 451, in _create_connection\n raise self._reap_last_connection_workflow_error(error)\n File "/usr/local/lib/python3.8/dist-packages/pika/adapters/utils/selector_ioloop_adapter.py", line 565, in _resolve\n result = socket.getaddrinfo(self._host, self._port, self._family,\n File "/usr/lib/python3.8/socket.py", line 918, in getaddrinfo\n for res in _socket.getaddrinfo(host, port, family, type, proto, flags):\nsocket.gaierror: [Errno -2] Name or service not known”, “ua”: “alephclient/2.6.0”, “session_id”: “ca9c79f6-2373-437c-9769-638e1c403e4d”, “begin_time”: “2024-12-20T22:10:21.614371”, “trace_id”: “0e69710a-3011-4b9f-8418-1a0124c64bc0”, “role_id”: 34, “method”: “POST”, “endpoint”: “collections_api.bulk”, “v”: “4.0.2”, “locale”: “en”, “url”: “http://localhost:8080/api/2/collections/2/_bulk”, “referrer”: null, “ip”: “172.18.0.1”, “path”: “/api/2/collections/2/_bulk?”, “message”: “InternalServerError: 500 Internal Server Error: The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.”, “severity”: “ERROR”}

I spoke to @jrcloutier and the issue was on our end. The docker-compose.yml on main was outdated. I fixed this with fix: docker-compose image tax syntax broken after merge by stchris · Pull Request #4096 · alephdata/aleph · GitHub just now.