Aleph Q2 Roadmap Update

Announcements
1

Welcome
Hey everyone. Welcome to the April update on the Aleph Roadmap. We want to take a moment to give you some progress updates on the features that we’ve been working on over the past few months and what you can expect in the near future.
Landing soon - 2024 Aleph questionnaire

Aleph Community Survey
It’s that time again. For the past couple of years now the Aleph team have been running a questionnaire for our community, and this year will be no different. Our goal with the questionnaire is to get a better understand of the things that you like, the things that you don’t like, and the things that you find are missing from the Aleph platform. Look out for further communication on Discourse, X (formerly Twitter) and in our Slack channel over the coming weeks.

The Aleph questionnaire is our attempt to get a feeling of what our community needs from us, and helps us to prioritize the work that ends up on our roadmap. We’d love to get your feedback on all things Aleph related, so when the questionnaire lands, please take a moment to fill it in.
Just Released - Security updates

Just Released - Security Updates
For those of you more active in the Aleph community you will have noticed that we just released some updates to Aleph and Ingest file, these updates contain fixes to security vulnerabilities that were identified during a routine security audit. The team recommends that you update your Aleph instance to the latest version to ensure that you have these fixes:

You can read a detailed breakdown of the issues on the following page: aleph 3.15.6, ingest-file 3.20.2 and servicelayer 1.22.2 fix security vulnerabilities

Releasing soon - Improved Metrics
As mentioned in the last update at the end of 2023 the Aleph team have been working on improved metrics for the platform. The work for this has now been completed and should be available in the next minor version of Aleph (3.16.x).

The metrics project aims to provide admins and those concerned with the performance and stability of Aleph access to a range of metrics that will help to better run and manage your instance. These metrics will use Prometheus which will allow you to import the information into tools such as Grafana and Datadog.

The metrics work will provide you with access to:

  • Which versions of Aleph, FtM and ingest-file are running.
  • Statistics on response times from Aleph
  • Performance metrics
  • Concurrent users and authentication requests
  • API request calls, counts, and timings

Releasing soon - RabbitMQ Integration
Our RabbitMQ project has been running for quite some time now. In fact, we initially launched the feature all the way back in 2021. Unfortunately, there were a number of shortcomings with the initial implementation which required us to roll things back and rethink our strategy.

We picked the project back up in late 2023 and since that time the Aleph team have been making good progess on our RabbitMQ implementation trying to resolve outstanding issues and ensure that new issues do not become apparent when we launch in the near future.

This change, which moves us away from our existing custom message broker will provide us with a number of benefits:

  • Reduced maintenance of our code
  • Reliable status reporting including the ability to see the timestamp of the latest update to the ingest, allowing interested parties to check the velocity of a dataset ingest.
  • Improved exception handling over the lifetime of a task
  • Improved message capture and retrieval
  • A shift away from a memory based queue to one stored on permenant disk, increasing reliability and scalability.

This work is an important first step in allowing us to progress on our next big project and should be landing shorty after 3.16 gets into your hands.

It should be noted that there will be a breaking changes with this feature. The status API has a breaking change that prevents backwards compatibility. The JSON structure used for response information has changed. For more information on this check out the “Other changes” section of the following PR: Release/1.23.0 by stchris · Pull Request #143 · alephdata/servicelayer · GitHub

In development - Improved Notifications
Our notifications project aims to tackle the existing shortcomings that exist on the Aleph status page. How many times have you ingested a dataset into Aleph only for it to get stuck at 100% with no way to ascertain what has happened, and nothing to help you decide what you should do?

Once completed this project will help solve some of the common problems faced around ingesting datasets by providing more information and context around the data being pulled into Aleph. This project builds off of work that the team have already done around RabbitMQ. Our goal is to create a better, more informative experience on the status page that communicates information such as:

  • An indication of the type of process running (ingest, xref, etc)
  • The start and last update timestamp for each ingest
  • A breakdown of entities that have failed to import into Aleph
  • Estimates on the time left to complete a running process for some (but not all) types of processes.

We’ll be sharing more on this project via Discourse in the coming weeks and months, so stay tuned for updates.

Later this year - Search improvements
Search is at the core of what makes Aleph such a useful tool. For a while now the team has been collecting feedback both internally and externally from users that feel that the Aleph search experience could be a lot better.

To address this, later on in the year, the team are going to be focused on implementing changes and features that we hope will make the search experience in Aleph better than ever. The precise list of work that we’re going to be doing is still being discussed but you may see:

  • Greater access to the full range of document search results
  • Improved user experience around previewing search results in a document
  • Better filtering of search results with languages and countries
  • Improved search results highlighting
  • Vector based semantic search
  • Audio and video search via transcription

On Hold - Profiles
In our December update we talked about our desire to want to tackle Alephs implementation on Profiles. We’ve now moved this project back as we felt that continuing to address the speed and quality of our search experience, and providing better insights into the state of Aleph were more valuable to our users than addressing profiles. At least for now.

Want to have input into our roadmap? Book an appointment, let’s chat!
You may not be aware but the Aleph team has a regular slot for you to book a call and talk to us about anything Aleph related. Whether that is an issue in deploying or managing your own Aleph instance, or if it’s some feedback on missing or desired features, the team would be more than happy to sit with you and discuss whatever Aleph issues are on your mind.

If you do want to chat please feel free to make an appointment using the link below

https://calendar.app.google/i5tLYgCGj2tpH7kP8

Speak soon
We’re actively working on better communication to our community and those that use and deploy Aleph. We’ll continue to put these quarterly updates out and are starting to experiment on more project focused communication for the initiatives that we’re working on. Keep an eye on discourse for more updates in the coming weeks and months.

3 Likes