Hi,
You have probably read on the rest of the website about what we do at Ethicrawl, what our ethos and values are and how we can help you. What you are probably wondering about is who we are and where we came from.
So, my name is Julien Nioche. I have been working in the field of web crawling, search and big data for a couple of decades now and I describe myself as an open-source veteran. I have been involved in many open source projects over the years, some at the Apache Software Foundation, of which I am a member. I am the original author of what is now Apache StormCrawler but also URLFrontier. I used to be a committer on Apache Nutch and Apache Tika.
I helped many organisations with their crawling needs through my consultancy, DigitalPebble. Quite recently, I became more active in digital sustainability and GreenOps, and the name of DigitalPebble has become associated with that field of work. This left me with no clear identity for my services in web crawling, so instead of mixing things and making it all confusing for my two different audiences, I decided to have Ethicrawl as a home for my web crawling activities, while keeping DigitalPebble for my sustainability work.
Two different worlds, two different names. I hope this clarifies things a bit.
What has happened since
Now, a lot has happened since the last time I blogged on web crawling. For a start, StormCrawler has gone through incubation at the ASF and is now Apache StormCrawler®. The project is very healthy and has a diverse community of contributors; it is also used by many organisations over the world. StormCrawler is as efficient as it is robust, and the fact that it has been running for ten years now at the Common Crawl Foundation to produce the News Dataset is testament to that.
Web crawling as an activity is also seeing a resurgence, driven in part by the need of AI labs to get training data but also to provide search results for AI agents. And this is what motivated me to spell out the values that drive Ethicrawl: facilitate the acquisition of web data in an open, respectful and resource efficient manner.
Open source, and Horizon
Most of our work is based on open source code, but we also have a commercial product called Horizon. This is pretty exciting, and you will see a blog post on its performance compared to other backends for StormCrawler. What it gives you is a URL frontier implementation which is fast but also robust and comes with extra features. A licence for Horizon also gives you some of my consultancy time for free; perhaps to work on your StormCrawler setup.
So, I hope you will follow us on this journey. Please get in touch at julien@ethicrawl.ai for any enquiries, questions or comments.
Happy crawling!
PS: my two professional worlds do overlap. We recently did some work on web crawling for the wonderful Green Web Foundation. Besides, minimising the environmental impact of running web crawls is an important aspect of Ethicrawl.