Yahoo is giving away crucial technology powering its internal searches to the world. The internet company has just made Vespa, a search tool used by the tech company for searches within its network has now been open-sourced, described to be the company’s biggest open-source code release since it made Hadoop open source in 2009.
Oath Inc., the Verizon company that has owned Yahoo since June, announced that Vespa is now available as open source on GitHub. According to a company blog post, making the big data processing and serving engine open source is a step further in Oath’s commitment to opening up its big data infrastructure to developers.
“By releasing Vespa, we are making it easy for anyone to build applications that can compute responses to user requests, over large datasets, at real time and at internet scale – capabilities that up until now, have been within reach of only a few large companies,” Jon Bratseth, a systems architect at Yahoo, explained via the blog post.
Vespa, Yahoo’s big data processing and serving engine, available as open source on GitHub!https://t.co/InVSKoSlZY
— Arkaitz Etxezarreta (@arkaitzo) September 29, 2017
Vespa powers Yahoo network’s internal search feature and is used to determine what recommendations to display as well as the company’s ad targeting system. Currently, Vespa is used in around 150 apps such as Yahoo’s main search engine, Yahoo Mail, and Flickr. In addition, the search tool also serves around 3 billion native ads daily.
Vespa’s roots can be traced back to AllTheWeb, a Norwegian search engine which was acquired by Yahoo in 2003, according to Wired. Since then, the tool has been modified as a general purpose engine that Yahoo could use internally in a host of different applications.
The release of Vespa is not the first time Yahoo offered some of its technology as open source. In 2006, the company released Hadoop which became a unique open source tool for Big Data. Apparently, the plan is to replicate the benefits of making Hadoop publicly available which allowed Yahoo to easily recruit people as it becomes widely used.
But Vespa’s reach has the potential to overshadow that of Hadoop. As Bratseth puts it, “Vespa is larger in scope and lines of code than any open source project we’ve ever released,” adding that it has already been battle-proven in Yahoo’s largest and most critical systems. It is attractive both for its scalability and versatility while it is touted to be better than Hadoop in serving results to end users.
However, there are doubts if Vespa could be successful outside of Oath. According to Wired, Hadoop became a success because it was born open source and arrived at a time when businesses needed it most. This time, however, most large companies have already addressed web search issues that Vespa is designed for.
[Featured Image via Yahoo]