Grub Next Generation is a distributed web crawling system (clients/servers) which helps to build and maintain free (as in freedom) index of the Web. This is the standalone, configurable upload server written in C# (Mono)
Search is part of the fundamental infrastructure of the Internet. And, it is currently broken.
Why is it broken? It is broken for the same reason that proprietary software is always broken: lack of freedom, lack of community, lack of accountability, lack of transparency. Here, we will start to change all that.
Grub started back in 2000 with a simple concept of distributing part of the search process pipeline: crawling. In a way, we were a bit ahead of our time, but our intention then was what it is now. We want to help fix search.
Now, with the help of Wikia, community members, contributers, and Open Source developers our time has come again. Come be part of something greater. Come help us change the World.
What is new in this release:
- fixed bug with not deleting temporary directories
- some code optimization
What is new in version 0.9:
- better handling .pid file
- fixed bug with crash uploader on connection failure to Solr
- better upload .arc files
- fixed bug with stop/restart server
What is new in version 0.8.6:
- updated server API
- checking if uploader works by .pid file
- fixed bug with crash server on file listing
- added informations about amount and total size of files during listing directories
- added option for disable directory listing via HTTP
- added support for compression methods: BZip2 and Zip
What is new in version 0.8.5:
- Setting server options via the API was fixed.
- The server now correctly encodes URLs for HTTP GET requests.
- Some code optimizations were made.
What is new in version 0.8.4:
- added option to work directly on disk instead of using memory
- added check for content type for URL's in .arc files
- added option to set limit connection speed for upload/download
- fixed bug with crash server on invalid PUT request
- some code optimization
What is new in version 0.8.3:
- fixed bug with crash server on received invalid .arc file
- added option to configure limit for max allowed MB for one URL in .arc files added option to configure size of good.arc and bad.arc files
- added option to ban connection by IP or username
- added remote control (configure/stop/restart)
- added ClamAV C# wrapper
- some code optimization
What is new in version 0.8.1:
- fixed bug with crash uploader on invalid HTML META tag
- fixed bug with crash uploader on invalid data in file
- fixed bug with crash uploader on connect to Solr
- fixed bug with crash server on invalid HTTP status code in .arc file
- fixed bug with crash server on prematurely closed connection
- added write to Solr only this pages which can be converted to UTF-8
- added remove all control characters from Solr UpdateXML
What is new in version 0.6:
- code cleanup and refactoring
- move uploading and checking .arc files to separate program
- fixed bug with count connected clients
- fixed server answer on invalid POST request
- fixed bug with crash server on copy not existing file
- fixed bug with crash server on unfinished files download
- fixed bug #377: Connection gets reset instead of closed after disk full error
- added simple connections log
- added option to move .arc files on disk (not only to Hadoop)
What is new in version 0.5:
- This is the first beta version.
- LZMA compression support was added.
- The server API was extended.
- Checking the correctness of sent .arc files was improved.
- Code optimization was done.
Comments not found