Javascript Results Crawler & Go Server

NullPointer · 2017-07-15T14:18:21+00:00

Leo matokeo ya ACSEE yametoka. Ilikua ngumu kidogo kuyapata maana server ilikua very busy. Sasa kama kubrush kidogo JavaScript knowledge nimeandika crawler fasta fasta kuvuta pages zote then kucreate our own webpage na kuserve matokeo yote. To see it in action fungua hapa, link itakua hewani for few hours only. https://goo.gl/4HNnhw const Request = require('request'); const fs = require('fs'); const path = require('path'); function getPage(number){ console.log("Getting page..." + number); Request.get(`http://www.necta.go.tz/matokeo/ACSEE2017/results/${number}.htm`, function(err,res, body){ if(err){ setTimeout(function(){ console.log("Trying again...." + number) //keep trying after 2 seconds coz the server is busy getPage(number) }, 2000); }else{ savePageToDisk(number, body) } }); } function savePageToDisk(number, body){ fs.writeFile(path.join(__dirname,'matokeo',`${number}.htm`), body, function(err){ if(err){ console.log("Error saving page") console.log(err) } }); } function startingPoint(){ let k=0; //coz schools go up to 5500, a better way is to get index then just regex urls instead of trying all combinations for(let i=101; i<5500; i++){ setTimeout(function(){ getPage(FormatNumberLength(i)) //execute every after 250 milliseconds }, k*250) k++ } } function FormatNumberLength(num) { var r = "" + num; while (r.length < 4) { r = "0" + r; } //change s into p to get private candidate schools return "s" + r; } startingPoint() Alafu kibishibishi ili kuwa na webpage ya kuyaserve yote. Tunarudi kwenye golang, code yake ipo hivi package main import ( "compress/gzip" "fmt" "io" "log" "net/http" "os" "path" "github.com/julienschmidt/httprouter" ) func main() { router := httprouter.New() router.GET("/", handleIndex) router.GET("/matokeo/:filename", handleFiles) log.Print("Listening on 8080") http.ListenAndServe(":8080", router) } func handleIndex(w http.ResponseWriter, r *http.Request, _ httprouter.Params) { file, err := os.Open("./index.html") if err != nil { io.WriteString(w, "Kuna tatizo jaribu tena") return } defer file.Close() w.Header().Set("Content-Encoding", "gzip") w.Header().Set("Content-Type", "text/html") gzipWriter := gzip.NewWriter(w) defer gzipWriter.Close() io.Copy(gzipWriter, file) } func handleFiles(w http.ResponseWriter, r *http.Request, params httprouter.Params) { filename := params.ByName("filename") file, err := os.Open(path.Join("matokeo", filename)) if err != nil { io.WriteString(w, "Kuna tatizo jaribu tena") return } defer file.Close() w.Header().Set("Content-Encoding", "gzip") w.Header().Set("Content-Type", "text/html") w.Header().Del("Content-Length") gzipWriter := gzip.NewWriter(w) defer gzipWriter.Close() io.Copy(gzipWriter, file) } The code could be better, ila it does its job. Mfano hiyo gzip compression writer, kwenye real production app inakua kama middleware inapitishwa kwenye handleFunc zote automatically na sio kurudiarudia the same thing over and over au an even better method ni kuacha kuhangle gzip compression within the app na kurusha hiyo kazi yote kwa nginx, its much faster. Ila tuache utani Go is beautiful. Another thing, badala ya kucrawl from 101 to 5000 kuna so many 404 pages ambazo ni unnecessary, a better way ni kupata index.html then kutumia regex kupata urls zote directly then ku-crawl hizo urls. Ila at the time I'm writing this index.html haikuwepo coz ilikua ni system ya kuandika inakutafutia. Kama upo curious, all this was done on a raspberry pi, hiyo go server nayo inarun from a raspberry pi, nimetumia ngrok kutunnel data through that domain, ndio maana unaona domain ya ajabu ajabu

Thread starter NullPointer
Start date Jul 15, 2017

NullPointer

JF-Expert Member

Jul 15, 2017

Mahamoud Salum

JamiiForums uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
By continuing to use this site, you are consenting to our use of cookies.

Accept Learn more…

Back

Top Bottom

Forums

New posts