← All projects

Distributed Scholarly Search Engine

A microservices search engine on a 3-node GCP cluster (Docker, Kafka, MapReduce, Terraform) indexing 10K+ IEEE abstracts, with 37% faster queries by conditionally skipping MapReduce jobs.

Docker Kafka MapReduce Terraform GCP Microservices
CLOUD / DISTRIBUTED SEARCH03 / SYSTEM STUDY
From abstracts to answersA frontend sends search requests to the API. Kafka distributes processing work. Crawled abstracts are stored in GCS and indexed by MapReduce on Dataproc; the API uses existing results when a new MapReduce job is unnecessary.PROCESSING / GCPSearchAPIKafkawork queueCrawlerGCSMapReduceDataprocResultsreuse existing results
From abstracts to answersKafka · GCS · Dataproc

Overview

A distributed search engine over scholarly abstracts, deployed as microservices across a 3-node GCP cluster and provisioned with infrastructure-as-code.

Continue to update… soon.