Featured in Perplexity AI Cookbook

Snap a poster, unlock insights

iOS app that transforms scientific posters into structured summaries, interactive chat, and validated research citations — using a 4-stage multi-API pipeline.

Swift · SwiftUI MVVM iOS 16+ MIT Perplexity Cookbook App Store
~17k
Lines of Swift
4
API Stages
11
Services
23
SwiftUI Views

The 4-Stage Pipeline

How a poster photo becomes an interactive research assistant — ~$0.0075 per scan

Stage 1
photo_camera
Capture + OCR
AVFoundation camera → Apple Vision framework on-device text extraction
FREE
→
Stage 2
psychology
AI Summary
OpenAI GPT-3.5-turbo generates structured 4-point summary + author questions
→
Stage 3
biotech
Research Discovery
Perplexity Search API with 14 academic domain filters finds 3-5 related papers
→
Stage 4
verified
Citation Validation
PubMed E-Utilities validates, enriches, and links every citation
FREE
→
+ Interactive
forum
Chat
Context-aware Q&A about the poster via Perplexity with conversation history
☁️ Evidence V2 — RAG Pipeline IN DEVELOPMENT

Cloud Functions backend with BigQuery vector search + Vertex AI embeddings for enhanced paper discovery. Deployed on GCP (europe-west2).

iOS App → Cloud Function (evidenceV2) → BigQuery (pubmed_rag.papers) → Vertex AI (text-embedding-004) → Re-ranked citations

Repository Structure

iOS app (Swift/SwiftUI) + GCP Cloud Functions (Python)

  PosterLens/PosterLens/
  • ⚡ PosterLensApp.swift entry
  • ⚙️ Config.swift API config
  • 📁 Services/ 3,819 lines
  • 👁 OCRService.swift 207
  • 🤖 OpenAIService.swift 358
  • 🔬 PerplexityService.swift 1,278
  • 🔬 PerplexityRelatedResearchService.swift 745
  • 📚 PubMedAPI.swift 558
  • ☁️ RAGEvidenceService.swift 52
  • 💬 ChatService.swift 209
  • 🏷 CategoryExtractionService.swift 121
  • 📄 PDFExportService.swift 189
  • 🖼 ImageOrientationFixer.swift 97
  • 🚩 FeatureFlags.swift 5
  • 📁 Models/ 1,594 lines
  • 📋 PosterScan.swift 118
  • 📷 CameraViewModel.swift 146
  • 📷 CameraOverlayManager.swift 223
  • 💬 ChatMessage.swift 81
  • 📝 Citation.swift 166+81
  • 💾 DataStore.swift 406
  • 🏷 PosterCategory.swift 110
  • 👋 OnboardingManager.swift 209
  • 📁 Views/ 8,146 lines
  • 📱 23 SwiftUI views
  • 📁 Utilities/ 1,379 lines
  • 🔧 8 utility modules
☁️ functions/ — GCP Cloud Functions
  • 🐍 main.py 769 lines
  • 📁 ingestion/
  • 🐍 embed_and_load.py 564
  • 🐍 prepare_pubmed_subset.py 400
  • 📊 pubmed_subset.jsonl data
  • 📄 openapi.yaml API spec
  • 📄 requirements.txt
🧪 Tests — 1,915 lines
  • ✅ PubMedParserTests.swift 494
  • ✅ LinkHealthTests.swift 415
  • ✅ QueryBuilderV2Tests.swift 400
  • ✅ VancouverFormatterTests.swift 339
  • ✅ PosterLensTestSuite.swift 186
  • ✅ PerplexityPayloadTests.swift 81
Stack
Swift 5.7+
SwiftUI
AVFoundation
Vision
PDFKit
Python 3.11
Vertex AI
BigQuery
Cloud Functions

Services Layer

11 services powering the multi-API pipeline — singletons with retry logic and caching

OCRService Vision

On-device text extraction using Apple Vision framework. Handles image orientation, preprocessing, and recognition confidence. No API key, no network — pure on-device ML.

Services/OCRService.swift · 207 lines

OpenAIService OpenAI

GPT-3.5-turbo integration for structured summarization (4-point format), author question generation, and category extraction. Temperature 0.3 for factual responses.

Services/OpenAIService.swift · 358 lines

PerplexityService Perplexity

Core Perplexity API client (sonar model). Powers chat, summarization, and author question generation. Singleton with context caching per poster ID.

Services/PerplexityService.swift · 1,278 lines

PerplexityRelatedResearchService Search

Perplexity Search API with search_domain_filter restricting to 14 trusted academic domains. Returns real citations with URLs, validated against PubMed.

Services/PerplexityRelatedResearchService.swift · 745 lines

PubMedAPI PubMed

NCBI E-Utilities client — esearch, esummary, efetch. Validates Perplexity citations by extracting PMIDs, fetching metadata, comparing title similarity (>70% match).

Services/PubMedAPI.swift · 558 lines

RAGEvidenceService RAG V2

Cloud Functions client for the Evidence V2 endpoint. BigQuery vector search + Vertex AI embeddings with deterministic re-ranking (recency, keyword overlap, specificity).

Services/RAGEvidenceService.swift · 52 lines + functions/main.py · 769 lines

ChatService Chat

Conversation management with per-poster context caching. Routes to Perplexity for context-aware Q&A with conversation history.

Services/ChatService.swift · 209 lines

CategoryExtractionService OpenAI

Extracts research categories from poster text via OpenAI. Colour-coded: Cancer Types (red), Research Focus (purple), Phases (green), Treatment (blue).

Services/CategoryExtractionService.swift · 121 lines

PDFExportService Export

Generates A4 PDFs with poster image, structured summary, citations, and categories using UIGraphicsPDFRenderer.

Services/PDFExportService.swift · 189 lines

ImageOrientationFixer Utility

Fixes image orientation by actually rotating pixel data (not just EXIF metadata) via CoreImage transforms.

Services/ImageOrientationFixer.swift · 97 lines

NetworkErrorHandler Utility

Centralised error handling: 3 retries with exponential backoff (1s, 2s, 4s), safe JSON parsing, user-friendly error messages for all API failures.

Utilities/NetworkErrorHandler.swift · 466 lines

Academic Domain Filtering

Perplexity Search API restricted to 14 trusted sources — the key innovation that made it into the Cookbook

pubmed.ncbi.nlm.nih.gov pmc.ncbi.nlm.nih.gov nih.gov arxiv.org scholar.google.com nature.com science.org sciencedirect.com springer.com wiley.com cell.com nejm.org pnas.org ieee.org

Views

23 SwiftUI views — 8,146 lines total

CameraPreviewViewController
AVFoundation camera with edge detection overlay
1,353 lines
CameraView
SwiftUI wrapper with scanning UI and guidelines
846 lines
RelatedResearchView
Citation cards with PubMed links and validation status
789 lines
HistoryView
Scan history with category filtering and search
723 lines
AboutView
App info with PharmaTools branding
691 lines
ButtonRowView
Action buttons for share, chat, research
668 lines
SimpleChatView
Poster-specific chat with message history
402 lines
ChatView
Full chat interface with typing indicators
407 lines
SummaryView
Structured 4-point AI summary display
304 lines
OpenAIChatView
Alternative chat powered by OpenAI directly
309 lines
ScanCardView
Poster scan thumbnail with category tags
255 lines
LiteratureContextView
Evidence context with RAG results
215 lines

Data Models

Core entities — 1,594 lines across 12 models

ModelPurposeKey PropertiesLines
PosterScanCore entity — a scanned posterid, title, rawText, summaryPoints[], categories[], image118
DataStorePersistence layerJSON encoding/decoding, image file management406
CameraOverlayManagerCamera edge detectionOverlay positioning, guide frame calculations223
OnboardingManagerFirst-launch flowStep tracking, completion state209
CitationAcademic referencetitle, authors[], journal, year, doi, pmid, url247
CameraViewModelCamera state managementCapture session, permission handling146
PosterCategoryResearch classificationname, type (enum), color, icon110
ChatMessageChat conversation unitrole, content, timestamp81
EvidenceResultRAG pipeline responsepapers[], confidence scores35
ResearchContextAuthor questions + directionsquestions[], futureDirections11

API Configuration

Rate limits, models, and error handling across all integrations

APIModel / EngineRate LimitCostRetry
Apple VisionOn-device MLUnlimitedFreeN/A
OpenAIgpt-3.5-turbo (temp 0.3)3,500 req/min~$0.0015/req3× exponential
Perplexitysonar (search mode)1,000 req/hr~$0.005/search3× exponential
PubMedE-Utilities (esearch/efetch)3 req/secFree3× exponential
Evidence V2text-embedding-004 + BigQueryCloud FunctionsGCP pricing3× exponential

Cost per Scan

Vision OCR
$0.00
OpenAI Summary
$0.0015
OpenAI Questions
$0.0010
Perplexity Search
$0.0050
PubMed Validation
$0.00
Total per Scan
~$0.0075

Cloud Backend — RAG Pipeline

GCP Cloud Functions + BigQuery + Vertex AI for enhanced evidence retrieval

functions/main.py — 769 lines

evidenceV2 Cloud Function endpoint. Takes poster text, generates embeddings via Vertex AI (text-embedding-004), runs vector similarity search against BigQuery pubmed_rag.papers table, then applies a deterministic re-ranking layer:

  • Recency boost: +0.03 for papers within 5 years
  • Keyword overlap: +0.04 for ≥5 matching terms
  • Generic penalty: −0.02 for review-heavy language

Ingestion Pipeline

Two-stage data pipeline for populating the BigQuery vector store:

1. prepare_pubmed_subset.py (400 lines) — Fetches and filters PubMed metadata into JSONL
2. embed_and_load.py (564 lines) — Generates Vertex AI embeddings, loads into BigQuery with vector index