Real-Time Data Streaming with GCP Pub/Sub
Introduction
Retail today is not just about selling products – it’s about instant insights. Customers expect personalized offers, faster checkouts, and always-available inventory. For that, retailers need real-time data processing.
In this tutorial, we’ll build a real-time data streaming pipeline for a retail company using Google Cloud Pub/Sub.
Use Case
A retail chain with 500+ stores wants to:
Track sales instantly.
Detect low inventory in real time.
Enable marketing campaigns while customers shop.
Architecture
Flow:
POS systems → Pub/Sub Topic → Subscription → Dataflow → BigQuery
Step-by-Step Implementation
Step 1: Create a Pub/Sub Topic
gcloud pubsub topics create retail-sales-events
This topic will receive events from all POS systems.
Step 2: Create a Subscription
gcloud pubsub subscriptions create retail-sales-sub \
–topic=retail-sales-events
A subscription delivers messages to consumers.
Step 3: Publish Test Events
gcloud pubsub topics publish retail-sales-events \
–message='{“store_id”:”101″,”product_id”:”P123″,”quantity”:2,”timestamp”:”2025-08-11T10:15:00Z”}’
In production, POS systems will send these events automatically.
Step 4: Pull Messages
gcloud pubsub subscriptions pull retail-sales-sub –auto-ack
Use this to verify events are flowing.
Integrating with Dataflow & BigQuery
Dataflow transforms and enriches events (e.g., adding product category, price, discount).
BigQuery stores the transformed data for real-time dashboarding in Looker Studio.
Example Dataflow Template Command:
gcloud dataflow jobs run retail-sales-stream \
–gcs-location gs://dataflow-templates/latest/PubSub_to_BigQuery \
–parameters inputTopic=projects/YOUR_PROJECT/topics/retail-sales-events,outputTableSpec=YOUR_PROJECT:retail_dataset.sales_table
Benefits
✅ Real-Time Inventory Management – Trigger restock orders before products sell out.
✅ Personalized Marketing – Offer discounts based on ongoing purchases.
✅ Fraud Detection – Identify suspicious transactions instantly.
Conclusion
With GCP Pub/Sub, retail companies can unlock the power of real-time decision-making. From inventory control to personalized offers, the possibilities are endless – and the business impact is massive.