Build a Competitive AI System: TensorFlow & Kubernetes

Your competitors are monitoring your prices and offers in real-time. While you’re still manually updating spreadsheets, they’re using automated systems processing thousands of data points per hour. The edge isn’t just technological but architectural. It’s within your reach without spending on third-party APIs.

Photo: Igor Omilaev on Unsplash

Building your own AI-powered competitive intelligence system isn’t just for large companies with massive teams. It’s practical engineering that combines TensorFlow for training models with Kubernetes for orchestrating distributed scraping. The best part? The architecture scales from 10 URLs to 100,000 without rewriting a single line. This is what few explain.

Why Build Your Own System Instead of Buying SaaS

Crater, Kompyte, and Klue charge between $800 and $2,400 monthly for dashboards you can’t customize. The issue isn’t just the price: you’re tied to their roadmap and integrations. You need to extract specific information from competitors’ PDFs or analyze sentiment in niche forums, but those platforms don’t always meet your needs.

In 2026, training a text classification model with TensorFlow costs less than $12 on Compute Engine if you use preemptible instances, while Kubernetes on GKE offers a free tier for small clusters. The key investment is development time: between 40 and 60 hours for a functional MVP. What surprises me most is the complete control you gain over what data you collect and how you expose it to your team.

There are three clear situations where the advantage is significant: processing documents in proprietary formats that SaaS doesn’t support, requiring latency under 2 seconds for real-time alerts, or having such a specific jargon that generic models fail. If any apply, building is more economical than adapting a tool not designed for your case.

System Architecture: Scraping, Classification, and Storage

Photo: Luke Jones on Unsplash

The flow has three layers. First layer: distributed scrapers running as CronJobs in Kubernetes. Each Job manages a subset of URLs, extracts raw HTML, and deposits it in Cloud Storage. Second layer: processing pipeline with TensorFlow that classifies content and extracts relevant entities. Third layer: REST API in FastAPI that exposes structured data for dashboards or alerts.

That said, never try to do everything in one container. Scrapers fail: some sites block, others change HTML structure overnight. If you mix scraping with ML in the same pod, an error in BeautifulSoup can crash your model. Separating responsibilities is crucial: you can scale scrapers without touching TensorFlow, and vice versa.

For storage, I prefer Cloud Storage over Postgres for raw data. One JSON file per monitored site is economical. Postgres comes later: store only extracted entities and metadata. This keeps your database agile and your queries fast.

# scraper-cronjob.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: competitor-scraper
spec:
  schedule: "0 */6 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: scraper
            image: gcr.io/your-project/scraper:latest
            env:
            - name: TARGET_URLS
              valueFrom:
                configMapKeyRef:
                  name: scraper-config
                  key: urls
          restartPolicy: OnFailure

This CronJob runs every 6 hours. Adjust the frequency as needed: pricing sites every hour, corporate blogs every 24 hours.

Training the Classification Model with TensorFlow

Do you really need GPT-4 to classify price or feature information? A binary classification model with pre-trained embeddings and two dense layers achieves 94% accuracy with just 2,000 labeled examples. The advantage is inference in under 50ms per document.

Use Universal Sentence Encoder from TensorFlow Hub for embeddings. It’s faster than BERT and precise enough for this case. The model takes text, converts it into a vector, and classifies it into categories you define. Each category needs at least 300 examples for decent training.

import tensorflow as tf
import tensorflow_hub as hub

# Load Universal Sentence Encoder
embed = hub.load("https://tfhub.dev/google/universal-sentence-encoder/4")

# Define model
model = tf.keras.Sequential([
    tf.keras.layers.Input(shape=(512,), dtype=tf.float32),
    tf.keras.layers.Dense(128, activation='relu'),
    tf.keras.layers.Dropout(0.3),
    tf.keras.layers.Dense(5, activation='softmax')  # 5 categories
])

model.compile(
    optimizer='adam',
    loss='sparse_categorical_crossentropy',
    metrics=['accuracy']
)

# Train with your labeled data
model.fit(embeddings_train, labels_train, epochs=10, validation_split=0.2)
model.save('gs://your-bucket/models/classifier-v1')

Dropout prevents overfitting. With small datasets, increase dropout. The model, at 12MB, deploys easily in a container with 512MB of RAM.

To label initial data, use Active Learning: train a base model with 200 examples, classify unlabeled data, and manually review only when confidence is low. Honestly, this saves you days of manual labeling.

Extracting Specific Entities: Prices, Features, Changes

Classifying isn’t enough. You need to extract concrete values: prices, new features, changes in positioning messaging.

For prices, regex works better than generic NER. Price patterns are common. With 15 patterns, you cover most cases. Train a specific NER model for the rest.

import re

def extract_pricing(text):
    patterns = [
        r'\$\d{1,5}(?:,\d{3})*(?:\.\d{2})?(?:/month|/mo|/year)?',
        r'€\d{1,5}(?:,\d{3})*(?:\.\d{2})?(?:/month|/mo|/year)?',
        r'from \$\d{1,5}',
        r'starting at \$\d{1,5}'
    ]
    
    prices = []
    for pattern in patterns:
        matches = re.findall(pattern, text, re.IGNORECASE)
        prices.extend(matches)
    
    return list(set(prices))

To detect significant changes, use paragraph embeddings. Compare embeddings of versions: if cosine similarity drops, something changed. This triggers an alert for manual review.

Store previous embeddings in Firestore or Redis. Calculate the embedding of current content, compare with historical data, and save the new one if changes are detected. This allows you to track evolution without storing full HTML.

Deploying on Kubernetes with Smart Autoscaling

Your scraper should handle from 20 to 5,000 URLs without intervention. Kubernetes Horizontal Pod Autoscaler (HPA) scales based on CPU, but scraping is I/O-bound, not CPU-bound. The solution: scale based on pending messages in a queue.

Use Cloud Tasks or RabbitMQ. Each URL is a message. Scrapers consume the queue. HPA scales based on queue size: more than 100 pending messages, spin up additional pods. Less than 20, reduce replicas.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: scraper-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: scraper
  minReplicas: 2
  maxReplicas: 20
  metrics:
  - type: External
    external:
      metric:
        name: pubsub.googleapis.com|subscription|num_undelivered_messages
      target:
        type: AverageValue
        averageValue: "30"

This HPA maintains 30 messages per pod. If you have 600 URLs pending, it spins up 20 pods. When the queue is cleared, it scales down to 2 replicas to save costs.

Deploy the TensorFlow model with TensorFlow Serving in a separate Deployment. TF Serving handles automatic batching: groups multiple requests in one pass, reducing average latency. With batches of 32, you process 1,000 classifications in less than 2 seconds.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tf-serving
spec:
  replicas: 3
  template:
    spec:
      containers:
      - name: tensorflow-serving
        image: tensorflow/serving:2.13.0
        args:
        - "--model_base_path=gs://your-bucket/models"
        - "--rest_api_port=8501"
        - "--enable_batching=true"
        - "--batching_parameters_file=/config/batching.txt"
        resources:
          requests:
            memory: "2Gi"
            cpu: "1000m"

With 3 replicas, you handle up to 5,000 classifications per minute. The cost on GKE is competitive.

Alerts and Dashboard: From Data to Decisions

Raw data is useless if your team doesn’t see it. Use Streamlit or Grafana for a dashboard that shows detected changes, new features, and price variations.

For real-time alerts, integrate Slack or Discord. Detect a significant change, trigger a webhook with context: what changed, in which competitor, and a visual diff.

import requests

def send_alert(competitor, change_type, old_content, new_content):
    webhook_url = "https://hooks.slack.com/services/..."
    
    message = {
        "text": f"🚨 Change detected in {competitor}",
        "blocks": [
            {
                "type": "section",
                "text": {
                    "type": "mrkdwn",
                    "text": f"*Type:* {change_type}\n*Competitor:* {competitor}"
                }
            },
            {
                "type": "section",
                "fields": [
                    {"type": "mrkdwn", "text": f"*Before:*\n{old_content[:200]}"},
                    {"type": "mrkdwn", "text": f"*Now:*\n{new_content[:200]}"}
                ]
            }
        ]
    }
    
    requests.post(webhook_url, json=message)

Note, the key is to filter noise. Not all alerts are useful. Define a scoring system: price changes have high priority, blog updates low. Only medium or high-priority alerts reach Slack; the rest await review.

Store all detections in BigQuery for historical analysis. Which competitor launches features faster? Do their price changes correlate with our conversions? BigQuery offers quick and cost-effective answers.

In Conclusion: Compete Without Corporate Spies or Inflated Budgets

Competitive intelligence doesn’t require research agencies or costly platforms. It requires intentional engineering: simple tools, clear responsibilities, and infrastructure that scales without breaking.

TensorFlow and Kubernetes aren’t just buzzwords. They’re architectural decisions that give you total control over data, models, and cost. The initial 50-hour development investment quickly pays off compared to any SaaS. Afterward, you only pay for infrastructure: between $40 and $150 monthly, depending on how many competitors you monitor.

Is your team already manually monitoring competitors, or are you still relying on Google alerts with a three-day delay?