AI competitor monitoring with weekly report

Introduction

Real-time competitor monitoring is crucial for any company striving to maintain a technological edge. Thanks to advances in artificial intelligence and automation, it is now possible to build a system that, with minimal manual work, tracks competitors’ products, prices, features, and communication and generates regular overviews. This article serves as a practical guide to creating a functional MVP of the “AI competitor monitoring with weekly report” project. The goal is to connect several existing services—web scraping, APIs, a language model (e.g. GPT‑5), cloud storage, and automated notifications—into a comprehensive solution.

Project goal

Try NordVPN

The output of the project is a system that:

• automatically collects data from competitors’ public websites and RSS feeds,
• analyzes changes in content (new articles, feature updates, price changes),
• generates a report every week in a clear format (PDF, email, or Slack message),
• allows managers to quickly evaluate trends without manually reading dozens of sources.

Prerequisites

Try Semrush

• Basic knowledge of Python (version ≥ 3.10)
• Created accounts for access to public APIs (e.g. SerpApi, OpenAI API, NewsAPI)
• Access to a task automation tool, e.g. Zapier or Apache Airflow
• A cloud account (e.g. Google Cloud or AWS)
• Basic experience with output formatting (Markdown, HTML, PDF)

Implementation steps

Try Make

Step 1: Define the list of competitors and sources

Action: Create a competitors_sources.csv table with the most important competitor websites and source URLs. Use Google Sheets or a CSV file with the following structure:

name,url,type
CompanyA,https://www.companya.cz/rss,news
CompanyB,https://www.companyb.cz/api/products,products

Input: a list of feed URLs, API links, or competitor sitemaps.

Output: a competitors_sources.csv file saved in the project.

Success metric: At least 90% of monitored sources must be available and provide data in an HTTP 200 test.

Step 2: Automated data collection

Action: Use the requests or httpx library to access URLs and BeautifulSoup for text extraction. Schedule daily scraping using Apache Airflow.

from bs4 import BeautifulSoup
import requests

url = "https://www.firmaa.cz/rss"
response = requests.get(url)
soup = BeautifulSoup(response.text, "xml")
items = soup.find_all("item")
news_data = [{"title": i.title.text, "link": i.link.text} for i in items]

Input: URLs from the previous step.

Output: a JSON data file data/raw/YYYY-MM-DD.json with extracted items.

Success metric: at least 95% of feeds processed without HTTP 4xx/5xx errors. Log exceptions to logs/scraper.log.

Step 3: Data storage and versioning

Action: Use a cloud database (e.g. Firestore or DynamoDB) to store new records and version previous ones. Content hashing is recommended so changes can be detected.

import hashlib
hash_val = hashlib.sha256(item['title'].encode() + item['link'].encode()).hexdigest()

Input: JSON with new data.

Output: Stored records with the fields hash, source, timestamp.

Success metric: duplicates < 2% measured by hash values per 1000 records.

Step 4: Change analysis and extraction of updates

Action: Compare the current dataset with the previous week. Use simple hash comparison or the pandas library to detect unique records. Then use the GPT‑5 API to summarize the updates.

import pandas as pd
from openai import OpenAI

client = OpenAI(api_key="YOUR_OPENAI_API_KEY")

new_items = df_current[~df_current['hash'].isin(df_previous['hash'])]
prompt = f"Summarize the main competitor changes: {new_items.to_json()}"
response = client.chat.completions.create(model="gpt-5", messages=[{"role": "user", "content": prompt}])
summary = response.choices[0].message.content

Input: Dataset with new items (JSON).

Output: Text summary weekly_summary.txt.

Success metric: The summary has accuracy > 85% in manual verification (evaluated by the marketing director or analyst).

Step 5: Generate the weekly report

Action: Convert the text output into PDF format using WeasyPrint or ReportLab. Add a chart of price changes from previous weeks using the matplotlib library.

from weasyprint import HTML
HTML('summary.html').write_pdf('weekly_report.pdf')

Input: Summary weekly_summary.txt + competitor pricing data from the database.

Output: File weekly_report.pdf.

Success metric: The report contains all monitored competitors and displays no empty sections (validated via regex text check).

Step 6: Report distribution and integration with team communication

Action: Use Slack Webhooks or the Gmail API to send the weekly PDF to all team members.

import requests
webhook_url = "https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXX"
msg = {"text": "The weekly AI report is available.", "attachments": [{"title": "View PDF", "title_link": "https://storage.cloud/report.pdf"}]}
requests.post(webhook_url, json=msg)

Input: Location of the PDF report in the cloud (e.g. Google Cloud Storage).

Output: Confirmation of message delivery to the Slack channel.

Success metric: 100% of team members receive the notification and report within 5 minutes of creation (verified by Slack API timestamp).

Testing

Try Hostinger

Verify functionality with both unit tests and end‑to‑end scenarios:

• Unit tests for download functions (e.g. a test that requests.get returns valid XML)
• Checking data types and empty values in pandas dataframes
• Comparison of two weeks of data—expected differences (e.g. number of new items)

Example of a unit test:

def test_fetch_data():
    response = requests.get("https://www.firmaa.cz/rss")
    assert response.status_code == 200

Save each test in the tests/ folder and run with the command pytest --maxfail=1 --disable-warnings -q.

Deployment

Try Fiverr

Implement deployment in two environments—testing and production:

1. Testing environment: Docker container with limited access to API keys and data. Use the official Python image.
2. Production environment: Cloud Run (Google Cloud) or AWS Lambda with a daily trigger. Set environment variables: OPENAI_API_KEY, NEWSAPI_KEY.
3. Set the cron scheduler, e.g. “0 8 * * 1" (Monday 8:00) to run report generation.

Before the final launch, check stability (logs without errors during 3 cycles) and verify that the report is generated even when some sources are partially unavailable—the system should recover after an HTTP error.

Limits

Try Adobe

• The GPT‑5 API and other tools may have a daily call limit (e.g. 1000 requests). Prompt caching is recommended.
• Website scraping may be restricted by robots.txt—it is necessary to respect the terms.
• Some pages change their HTML structure, so it is advisable to validate xpath selectors at least once a month.
• The accuracy of GPT‑5 summaries depends on the input format; it is recommended to keep the maximum text length to 6,000 tokens per request.

FAQ

Question: Can this system be extended to monitor social media?

Answer: Yes, for example by integrating with the Meta Graph API or the Twitter API, while taking limits and OAuth 2.0 authentication into account.

Question: How can API keys be secured?

Answer: Use a dotenv file stored outside the Git repo, and set environment variables in production.

Question: How long does report generation take?

Answer: Usually 2–5 minutes depending on the number of sources (measured on a VM with 2 vCPU, 4 GB RAM).

Question: Will the system maintain itself?

Answer: With proper Airflow DAG configuration, maintenance is minimal; recommended supervision is once a week.

Conclusion

This guide presents a complete process for creating an AI competitor monitoring system with autonomous weekly report generation. It combines open APIs, cloud automation tools, and GPT‑5 capabilities for data processing and clear summarization. After completing all steps, you will have an MVP solution that can be further developed—for example with trend charts, price trend prediction, or sentiment analysis integration. The key benefit is significant time savings for analysts and an immediate overview of the market situation, which can be scaled without the need for manual intervention.

Recommended AI stack for implementation

Select tools according to your budget and level of automation. Below is a direct overview of services for implementing the project.

Recommended next step

Try NordVPN

Links in the article

• NordVPN
• Semrush
• Make
• Hostinger
• Fiverr
• Adobe
• Canva
• Jasper

Sources of illustrative images

• Stock photo: source

The custom illustrative image was created using the OpenAI Images API.

Note: We use affiliate links for listed services. If you purchase through them, we may earn a commission at no extra cost to you.

Doporučení ke čtení

AI machine for processing inquiries from the web and email step by step

Alternatives to Make AI by team type

The most important news in AI tools: a practical overview

Deep Research workflow: from assignment to verified briefing in 30 minutes