Price Tracker System

A minimalist web application for tracking product prices across multiple online stores. Built with Go for simplicity, performance, and easy deployment.

Features

• Multi-store price tracking: Monitor prices from Belgian and French online stores
• Generic food categorization: Organize products into logical categories
• Price history & trends: Track price changes over time with percentage calculations
• Ingredient monitoring: Get notified when product ingredients change
• Favorites system: Mark frequently monitored products as favorites
• Unified search: Quickly locate foodstuffs, products, or stores from the homepage or header search bar
• Web scraping: Automated price collection using configurable CSS selectors
• Notifications: Alert system for significant price changes and scraping issues
• Responsive UI: Clean, minimal interface that works on mobile and desktop

Authentication & Security

• Single-admin model: There is exactly one privileged account named admin. The server refuses to start until that account exists and has a password.

• Bootstrap flow: On first launch the application checks for the admin user:

  • If PWD_ADMIN_PASSWORD is set, the user is created (or the password is rotated) automatically.
  • In interactive runs (e.g. go run ... from a terminal) the server prompts for the password if the environment variable is missing.
  • In non-interactive environments (Docker, systemd, Kubernetes) you must set PWD_ADMIN_PASSWORD; otherwise startup aborts with an error.

• Environment variables:

  • PWD_ADMIN_PASSWORD (required) ― password for the admin account. Changing the value rotates the password on the next restart and invalidates existing sessions.
  • PWD_ADMIN_EMAIL (optional) ― email associated with the admin account; defaults to admin@localdomain.
  • PWD_SECURITY_LOG (optional) ― full path to the authentication audit log. Defaults to data/logs/security.log inside the working directory.

• Login rate limiting & sessions: Failed attempts are rate-limited (5 tries per 5 minutes with a 15 minute lockout). Sessions use secure, HttpOnly cookies valid for 30 days and are purged automatically when the password rotates.

• Security log: Login successes, failures, and rate-limit events are written in a fail2ban-friendly format:

  2024-10-25T18:43:12Z AUTH_FAIL user=admin ip=203.0.113.42 reason=invalid_credentials
  2024-10-25T18:43:18Z AUTH_SUCCESS user=admin ip=203.0.113.42
  2024-10-25T18:43:21Z AUTH_RATE_LIMIT user=- ip=203.0.113.42 rate_limited_for=15m0s
  

  Configure PWD_SECURITY_LOG to point at a host-mounted path if you want tools such as fail2ban to consume it.

• fail2ban integration: Use a filter that matches AUTH_FAIL lines (see docs/guides/security/authentication.md for a full example). Start with maxretry=5, findtime=600, and bantime=900 to mirror the built-in rate limiter.

Architecture

• Backend: Go with built-in net/http server
• Database: SQLite for data persistence
• Frontend: Server-side templates with minimal JavaScript
• Scraping: goquery for HTML parsing
• Configuration: TOML files for store settings

Quick Start

Prerequisites

• Go 1.21 or later
• Git

Installation

1. Clone the repository:

git clone <repository-url>
cd price-tracker-system

2. Install dependencies:

go mod download

3. Create the data directory:

mkdir -p data

4. Run the application:

go run cmd/server/main.go

5. Open your browser and navigate to:

http://localhost:3000

Configuration

Port Configuration

The server port can be configured in multiple ways (in order of priority):

1. Command-line flag: --port or -port

go run cmd/server/main.go -port 8090

2. Environment variable: PORT

PORT=8090 go run cmd/server/main.go

3. Default: Port 3000 if neither flag nor environment variable is set

You can also specify the host to bind to:

go run cmd/server/main.go -host 0.0.0.0 -port 8090

Application Configuration

The application uses TOML configuration files:

• config/stores.toml: Store definitions and scraping selectors
• Modify rate limits, user agents, and CSS selectors as needed

Database

The SQLite database is automatically created at data/price-tracker.db on first run. The schema includes:

• Generic foodstuffs (categories)
• Stores and their configurations
• Products with scraping selectors
• Price history tracking
• Change notifications
• Scraping job queue

Project Structure

price-tracker-system/
├── cmd/server/          # Application entry point
├── internal/
│   ├── database/        # Database connection and schema
│   ├── handlers/        # HTTP request handlers
│   ├── models/          # Data models and structures
│   └── scraper/         # Web scraping functionality
├── web/
│   ├── templates/       # HTML templates
│   └── static/          # CSS and JavaScript assets
├── config/              # Configuration files
├── data/                # SQLite database location
└── scripts/             # Utility scripts

Usage

Searching the catalog

Use the search bar on the homepage (or in the global header) to look up foodstuffs, individual products, or stores. Results are grouped by entity type, with quick links to jump into detailed pages or drill down to full listings when more matches are available.

Adding a Generic Foodstuff

1. Navigate to the "Add Product" page
2. Create a new generic category (e.g., "Organic Milk")
3. Specify the default unit (kg, L, pieces)
4. Add tags for better organization

Adding Store Products

1. Find the product URL from a supported store
2. Add the product to an existing generic foodstuff
3. Configure CSS selectors for:
   • Product name
   • Price
   • Amount/weight
   • Ingredients list
4. Test the scraping configuration

Monitoring Prices

The system automatically:

• Scrapes product pages based on configured intervals
• Calculates price per unit for comparison
• Detects significant price changes
• Tracks ingredient modifications
• Monitors product availability

Notifications

Configure notification thresholds in config/stores.toml:

• Price change percentage (default: 10%)
• Ingredient change alerts
• Availability change notifications
• Scraping failure alerts

Supported Stores

Currently configured for:

Belgium

• Delhaize: General grocery store
• Colruyt: Discount grocery chain
• Bio-Planet: Organic products specialist
• Carrefour: International hypermarket chain
• The Barn Biomarket: Organic specialty store

France

• La Fourche: Organic products online
• KoRo: Bulk and specialty foods

Development

Running in Development

# Run with default port (3000)
just run

# Run with custom port
just run-port 8090

# Run with custom host and port
just run-host 0.0.0.0 8090

# Run with auto-reload (install air first: go install github.com/cosmtrek/air@latest)
air

# Or run directly with flags
go run cmd/server/main.go -port 8090

Adding New Stores

1. Add store configuration to config/stores.toml
2. Define default CSS selectors for the store
3. Test scraping with sample products
4. Adjust rate limits and user agents as needed

Database Migrations

The application automatically creates and updates the database schema. For manual database access:

sqlite3 data/price-tracker.db

Testing Scrapers

# Test individual product scraping
go run scripts/test-scraper.go <product-url>

API Documentation

The Price Tracker provides several endpoints for monitoring and integration:

Health Check Endpoint

Check if the application is running and database is accessible:

GET /health

Response:

{
  "status": "ok",
  "database": "connected",
  "timestamp": "2024-01-01T12:00:00Z"
}

Status Codes:

• 200 OK: Application is healthy
• 503 Service Unavailable: Database connection issues

Usage:

• Monitoring systems (Prometheus, Grafana)
• Load balancer health checks
• CI/CD deployment verification

Version Endpoint

Get application version information:

GET /version

Deployment

Single Binary Deployment

Build a single binary for deployment:

go build -o price-tracker cmd/server/main.go

Copy the binary along with:

• config/ directory
• web/ directory
• data/ directory (or create empty)

Environment Variables

• PORT: Server port (default: 3000)
• DATABASE_PATH: Custom database location
• CONFIG_PATH: Custom configuration directory

Command Line Options

• -port: Server port (overrides PORT env var)
• -host: Host to bind to (default: localhost)
• -h or -help: Show help message

Systemd Service

Create /etc/systemd/system/price-tracker.service:

[Unit]
Description=Price Tracker Service
After=network.target

[Service]
Type=simple
User=price-tracker
WorkingDirectory=/opt/price-tracker
ExecStart=/opt/price-tracker/price-tracker
Restart=always

[Install]
WantedBy=multi-user.target

Contributing

1. Follow the project's design philosophy: simplicity and minimalism
2. Use standard Go conventions and built-in libraries when possible
3. Keep the UI functional and data-dense
4. Test scraping configurations with real store pages
5. Update documentation for new features

License

[Specify your license here]

Troubleshooting

Common Issues

Scraping failures:

• Check CSS selectors against current store layouts
• Verify rate limits aren't being exceeded
• Update user agents if blocked

Database errors:

• Ensure write permissions to the data directory
• Check disk space availability
• Verify SQLite installation

Performance issues:

• Reduce concurrent scraping limit
• Increase rate limiting delays
• Optimize database queries

Logs

Application logs are written to stdout. For production deployment, redirect to files:

./price-tracker > price-tracker.log 2>&1