Designing a silver market data pipeline for WSS
·
Wall Street Silver needs reliable market data: CFTC commitment of traders reports, COMEX inventory levels, LBMA prices, and community sentiment signals from Reddit.
The challenge is building a pipeline that ingests from multiple sources with different schemas, normalizes timestamps, handles failures gracefully, and stores everything in a queryable time-series format.
Pipeline components
- Reddit API ingestion for sentiment extraction
- CFTC data parsing and normalization
- COMEX/LBMA price feeds
- Time-series database for storage
- Analytics layer for signal extraction
- Visualization dashboard
This pipeline supports research, community insights, and market analysis tools that help understand silver market dynamics.