Whisker watch

Inspiration

Stock market analysis can quickly become overwhelming, with the overflow of information and the need to spot instantaneous changes in price. Our project aims to make it easier with an intuitive web-based tool that combines data visualization, anomaly detection, and machine learning to identify valuable patterns from stock prices. It motivated us to blend interactive visualizations with deep predictive analytics into an enabling solution that allowed traders, analysts, and researchers to make wiser decisions.

What it does

-The following is a breakdown of four major components involved in our stock analysis and prediction system:

Overview: It provides interactive visualizations of stock data, including the evolution of bid and ask prices, or mid prices, over time. It allows zooming in by selecting specific time ranges, and the user can visually detect trends.

DataParser: Automation of raw stock data cleaning and preprocessing into structured data. The preprocessed tasks it does are aligning timestamps, calculating mid-prices, and preparing the data further for analysis.

Predicter: It uses a sliding window to scan the stock data at 1-second intervals. It identifies sudden price movements up or down by computing the difference between the first and last prices in the window. The sharp changes are flagged and visualized.

ML Page: This will be predicted with the help of a machine learning method for sharp movement estimation within the price fluctuation from historical data. It prepares or trains key features: rolling average, standard deviations, and momentum. It flags up the outcome along with all future potential outliers.

How we built it

Frontend: The application was developed using Streamlit, allowing users to interact interactively and intuitively with the stock data and machine-learning models.

Backend:

Data Processing: This has been done using Pandas, allowing us to manipulate and preprocess the data efficiently.
Sliding Window Algorithm: The Python-based algorithm calculates sharp changes based on thresholds provided by the user.
Machine Learning: Trained models using Scikit-Learn and XGboost on historical stock data. Visualization: Integrated Bokeh for dynamic and interactive time-series charts.

Challenges we ran into

Small team with limited experience: Our team only had two members, one of whom was a first-year student participating in their first hackathon. Balancing the workload across such a large-scale project was highly challenging.

Backend development burden: This means only one teammate needed to work on the construction and optimization of the backend infrastructure necessary for data processing and visualization, thus adding even more challenges to their capability to meet deadlines.

Computational limitations: Above all, testing and running our machine learning sections of the code demanded a lot and were quite unfriendly due to computationally bounded processors. It got frustrating not to be able to verify our algorithm results quickly or present some of their essential outputs efficiently.

Accomplishments that we're proud of

Realized real-time integration of sharp change detection using visually appealing time series charts.

Developed a modular system to simplify the complex processes of stock analysis for the user.

Carried out large projects of high magnitude with a small team of only two members and worked through challenges regarding experience and computational powers.

Delivered a functional prototype to understand through interactive visualization.

What we learned

Preprocessing Data preprocessing is crucial in ensuring efficiency during analysis and model training. Properly preprocessed data leads to more accurate models and streamlined workflows. Handling missing values, aligning timestamps, and feature engineering are key steps that improve data quality for further processing.

Efficiency in Sliding Window Algorithms Sliding window-based algorithms can be optimized for anomaly detection in large datasets by ensuring efficient data resampling, batch-processing techniques, and leveraging parallel computation. This helps to reduce the runtime when detecting sharp changes or anomalies over large-scale stock data.

Intuitive Visualizations Creating intuitive visualizations with tools like Bokeh and Streamlit involves using clean, interactive charts with responsive features such as tooltips and zooming. Following best practices, like presenting data in a visually digestible format and ensuring the charts are fast and responsive, enhances the user experience.

Team Collaboration and Challenges Balancing the workload within a small team is critical for project success. Effective communication and assigning responsibilities based on strengths help overcome technical and computational challenges, such as managing backend infrastructure or handling large-scale datasets.

What's next for Whisker watch

Integration with Live Market Data: The system will be extended to fetch and process live stock market data, enabling real-time predictions and visualizations.

Advanced Anomaly Detection: Deep learning models will be incorporated to identify and extract complex patterns in stock price fluctuations, enhancing the anomaly detection capabilities.

Portfolio Analysis: The system will be expanded to analyze and provide actionable recommendations for multi-stock portfolios, allowing users to manage diverse investments effectively.