ETL system for Complex Data Analysis
WeSoftYou developed an ETL Data feed management system for e-Commerce retail company
Industry
Retail & E-Commerce
Project duration
June 2018 - February 2019
Country state
Australia
Team
4
Technology
Productify is a data feed management company that processes complex data provided by retailers. 🏃
The main idea of Productify is to help people get an enormous amount of data providing the smart mapping engine and complex data analysis. The platform takes the complex data provided by retailers and affiliates and alleviates the headache of having to integrate with hundreds of different feed formats. Productify setups, maintains, and normalizes all product data, and provides it to customers in a standard format across all retailers.
WeSoftYou helped to develop a web platform, and now Productify is a system that analyzes the complex data provided by retailers and further integrates it with hundreds of different feed formats. The system counts more than 100 shops in a database.
Results ✨
Simplified Data Integration
The platform successfully alleviates the complexities associated with integrating data from various retailers and affiliates by setting up, maintaining, and normalizing all product data.
Standardized Product Data
Productify now provides customers with product data in a standardized format across all retailers, ensuring consistency and ease of use.
Expanded Retailer Database
The system has evolved to analyze complex data from retailers and integrate it with over 100 different feed formats, demonstrating significant growth in the retailer database.
Enhanced Data Analysis
WeSoftYou implemented a system capable of analyzing complex data provided by retailers and further integrating it with hundreds of different feed formats.
Project goals ⚡️
Develop a Custom Web Platform: Create a robust custom web application surpassing template-based sites, tailored to the startup's unique needs.
Enhance Data-Feed Management System: Expand the existing data-feed management system by integrating a fintech's assignment management tool, improving functionality for efficient data feed management.
Challenges ⛰
01.
Data Aggregation and Scraping: Gathering data from diverse trade points within a shopping mall posed challenges in ensuring accurate and consistent information retrieval.
02.
Data Conversion and Standardization: Converting and normalizing data into a uniform format across retailers required addressing inconsistencies and discrepancies, presenting significant hurdles for data processing.
03.
Integration with Multiple Feed Formats: Integrating data from numerous feed formats necessitated robust solutions to ensure seamless compatibility and overcome technical complexities.
04.
Scalability and Maintenance: With over 100 shops in the database and expectations for further growth, ensuring scalability and efficient maintenance became critical challenges requiring ongoing optimization.
The process 🚧
The Productify project’s development process involved meticulous planning, prototype development for data scraping, expansion into web application development, and rigorous testing to ensure reliability and usability. Following deployment, ongoing maintenance and support were provided to ensure functionality, optimization, and scalability, delivering a comprehensive solution for data aggregation and normalization across multiple retailers.
Stages 🎢
01.
Requirement Analysis: WeSoftYou conducted a comprehensive analysis of project objectives and constraints, collaborating closely with stakeholders to define clear development goals.
02.
Data Scraping Prototype: WeSoftYou developed a prototype for data scraping, testing the feasibility of aggregating and processing data from various trade points within a shopping mall.
03.
Expansion to Web Application Development: Upon successful validation of the prototype, the project expanded to include the development of a web application aimed at centralizing and standardizing product data across retailers.
04.
Architecture and Infrastructure Design: WeSoftYou meticulously designed the architecture and infrastructure of the web application, determining the optimal technologies and database structures to handle data complexities.
05.
Core Functionality Development: The development team built the core functionalities of the web application, implementing mechanisms for data aggregation, scraping, and normalization.
06.
Testing and Quality Assurance: Rigorous testing procedures were implemented to validate the reliability and performance of the web application, ensuring accuracy and usability.
07.
Deployment and Post-Deployment Support: WeSoftYou deployed the Productify web application, providing ongoing maintenance and support to ensure functionality, optimization, and scalability, and address client requests.
Team composition 👨💻
Front-End developer
Back-End developer
Project Manager
Quality Assurance specialist
Technology stack ⚙️
Front-end: : Python, Django, React.js, Redux
Core features 💻
- Complex data analysis engine
- ETL system implemented
- Easy Scalable architecture
- Search and filtering functionality
Do you want to start a project?
Meet us across the globe
United States
66 W Flagler st Unit 900 Miami, FL, 3313016 E 34th St, New York, NY 10016
Europe
109 Borough High St, London SE1 1NL, UKProsta 20/00-850, 00-850 Warszawa, Poland
Vasyl Tyutyunnik St, 5A, Kyiv, Ukraine
Av. da Liberdade 10, 1250-147 Lisboa, Portugal
