Grocery List Plus is a web application where you can create, and manage grocery lists.

You can search for the items in your grocery lists across three major supermarkets we have here in New Zealand PAK'nSAVE, New World, and Countdown.

Regarding the frontend side of the application, I decided to go with Angular as the frontend framework as I hadn’t worked with Angular before

In developing the backend, I decided to 1) Try out a NoSQL database, and 2) Develop a REST API using Python. Ultimately this led me to go with MongoDB for the database, and Flask web framework for the REST API.

The API needed to be able to search across the three major supermarket chains for grocery list items, this meant I needed to implement some sort of web scraping to get the grocery item information I needed from the websites.

After some research on web scraping, I decided to use a headless browser to access the supermarkets’ websites, I went with selenium and got to the point where the API could receive a query i.e “Chocolate”, as well as the supermarket chains that you wanted to search across i.e “PakNSave”, “Countdown” and the API would open up a headless browser, and make a search on the Supermarket's website as a human would, then once the results have been returned, it would get the information of each grocery item (e.g. price, name, unit, weight) by grabbing elements using CSS selectors then returning their text.

Upon doing a bit more research into web scraping, I decided to search the supermarkets for grocery items not by imitating a user, but by accessing the endpoints for making search queries directly. Some of the supermarket websites return an authentication token used to identify the user and make requests to the endpoints for searching grocery items, I simply go onto the supermarket’s website, grab the authentication token, and then make the needed requests to the endpoint. I decided to switch to Playwright as the headless browser, as intercepting the HTTP requests seemed much easier to do using Playwright than in Selenium

The REST API is hosted on an AWS EC2 instance, I do this by building a docker image from the Flask App, then pushing the image to my private docker repo, and running a container from the image on my EC2 instance, with secrets imported from AWS Secrets Manager, and for the production MongoDB database, I went with MongoDB atlas.

I hosted the Angular App on Google App Engine and pointed my subdomain (list.mustapha-conteh.me) to the one provided by Google App Engine.

I setup workflows using Github Actions, to build and deploy my Angular App to Google App Engine, and to build a docker image of my Rest API (Flask App), and push it to my private docker repo, I then run a bash script on my ec2 instance to pull the new docker image, and run a container with secrets (e.g api keys, db credentials) loaded from AWS Secrets Manager.