While doing web scraping I came across with a website who has implemented Cloudfare DDoS (Distributed Denial of Service) protection. DDoS is an attempt where a target host is attacked by multiple sources commonly to bring it down. Wikipedia. Cloudfare, apart from being a usual CDN also provides security features to the websites. One of which is the … Continue reading Bypassing Cloudfare DDoS in Scrapy
Pandas gives you the ability to read large csv in chunks using a iterator. This way you don't have to load the full csv file into memory before you start processing. https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html My objective was to extract, transform and load (ETL) CSV files that is around 15GB. Here is the code snippter that can be … Continue reading Process CSV files with multiprocessing in Pandas
Often times we have to create command line utilities and it doesn't make sense every time to put one command in a separate file specially when you have a bunch of related functionalities. This is a quick tutorial for handling sub commands in python which shows how we can write multiple functions in one module … Continue reading Python argparse and subparsers
PHP - People Helping People Following the twitter trend of #20yearsofphp, I could not resist to share my experience of my little PHP career 🙂 I am a commerce graduate and spend almost 5 years of my professional career in areas like accounts, customer service, BPO. I learned HTML during my school days in early … Continue reading My PHP Story – 20 Years of PHP
During the development of an API for a client i feel the need to have one class that can return me instance of different databases. I made a singleton pattern to achieve this. Below is the class that you can utilize if needed.