I came, I saw, I scraped

Posted by Randy Taylor on September 15, 2019

I just finished my CLI project. It was a lot of work! I got stuck, I cried, I stayed up late, I was late to work, I didn’t sleep, and I eventually won. I beat the errors, I built the code, I fixed stuff that was missing, I dryed out the code, etc etc etc. I feel like I learned so much and really that is how you learn to code. With sweat, tears, blood, and headaches I managed to shove so much code info into my brain and come out code smarter. I am totally loving the understanding of Object Oriented programming. The best thing I can do here is point out in Object Oriented Programming you need to pass around Objects to other Objects. That is Key to how to really use the language to it’s full potential. Otherwise you are just kinda doing simple math problems you can never use. The second thing I can point out is that you can encapsulate your code and reuse it over and over. I had this problem with my CLI where to keep it simple for the user I had to create multiple gets for different parts of the program. That way the user could just keep typing the same number to go deeper into the website I was scrapping. This created a problem with multiple streams of gets variables running and being difficult to track and close out. Then I realized that I could create a def method that would put out a variable that used the gets method and boom I was good to go. Now I just had to use that method in different places with different variable names! It made the code so much dryer! Is my code perfect? NO. Most definitely not. I already have a few ideas to make it better (e.g. fix the custom_error I tried to create when the internet was slow or down). I really feel much better than two months ago when I first started! Before I end this blog post I’m going to point out a huge error I had with Open-uri and nonsmoking that was very simple and misleading to debug. Open-uri and nonsmoking need to http:// at the beginning of a url to pull out HTML and scrape it. Now when you scrape a website and pull out links you may just be pulling out the tail end of the url. With binging.pry you will see the link and put it in the browser. That will mislead you because the browser will ADD THE HTTP://!!!! It will appear as if the link is valid despite the error telling you it does not exist. I spend two days on that problem. I thought It was me and something wrong with my code (part of the error had something about initialized). I coded and recoded and triple coded the same things in different ways! Errors can only guess what is wrong. Again. Open-uri and nonsmoking use http:// and without that initial http:// it does not exist to either!