The Undocumented Web: scraping, private APIs, proxies and “alternative solutions”

What is the undocumented web? Scott and Wes dive into it, discussing APIs, faking, scraping, automation, proxies as well as tips and tricks for best practices. Kyle Prinsloo’s Freelancing & Beyond — Sponsor Kyle Prinsloo teaches you everything you need to know about freelancing, including how to quit your job, earn a side-income and start taking control of your life. Check it out at https://studywebdevelopment.com/freelancing and use the coupon code “syntax” at checkout to get 25% off. Freshbooks - Sponsor Get a 30 day free trial of Freshbooks at freshbooks.com/syntax and put SYNTAX in the “How did you hear about us?” section. Show Notes 3:05 - What Is the undocumented web? • When traditional methods don’t give you what you want • Wild West 6:10 - Undocumented APIS • Instagram’s private API • Use dev tools’ XHR Request • React Dev Tools will also have the data Ep 008 - Wes Bos Origin Story Realtor.ca - MLS Mapping Simon Willison’s Zeit Day Talk about XML files Charles Proxy App • - also good for debugging requests 17:34 - Faking • Faking User Agents • Sending headers from requests • Free Blackberry wifi on plane • 23:52 - Scraping • What is craping? Cheerio • - jQuery like node package 27:30 - Automation • Cron Jobs • Filling out forms instantly • Headless browsers Puppeteer PhantomJS • 33:30 - Proxies • Need a fresh IP address • Need a different country • Side note: Canadians buying stuff in USA M4S1P1 === 41100 • API throttling 39:00 - Tips and Tricks • Multiple API keys diskDB • - Text files as databases • Search Github for API keys • Inspect Element and Delete Overlays • Delete Cookiesokay 44:45 - Stories • Photo Voting • MLS Listing • Top Entrepreneur voting cheating • Vanity Plate Checker My Hockey Team Sucks Vehicle auction bidding app youtube-dl ××× SIIIIICK ××× PIIIICKS ××× • Scott: Octopath Traveler & OST • Wes 1: AvE Youtube Channel • Wes 2: Mustie1 Youtube Channel Shameless Plugs Scott’s Level Up Tuts Pro Wes’ Courses Tweet us your tasty treats! Scott’s Instagram LevelUpTutorials Instagram Wes’ Instagram Wes’ Twitter Wes’ Facebook Scott’s Twitter • Make sure to include @SyntaxFM • in your tweets