API Reference

The Spider API to get the data you need from any app.


The Spider API is based on REST. Our API is predictable, returns JSON-encoded responses, uses standard HTTP response codes, authentication, and verbs. Set your access token in your authentication header to commence.

Base Url

Start crawling a website or multiple websites to collect the resources. This endpoint returns all the content either formatted as raw text or raw resource like HTML streamed with JSONL responses by default. You can use the content-type header with application/xml, application/json, and text/csv for shaping the response.


  • url required

    The URI resource to crawl. This can be a comma split list for multiple urls.

  • authorization required

    The JWT access token string can also be passed in the body to take precedence over the header.

Authenticated Request
curl --location 'https://spider.a11ywatch.com/crawl' \
      --header 'Authorization: REPLACE_WITH_MY_ACCESS_TOKEN' \
      --header 'Content-Type: application/json' \
      --data '{"url":"https://jeffmendez.com"}'
Example Output
{"url":"https://jeffmendez.com/", "content":"Jeff Mendez About Projects Hi, my name is Jeff Mendez and I am software engineer residing in Florida. Checkout  A11yWatch  to improve your web inclusion. Use the  website accessibility checker  for free to evaluate your page. Blog posts You can view some of my blog content at  A11yWatch Blog . Open Source Work One of the main projects I contribute to the OSS community is  Spider  which is the fastest web crawler to gather data. It is a huge foundation block for A11yWatch allowing audits to be done at impressive speeds concurrently. Github", "error": null, "status": 200 }
{"url":"https://jeffmendez.com/tags", "content":"Jeff Mendez About Projects Roles in projects A couple of projects that I got to work on over the years. Software from different stacks along the way. Most of the projects show cased are mobile apps since that was a big area of focus. A11yWatch Improve your website inclusiveness and prevent your page from having drastic accessibility problems. About the Project A11ywatch is a SaaS product that allows users to attach a monitoring service on their website that will notify you as accessibility issues arise. The project provides a helper CDN that you include to your website that will automatically link the fixes you provide from a11ywatch. The cdn even has the ability to automatically fix the issues with the power of machine learning and AI. If you want more control on fixing issues there's also scripts that you can include directly into your project as an alternative. The main reason for this project as after trying to make a big project fully ADA compliant and realizing the amount of time and effort it takes. You can spend days/weeks/months depending on how big your product is to become ADA compliant and it is also against the law in the United States to not comply section 508. in the United States to not comply. I wanted to make this tasks simple instead of something that takes experienced engineers to come in and do. Creator Made using React Next.js GraphQL Node TensorFlow Rust WebAssembly Visit A11yWatch Carsight Never lose where your car is parked again. About the Project Carsight is a mobile app that allows you to find your car in augmented reality without having to press a manual park button. Useful for when you are in a city and need to park in elevated parking lots. Personal Project Made using React-Native Swift Zeemee Apply with confidence Join the #1 app to discover colleges, stand out in your applications, and connect with other students. About the Project Zeeme is a application that allows students to connect with their classmates and teachers before it starts. Part in project I mainly focused on the web application and backend doing ruby on rails with graphql work. While working on this project, I was able to fully automate and bootstrap the project into micro services using Docker. The main focus initially was to make the web application fully ADA compliant. Made using React GraphQL MYSQL Rails Redis View on App Store View on Play Store Arcus Track and manage your truck freights across your company. About the Project Arcus is a freight management and navigation TBT(turn-by-turn) application for truck drivers and brokers to co-exist and work together easier. Has custom themes to tailor app experience. Part in project The API and web application was already set. I joined the team to create the iOS and android applications. Will show the driver part soon its comparable to Uber. Made using React-Native React Java Obj-C MySql View on App Store View on Play Store Knack Find a tutor for any college course. About the Project Knack allows users to find a tutor for any subject that they need help with on your own time. Part in project I mainly focused on the front-end part of the application. The app was already done in obj-C so we recreated it from scratch in react-native. Made using React-Native React PHP Relay GraphQL MySql Realm", "error": null, "status": 200}
{"url":"https://jeffmendez.com/sitemap-index.xml", "content":"https://jeffmendez.com/sitemap-0.xml", "error": null, "status": 200}
{"url":"https://jeffmendez.com/favicon.ico", "content":"", "error": null, "status": 200}
{"url":"https://jeffmendez.com/about", "content":"Jeff Mendez About Projects About myself and personal life, sorta Florida based software engineer that loves everything about building things. I specialize in security, devops, data intensive applications, and UI/UX (all platforms from no OS, web, mobile, devops, cloud, and native applications). If you are looking for an experienced software engineer to create a new application, write new features into an existing application, help you make your application better by creating tests or refactoring code then we should get in touch! Contact me at jeffmendez19@gmail.com My github is  j-mendez .", "error": null, "status": 200}
Your API Key

Your API key is included in all the examples here, so you can test any example right away. Only you can see this value.

API Tokens

Access Token
Authenticate to get access token
Refresh Token
Authenticate to get refresh token

The access token is valid for 168 hours. You can use the API endpoint https://spider.a11ywatch.com/jwt with your authorization and refresh-token in your HTTP headers to get a new access token and refresh token. The API is available with a paid subscription while it is in Beta. We may move the API to a standalone subscription. Headless Chrome connectivity is not available via API at this moment.

Crawl Logs Beta

You can view the last hour of crawls performed.

No results.
0 of 0 row(s) selected.