I use Screaming Frog a lot. Just to see how my sites are doing. As you might have read in several of my blog posts (like: 5 reasons to stop with Wordpress, or Medium stopped offering custom domains), I could not really get used to some blog frameworks. And by switching a lot my sites tend to fluctuate in Google Search a lot.

So every now and then I use Screaming Frog to test my sites and see what I can do to improve them. Now, I'm not using the GUI that Screaming Frog has, but I use the command line.

With this command line I export the Bulk Export and the Export Tabs and process them with Python to an actually readable Word Document. I build that in the past just for fun.

The ‘Bulk Export’ in the Screaming Frog GUI is located under the top level menu and allows bulk exporting of all data. You can export all instances of a link found in a crawl via the ‘All inlinks’ option, or export all inlinks to URLs with specific status codes such as 2XX, 3XX, 4XX or 5XX responses.

As I'm using it in Python I wanted to know the names of all the Bulk Export and the filenames that Screaming Frog would spit out.

And Guess what.... there is no complete list! Not even the Screaming Frog people could tell me what Bulk Export items there are.

I would really like an over view (for command line) with -bulk-export and -export-tabs.
We don't have one I am afraid.

We may look at writing them all out at some stage, but we don't have one now, or one I can offer you at a set timescale unfortunately.

Bummerrrrrrr

Python OCR the Bulk Export tab images

So..... no way to get the Screaming Frog Bulk Export List. I can tell you I did not want to type 60 Bulk Export names and file names.

What I could do was to make a copy of the names in the GUI overview and retrieve the text. So, I did some  coding in python to actually get the text from the overview. (yes... I might have as well typed the names myself. But I just love coding :-)

So I made screenshots, saved the screen shots in a folder and ran my python script

Screaming Frog Bulk Export List

So here is the list. Where name is the actual name you can use in the command line option from screaming frog "--bulk-export" and file is the filename coming out of the command line processing.

YML output can be found here.


file: all_inlinks.csv
name: All Inlinks

file: all_outlinks.csv
name: All Outlinks

file: queued_urls.csv
name: Queued URLs

file: all_anchor_text.csv
name: All Anchor Text

file: all_images.csv
name: All Images

file: null
name: Screenshots

file: null
name: All Page Source

file: external_links.csv
name: External Links

file: response_codes_blocked_by_robots_txt.csv
name: Response Codes:Blocked by Robots.txt Inlinks

file: response_codes_blocked_resource.csv
name: Response Codes:Blocked Resource Inlinks

file: response_codes_no_response.csv
name: Response Codes:No Response Inlinks

file: response_codes_success_(2xx).csv
name: Response Codes:Success (2xx) Inlinks

file: response_codes_redirection_(3xx).csv
name: Response Codes:Redirection (3xx) Inlinks

file: response_codes_redirection_(javascript).csv
name: Response Codes:Redirection (JavaScript) Inlinks

file: response_codes_redirection_(meta_refresh).csv
name: Response Codes:Redirection (Meta Refresh) Inlinks

file: response_codes_client_error_(4xx).csv
name: Response Codes:Client Error (4xx) Inlinks

file: response_codes_server_error_(5xx).csv
name: Response Codes:Server Error (5xx) Inlinks

file: directives_index.csv
name: Directives:Index Inlinks

file: directives_noindex.csv
name: Directives:Noindex Inlinks

file: directives_follow.csv
name: Directives:Follow Inlinks

file: directives_nofollow.csv
name: Directives:Nofollow Inlinks

file: directives_none.csv
name: Directives:None Inlinks

file: directives_noarchive.csv
name: Directives:NoArchive Inlinks

file: directives_nosnippet.csv
name: Directives:NoSnippet Inlinks

file: directives_maxsnippet.csv
name: Directives:Max-Snippet Inlinks

file: directives_maximagepreview.csv
name: Directives:Max-Image-Preview Inlinks

file: directives_maxvnameeopreview.csv
name: Directives:Max-Vnameeo-Preview Inlinks

file: directives_noodp.csv
name: Directives:NoODP Inlinks

file: directives_noydir.csv
name: Directives:NoYDIR Inlinks

file: directives_noimageindex.csv
name: Directives:NoImageIndex Inlinks

file: directives_notranslate.csv
name: Directives:NoTranslate Inlinks

file: directives_unavailable_after.csv
name: Directives:Unavailable_After Inlinks

file: directives_refresh.csv
name: Directives:Refresh Inlinks

file: canonicals_contains_canonical.csv
name: Canonicals:Contains Canonical Inlinks

file: canonicals_self_referencing.csv
name: Canonicals:Self Referencing Inlinks

file: canonicals_canonicalised.csv
name: Canonicals:Canonicalised Inlinks

file: canonicals_missing.csv
name: Canonicals:Missing Inlinks

file: canonicals_multiple.csv
name: Canonicals:Multiple Inlinks

file: canonicals_nonindexable_canonical.csv
name: Canonicals:Non-Indexable Canonical Inlinks

file: amp_all.csv
name: AMP:All Inlinks

file: amp_non200_response.csv
name: AMP:Non-200 Response Inlinks

file: amp_nonconfirming_canonical.csv
name: AMP:Non-Confirming Canonical Inlinks

file: amp_missing_nonamp_canonical.csv
name: AMP:Missing Non-AMP Canonical Inlinks

file: amp_nonindexable_canonical.csv
name: AMP:Non-Indexable Canonical Inlinks

file: amp_indexable.csv
name: AMP:Indexable Inlinks

file: amp_nonindexable.csv
name: AMP:Non-Indexable Inlinks

file: structured_data_contains_structured_data.csv
name: Structured Data:Contains Structured Data

file: structured_data_valnameation_errors.csv
name: Structured Data:Valnameation Errors

file: structured_data_valnameation_warnings.csv
name: Structured Data:Valnameation Warnings

file: structured_data_jsonld_urls.csv
name: Structured Data:JSON-LD URLs

file: structured_data_microdata_urls.csv
name: Structured Data:Microdata URLs

file: structured_data_rdfa_urls.csv
name: Structured Data:RDFa URLs

file: images_missing_alt_text_inlinks.csv
name: Images:Images Missing Alt Text Inlinks

file: images_over_100_kb.csv
name: Images:Images over X KB Inlinks

file: sitemaps_urls_in_sitemap.csv
name: Sitemaps:URLs in Sitemap Inlinks

file: search_console_orphan_urls.csv
name: Sitemaps:Orphan URLs Inlinks

file: sitemaps_nonindexable_urls_in_sitemap.csv
name: Sitemaps:Non-Indexable URLs in Sitemap Inlinks

file: sitemaps_urls_in_multiple_sitemaps.csv
name: Sitemaps:URLs in Multiple Sitemaps Inlinks

file: custom_search_all.csv
name: Custom Search:All Inlinks

file: custom_extraction_all.csv
name: Custom Extraction:All Inlinks


Like it? Don't forget to buy me a Coffee!!