{"id":5821,"date":"2020-06-03T19:24:29","date_gmt":"2020-06-03T19:24:29","guid":{"rendered":"http:\/\/41j.com\/blog\/?p=5821"},"modified":"2020-06-03T19:24:30","modified_gmt":"2020-06-03T19:24:30","slug":"scripts-to-download-sars-cov-2-replacements","status":"publish","type":"post","link":"https:\/\/41j.com\/blog\/2020\/06\/scripts-to-download-sars-cov-2-replacements\/","title":{"rendered":"Scripts to download SARS-CoV-2 replacements"},"content":{"rendered":"\n

I wanted to download a set of mutations in SARS-CoV-2. CoV-GLUE<\/a> seems to be a reasonable database of mutations in SARS-CoV-2. However the web interface doesn’t seem to have an option to download a dataset. And there isn’t a published API. So I threw together some ugly bash\/awk to get what I wanted. I don’t imagine this will work for long, as the website appears to be under active development. But here are my notes anyway.<\/p>\n\n\n\n

The website works off a (undocumented?) JSON API. I used the follow JSON template to get replacements (non-synonymous substitutions) which occur in 2 or more sequences:<\/p>\n\n\n

\n{"multi-render":{"tableName":"cov_replacement","allObjects":false,"whereClause":"(true) and  (((num_seqs >= 2)))","rendererModuleName":"covListReplacementsRenderer","pageSize":500,"fetchLimit":500,"fetchOffset":FETCHOFFSET,"sortProperties":"-num_seqs,+variation.featureLoc.feature.name,+codon_label_int,+replacement_aa"}}\n\n<\/pre><\/div>\n\n\n
The above goes in a file called templ. I then just modify “FETCHOFFSET” using sed and download the first 4500 mutations (at the time of writing there are 4000 odd mutations. You’d want to stick this all in a loop… but I didn’t bother:<\/p>\n\n\n
\nrm all\nrm index.html;cp templ c;sed -i 's\/FETCHOFFSET\/0\/g' c;wget http:\/\/cov-glue.cvr.gla.ac.uk\/gluetools-ws\/project\/cov\/ --post-file=.\/c --header "Cookie: _cle=accepted" --header "Content-Type: application\/json";cat index.html >> all\nrm index.html;cp templ c;sed -i 's\/FETCHOFFSET\/500\/g' c;wget http:\/\/cov-glue.cvr.gla.ac.uk\/gluetools-ws\/project\/cov\/ --post-file=.\/c --header "Cookie: _cle=accepted" --header "Content-Type: application\/json";cat index.html >> all\nrm index.html;cp templ c;sed -i 's\/FETCHOFFSET\/1000\/g' c;wget http:\/\/cov-glue.cvr.gla.ac.uk\/gluetools-ws\/project\/cov\/ --post-file=.\/c --header "Cookie: _cle=accepted" --header "Content-Type: application\/json";cat index.html >> all\nrm index.html;cp templ c;sed -i 's\/FETCHOFFSET\/1500\/g' c;wget http:\/\/cov-glue.cvr.gla.ac.uk\/gluetools-ws\/project\/cov\/ --post-file=.\/c --header "Cookie: _cle=accepted" --header "Content-Type: application\/json";cat index.html >> all\nrm index.html;cp templ c;sed -i 's\/FETCHOFFSET\/2000\/g' c;wget http:\/\/cov-glue.cvr.gla.ac.uk\/gluetools-ws\/project\/cov\/ --post-file=.\/c --header "Cookie: _cle=accepted" --header "Content-Type: application\/json";cat index.html >> all\nrm index.html;cp templ c;sed -i 's\/FETCHOFFSET\/2500\/g' c;wget http:\/\/cov-glue.cvr.gla.ac.uk\/gluetools-ws\/project\/cov\/ --post-file=.\/c --header "Cookie: _cle=accepted" --header "Content-Type: application\/json";cat index.html >> all\nrm index.html;cp templ c;sed -i 's\/FETCHOFFSET\/3000\/g' c;wget http:\/\/cov-glue.cvr.gla.ac.uk\/gluetools-ws\/project\/cov\/ --post-file=.\/c --header "Cookie: _cle=accepted" --header "Content-Type: application\/json";cat index.html >> all\nrm index.html;cp templ c;sed -i 's\/FETCHOFFSET\/3500\/g' c;wget http:\/\/cov-glue.cvr.gla.ac.uk\/gluetools-ws\/project\/cov\/ --post-file=.\/c --header "Cookie: _cle=accepted" --header "Content-Type: application\/json";cat index.html >> all\nrm index.html;cp templ c;sed -i 's\/FETCHOFFSET\/4000\/g' c;wget http:\/\/cov-glue.cvr.gla.ac.uk\/gluetools-ws\/project\/cov\/ --post-file=.\/c --header "Cookie: _cle=accepted" --header "Content-Type: application\/json";cat index.html >> all\n\n\n<\/pre><\/div>\n\n\n
The we extract all the mutation IDs:<\/p>\n\n\n
\nawk  'BEGIN{RS="id\\":\\"";FS="\\""}{print $1}' all > allmuts\n<\/pre><\/div>\n\n\n
And then fetch them from the server, this will create 4000 odd files, which we can then parse further:<\/p>\n\n\n
\n#awk '{print "wget --header \\"Cookie: _cle=accepted\\" --header \\"Content-Type: application\/json\\" --post-file=.\/info.json http:\/\/cov-glue.cvr.gla.ac.uk\/gluetools-ws\/project\/cov\/custom-table-row\/cov_replacement\/" $1}' allmuts  > allmuts.get\n<\/pre><\/div>\n\n\n
The CoV-GLUE database seems like a great resource. I hope they add a feature to download sequences\/results soon. I’ve seen database results presented in a few preprints. It would be nice it those papers could also include the raw data, otherwise they’re unfortunately going to end up being difficult to replicate…<\/p>\n","protected":false},"excerpt":{"rendered":"
I wanted to download a set of mutations in SARS-CoV-2. CoV-GLUE seems to be a reasonable database of mutations in SARS-CoV-2. However the web interface doesn’t seem to have an option to download a dataset. And there isn’t a published API. So I threw together some ugly bash\/awk to get what I wanted. I don’t […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[1],"tags":[],"class_list":["post-5821","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/p1RRoU-1vT","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/41j.com\/blog\/wp-json\/wp\/v2\/posts\/5821","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/41j.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/41j.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/41j.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/41j.com\/blog\/wp-json\/wp\/v2\/comments?post=5821"}],"version-history":[{"count":1,"href":"https:\/\/41j.com\/blog\/wp-json\/wp\/v2\/posts\/5821\/revisions"}],"predecessor-version":[{"id":5822,"href":"https:\/\/41j.com\/blog\/wp-json\/wp\/v2\/posts\/5821\/revisions\/5822"}],"wp:attachment":[{"href":"https:\/\/41j.com\/blog\/wp-json\/wp\/v2\/media?parent=5821"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/41j.com\/blog\/wp-json\/wp\/v2\/categories?post=5821"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/41j.com\/blog\/wp-json\/wp\/v2\/tags?post=5821"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}