{"id":15553,"date":"2018-08-01T16:17:11","date_gmt":"2018-08-01T10:47:11","guid":{"rendered":"http:\/\/blog.taragana.com\/?p=15553"},"modified":"2018-08-01T16:17:11","modified_gmt":"2018-08-01T10:47:11","slug":"how-to-incrementally-mirror-ftp-site-regularly","status":"publish","type":"post","link":"https:\/\/blog.taragana.com\/how-to-incrementally-mirror-ftp-site-regularly-15553","title":{"rendered":"How to incrementally mirror FTP site regularly"},"content":{"rendered":"<p>Daily Business Data, like NAV, is often made available via FTP servers which stores and makes available the data for limited time duration (like a week). We need to not only download the data for the day but also want to ensure that the historical data remains intact. Today I will show you how you can download and mirror the data from an ftp server without losing access to old data and how you can schedule it via cron to run periodically. Only changes are downloaded.<\/p>\n<pre>wget -m --retry-connrefused --password='<strong>password<\/strong>' ftp:\/\/<strong>login<\/strong>@<strong>address<\/strong>\/ -o <strong>log<\/strong><\/pre>\n<p><strong>login<\/strong> -&gt; Login<br \/>\n<strong>password<\/strong> -&gt; Password<br \/>\n<strong>address<\/strong> -&gt; IP Address or Hostname of the FTP server<br \/>\n<strong>log<\/strong> -&gt; Log file name<br \/>\nSave the above in a file named <strong>mirror-data<\/strong> and make it executable:<\/p>\n<pre>chmod 755 mirror-data<\/pre>\n<p>Run c<strong>rontab -e<\/strong> to edit the cron (scheduler) and add the following line after making necessary changes:<\/p>\n<pre>0 2 * * 1-5 \/path\/<strong>mirror-data<\/strong><\/pre>\n<p>This runs the executable every week day at 2am.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Daily Business Data, like NAV, is often made available via FTP servers which stores and makes available the data for limited time duration (like a week). We need to not only download the data for the day but also want to ensure that the historical data remains intact. Today I will show you how you [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[12],"tags":[],"_links":{"self":[{"href":"https:\/\/blog.taragana.com\/wp-json\/wp\/v2\/posts\/15553"}],"collection":[{"href":"https:\/\/blog.taragana.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.taragana.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.taragana.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.taragana.com\/wp-json\/wp\/v2\/comments?post=15553"}],"version-history":[{"count":0,"href":"https:\/\/blog.taragana.com\/wp-json\/wp\/v2\/posts\/15553\/revisions"}],"wp:attachment":[{"href":"https:\/\/blog.taragana.com\/wp-json\/wp\/v2\/media?parent=15553"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.taragana.com\/wp-json\/wp\/v2\/categories?post=15553"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.taragana.com\/wp-json\/wp\/v2\/tags?post=15553"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}